With bug 30280 and subsequent patches we attempt to respect the
thesaurus indication when linking with LinkerConsiderThesaurus on.
However, there are still cases when we link wrong.
This happens when thesaurus in bibliographic field 6XX is defined by
subfield $2. In such a case the authority record should have 008/11 = 'z'
and, according to https://www.loc.gov/marc/authority/ad008.html
(cf. comment for 008/11 = 'z': "A MARC code for the conventions used to
formulate the heading *may* be contained in subfield $f"), may or may have
thesaurus declared in 040 $f.
This is why we make a second search attempt in C4::Heading::_search,
but doing so we have to make sure that the resulting authority
record has not a different thesaurus declared in 040 $f. (Note
that 008/11 and 040 $f are indexed with the same search
field: subject-heading-thesaurus. So we search for the
second time in the same search field but with
'notdefined' == 008/11='z' instead of the original
thesaurus taken from subfield $2 of the bibliographic
field 6XX.) This means we have to explicitly check the
content of 040 $f of the resulting authority record.
Test plan:
==========
1. Turn on the LinkerConsiderThesaurus systempreference.
2. Create an authority record (e.g. TOPIC_TERM) with 008/11 = 'z' and
040 $f containing a thesaurus indication (e.g. 'kaba'), and a term
in field 150 $a (e.g. Early music).
3. Edit a bibliographic record: in field 650 put 2nd indicator '7',
the term 'Early music' in $a, and a different thesaurus code in $2
(e.g. 'dbn'). Click the 'Link authorities automatically' button.
Note that 650 'Early music' (dbn) has been linked with and authority
record 'Early music' (kaba). This is wrong.
4. Apply the patch ; restart_all ; reindex ES for authorities with:
koha-elasticsearch --rebuild -d -a kohadev
5. Repeat p. 3. Now the 650 'Early music' (dbn) has not been linked to
an authority record (650 - No matching authority found.)
Signed-off-by: Janusz Kaczmarek <januszop@gmail.com>
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
When building queries for matchign Zebra uses OR to combine searches - Elasticsearch
has been effectively using 'AND' - this patch checks whether and_or has been passed
and follows the variable when present.
To test:
1 - Have an import file of auths with multiple 035
2 - Setup an authority matching rule
Search index: other-control-number
Score: 1000
Tag: 035
Subfields: a
3 - Make sure Koha is using Elasticsearch
4 - Stage file without matching, import file
5 - Stage file again, using your matching rule
6 - Note all records match
7 - Setup Cataloging->Marc modification template
8 - Add an action to delete first 035 field
9 - Run it against the records you just imported
You can use a report:
SELECT authid FROM auth_header WHERE authid > #### (find the lowest new record number in batch you imported)
10 - Change matching rule in batch to none, then back to your rule
11 - No matches!
12 - Apply patch, restart all
13 - Set matchign rule to none, then back to your rule
14 - Matches!
Signed-off-by: Magnus Enger <magnus@libriotech.no>
Works as advertised!
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
to keep the syspref names consistent with the existings values. Also "ES" is not as self-documenting as "Elasticsearch".
No code change, so the test plan stays the same.
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
What this patch does is:
1 - Wraps the existing search code in a a "bool" compound query as a "must". This should not affect
relevancy or results of the existing searches.
2 - Before we clean/truncate terms, loop through the passed in search terms and indexes to build a
new 'should' query, using the 'match' on the specified index/field that is added to the 'bool' query from above.
This means that if a result from the original query is also returned here, that item will be boosted in the result.
For searches on 'keyword' or 'title', or if no index is set, we use 'title-cover' as the most narrow form of title
This query isn't going to help when users enter CCL (i.e. ti:To die for) and it won't boost titles from 505, series, etc
when doing a general search. Nor will it have a detrimental effect, it will only boost field matches
To test:
1 - Add a record with
245 $a novel
2 - Add a record with
245 $a A novel : $b about things / $c by me
3 - Search for: novel
- Above records are returned lower in the list (results #6 and #14 for me)
4 - Search for: a novel
- Above records are returned lower in the list (results #6 and #14 for me)
5 - Apply patch, restart all, enable ESBoostFieldMatch option
6 - Repeat searches, note exact titles are boosted
- For 'novel' record from #1 is #1 result and other is second
- For 'a novel' record form #2 is #1 result and other is third
7 - Disable the new pref and repeat steps 3 and 4 and get results as before patch
8 - Experiment with other searches, turning pref on and off to verify relevant titles are boosted when enabled
9 - Search results when disabled should return as before the patch
Signed-off-by: Matthias Meusburger <matthias.meusburger@biblibre.com>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Added head2 right before sub with missing POD
Signed-off-by: Baptiste Wojtkowski <baptiste.wojtkowski@biblibre.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
To test:
1 - Set search engine to ES
2 - Search the catalog for "*"
3 - Note results
4 - Edit first result
5 - Repeat search, results are reordered
6 - Apply patch
7 - Search for "*"
8 - Highest biblionumbers are returned first
9 - Edit first result, repeat search
10 - Order does not change
11 - Confirm sorting by author, title, etc passes the same trials
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Add a new hook elasticsearch_to_document in order to allow completing
the 'document' to be sent to Elasticsearch. Usage example: (1) add new
search fields, (2) normalize/sanitize existing fields, (3) use id fields
to retrieve info via web services in other repositories (say VIAF), (4)
download file referenced in a field, and index its content.
To test:
1. Apply the unit tests patch, and run it:
prove t/db_dependent/Koha/Plugins/Biblio_and_Items_plugin_hooks.t
=> FAIL: Tests fail! The hook is not implemented so the desired
2. Apply this patch, and then repeat 1:
=> SUCCESS: It works!
Signed-off-by: Shi Yao Wang <shi-yao.wang@inlibro.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds a new option to Elasticsearch to skip facets in build_query and
build_query_compat.
There should be no behavior change, however, queries to ES should be simpler
To test:
1 - Export some record from Koha
2 - Stage the records for import, matching on 999c
3 - Confirm expected matches
4 - Switch batch to use ISBN for matching
5 - Confirm expected matches
6 - Apply patches, restart all
7 - Switch batch to use 999c
8 - Confirm matches
9 - Switch batch to use ISBN
10 - Confirm matches
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
As elsewhere in the code, we should use a state variable for our
'bib_searcher' object to avoid overhead of extra ES connections.
Worth noting - we don't actually use the 'bib_searcher' at all during
record auto linking as we pass skipmetadata
To test:
0 - Apply patch
1 - Enable biblio auto linking
AutoLinkBiblios, CatalogModuleRelink, LinkerRelink
2 - Stage and import a marc file
3 - Confirm record authorities link as expected
4 - Search authorities in the staff interface
5 - Confirm usage counts are shown for results
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This commit is generated using:
% perl misc/devel/tidy.pl
*within* ktd, to get the same version of perltidy than what will be used
by our CI (currently v20230309).
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
After Bug 38416 Elasticsearch indexing explodes with oversized
records, especially with UTF encoded data.
In Koha::SearchEngine::Elasticsearch::marc_records_to_documents a
following snippet has been introduced:
my $usmarc_record = $record->as_usmarc();
my $decoded_usmarc_record = MARC::Record->new_from_usmarc($usmarc_record);
But if $record is oversized (> 99999 bytes), it is OK for MARC::Record
object, but not for $record->as_usmarc. The produced ISO 2709 string
is not correct and hence cannot be properly converted back to
MARC::Record object by new_from_usmarc.
The result in this case can be like:
UTF-8 "\x85" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35.
Since it is done without any eval / try, the whole reindex procedure
(for instance rebuild_elasticsearch.pl) is being randomly interrupted
with no explanation.
Test plan:
==========
Hard to reproduce. But the explanation together with discussion in Bug
38416 (from 2024-12-15) explains and justifies the need of this added
eval.
1. Have a standard KTD installation with Elasticsearch.
2. Use the provided test record - add it to Koha with
./misc/migration_tools/bulkmarcimport.pl -b -file test.xml -m=MARCXML
(have patience).
During load process you should see a message like:
UTF-8 "\xC4" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35.
3. The record should get biblionumber 439. Check in librarian interface with
http://<your_addreess>:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=439
that the record has been imported.
However, you should not be able to make a search for this record.
4. Try to reindex with:
./misc/search_tools/rebuild_elasticsearch.pl -b -bn 439
You should get a message like:
UTF-8 "\xC4" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35.
Again, no search results.
5. Apply the patch ; restart_all.
6. Repeat reindex with:
./misc/search_tools/rebuild_elasticsearch.pl -b -bn 439
There should be no warning now and you should be able to find the record.
Signed-off-by: Magnus Enger <magnus@libriotech.no>
Followed the test plan. Works as advertised.
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch makes the `_process_mappings()` method split the index values
in the event of them being bigger than the allowed 32766 bytes size.
To test:
1. Have KTD running with ES:
$ ktd --proxy --es7 up -d
2. Perform a search
3. Pick the first result for edition
4. Find a cool Wiki page with lots of paragraphs
5. Copy all of the paragraphs and put them on a 500$a field for the record.
6. Repeat 2
=> FAIL: The record is not found
7. Reindex manually:
$ ktd --shell
k$ perl misc/search_tools/rebuild_elasticsearch.pl --biblios --where "biblionumber=3" -v -v
=> FAIL: You get something like:
```
[22229] Committing final records...
One or more ElasticSearch errors occurred when indexing documents at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Indexer.pm line 148.
[22229] There were errors during indexing
Record #3 Document contains at least one immense term in field="note.raw" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[10, 109, 117, 115, 116, 97, 102, 97, 32, 102, 117, 101, 32, 101, 108, 32, 115, 101, 103, 117, 110, 100, 111, 32, 104, 105, 106, 111, 32, 100]...', original message: bytes can be at most 32766 in length; got 32771 (illegal_argument_exception) : max_bytes_length_exceeded_exception (bytes can be at most 32766 in length; got 32771)
[22229] Total 1 records indexed
```
8. Apply this patch
9. Repeat 7
=> SUCCESS: No error!
10. Repeat 2
=> SUCCESS: The record is indexed and can be found!
11. Sign off :-D
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This change failsover to MARCXML from USMARC if there are any
warnings generated by MARC::File::USMARC::decode when trying to
roundtrip the record.
Test plan:
0. Apply the patch
1. Setup your koha-testing-docker to use Elasticsearch
2. Create a new record with 15,000 characters in the 500$a field
3. Index that record
(e.g. perl misc/search_tools/rebuild_elasticsearch.pl --biblios -v -v)
4. Note that a warning saying the following appears:
"Warnings encountered while roundtripping a MARC record to/from USMARC.
Failing over to MARCXML"
5. View the "Elasticsearch record" on the detail page and note that the
marc_format is MARCXML
6. Perform a search for the record (the keyword should be something that
brings up other results too)
7. Note that the record appears correctly in the search results
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
The patch was corrupted, and was rewritten as the original one by
Hammat
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Test plan:
- Enable elasticsearch
- git to labels/label-item-search.pl
- search using after date only
=> check you get items with date-of-acquisition greater than after
date
- search using before date only
=> check you get items with date-of-acquisition less than before
date
- search using after and before date
=> check you get items with date-of-acquisition between after and
before
- Combine this searches with a specified index
- enable zebra and repeat these tests
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
https://bugs.koha-community.org/show_bug.cgi?id=30845
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
And copy fix across to Zebra search as well.
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch ensures the ShowHeadingUse system preference and feature works as expected when using Elasticsearch as the searchengine.
To test, follow the test plan at Bug 29990 Comment 9. Ensure the SearchEngine system preference is set to Elasticsearch.
Sponsored-by: Education Services Australia SCIS
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Some libraries ask for the ability to scan/browse the call numbers index.
Test plan:
==========
1. Have an installation with items with callnumbers (in ktd with standard
test data set: add some callnumbers to the items).
2. Go to Advanced search > More options; choose Call number in the first
drop list and mark Scan indexes. Enter one of the call numbers you
assigned to the items and perform the search. You should get no
results.
3. Apply the pach; restart all; when using ES:
koha-elasticsearch --rebuild -r -d kohadev
4. Repeat p. 2. You should get a list with at least one call number.
The link should lead you to the record(s) with items with the selected
call number.
This should work for ES and for Zebra, but for Zebra you would need to
replace /etc/koha/zebradb/ccl.properties with the repository version; and
you will get one token results on the list, like for other indexes.
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
We can now call metadata_record directly on the Koha::Biblio object.
This aptch updates all modules and controllers to use
Koha::Biblio->metadata_record directly where appropriate. The
exceptions are where we don't require any filtering or the filtering
makes sense to do after the initial fetch.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This change filters the values from the OpacHiddenItems syspref
out of their corresponding search results facets.
Note: This does not mean that all values from hidden items are filtered out of facets.
Rather, it just means that facet data that matches OpacHiddenItems is filtered out. This is
an imperfect fix, but it is a practical fix that has been requested by more than one library.
Test plan:
0. Apply the patch and koha-plack --restart kohadev
1. Go to http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=29
2. Add an item with an item type of "Maps"
3. Go to http://localhost:8081/cgi-bin/koha/admin/preferences.pl?op=search&searchfield=OpacHiddenItems
4. Fill in OpacHiddenItems with the following:
itype: ['MP']
5. Go to http://localhost:8080/cgi-bin/koha/opac-search.pl?idx=&q=test&weight_search=1
6. Note that "Maps" does not appear in the "Item types" facet
7. Go to http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=test
8. Note that "Maps" does appear in the "Item types" facet
9. ***Repeat this test plan using both Elasticsearch and Zebra indexing***
10. prove -v t/Koha/SearchEngine/Search.t
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
With the Bug 36574 we added a new ES search field isbn-all which makes it
possible to find records with the content of the subfield 020 $z
(invalid/cancelled ISBN) in general search (by keyword). But still, when
performing a search with ISBN-search these records will not be found.
Since some libraries deliberately want to find records with only 020 $a
with ISBN-search, a system preference would be needed here to be able to
customize the ISBN search.
The same for ISSNs, with one system preference.
Test plan:
==========
(ISBN)
------
1. Have the ktd with with ES enabled.
2. With standard ktd test data, try to perform a ISBN search from Advanced
search for 9780141930848. You should get no results.
In the general search input field (on the main page) try a search
nb:9780141930848 and/or isbn:9780141930848 - you should get no results.
3. Apply the patch, restart_all.
4. Repeat p. 2 -- you should still get no results.
5. Enable SearchCancelledAndInvalidISBNandISSN syspref.
6. Repeat p. 2 -- now you should get one record (having 9780141930848
in 020 $z).
(ISSN)
------
0. Return to main, delete the bug branch, reset_all == start from fresh.
1. Have the ktd with with ES enabled.
2. With standard ktd test data, try to perform a ISSN search from Advanced
search for 1089-6891. You should get no results.
In the general search input field (on the main page) try a search
ns:1089-6891 and/or issn:1089-6891 - you should get no results.
3. Apply the patch, restart_all.
If you have had to apply also the bug 36727, reindex ES with:
koha-elasticsearch --rebuild -r -d kohadev
(there is a modification of mappings.yaml)
4. Repeat p. 2 -- you should still get no results.
5. Enable SearchCancelledAndInvalidISBNandISSN syspref.
6. Repeat p. 2 -- now you should get one record (having 1089-6891
in 022 $y).
Sponsored-by: Ignatianum University in Cracow
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Tiny adjustment for very special and malicious cases needed in
Koha::SearchEngine::Elasticsearch::QueryBuilder::clean_search_term.
Extended unit tests added following Nick's suggestions in
Bug 33407 discussion.
Test plan
=========
1. Use ktd with Elasticsearch and ktd's test data.
2. Edit the 4th record (The C programming language) and add a semicolon
followed by a space before the slash in 245 $a to have:
The C programming language ; /
3. Make a search for the first ISBD zone taken verbatim from the forth
record:
The C programming language ; / Brian W. Kernighan, Dennis M. Ritchie
There should be no result.
4. Apply the patch, restart plack.
5. Repeat the search. You should get the record.
Sponsored-by: Ignatianum University in Cracow
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Test plan:
With ElasticSearch enabled,
* Perform a search using the default sort order
(i.e. 'relevance').
* Verify that no warnings are generated in
plack-intranet-error.log
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Test plan:
1. Apply patch
2. Rerun and make sure the error "Use of uninitialized value $sub6 in pattern match (m//) at /usr/share/koha/lib/Koha/SearchEngine/Elastricsearch.pm" is no longer present.
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Initial testing done, it works well, needs an additional sign off from an external party
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
The highlight logged-in library in facets feature does not work with
Elasticsearch. This is because facet_title_value contains an additional
" (count)" information and so gives negative results in comparisons made
in facets.inc (line 107).
Test plan:
==========
1. Have a test installation with Elastiscsearch.
2. Set system preference 'DisplayLibraryFacets' to 'both home and holding libraries'
3. Perform a seach that will find items of your logged-in library.
In KTD, search for 'branch:CPL'
4. Check the facet (home and holding libraries) of your logged-in
library is not highlighted
5. Apply the patch ; restart_all.
6. Repeat p. 3. Check the facet (home and holding libraries)--you
should see your logged-in branch highlighted now.
Sponsored-by: Ignatianum University in Cracow
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Like IncludeSeeFromInSearches adds see from heading in search (4xx),
new system preference IncludeSeeAlsoFromInSearches adds see also from heading (5xx).
Test plan:
Test on both Elasticsearch and Zebra:
1)
1.1) Create an authority record with heading, see from heading 4xx and see also from heading 5xx
1.2) Use this authority in a biblio record
2)
2.1) Set IncludeSeeFromInSearches and IncludeSeeAlsoFromInSearches to "Don't include"
2.3) Rebuild search engine
2.4) Perform a search on heading text => you find the biblio record
2.5) Perform a search on see from heading => you do not find the biblio record
2.6) Perform a search on see also from heading => you do not find the biblio record
3)
3.1) Set IncludeSeeFromInSearches and IncludeSeeAlsoFromInSearches to "Include"
3.2) Rebuild search engine
3.3) Perform a search on heading text => you find the biblio record
3.4) Perform a search on see from heading => you find the biblio record
3.5) Perform a search on see also from heading => you find the biblio record
Signed-off-by: Frédéric Demians <f.demians@tamil.fr>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
With ES and QueryAutoTruncate on, a search with punctuation surrounded
by spaces (like ISBD punctuation copied from other catalogue or
bibliography) returns no results. E.g.: in a search for "Maria Stuart ;
Die Jungfrau von Orleans / Schiller" (coded in record as 245 10 $a Maria
Stuart ; $b Die Jungfrau von Orleans / $c Schiller) -- both semicolon
and slash cause problems. One had to remove them manually to get
results which is not what is desired.
Test plan
=========
1. Use ktd with Elasticsearch and ktd's test data
(http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=1):
2. Make a search for the first ISBD zone taken verbatim from the first record:
E Street shuffle : the glory days of Bruce Springsteen & the E Street Band / Clinton Heylin
There should be no result.
3. Apply teh patch, restart plack.
4. Repeat the search. You should get the record (and onother one).
Sponsored-by: Ignatianum University in Cracow
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Test plan:
Export the ES fields/mappings using
misc/search_tools/export_elasticsearch_mappings.pl > mappings.yaml
Copy this file to admin/searchengine/elasticsearch/mappings.yaml
Reset the mapping from the UI
Notice that the links to the AV categories have been preserved
Sponsored-by: The Research University in the Helmholtz Association (KIT)
Signed-off-by: Lukas Koszyk <lukasz.koszyk@kit.edu>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This new feature adds the ability to link an ES facet to an authorised
value category, in order to display a description instead of the code.
Test plan:
1. Go to the ES mapping admin page
2. Notice the new "Authorized value category" column in the facets
config table
3. Create a new facet that is linked to a search field containing values
from an authorised value category. Pick this category in the list
4. Save
5. Reindex
6. Notice that the descriptions are now displayed (staff and OPAC) in
the search result page.
QA: See the related bug report for possible improvements. It's not
easy at all to deal with the "specials" here, a lot of changes must be
done in how we fetch/cache the AVs.
Sponsored-by: The Research University in the Helmholtz Association (KIT)
Signed-off-by: Lukas Koszyk <lukasz.koszyk@kit.edu>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Currently we only remove non-filing characters for sort fields, however, this can make searching difficult.
This patch adds the filing form to the index as well to aid in searching.
To test:
0 - Setup Koha with Elasticsearch
1 - Import the sample record on this report: "L'amour de l'art"
2 - Search for "amour de l'art" - no results
3 - Apply patch
4 - Reindex
5 - Search for "amour de l'art" - result!
6 - Search for "title:amour de l'art" - result!
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
We can know add/remove homelibrary and holdinglibray from the config
page, this syspref is no longer needed for ES
Sponsored-by: The Research University in the Helmholtz Association (KIT)
Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This new feature allows to manage facets for Elasticsearch from the
administration page.
Prior to this the facet fields were hardcoded in the codebase.
Librarians can then add/remove facet fields and modify their label.
Test plan:
1. Create a new search field and set it a label different than its name.
2. Save
3. Go the bibliographic mapping tab
4. Add 1+ mapping for this search field (Searchable and facetable must be "Yes")
5. Add, reorder, remove new facets
6. Save and reindex your records
7. Search and notice the new facet list
QA: There are several wrong things in this area (ES + facets code,
everything, pm, pl, tt AND on this administration page). I have done my
best to clean the code as much as I could and keep the code cleaner
after than before. But there are still a lot to do.
There are still inconsistencies on this page (like we need to save to
see the changes applied to the other tables), but this is clearly out of
the scope of this bug report.
Another enhancement would be to move the facet list to a different DB
table, that could bring more flexibility:
* display or not (could be opac/staff/both/none)
* define the size per field (number of facet to display)
* order: move search_field.facet_order to this new table.
But, again, it's a lot more work.
More work is done in this area, please see related bug reports.
Sponsored-by: The Research University in the Helmholtz Association (KIT)
Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds geosearch to Koha (using Elasticsearch 7). ElasticSearch
search_mappings get new types to store lat/lon, which can be indexed
from MARC 034$s and 034$t. There is a small change to the DB to allow a
new value in search_field.type ENUM.
The QueryBuilder is extended to allow for building advanced
ElasticSearch Querys (eg geo_distance) that cannot be represented in a
simple string query. The UI for searching (including showing the results
on a OSM/Leaflet map) is implemented in a separate plugin
(https://github.com/HKS3/HKS3GeoSearch)
Test Plan:
* make sure you're running ElasticSearch 7
(eg via `curl http://es:9200?pretty | grep number`)
* apply patch
* got to a Framework, check Editor for 034$s and 034$t and save
* got to some books (in the correct framework) and enter some lat and lon into 034$s and 034$t (for example lat=48.216, lon=16.395)
* Run the elasticsearch indexer, maybe limited on the books you edited (-bn 123 -bn 456):
misc/search_tools/rebuild_elasticsearch.pl -b -v
* You can check if the indexing worked by inspecting the document in elasticsearch:
* get the biblionumber (eg 123)
* curl http://es:9200/koha_kohadev_biblios/_doc/123?pretty | grep -A5 geolocation
* You should get back a JSON fragment containing the lat/lon you stored
* You can query elasticsearch directly:
* Run the following curl command, but adapt the value for lat/lng and/or the distance (in meters)
* curl -X GET "http://es:9200/koha_kohadev_biblios/_search?pretty" -H 'Content-Type: application/json' -d '{"query": {"bool":{"must":{"match_all":{}},"filter":{"geo_distance":{"distance":100000,"geolocation":{"lat":48.2,"lon":16.4}}}}}}'
* To run the search via Koha, you need to either install and use https://github.com/HKS3/HKS3GeoSearch or create a handcrafted query string:
* handcrafted query string:
* /cgi-bin/koha/opac-search.pl?advsearch=1&idx=geolocation&q=lat:48.25+lng:18.35+distance:100km&do=Search
* HKS3GeoSearch
* install the plugin and enable it
* got to OPAC / Advanced Search
* There is a new input box "Geographic Search" where you can enter lat/long/radius
* On the search result page a map is shown with pins for each found biblioitem
Sponsored-by: ZAMG - Zentralanstalt für Meterologie und Geodynamik, Austria - https://www.zamg.ac.at/
Sponsored-by: Geosphere - https://www.geosphere.at/
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Additional finetuning:
- Fix update and remove fixed fixme
- Update test count as well
- fix last small issues raised in Comment 23
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Koha with Zebra prevented auto truncation in some circumstances -- see
the first return for ccl inside C4::Search:: buildQuery, before applying
auto truncation, and setting $auto_truncation = 0 for some search
fields.
Koha with ElasticSearch applies auto truncation for all search fields,
not paying attention to these special cases when it should not be done.
This leads to various problems as described in bug 26508, 26608, etc.
The solution would be to prevent auto truncation for certain search
fields, above all – the identifiers. In addition, under no
circumstances should a search field other than of text type be truncated
(an attempt to truncate would generate an exception from ElasticSearch,
e.g. number_format_exception for integer).
Test plan
=========
0. Use the test data sample provided in the bug report. Comment 16 in
the ticket.
Scenario A (authority)
----------------------
1. Enable Elasticsearch engine.
2. Have like 100+ bibliographic records and properly linked authority
records.
3. Reindex ElasticSearch if needed.
4. Enable QueryAutoTruncate systempreference.
5. Find the authority record #1 and note the number of linked biblio
records.
6. Show the detail of authority #1 and compare the number of linked
biblio records.
7. If in the database there are authority records with ids =~ /^1/ (i.e.
10, 11, 12, ..., 100, 101, ...) linked to the biblio records you get
two different numbers of linked records.
8. Also, as lists of linked biblio records (via link: Used in N
record(s) from results view and detail view) you will get two
different sets of biblio records. In particular, on the list generated
from detail view (authorities/detail.pl?authid=1) you will get biblio
records that are in fact not linked to the auth #1 (the list is
generated with &q=an:1).
8.99. Skip to scenario B and come back here after finishing to not have
to unapply the patches and restart services for nothing.
9. Apply this patch.
10. Counts and list of linked biblios should be ok.
Scenario B (analytics)
----------------------
1. Enable Elasticsearch engine.
2. Have three monographic bibliographic records with 001 = 1, 10, 100
(i.e. =~ /^1/).
3. Have an analytical record with 773 $w = 1 (in the test data set - biblio 896).
4. Enable QueryAutoTruncate and UseControlNumber systempreference.
6. Find the analytical record and click on the link generated from 773,
i.e. (In: ... --> catalogue/search.pl?q=Control-number:1). You should see
a list of 100+ records (001 = 1, 10, 100) instead of one (001 = 1).
7. From the biblio # 1 try to go to the analytice records (with Show analytics
link) - you should get 60+ records with from different host records (773 $t)
-- instead of one.
8. Apply this patch. Make sure that control-number and record-control-number
are included in ESPreventAutoTruncate syspref.
9. Repeat steps 6 and 7. You should arrive at the right host record in p. 6
and at one analytical records when looking for analytical records in p. 7
(or more than one, but right, if you modified the test data set).
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Whne performing batch operations we can send a large numebr of records for reindexing at once.
Currently this can create requetss that are too large for Elasticsearch to process. We need
to break these requests into chunks/
This patch adds a chunk_size configuration to the elasticsearch stanza in koha-conf.xml
If blank we default to 5000.
To test:
0 - Have Koha using Elasticsearch
1 - Create and download a report of all barcodes:
SELECT barcode FROM items
2 - Batch modify these items
3 - Note a single ESindexing job is created
4 - Create and download a report of all authority ids:
SELECT auth_header.authid FROM auth_header
5 - Setup a marc modification template, and batch modify all the authorities
6 - Again note a single ES backgorund job is created
7 - Apply patch
8 - Repeat the modifications above - you still get a single job
9 - Edit koha-conf.xml and add <chunk_size>250</chunk_size> to elasticsearch stanza
10 - Repeat modifications - you now get several background ES jobs
11 - prove -v t/db_dependent/Koha/SearchEngine/Elasticsearch/Indexer.t
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
For some reasons, with Search::Elasticsearch 8.0 JSON::true is
interpreted as 1 and Elasticsearch will always respond that it found "at
least 1 record". Something like this:
{
total => {
value => 1,
relation => 'gte'
}
}
Replacing JSON::true by \1 works with every version of
Search::Elasticsearch I tested (6.80, 7.717, 8.0)
Also worth noting:
Search::Elasticsearch will stop being supported by Elastic
https://github.com/elastic/elasticsearch-perl/issues/220
We might need to find an alternative for future versions of
Elasticsearch
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Philip Orr <philip.orr@lmscloud.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Elasticsearch 8 removed the support of types. This patch adapts the
requests accordingly.
With this patch, Koha will still be compatible with Elasticsearch 7.x
but will no longer work with Elasticsearch 6.x
Since Elasticsearch 6.x is no longer maintained, this should not be a
problem.
https://www.elastic.co/guide/en/elasticsearch/reference/7.17/removal-of-types.html
Tested with Search::Elasticsearch 6.80 and 7.717
Test plan:
1. Install Elasticsearch 8
(use docker image: docker.io/koha/elasticsearch-icu:8.x)
2. Change the elasticsearch server location in $KOHA_CONF
3. Run misc/search_tools/rebuild_elasticsearch.pl -r -b -v
It should fail
4. Apply the patch
5. Run misc/search_tools/rebuild_elasticsearch.pl -r -b -v
It should end with "Total XXX records indexed"
6. Try to search some biblios and verify that it works the same as
before
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Philip Orr <philip.orr@lmscloud.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch uses the filter option from the mappings to add a value_callback
to reord processing for indexing.
Fields defined with 'punctuation' filter will have all punctuation stripped when
conveted to documents.
Tests are updated.
Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Test plan:
1) Launch k-t-d with ES:
-- ktd --es7 pull
-- ktd --es7 up
2) Visit /cgi-bin/koha/authorities/authorities-home.pl and search for 'tim'
3) Edit that, add a dash '-' to 100$a, like 'Bunce, Tim - name'"
4) Do a authority search for 'tim - name', notice it errors out
5) Apply patch
6) Perform the same search, notice it doesn't error and returns the correct record
Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Instead of typing the case sensitive Control-number each time.
4 strikes instead of 15 on your keyboard. Wow! Gain of 73%.
Test plan:
Copy ccl.properties to /etc/koha/zebradb, restart Zebra and
search for cnum=SOME_ID in opac or intranet.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Nick Clemens
Jonathan Druart
Thomas Klausner
Frédéric Demians
Koha Development Team
Janusz Kaczmarek
Tomas Cohen Arazi
David Cook
Olivier V
Alex Arnaud
Aleisha Amohia
Martin Renvoize
Andreas Jonsson
Catalyst Bug Squasher
Kyle M Hall
Fridolin Somers
Lari Strand
Katrin Fischer
Mark Hofstetter
Julian Maurice
Marcel de Rooy