Add explicit `use JSON::PP ()` to Elasticsearch.pm and the test file
to make the dependency on JSON::PP::false visible rather than relying
on YAML::XS loading it as a side effect.
Extend the get_elasticsearch_mappings() test subtest to assert that
date_detection and numeric_detection are disabled at the mapping root.
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
When ElasticsearchMARCFormat is set to ARRAY, the marc_data_array
field uses dynamic:true mapping. Elasticsearch auto-detects field
types based on the first document indexed. If a MARC subfield value
looks like a date (e.g. 2024-01-15 in 952$e), ES locks that
subfield to type date. Subsequent records with non-date values in
the same subfield fail to index with a document_parsing_exception.
This patch sets date_detection and numeric_detection to false at
the mapping root level, so ES treats all dynamically mapped fields
as text instead of guessing types.
Test plan:
1. $ ktd --search-engine es8 --name test-es --proxy up -d
$ ktd --name test-es --wait-ready 180
$ ktd --name test-es --shell
2. k$ koha-mysql kohadev -e "UPDATE systempreferences
SET value='ARRAY'
WHERE variable='ElasticsearchMARCFormat';"
k$ echo 'flush_all' | nc -q1 memcached 11211
k$ koha-mysql kohadev -e "DELETE FROM biblio_metadata;
DELETE FROM items; DELETE FROM biblioitems;
DELETE FROM biblio;"
k$ rebuild_elasticsearch.pl -d -r --biblios
3. Add a record with date-like 952$e and index it:
k$ koha-shell kohadev -p -c 'perl -MC4::Biblio
-MMARC::Record -MMARC::Field -MKoha::Item -e "
my \$r = MARC::Record->new();
\$r->append_fields(
MARC::Field->new(q{245},q{1},q{0},a=>q{Record one}),
MARC::Field->new(q{942},q{ },q{ },c=>q{BK}));
my (\$bn) = C4::Biblio::AddBiblio(\$r, q{});
Koha::Item->new({biblionumber=>\$bn,
biblioitemnumber=>\$bn, homebranch=>q{CPL},
holdingbranch=>q{CPL}, itype=>q{BK},
booksellerid=>q{2024-01-15}})->store;
print qq{Added biblio \$bn\n};"'
k$ rebuild_elasticsearch.pl --biblios -bn <biblionumber>
4. Check the mapping for 952$e:
k$ curl -s es:9200/koha_kohadev_biblios/_mapping/field/\
marc_data_array.fields.952.subfields.e
=> FAIL: type is "date" (should be "text")
5. Add a record with non-date 952$e and index it:
k$ koha-shell kohadev -p -c 'perl -MC4::Biblio
-MMARC::Record -MMARC::Field -MKoha::Item -e "
my \$r = MARC::Record->new();
\$r->append_fields(
MARC::Field->new(q{245},q{1},q{0},a=>q{Record two}),
MARC::Field->new(q{942},q{ },q{ },c=>q{BK}));
my (\$bn) = C4::Biblio::AddBiblio(\$r, q{});
Koha::Item->new({biblionumber=>\$bn,
biblioitemnumber=>\$bn, homebranch=>q{CPL},
holdingbranch=>q{CPL}, itype=>q{BK},
booksellerid=>q{My Vendor Name}})->store;
print qq{Added biblio \$bn\n};"'
k$ rebuild_elasticsearch.pl --biblios -bn <biblionumber> -v -v -v
=> FAIL: document_parsing_exception - failed to parse date field
[My Vendor Name]
6. Apply this patch, repeat from step 2
7. Repeat step 4:
=> SUCCESS: type is "text"
8. Repeat step 5:
=> SUCCESS: No indexing errors
9. Sign off :-D
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Refactor marc_record_to_documents to avoid code duplicaiton.
To test:
1) Verify tests in t/db_dependant/Koha/SearchEngine/Elasticsearch.t all
pass
2) Apply patch
3) Veriy tests in t/db_dependant/Koha/SearchEngine/Elasticsearch.t
still pass
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Test plan:
1- Change to a language other than english
2- Search in the catalog with "!"
3- Notice the string "Unable to understand your search query, please
rephrase and try again." is not translated
4- Apply the patch
5- Add translation in .po file of used language for "Unable to
understand your search query, please rephrase and try again.\n"
6- Update translation
7- Redo the search and notice the string is now translated
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
The Koha_Main_OS1 and Koha_Main_OS2 jobs are currently failing on
Jenkins.
```
[NoNodes] ** No nodes are available: [http://es:9200], called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Indexer.pm line 417. # Looks like your test exited with 11 just after 3.
```
We need to workaround Elasticsearch/OpenSearch product check
incompatibility.
This patch suggests to monkey-patch
Search::Elasticsearch::Role::Cxn::process_response to disable the strict
product header check at runtime. This allows the same codebase to talk
to both Elasticsearch 8+ and OpenSearch without forking or patching
upstream.
Test plan:
0. Set KOHA_ELASTICSEARCH=yes in ktd's .env
1. Startup ktd with the compose file for OS1
ktd -f docker-compose.os1.yml up
2. Set SearchEngine=Elasticsearch
3. reindex
ktd --shell
perl misc/search_tools/rebuild_elasticsearch.pl -a -b
4. run the tests
prove -r t/db_dependent/Koha/SearchEngine/Elasticsearch
5. search!
http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=perl
should show results
6. ktd down then repeat using the compose file for OS2 (docker-compose.os2.yml), ES6 and ES7
for ES6 and ES7 you can use the --es6 and --es7 ktd flag.
IF you get a 403 during the reindex you can try to add
cluster.routing.allocation.disk.threshold_enabled: "false"
to the environment section of the compose file.
Or
```
curl -X PUT http://es:9200/_cluster/settings \
-H 'Content-Type: application/json' \
-d '{
"persistent": {
"cluster.blocks.create_index": false
}
}'
```
OS thinks we lack disk space and force all indices into read-only.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
With bug 30280 and subsequent patches we attempt to respect the
thesaurus indication when linking with LinkerConsiderThesaurus on.
However, there are still cases when we link wrong.
This happens when thesaurus in bibliographic field 6XX is defined by
subfield $2. In such a case the authority record should have 008/11 = 'z'
and, according to https://www.loc.gov/marc/authority/ad008.html
(cf. comment for 008/11 = 'z': "A MARC code for the conventions used to
formulate the heading *may* be contained in subfield $f"), may or may have
thesaurus declared in 040 $f.
This is why we make a second search attempt in C4::Heading::_search,
but doing so we have to make sure that the resulting authority
record has not a different thesaurus declared in 040 $f. (Note
that 008/11 and 040 $f are indexed with the same search
field: subject-heading-thesaurus. So we search for the
second time in the same search field but with
'notdefined' == 008/11='z' instead of the original
thesaurus taken from subfield $2 of the bibliographic
field 6XX.) This means we have to explicitly check the
content of 040 $f of the resulting authority record.
Test plan:
==========
1. Turn on the LinkerConsiderThesaurus systempreference.
2. Create an authority record (e.g. TOPIC_TERM) with 008/11 = 'z' and
040 $f containing a thesaurus indication (e.g. 'kaba'), and a term
in field 150 $a (e.g. Early music).
3. Edit a bibliographic record: in field 650 put 2nd indicator '7',
the term 'Early music' in $a, and a different thesaurus code in $2
(e.g. 'dbn'). Click the 'Link authorities automatically' button.
Note that 650 'Early music' (dbn) has been linked with and authority
record 'Early music' (kaba). This is wrong.
4. Apply the patch ; restart_all ; reindex ES for authorities with:
koha-elasticsearch --rebuild -d -a kohadev
5. Repeat p. 3. Now the 650 'Early music' (dbn) has not been linked to
an authority record (650 - No matching authority found.)
Signed-off-by: Janusz Kaczmarek <januszop@gmail.com>
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Added head2 right before sub with missing POD
Signed-off-by: Baptiste Wojtkowski <baptiste.wojtkowski@biblibre.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Add a new hook elasticsearch_to_document in order to allow completing
the 'document' to be sent to Elasticsearch. Usage example: (1) add new
search fields, (2) normalize/sanitize existing fields, (3) use id fields
to retrieve info via web services in other repositories (say VIAF), (4)
download file referenced in a field, and index its content.
To test:
1. Apply the unit tests patch, and run it:
prove t/db_dependent/Koha/Plugins/Biblio_and_Items_plugin_hooks.t
=> FAIL: Tests fail! The hook is not implemented so the desired
2. Apply this patch, and then repeat 1:
=> SUCCESS: It works!
Signed-off-by: Shi Yao Wang <shi-yao.wang@inlibro.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This commit is generated using:
% perl misc/devel/tidy.pl
*within* ktd, to get the same version of perltidy than what will be used
by our CI (currently v20230309).
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
After Bug 38416 Elasticsearch indexing explodes with oversized
records, especially with UTF encoded data.
In Koha::SearchEngine::Elasticsearch::marc_records_to_documents a
following snippet has been introduced:
my $usmarc_record = $record->as_usmarc();
my $decoded_usmarc_record = MARC::Record->new_from_usmarc($usmarc_record);
But if $record is oversized (> 99999 bytes), it is OK for MARC::Record
object, but not for $record->as_usmarc. The produced ISO 2709 string
is not correct and hence cannot be properly converted back to
MARC::Record object by new_from_usmarc.
The result in this case can be like:
UTF-8 "\x85" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35.
Since it is done without any eval / try, the whole reindex procedure
(for instance rebuild_elasticsearch.pl) is being randomly interrupted
with no explanation.
Test plan:
==========
Hard to reproduce. But the explanation together with discussion in Bug
38416 (from 2024-12-15) explains and justifies the need of this added
eval.
1. Have a standard KTD installation with Elasticsearch.
2. Use the provided test record - add it to Koha with
./misc/migration_tools/bulkmarcimport.pl -b -file test.xml -m=MARCXML
(have patience).
During load process you should see a message like:
UTF-8 "\xC4" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35.
3. The record should get biblionumber 439. Check in librarian interface with
http://<your_addreess>:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=439
that the record has been imported.
However, you should not be able to make a search for this record.
4. Try to reindex with:
./misc/search_tools/rebuild_elasticsearch.pl -b -bn 439
You should get a message like:
UTF-8 "\xC4" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35.
Again, no search results.
5. Apply the patch ; restart_all.
6. Repeat reindex with:
./misc/search_tools/rebuild_elasticsearch.pl -b -bn 439
There should be no warning now and you should be able to find the record.
Signed-off-by: Magnus Enger <magnus@libriotech.no>
Followed the test plan. Works as advertised.
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch makes the `_process_mappings()` method split the index values
in the event of them being bigger than the allowed 32766 bytes size.
To test:
1. Have KTD running with ES:
$ ktd --proxy --es7 up -d
2. Perform a search
3. Pick the first result for edition
4. Find a cool Wiki page with lots of paragraphs
5. Copy all of the paragraphs and put them on a 500$a field for the record.
6. Repeat 2
=> FAIL: The record is not found
7. Reindex manually:
$ ktd --shell
k$ perl misc/search_tools/rebuild_elasticsearch.pl --biblios --where "biblionumber=3" -v -v
=> FAIL: You get something like:
```
[22229] Committing final records...
One or more ElasticSearch errors occurred when indexing documents at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Indexer.pm line 148.
[22229] There were errors during indexing
Record #3 Document contains at least one immense term in field="note.raw" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[10, 109, 117, 115, 116, 97, 102, 97, 32, 102, 117, 101, 32, 101, 108, 32, 115, 101, 103, 117, 110, 100, 111, 32, 104, 105, 106, 111, 32, 100]...', original message: bytes can be at most 32766 in length; got 32771 (illegal_argument_exception) : max_bytes_length_exceeded_exception (bytes can be at most 32766 in length; got 32771)
[22229] Total 1 records indexed
```
8. Apply this patch
9. Repeat 7
=> SUCCESS: No error!
10. Repeat 2
=> SUCCESS: The record is indexed and can be found!
11. Sign off :-D
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This change failsover to MARCXML from USMARC if there are any
warnings generated by MARC::File::USMARC::decode when trying to
roundtrip the record.
Test plan:
0. Apply the patch
1. Setup your koha-testing-docker to use Elasticsearch
2. Create a new record with 15,000 characters in the 500$a field
3. Index that record
(e.g. perl misc/search_tools/rebuild_elasticsearch.pl --biblios -v -v)
4. Note that a warning saying the following appears:
"Warnings encountered while roundtripping a MARC record to/from USMARC.
Failing over to MARCXML"
5. View the "Elasticsearch record" on the detail page and note that the
marc_format is MARCXML
6. Perform a search for the record (the keyword should be something that
brings up other results too)
7. Note that the record appears correctly in the search results
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Test plan:
1. Apply patch
2. Rerun and make sure the error "Use of uninitialized value $sub6 in pattern match (m//) at /usr/share/koha/lib/Koha/SearchEngine/Elastricsearch.pm" is no longer present.
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Initial testing done, it works well, needs an additional sign off from an external party
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Like IncludeSeeFromInSearches adds see from heading in search (4xx),
new system preference IncludeSeeAlsoFromInSearches adds see also from heading (5xx).
Test plan:
Test on both Elasticsearch and Zebra:
1)
1.1) Create an authority record with heading, see from heading 4xx and see also from heading 5xx
1.2) Use this authority in a biblio record
2)
2.1) Set IncludeSeeFromInSearches and IncludeSeeAlsoFromInSearches to "Don't include"
2.3) Rebuild search engine
2.4) Perform a search on heading text => you find the biblio record
2.5) Perform a search on see from heading => you do not find the biblio record
2.6) Perform a search on see also from heading => you do not find the biblio record
3)
3.1) Set IncludeSeeFromInSearches and IncludeSeeAlsoFromInSearches to "Include"
3.2) Rebuild search engine
3.3) Perform a search on heading text => you find the biblio record
3.4) Perform a search on see from heading => you find the biblio record
3.5) Perform a search on see also from heading => you find the biblio record
Signed-off-by: Frédéric Demians <f.demians@tamil.fr>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Test plan:
Export the ES fields/mappings using
misc/search_tools/export_elasticsearch_mappings.pl > mappings.yaml
Copy this file to admin/searchengine/elasticsearch/mappings.yaml
Reset the mapping from the UI
Notice that the links to the AV categories have been preserved
Sponsored-by: The Research University in the Helmholtz Association (KIT)
Signed-off-by: Lukas Koszyk <lukasz.koszyk@kit.edu>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Currently we only remove non-filing characters for sort fields, however, this can make searching difficult.
This patch adds the filing form to the index as well to aid in searching.
To test:
0 - Setup Koha with Elasticsearch
1 - Import the sample record on this report: "L'amour de l'art"
2 - Search for "amour de l'art" - no results
3 - Apply patch
4 - Reindex
5 - Search for "amour de l'art" - result!
6 - Search for "title:amour de l'art" - result!
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This new feature allows to manage facets for Elasticsearch from the
administration page.
Prior to this the facet fields were hardcoded in the codebase.
Librarians can then add/remove facet fields and modify their label.
Test plan:
1. Create a new search field and set it a label different than its name.
2. Save
3. Go the bibliographic mapping tab
4. Add 1+ mapping for this search field (Searchable and facetable must be "Yes")
5. Add, reorder, remove new facets
6. Save and reindex your records
7. Search and notice the new facet list
QA: There are several wrong things in this area (ES + facets code,
everything, pm, pl, tt AND on this administration page). I have done my
best to clean the code as much as I could and keep the code cleaner
after than before. But there are still a lot to do.
There are still inconsistencies on this page (like we need to save to
see the changes applied to the other tables), but this is clearly out of
the scope of this bug report.
Another enhancement would be to move the facet list to a different DB
table, that could bring more flexibility:
* display or not (could be opac/staff/both/none)
* define the size per field (number of facet to display)
* order: move search_field.facet_order to this new table.
But, again, it's a lot more work.
More work is done in this area, please see related bug reports.
Sponsored-by: The Research University in the Helmholtz Association (KIT)
Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds geosearch to Koha (using Elasticsearch 7). ElasticSearch
search_mappings get new types to store lat/lon, which can be indexed
from MARC 034$s and 034$t. There is a small change to the DB to allow a
new value in search_field.type ENUM.
The QueryBuilder is extended to allow for building advanced
ElasticSearch Querys (eg geo_distance) that cannot be represented in a
simple string query. The UI for searching (including showing the results
on a OSM/Leaflet map) is implemented in a separate plugin
(https://github.com/HKS3/HKS3GeoSearch)
Test Plan:
* make sure you're running ElasticSearch 7
(eg via `curl http://es:9200?pretty | grep number`)
* apply patch
* got to a Framework, check Editor for 034$s and 034$t and save
* got to some books (in the correct framework) and enter some lat and lon into 034$s and 034$t (for example lat=48.216, lon=16.395)
* Run the elasticsearch indexer, maybe limited on the books you edited (-bn 123 -bn 456):
misc/search_tools/rebuild_elasticsearch.pl -b -v
* You can check if the indexing worked by inspecting the document in elasticsearch:
* get the biblionumber (eg 123)
* curl http://es:9200/koha_kohadev_biblios/_doc/123?pretty | grep -A5 geolocation
* You should get back a JSON fragment containing the lat/lon you stored
* You can query elasticsearch directly:
* Run the following curl command, but adapt the value for lat/lng and/or the distance (in meters)
* curl -X GET "http://es:9200/koha_kohadev_biblios/_search?pretty" -H 'Content-Type: application/json' -d '{"query": {"bool":{"must":{"match_all":{}},"filter":{"geo_distance":{"distance":100000,"geolocation":{"lat":48.2,"lon":16.4}}}}}}'
* To run the search via Koha, you need to either install and use https://github.com/HKS3/HKS3GeoSearch or create a handcrafted query string:
* handcrafted query string:
* /cgi-bin/koha/opac-search.pl?advsearch=1&idx=geolocation&q=lat:48.25+lng:18.35+distance:100km&do=Search
* HKS3GeoSearch
* install the plugin and enable it
* got to OPAC / Advanced Search
* There is a new input box "Geographic Search" where you can enter lat/long/radius
* On the search result page a map is shown with pins for each found biblioitem
Sponsored-by: ZAMG - Zentralanstalt für Meterologie und Geodynamik, Austria - https://www.zamg.ac.at/
Sponsored-by: Geosphere - https://www.geosphere.at/
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Additional finetuning:
- Fix update and remove fixed fixme
- Update test count as well
- fix last small issues raised in Comment 23
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch uses the filter option from the mappings to add a value_callback
to reord processing for indexing.
Fields defined with 'punctuation' filter will have all punctuation stripped when
conveted to documents.
Tests are updated.
Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
In MARC 21, having title search field defined in mappings.yaml as 245abp
(more than one subfield), the Koha generated title__sort will have not
only $a, but also other subfields cut by ind2 characters, i.e.:
245 04 $a Die Renaissance : $b Architektur, Plastik, Malerei, Illustrationen, Zeichnungen
becomes as title__sort:
Renaissance : itektur, Plastik, Malerei, Illustrationen, Zeichnungen
instead of:
Renaissance : Architektur, Plastik, Malerei, Illustrationen, Zeichnungen
This is because copying an array of pointers in Perl causes you to
point to the same (anonymous) element. Therefore, the intended
modification only for subfield "a"
(Koha/SearchEngine/Elasticsearch.pm, ~line 1157) causes in fact
modification for all the subfields sharing the same mapping
definition and introduction of the nonfiling_characters_indicator to
all subfields 245 when processing 245abp. And this is obviously not
what is intended and results (in some cases) with bad sort order
when sorting by titles.
Test plan
=========
1. Have a standard Koha ES installation. In mappings.yaml
you should have 245abp for title search field.
2. Insert / download / modify a record with 245 $b or $p,
an article at the beginning of the title, and correctly
set 2nd indicator. For instance:
245 04 $a Die Renaissance : $b Architektur, Plastik, Malerei, Illustrationen, Zeichnungen
3. In ES, with Kibana or culr, control the content of
title__sort field for the related document. It will be:
Renaissance : itektur, Plastik, Malerei, Illustrationen, Zeichnungen
4. Apply the patch, full reindex ES (--rebuild -r -d -a -b)
5. In ES, with Kibana or culr, control the content of
title__sort field for the related document. It should
be now correct, i.e.:
Renaissance : Architektur, Plastik, Malerei, Illustrationen, Zeichnungen
Signed-off-by: Nick <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Before this patch we used two indexes for the thesaurus values, we can
simply index both needed fields into a single index and just form the
search correctly.
This patch also ensures we pass the 'thesaurus' vlaue for the heading
directly to the query builder - for zebra it goes through, and for ES
we convert it to the expected code.
This patch also moves the necessary mappings out of the user definable
mappings and hardcodes them. There is precedent for this with
'match-heading', it ensures matching works as expected
To test:
1 - Follow previous test plan in Zebra and ES
Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Previous commit correctly lists Zebra availability test, but was a
bad edit. This fixes the availability test to use itemlost
Additiionally, tests were written before limits were agreed. Adjusted
for the current settings
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
In Zebra the availability limit code states:
'available' is defined as (items.onloan is NULL) and (items.itemlost = 0)
We should make ES follow the same behaviour so that changing the engine does not
change this.
To test:
1 - Find or create several records with two items each
2 - For those records setup items as:
a - one checked out, one available
b - both marked lost
c - both damaged
d - both withdrawn
e - both notforloan
3 - Enable ES and reindex to confirm everything is set
4 - Perform an advanced search to return those records and 'limit to available items'
5 - Confirm you see records b,c,d,e and records say '2 items, none available'
6 - Apply patch
7 - Run `misc/search_tools/rebuild_elasticsearch.pl -d -b`
8 - Perform an advanced search to return those records and 'limit to available items'
9 - Confirm you see records a,b,c,d and al but a say '2 items, none available'
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Add a new boolean ES field named 'available', which is true if at least
one item is available, which means the item is not on loan, not
"notforloan", not withdrawn, not lost and not damaged
A full indexation is required
Test plan:
1. Apply patch and run updatedatabase.pl
2. Run `misc/search_tools/rebuild_elasticsearch.pl -d -b`
3. Make sure you have some biblios whose items are all unavailable, some
biblios whose items are all available, and some biblios with at least
one item available and at least one item unavailable
4. Use the 'available' filter on both opac and intranet and make sure it
works as expected.
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
When defining our sort fields in we defined all as 'numeric'
For other string containing numbers this is likely correct, however,
for callnumbers it is not. e.g. E45 should sort before E7
This patch adds a new 'callnumber' type and deifnes this for cn-sort and
adds to the field maping a sort without numeric set
To test:
0 - Be using ES with Koha
1 - On records with single item, add callnumbers:
VA65 E7 R63 1984
VA65 E7 T35 1990
VA65 E45 R67 1985
2 - Add public note 'shrimp' or something to make them easily searchable as a group
3 - Search for 'shrimp', sort by callnumber
4 - Note E45 comes last, it should come first
5 - Apply patch
6 - Reset ES mappings
7 - Reindex ES
8 - Repeat search
9 - Sorting should be correct when set to callnumber
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Michal Urban <michalurban177@gmail.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch includes the parameter:
include_type_name
on our ES calls
It removes the deprecation warning we are seeing, and should allow using ES7 with no other chnages.
For ES8 we will need to remove the hardcoded type,
I would suggest a second patch, with a new syspref "ElasticsearchLegacyType" set to 'True' by default.
Description can explain that this must be set to false, and data reindexed after upgrading to ES7 and before
upgrading to ES8 - then we can drop this pref when we drop ES7 support
When we drop ES6 support the pref can be set default true for upgrades, default 'false' for new installs
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
and some more...
There are lot of inconsistencies in our ->search calls. We could
simplify some of them, but not in this patch. Here we want to prevent
regressions as much as possible and so don't add unecessary changes.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
I need to add a 'request_timeout' entry to the ES configuration to fix some timeout problems on the sandboxes.
Instead of hard coding this new option it seems preferable to be
flexible and allow different options to be passed from the config file.
Test plan:
Add to the $KOHA_CONF, inside the elasticsearch section
<request_timeout>60</request_timeout>
Rebuild the ES index
koha-elasticsearch --rebuild kohadev
Change the value of the timeout to 1
Rebuild the index
It should fail (with a quite bad error 'Bad response received when
submitting request to Elasticsearch')
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Test plan:
1. Create an authority with at least a 1XX$a and a 4XX$a, for instance:
100 $a Foo
400 $a Bar
2. Create a biblio and add a link to this authority using the
cataloguing plugin
3. Disable syspref IncludeSeeFromInSearches
4. Reindex the biblio. You should be able to find it when searching
'Foo' but not when searching 'Bar'
5. Enable syspref IncludeSeeFromInSearches
6. Reindex the biblio. You should be able to find it when searching
'Foo' and also when searching 'Bar'
without the patch, 'Bar' doesn't yeld results. With it, it does.
7. prove t/db_dependent/Koha/SearchEngine/Elasticsearch.t
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Lucy Vaux-Harvey <lucy.vaux-harvey@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
There is existing error message in the code stating:
"Unable to understand your search query, please rephrase and try again."
which fits perfectly but because it looks for "ParseException" in the
warning output it doesn't show up on this page as it's actually
"parse_exception".
This patch makes that it's also checked if "parse_exception" is present
in the warning output.
Side note:
"ParseException" reaction code was added here:
e0f6c4dc Bug 12478: improve error reporting a bit
Search::Elasticsearch seems propagates clean ES JSON answer,
and in current ES version inside of $@ it contains "parse_exception"
string in dumped JSON answer ("'type' => 'parse_exception'").
Old seeked phrase "ParseException" wasn't reproduced, only in ES logs
("Caused by: org.apache.lucene.queryparser.classic.ParseException:
Cannot parse ..."). Check for both phrases won't complicate future
changes, but this note added for reference and code cleanup if needed.
To reproduce:
1) using ES search for something like "// ^ ! { } [ ] .. , <>" that
will for sure break the syntax of ES.
2) after the search query fails note that the error is
"Unable to perform your search. Please try again."
3) apply the patch
4) search for the same thing again
5) error message should be "Unable to understand your search query,
please rephrase and try again." now.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Andrew Nugged <nugged@gmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
On bug 17591 we discovered that there was something weird going on with
the way we export and use subroutines/modules.
This patch tries to standardize our EXPORT to use EXPORT_OK only.
That way we will need to explicitely define the subroutine we want to
use from a module.
This patch is a squashed version of:
Bug 17600: After export.pl
Bug 17600: After perlimport
Bug 17600: Manual changes
Bug 17600: Other manual changes after second perlimports run
Bug 17600: Fix tests
And a lot of other manual changes.
export.pl is a dirty script that can be found on bug 17600.
"perlimport" is:
git clone https://github.com/oalders/App-perlimports.git
cd App-perlimports/
cpanm --installdeps .
export PERL5LIB="$PERL5LIB:/kohadevbox/koha/App-perlimports/lib"
find . \( -name "*.pl" -o -name "*.pm" \) -exec perl App-perlimports/script/perlimports --inplace-edit --no-preserve-unused --filename {} \;
The ideas of this patch are to:
* use EXPORT_OK instead of EXPORT
* perltidy the EXPORT_OK list
* remove '&' before the subroutine names
* remove some uneeded use statements
* explicitely import the subroutines we need within the controllers or
modules
Note that the private subroutines (starting with _) should not be
exported (and not used from outside of the module except from tests).
EXPORT vs EXPORT_OK (from
https://www.thegeekstuff.com/2010/06/perl-exporter-examples/)
"""
Export allows to export the functions and variables of modules to user’s namespace using the standard import method. This way, we don’t need to create the objects for the modules to access it’s members.
@EXPORT and @EXPORT_OK are the two main variables used during export operation.
@EXPORT contains list of symbols (subroutines and variables) of the module to be exported into the caller namespace.
@EXPORT_OK does export of symbols on demand basis.
"""
If this patch caused a conflict with a patch you wrote prior to its
push:
* Make sure you are not reintroducing a "use" statement that has been
removed
* "$subroutine" is not exported by the C4::$MODULE module
means that you need to add the subroutine to the @EXPORT_OK list
* Bareword "$subroutine" not allowed while "strict subs"
means that you didn't imported the subroutine from the module:
- use $MODULE qw( $subroutine list );
You can also use the fully qualified namespace: C4::$MODULE::$subroutine
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
koha-elastic --rebuild was failing with
Unable to update mappings for index "koha_kohadev_biblios". Reason was: "Could not convert [marc_data.index] to boolean"
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Subdivision authorities are not used for linking, however, they are recognized by
C4::AuthoritiesMarc
While these records are not used for linking, they could provide reference and
should be allowed to exist in the catalog without breaking ES indexing
THis patch simply skips the step of parsing the authorities into the linking form
if the type contains '_SUBD'
To test:
1 - Import a subdivision authority record via Z39 or use the one attached to this bug
2 - perl misc/search_tools/rebuild_elasticsearch.pl -v -d
3 - Authority indexing dies:
Use of uninitialized value $tag in hash element at /usr/share/perl5/MARC/Record.pm line 202.
Use of uninitialized value $tag in regexp compilation at /usr/share/perl5/MARC/Record.pm line 206.
Use of uninitialized value $tag in hash element at /usr/share/perl5/MARC/Record.pm line 207.
Can't call method "tag" on an undefined value at /kohadevbox/koha/C4/Heading.pm line 71.
4 - Apply patches
5 - reindex
6 - Success!
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Pedro Amorim
Martin Renvoize
Tomás Cohen Arazi
Thomas Klausner
David Gustafsson
Shi Yao Wang
Jonathan Druart
Nick Clemens
Frédéric Demians
Koha Development Team
Janusz Kaczmarek
David Cook
Catalyst Bug Squasher
Kyle M Hall
Fridolin Somers
Katrin Fischer
Mark Hofstetter
Julian Maurice
Petro Vashchuk