Test plan:
1- Change to a language other than english
2- Search in the catalog with "!"
3- Notice the string "Unable to understand your search query, please
rephrase and try again." is not translated
4- Apply the patch
5- Add translation in .po file of used language for "Unable to
understand your search query, please rephrase and try again.\n"
6- Update translation
7- Redo the search and notice the string is now translated
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
The ES query builder correctly handles this parameters (passed from
search.pl and opac-search.pl) when it builds the ES query. But the
returned 'query_cgi' that is used later for keeping the search results
query context for building things like facets, just doesn't pass this
values around, leading to a loss of them on the generated links.
This patch makes sure they are passed around.
To test:
1. Apply the regression tests
2. Run:
$ ktd --shell
k$ prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
=> FAIL: Tests fail! These params are not passed around!
3. Set ElasticsearchMARCFormat="Searchable array"
4. Reindex all the things:
k$ koha-elasticsearch --rebuild -d -v kohadev
5. Go to advanced search, using the 'More options' option.
6. Check 'Search entire MARC record' and 'Apply weights to search'
7. Perform a search on 'FULL TEXT'
=> SUCCESS: The URL contains `whole_record=on` and `weighted_fields=on`
8. Hover on any facet and look at the link
=> FAIL: It doesn't contain them!
9. Apply this patch
10. Restart all
k$ restart_all
11. Refresh the search results
12. Repeat 6
=> SUCCESS: Links have the needed parameters!
13. Repeat 2
=> SUCCESS: Tests pass! the parameters are passed around!
14. Sign off :-D
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch restores the previous perl sorting to restore speed. For English language libraries
this will be correct in most instances, and is much more performant.
To test:
1 - Apply patch, update database
2 - Perform some searches, note ordering of facets
3 - Update system preference 'FacetOrder' to 'simple alphabetical'
4 - Peform some searches, note ordering of facets, should be correct unless you have unicode characters
5 - Sign off
NOTE: Benchmarks attached as a comment, you can also enable NYTProf and view results for yourself:
https://wiki.koha-community.org/wiki/Profiling_with_Devel::NYTProf#NYTProf_with_Plack
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
The Koha_Main_OS1 and Koha_Main_OS2 jobs are currently failing on
Jenkins.
```
[NoNodes] ** No nodes are available: [http://es:9200], called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Indexer.pm line 417. # Looks like your test exited with 11 just after 3.
```
We need to workaround Elasticsearch/OpenSearch product check
incompatibility.
This patch suggests to monkey-patch
Search::Elasticsearch::Role::Cxn::process_response to disable the strict
product header check at runtime. This allows the same codebase to talk
to both Elasticsearch 8+ and OpenSearch without forking or patching
upstream.
Test plan:
0. Set KOHA_ELASTICSEARCH=yes in ktd's .env
1. Startup ktd with the compose file for OS1
ktd -f docker-compose.os1.yml up
2. Set SearchEngine=Elasticsearch
3. reindex
ktd --shell
perl misc/search_tools/rebuild_elasticsearch.pl -a -b
4. run the tests
prove -r t/db_dependent/Koha/SearchEngine/Elasticsearch
5. search!
http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=perl
should show results
6. ktd down then repeat using the compose file for OS2 (docker-compose.os2.yml), ES6 and ES7
for ES6 and ES7 you can use the --es6 and --es7 ktd flag.
IF you get a 403 during the reindex you can try to add
cluster.routing.allocation.disk.threshold_enabled: "false"
to the environment section of the compose file.
Or
```
curl -X PUT http://es:9200/_cluster/settings \
-H 'Content-Type: application/json' \
-d '{
"persistent": {
"cluster.blocks.create_index": false
}
}'
```
OS thinks we lack disk space and force all indices into read-only.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
We currently have a sysprefs named "Elasticsearch*" and some name only "ES*". This patch unifies the names to always start with "Elasticsearch".
Test plan:
* Start koha-testing-docker, open the staff interface and go to Administration.
* Enter "Elasticsearch" into the System Preference search box and click "search"
* You will find a bunch of sysprefs
* Now search for "ESPreventAutoTruncate", which should find one match
* Apply the patch
* run (in ktd) the script `updatedatabase`
* You should see output like
DEV atomic update /kohadevbox/koha/installer/data/mysql/atomicupdate/bug_39526_unify_syspref_variable_names_for_elasticsearch.pl [09:35:03]: Bug 39526 - Unify system preference variable names for Elasticsearch
Renamed system preference 'ES..' to 'Elasticsearch..'
* Go again to to sysprefs and search again for "Elasticsearch"
* you should now find "ElasticsearchPreventAutoTruncate"
* Search for "ESPreventAutoTruncate", which should show "No system preferences matched your search for .." (and the list of ALL sysprefs)
Please note that the qa script explains about installer/data/mysql/mandatory/sysprefs.sql, but the last line is not touched by this patch, so not sure why this happens.
Sponsored-by: HKS3
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Use of uninitialized value in string comparison (cmp) at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Search.pm line 536.
I think this is the correct fix. We generate a random av category that
is used in the data, so $label get undef:
521 $label = $authorised_values{$t};
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This follow-up adds two major enhancements to the locale-based facet sorting
implementation:
1. **System Preference Integration**
- Adds FacetSortingLocale system preference for configurable locale selection
- Dynamically detects available system locales using Koha::I18N
- Provides user-friendly dropdown with locale descriptions
- Supports both Elasticsearch and Zebra search engines
2. **Zebra Search Engine Support**
- Extends locale-based sorting to Zebra (in addition to existing Elasticsearch)
- Implements _sort_facets_zebra() function in C4::Search
- Maintains consistent Unicode-aware sorting behavior across search engines
Key Features:
- Dynamic locale detection via Koha::I18N::available_locales()
- Proper module architecture with comprehensive unit tests
- Clean fallback chain: preference → system LC_COLLATE → default
- Admin UI integration in System Preferences → I18N/L10N
- Universal facet sorting for international library systems
Test plan:
1. Apply patch and restart services
2. Install/run database update
3. Navigate to Administration → System Preferences → I18N/L10N
4. Verify "Sort facet names using" dropdown shows available system locales
5. Test facet sorting with different locales and both search engines
6. Run: prove t/Koha/I18N.t t/Koha/SearchEngine/Elasticsearch/Search.t t/db_dependent/Search_FacetSorting.t
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This change uses a configurable locale-based collator to sort
the ES facet names.
Test plan:
0. Apply the patch
1. vi /etc/locale.gen
2. Uncomment the locale you want to generate (e.g. fi_FI.UTF-8 UTF-8)
3. locale-gen
4. vi "/etc/default/koha-common"
5. Add the following to the bottom of the file:
export LC_ALL=fi_FI.UTF-8
6. koha-plack --restart koha-common
7. Setup some test records with authors with accented and unaccented names,
and different cases for the lead letter
e.g. Aa author, Åa author2, aa author, étienne
8. Switch to using Elasticsearch and reindex
koha-elasticsearch -b -v --rebuild kohadev
9. Do a test search
e.g. http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=test
10. Confirm the facet names are sorted in ascending order following
Finnish collation rules
e.g.
aa author
Aa author
étienne
Farley, David
Humble, Jez
Martin, Robert C.
Åa author
NOTE: Any collation and language can be used. Finnish is just
an example of a Latin-based script which has a different
alphabetical ordering than just A-Z
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
With bug 30280 and subsequent patches we attempt to respect the
thesaurus indication when linking with LinkerConsiderThesaurus on.
However, there are still cases when we link wrong.
This happens when thesaurus in bibliographic field 6XX is defined by
subfield $2. In such a case the authority record should have 008/11 = 'z'
and, according to https://www.loc.gov/marc/authority/ad008.html
(cf. comment for 008/11 = 'z': "A MARC code for the conventions used to
formulate the heading *may* be contained in subfield $f"), may or may have
thesaurus declared in 040 $f.
This is why we make a second search attempt in C4::Heading::_search,
but doing so we have to make sure that the resulting authority
record has not a different thesaurus declared in 040 $f. (Note
that 008/11 and 040 $f are indexed with the same search
field: subject-heading-thesaurus. So we search for the
second time in the same search field but with
'notdefined' == 008/11='z' instead of the original
thesaurus taken from subfield $2 of the bibliographic
field 6XX.) This means we have to explicitly check the
content of 040 $f of the resulting authority record.
Test plan:
==========
1. Turn on the LinkerConsiderThesaurus systempreference.
2. Create an authority record (e.g. TOPIC_TERM) with 008/11 = 'z' and
040 $f containing a thesaurus indication (e.g. 'kaba'), and a term
in field 150 $a (e.g. Early music).
3. Edit a bibliographic record: in field 650 put 2nd indicator '7',
the term 'Early music' in $a, and a different thesaurus code in $2
(e.g. 'dbn'). Click the 'Link authorities automatically' button.
Note that 650 'Early music' (dbn) has been linked with and authority
record 'Early music' (kaba). This is wrong.
4. Apply the patch ; restart_all ; reindex ES for authorities with:
koha-elasticsearch --rebuild -d -a kohadev
5. Repeat p. 3. Now the 650 'Early music' (dbn) has not been linked to
an authority record (650 - No matching authority found.)
Signed-off-by: Janusz Kaczmarek <januszop@gmail.com>
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
When building queries for matchign Zebra uses OR to combine searches - Elasticsearch
has been effectively using 'AND' - this patch checks whether and_or has been passed
and follows the variable when present.
To test:
1 - Have an import file of auths with multiple 035
2 - Setup an authority matching rule
Search index: other-control-number
Score: 1000
Tag: 035
Subfields: a
3 - Make sure Koha is using Elasticsearch
4 - Stage file without matching, import file
5 - Stage file again, using your matching rule
6 - Note all records match
7 - Setup Cataloging->Marc modification template
8 - Add an action to delete first 035 field
9 - Run it against the records you just imported
You can use a report:
SELECT authid FROM auth_header WHERE authid > #### (find the lowest new record number in batch you imported)
10 - Change matching rule in batch to none, then back to your rule
11 - No matches!
12 - Apply patch, restart all
13 - Set matchign rule to none, then back to your rule
14 - Matches!
Signed-off-by: Magnus Enger <magnus@libriotech.no>
Works as advertised!
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
to keep the syspref names consistent with the existings values. Also "ES" is not as self-documenting as "Elasticsearch".
No code change, so the test plan stays the same.
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
What this patch does is:
1 - Wraps the existing search code in a a "bool" compound query as a "must". This should not affect
relevancy or results of the existing searches.
2 - Before we clean/truncate terms, loop through the passed in search terms and indexes to build a
new 'should' query, using the 'match' on the specified index/field that is added to the 'bool' query from above.
This means that if a result from the original query is also returned here, that item will be boosted in the result.
For searches on 'keyword' or 'title', or if no index is set, we use 'title-cover' as the most narrow form of title
This query isn't going to help when users enter CCL (i.e. ti:To die for) and it won't boost titles from 505, series, etc
when doing a general search. Nor will it have a detrimental effect, it will only boost field matches
To test:
1 - Add a record with
245 $a novel
2 - Add a record with
245 $a A novel : $b about things / $c by me
3 - Search for: novel
- Above records are returned lower in the list (results #6 and #14 for me)
4 - Search for: a novel
- Above records are returned lower in the list (results #6 and #14 for me)
5 - Apply patch, restart all, enable ESBoostFieldMatch option
6 - Repeat searches, note exact titles are boosted
- For 'novel' record from #1 is #1 result and other is second
- For 'a novel' record form #2 is #1 result and other is third
7 - Disable the new pref and repeat steps 3 and 4 and get results as before patch
8 - Experiment with other searches, turning pref on and off to verify relevant titles are boosted when enabled
9 - Search results when disabled should return as before the patch
Signed-off-by: Matthias Meusburger <matthias.meusburger@biblibre.com>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Added head2 right before sub with missing POD
Signed-off-by: Baptiste Wojtkowski <baptiste.wojtkowski@biblibre.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
To test:
1 - Set search engine to ES
2 - Search the catalog for "*"
3 - Note results
4 - Edit first result
5 - Repeat search, results are reordered
6 - Apply patch
7 - Search for "*"
8 - Highest biblionumbers are returned first
9 - Edit first result, repeat search
10 - Order does not change
11 - Confirm sorting by author, title, etc passes the same trials
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Add a new hook elasticsearch_to_document in order to allow completing
the 'document' to be sent to Elasticsearch. Usage example: (1) add new
search fields, (2) normalize/sanitize existing fields, (3) use id fields
to retrieve info via web services in other repositories (say VIAF), (4)
download file referenced in a field, and index its content.
To test:
1. Apply the unit tests patch, and run it:
prove t/db_dependent/Koha/Plugins/Biblio_and_Items_plugin_hooks.t
=> FAIL: Tests fail! The hook is not implemented so the desired
2. Apply this patch, and then repeat 1:
=> SUCCESS: It works!
Signed-off-by: Shi Yao Wang <shi-yao.wang@inlibro.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds a new option to Elasticsearch to skip facets in build_query and
build_query_compat.
There should be no behavior change, however, queries to ES should be simpler
To test:
1 - Export some record from Koha
2 - Stage the records for import, matching on 999c
3 - Confirm expected matches
4 - Switch batch to use ISBN for matching
5 - Confirm expected matches
6 - Apply patches, restart all
7 - Switch batch to use 999c
8 - Confirm matches
9 - Switch batch to use ISBN
10 - Confirm matches
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
As elsewhere in the code, we should use a state variable for our
'bib_searcher' object to avoid overhead of extra ES connections.
Worth noting - we don't actually use the 'bib_searcher' at all during
record auto linking as we pass skipmetadata
To test:
0 - Apply patch
1 - Enable biblio auto linking
AutoLinkBiblios, CatalogModuleRelink, LinkerRelink
2 - Stage and import a marc file
3 - Confirm record authorities link as expected
4 - Search authorities in the staff interface
5 - Confirm usage counts are shown for results
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This commit is generated using:
% perl misc/devel/tidy.pl
*within* ktd, to get the same version of perltidy than what will be used
by our CI (currently v20230309).
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
After Bug 38416 Elasticsearch indexing explodes with oversized
records, especially with UTF encoded data.
In Koha::SearchEngine::Elasticsearch::marc_records_to_documents a
following snippet has been introduced:
my $usmarc_record = $record->as_usmarc();
my $decoded_usmarc_record = MARC::Record->new_from_usmarc($usmarc_record);
But if $record is oversized (> 99999 bytes), it is OK for MARC::Record
object, but not for $record->as_usmarc. The produced ISO 2709 string
is not correct and hence cannot be properly converted back to
MARC::Record object by new_from_usmarc.
The result in this case can be like:
UTF-8 "\x85" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35.
Since it is done without any eval / try, the whole reindex procedure
(for instance rebuild_elasticsearch.pl) is being randomly interrupted
with no explanation.
Test plan:
==========
Hard to reproduce. But the explanation together with discussion in Bug
38416 (from 2024-12-15) explains and justifies the need of this added
eval.
1. Have a standard KTD installation with Elasticsearch.
2. Use the provided test record - add it to Koha with
./misc/migration_tools/bulkmarcimport.pl -b -file test.xml -m=MARCXML
(have patience).
During load process you should see a message like:
UTF-8 "\xC4" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35.
3. The record should get biblionumber 439. Check in librarian interface with
http://<your_addreess>:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=439
that the record has been imported.
However, you should not be able to make a search for this record.
4. Try to reindex with:
./misc/search_tools/rebuild_elasticsearch.pl -b -bn 439
You should get a message like:
UTF-8 "\xC4" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35.
Again, no search results.
5. Apply the patch ; restart_all.
6. Repeat reindex with:
./misc/search_tools/rebuild_elasticsearch.pl -b -bn 439
There should be no warning now and you should be able to find the record.
Signed-off-by: Magnus Enger <magnus@libriotech.no>
Followed the test plan. Works as advertised.
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch makes the `_process_mappings()` method split the index values
in the event of them being bigger than the allowed 32766 bytes size.
To test:
1. Have KTD running with ES:
$ ktd --proxy --es7 up -d
2. Perform a search
3. Pick the first result for edition
4. Find a cool Wiki page with lots of paragraphs
5. Copy all of the paragraphs and put them on a 500$a field for the record.
6. Repeat 2
=> FAIL: The record is not found
7. Reindex manually:
$ ktd --shell
k$ perl misc/search_tools/rebuild_elasticsearch.pl --biblios --where "biblionumber=3" -v -v
=> FAIL: You get something like:
```
[22229] Committing final records...
One or more ElasticSearch errors occurred when indexing documents at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Indexer.pm line 148.
[22229] There were errors during indexing
Record #3 Document contains at least one immense term in field="note.raw" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[10, 109, 117, 115, 116, 97, 102, 97, 32, 102, 117, 101, 32, 101, 108, 32, 115, 101, 103, 117, 110, 100, 111, 32, 104, 105, 106, 111, 32, 100]...', original message: bytes can be at most 32766 in length; got 32771 (illegal_argument_exception) : max_bytes_length_exceeded_exception (bytes can be at most 32766 in length; got 32771)
[22229] Total 1 records indexed
```
8. Apply this patch
9. Repeat 7
=> SUCCESS: No error!
10. Repeat 2
=> SUCCESS: The record is indexed and can be found!
11. Sign off :-D
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This change failsover to MARCXML from USMARC if there are any
warnings generated by MARC::File::USMARC::decode when trying to
roundtrip the record.
Test plan:
0. Apply the patch
1. Setup your koha-testing-docker to use Elasticsearch
2. Create a new record with 15,000 characters in the 500$a field
3. Index that record
(e.g. perl misc/search_tools/rebuild_elasticsearch.pl --biblios -v -v)
4. Note that a warning saying the following appears:
"Warnings encountered while roundtripping a MARC record to/from USMARC.
Failing over to MARCXML"
5. View the "Elasticsearch record" on the detail page and note that the
marc_format is MARCXML
6. Perform a search for the record (the keyword should be something that
brings up other results too)
7. Note that the record appears correctly in the search results
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
The patch was corrupted, and was rewritten as the original one by
Hammat
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Test plan:
- Enable elasticsearch
- git to labels/label-item-search.pl
- search using after date only
=> check you get items with date-of-acquisition greater than after
date
- search using before date only
=> check you get items with date-of-acquisition less than before
date
- search using after and before date
=> check you get items with date-of-acquisition between after and
before
- Combine this searches with a specified index
- enable zebra and repeat these tests
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
https://bugs.koha-community.org/show_bug.cgi?id=30845
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
And copy fix across to Zebra search as well.
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch ensures the ShowHeadingUse system preference and feature works as expected when using Elasticsearch as the searchengine.
To test, follow the test plan at Bug 29990 Comment 9. Ensure the SearchEngine system preference is set to Elasticsearch.
Sponsored-by: Education Services Australia SCIS
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Some libraries ask for the ability to scan/browse the call numbers index.
Test plan:
==========
1. Have an installation with items with callnumbers (in ktd with standard
test data set: add some callnumbers to the items).
2. Go to Advanced search > More options; choose Call number in the first
drop list and mark Scan indexes. Enter one of the call numbers you
assigned to the items and perform the search. You should get no
results.
3. Apply the pach; restart all; when using ES:
koha-elasticsearch --rebuild -r -d kohadev
4. Repeat p. 2. You should get a list with at least one call number.
The link should lead you to the record(s) with items with the selected
call number.
This should work for ES and for Zebra, but for Zebra you would need to
replace /etc/koha/zebradb/ccl.properties with the repository version; and
you will get one token results on the list, like for other indexes.
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
We can now call metadata_record directly on the Koha::Biblio object.
This aptch updates all modules and controllers to use
Koha::Biblio->metadata_record directly where appropriate. The
exceptions are where we don't require any filtering or the filtering
makes sense to do after the initial fetch.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This change filters the values from the OpacHiddenItems syspref
out of their corresponding search results facets.
Note: This does not mean that all values from hidden items are filtered out of facets.
Rather, it just means that facet data that matches OpacHiddenItems is filtered out. This is
an imperfect fix, but it is a practical fix that has been requested by more than one library.
Test plan:
0. Apply the patch and koha-plack --restart kohadev
1. Go to http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=29
2. Add an item with an item type of "Maps"
3. Go to http://localhost:8081/cgi-bin/koha/admin/preferences.pl?op=search&searchfield=OpacHiddenItems
4. Fill in OpacHiddenItems with the following:
itype: ['MP']
5. Go to http://localhost:8080/cgi-bin/koha/opac-search.pl?idx=&q=test&weight_search=1
6. Note that "Maps" does not appear in the "Item types" facet
7. Go to http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=test
8. Note that "Maps" does appear in the "Item types" facet
9. ***Repeat this test plan using both Elasticsearch and Zebra indexing***
10. prove -v t/Koha/SearchEngine/Search.t
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
With the Bug 36574 we added a new ES search field isbn-all which makes it
possible to find records with the content of the subfield 020 $z
(invalid/cancelled ISBN) in general search (by keyword). But still, when
performing a search with ISBN-search these records will not be found.
Since some libraries deliberately want to find records with only 020 $a
with ISBN-search, a system preference would be needed here to be able to
customize the ISBN search.
The same for ISSNs, with one system preference.
Test plan:
==========
(ISBN)
------
1. Have the ktd with with ES enabled.
2. With standard ktd test data, try to perform a ISBN search from Advanced
search for 9780141930848. You should get no results.
In the general search input field (on the main page) try a search
nb:9780141930848 and/or isbn:9780141930848 - you should get no results.
3. Apply the patch, restart_all.
4. Repeat p. 2 -- you should still get no results.
5. Enable SearchCancelledAndInvalidISBNandISSN syspref.
6. Repeat p. 2 -- now you should get one record (having 9780141930848
in 020 $z).
(ISSN)
------
0. Return to main, delete the bug branch, reset_all == start from fresh.
1. Have the ktd with with ES enabled.
2. With standard ktd test data, try to perform a ISSN search from Advanced
search for 1089-6891. You should get no results.
In the general search input field (on the main page) try a search
ns:1089-6891 and/or issn:1089-6891 - you should get no results.
3. Apply the patch, restart_all.
If you have had to apply also the bug 36727, reindex ES with:
koha-elasticsearch --rebuild -r -d kohadev
(there is a modification of mappings.yaml)
4. Repeat p. 2 -- you should still get no results.
5. Enable SearchCancelledAndInvalidISBNandISSN syspref.
6. Repeat p. 2 -- now you should get one record (having 1089-6891
in 022 $y).
Sponsored-by: Ignatianum University in Cracow
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Tiny adjustment for very special and malicious cases needed in
Koha::SearchEngine::Elasticsearch::QueryBuilder::clean_search_term.
Extended unit tests added following Nick's suggestions in
Bug 33407 discussion.
Test plan
=========
1. Use ktd with Elasticsearch and ktd's test data.
2. Edit the 4th record (The C programming language) and add a semicolon
followed by a space before the slash in 245 $a to have:
The C programming language ; /
3. Make a search for the first ISBD zone taken verbatim from the forth
record:
The C programming language ; / Brian W. Kernighan, Dennis M. Ritchie
There should be no result.
4. Apply the patch, restart plack.
5. Repeat the search. You should get the record.
Sponsored-by: Ignatianum University in Cracow
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Test plan:
With ElasticSearch enabled,
* Perform a search using the default sort order
(i.e. 'relevance').
* Verify that no warnings are generated in
plack-intranet-error.log
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Test plan:
1. Apply patch
2. Rerun and make sure the error "Use of uninitialized value $sub6 in pattern match (m//) at /usr/share/koha/lib/Koha/SearchEngine/Elastricsearch.pm" is no longer present.
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Initial testing done, it works well, needs an additional sign off from an external party
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
The highlight logged-in library in facets feature does not work with
Elasticsearch. This is because facet_title_value contains an additional
" (count)" information and so gives negative results in comparisons made
in facets.inc (line 107).
Test plan:
==========
1. Have a test installation with Elastiscsearch.
2. Set system preference 'DisplayLibraryFacets' to 'both home and holding libraries'
3. Perform a seach that will find items of your logged-in library.
In KTD, search for 'branch:CPL'
4. Check the facet (home and holding libraries) of your logged-in
library is not highlighted
5. Apply the patch ; restart_all.
6. Repeat p. 3. Check the facet (home and holding libraries)--you
should see your logged-in branch highlighted now.
Sponsored-by: Ignatianum University in Cracow
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Like IncludeSeeFromInSearches adds see from heading in search (4xx),
new system preference IncludeSeeAlsoFromInSearches adds see also from heading (5xx).
Test plan:
Test on both Elasticsearch and Zebra:
1)
1.1) Create an authority record with heading, see from heading 4xx and see also from heading 5xx
1.2) Use this authority in a biblio record
2)
2.1) Set IncludeSeeFromInSearches and IncludeSeeAlsoFromInSearches to "Don't include"
2.3) Rebuild search engine
2.4) Perform a search on heading text => you find the biblio record
2.5) Perform a search on see from heading => you do not find the biblio record
2.6) Perform a search on see also from heading => you do not find the biblio record
3)
3.1) Set IncludeSeeFromInSearches and IncludeSeeAlsoFromInSearches to "Include"
3.2) Rebuild search engine
3.3) Perform a search on heading text => you find the biblio record
3.4) Perform a search on see from heading => you find the biblio record
3.5) Perform a search on see also from heading => you find the biblio record
Signed-off-by: Frédéric Demians <f.demians@tamil.fr>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
With ES and QueryAutoTruncate on, a search with punctuation surrounded
by spaces (like ISBD punctuation copied from other catalogue or
bibliography) returns no results. E.g.: in a search for "Maria Stuart ;
Die Jungfrau von Orleans / Schiller" (coded in record as 245 10 $a Maria
Stuart ; $b Die Jungfrau von Orleans / $c Schiller) -- both semicolon
and slash cause problems. One had to remove them manually to get
results which is not what is desired.
Test plan
=========
1. Use ktd with Elasticsearch and ktd's test data
(http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=1):
2. Make a search for the first ISBD zone taken verbatim from the first record:
E Street shuffle : the glory days of Bruce Springsteen & the E Street Band / Clinton Heylin
There should be no result.
3. Apply teh patch, restart plack.
4. Repeat the search. You should get the record (and onother one).
Sponsored-by: Ignatianum University in Cracow
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Test plan:
Export the ES fields/mappings using
misc/search_tools/export_elasticsearch_mappings.pl > mappings.yaml
Copy this file to admin/searchengine/elasticsearch/mappings.yaml
Reset the mapping from the UI
Notice that the links to the AV categories have been preserved
Sponsored-by: The Research University in the Helmholtz Association (KIT)
Signed-off-by: Lukas Koszyk <lukasz.koszyk@kit.edu>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This new feature adds the ability to link an ES facet to an authorised
value category, in order to display a description instead of the code.
Test plan:
1. Go to the ES mapping admin page
2. Notice the new "Authorized value category" column in the facets
config table
3. Create a new facet that is linked to a search field containing values
from an authorised value category. Pick this category in the list
4. Save
5. Reindex
6. Notice that the descriptions are now displayed (staff and OPAC) in
the search result page.
QA: See the related bug report for possible improvements. It's not
easy at all to deal with the "specials" here, a lot of changes must be
done in how we fetch/cache the AVs.
Sponsored-by: The Research University in the Helmholtz Association (KIT)
Signed-off-by: Lukas Koszyk <lukasz.koszyk@kit.edu>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Currently we only remove non-filing characters for sort fields, however, this can make searching difficult.
This patch adds the filing form to the index as well to aid in searching.
To test:
0 - Setup Koha with Elasticsearch
1 - Import the sample record on this report: "L'amour de l'art"
2 - Search for "amour de l'art" - no results
3 - Apply patch
4 - Reindex
5 - Search for "amour de l'art" - result!
6 - Search for "title:amour de l'art" - result!
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
We can know add/remove homelibrary and holdinglibray from the config
page, this syspref is no longer needed for ES
Sponsored-by: The Research University in the Helmholtz Association (KIT)
Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This new feature allows to manage facets for Elasticsearch from the
administration page.
Prior to this the facet fields were hardcoded in the codebase.
Librarians can then add/remove facet fields and modify their label.
Test plan:
1. Create a new search field and set it a label different than its name.
2. Save
3. Go the bibliographic mapping tab
4. Add 1+ mapping for this search field (Searchable and facetable must be "Yes")
5. Add, reorder, remove new facets
6. Save and reindex your records
7. Search and notice the new facet list
QA: There are several wrong things in this area (ES + facets code,
everything, pm, pl, tt AND on this administration page). I have done my
best to clean the code as much as I could and keep the code cleaner
after than before. But there are still a lot to do.
There are still inconsistencies on this page (like we need to save to
see the changes applied to the other tables), but this is clearly out of
the scope of this bug report.
Another enhancement would be to move the facet list to a different DB
table, that could bring more flexibility:
* display or not (could be opac/staff/both/none)
* define the size per field (number of facet to display)
* order: move search_field.facet_order to this new table.
But, again, it's a lot more work.
More work is done in this area, please see related bug reports.
Sponsored-by: The Research University in the Helmholtz Association (KIT)
Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds geosearch to Koha (using Elasticsearch 7). ElasticSearch
search_mappings get new types to store lat/lon, which can be indexed
from MARC 034$s and 034$t. There is a small change to the DB to allow a
new value in search_field.type ENUM.
The QueryBuilder is extended to allow for building advanced
ElasticSearch Querys (eg geo_distance) that cannot be represented in a
simple string query. The UI for searching (including showing the results
on a OSM/Leaflet map) is implemented in a separate plugin
(https://github.com/HKS3/HKS3GeoSearch)
Test Plan:
* make sure you're running ElasticSearch 7
(eg via `curl http://es:9200?pretty | grep number`)
* apply patch
* got to a Framework, check Editor for 034$s and 034$t and save
* got to some books (in the correct framework) and enter some lat and lon into 034$s and 034$t (for example lat=48.216, lon=16.395)
* Run the elasticsearch indexer, maybe limited on the books you edited (-bn 123 -bn 456):
misc/search_tools/rebuild_elasticsearch.pl -b -v
* You can check if the indexing worked by inspecting the document in elasticsearch:
* get the biblionumber (eg 123)
* curl http://es:9200/koha_kohadev_biblios/_doc/123?pretty | grep -A5 geolocation
* You should get back a JSON fragment containing the lat/lon you stored
* You can query elasticsearch directly:
* Run the following curl command, but adapt the value for lat/lng and/or the distance (in meters)
* curl -X GET "http://es:9200/koha_kohadev_biblios/_search?pretty" -H 'Content-Type: application/json' -d '{"query": {"bool":{"must":{"match_all":{}},"filter":{"geo_distance":{"distance":100000,"geolocation":{"lat":48.2,"lon":16.4}}}}}}'
* To run the search via Koha, you need to either install and use https://github.com/HKS3/HKS3GeoSearch or create a handcrafted query string:
* handcrafted query string:
* /cgi-bin/koha/opac-search.pl?advsearch=1&idx=geolocation&q=lat:48.25+lng:18.35+distance:100km&do=Search
* HKS3GeoSearch
* install the plugin and enable it
* got to OPAC / Advanced Search
* There is a new input box "Geographic Search" where you can enter lat/long/radius
* On the search result page a map is shown with pins for each found biblioitem
Sponsored-by: ZAMG - Zentralanstalt für Meterologie und Geodynamik, Austria - https://www.zamg.ac.at/
Sponsored-by: Geosphere - https://www.geosphere.at/
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Additional finetuning:
- Fix update and remove fixed fixme
- Update test count as well
- fix last small issues raised in Comment 23
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Shi Yao Wang
Tomás Cohen Arazi
Nick Clemens
Jonathan Druart
Thomas Klausner
Martin Renvoize
David Cook
Frédéric Demians
Koha Development Team
Janusz Kaczmarek
Olivier V
Alex Arnaud
Aleisha Amohia
Martin Renvoize
Andreas Jonsson
Catalyst Bug Squasher
Kyle M Hall
Fridolin Somers
Lari Strand
Katrin Fischer
Mark Hofstetter