This patch makes sure we inject the __sort form of local-number, rather
than the field directly - this ensurs searching works even if mappings mayu
be different and local-number itself is not numeric
To test:
1 - Edit mappings.yaml and set local-number from 'number' to 'string'
NOTE: There are two copies of local-number - one in auth and one in biblio - the latter is the important one here
2 - perl misc/search_tools/rebuild_elasticsearch.pl -r -v
3 - Confirm in staff interface Admin -> Search Engine Configuration that local-number is now a string
4 - Search and get an error
Error: Unable to perform your search. Please try again.
In the logs:
[illegal_argument_exception] Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [local-number] in order to load field data by uninverting the inverted index. Note that this can use significant memory.
5 - Apply patch, restart all
6 - Search again
7 - Success!
Signed-off-by: Bernard Scaife <bernard.scaife@openfifth.co.uk>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Previously, Koha always added items to the Zebra queue when indexing via Elasticsearch. This adds a new system preference ElasticsearchEnableZebraQueue that allows disabling this behavior.
Unit Test:
* run the unit test t/db_dependent/Koha/SearchEngine/Elasticsearch/Indexer.t
* there should be a new subtest like this:
1..5
ok 1 - We start with an empty zebra queue
ok 2 - new entry in zebraqueue for auth
ok 3 - new entry in zebraqueue for biblio
ok 4 - no new entry in zebraqueue for auth
ok 5 - no new entry in zebraqueue for biblio
ok 5 - syspref ElasticsearchEnableZebraQueue
Manual Test Plan:
* Start ktd with Elasticsearch enabled
* As there is no interface to the zebraqueue, you need to connect to the DB and use SQL directly
* ktd --dbshell
* `select count(*) from zebraqueue;`
* remember that number
* Edit any biblio or auth
* Check the zebraqueue again via SQL
* There should be one more entry
* Now apply the patch
* Then got to Sysprefs and set ElasticsearchEnableZebraQueue to `Disable`
* Edit another biblio or auth
* has to be a different one, because Koha will prevent the same ID to be added again to zebraqueue
* Again check zebraqueue via SQL
* There should be *no* new entry
QA Note:
the qa tool reports
[FAIL] Koha/SearchEngine/Elasticsearch/Indexer.pm
FAIL pod_coverage
sub _get_record has no pod coverage.
But this patch does not touch this code, so I guess this fail is caused by a stricter qa tool
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Refactor marc_record_to_documents to avoid code duplicaiton.
To test:
1) Verify tests in t/db_dependant/Koha/SearchEngine/Elasticsearch.t all
pass
2) Apply patch
3) Veriy tests in t/db_dependant/Koha/SearchEngine/Elasticsearch.t
still pass
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch ensures the scores from ES are passed through and added to the
search results in the staff side.
Initially hidden, the scores can be shown by entering the line below into the
JS console in developer tools in the browser, or added to IntranetUserJS to
make them always show:
$(".search_score").toggle();
To test:
1 - Have Koha using ES
2 - Do some searches, note results
3 - Apply patch, restart all
4 - Confirm results don't change
5 - Enter code from description into IntranetUserJS
6 - Do more searches
7 - Note the score is visible
To really note scores, you will need to add some weights to the ES config:
Set weights like:
title 32
author 16
subject 8
Signed-off-by: Andrew Auld <andrew.auld@openfifth.co.uk>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
With facets generated from Authorized values, it may happen that the
list of authorized values lacks an entry for a code used in
a record. In this case, an empty entry will appear in the
facet list, with the number of uses but without any label.
It would be desirable to show at least the raw code, rather
than an empty label. This makes it easier to see which code has no
entry in the respective AV and to either add it or correct an
erroneous code.
Test plan:
==========
1. Have a standard KTD installation with Elasticsearch.
2. Add AV COUNTRY_CODES category and add two pairs (Authorized
value, Description):
* enk - England
* xxu - United States
3. In the admin/searchengine/elasticsearch/mappings.yaml file,
correct the definition of the pl search field as follows:
pl:
- label: pl
+ facet_order: 24
+ label: Publication place
mandatory: ~
+ authorised_value_category: COUNTRY_CODES
mappings:
- - facet: ''
+ - facet: 1
marc_field: 008_/15-17
marc_type: marc21
sort: 1
4. Reindex ES with:
koha-elasticsearch --rebuild -r -d -b kohadev
5. Set the displayFacetCount systempreference to 'Show'
6. Perform a general search in the OPAC or staff interface: *
View the 'Publication place' facet and notice a dozen empty
entries, with only the number of occurrences.
7. Apply the patch ; restart_all
8. Repeat p. 6. You will see country codes that you can now add to AV
or correct.
BTW, note that the tiny correction to Search.pm is analogous to
setting the label for 'special' types (i.e. itype, location, ccode,
holdingbranch, homebranch) two lines above.
Sponsored-by: Ignatianum University in Cracow
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Test plan:
==========
1. Have a ktd instance with Elasticsearch enabled.
2. In standard ktd installation there is a record with a title
"Typical girls? : the story of the Slits." (with question mark
attached to a word) - biblio # 229. We will try to find it.
3. Go to advanced search (OPAC or intranet) and try a *title* search
with:
Typical girls? the story of the Slits
You should get no results.
From the main page try a ccl-style search:
ti:Typical girls? the story of the Slits
You should get no results too.
4. Apply the patch ; updatedatabase ; restart_all.
5. Put ? in OpacElasticsearchEscapeCharacters and/or
ElasticsearchEscapeCharacters system preference.
4. Repeat the searches from p. 3. You should get the record if a
system preference was set for the given interface.
Sponsored-by: Pontificia Università di San Tommaso d'Aquino (Angelicum)
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
This adds a new system preference to control the amount of boost. The default of 0 will act as before this patch.
You can add any positive number, or a number between 0 and 1 for negative boost
To test:
1 - Add a record with
245 $a novel
2 - Add a record with
245 $a A novel : $b about things / $c by me
3 - Search for: novel
4 - Search for: a novel
5 - Apply patch, restart all
6 - Repeat searches, note exact titles are boosted
7 - Test with more titles as you are able
Signed-off-by: Jason Robb <jrobb@sekls.org>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Test plan:
1- Change to a language other than english
2- Search in the catalog with "!"
3- Notice the string "Unable to understand your search query, please
rephrase and try again." is not translated
4- Apply the patch
5- Add translation in .po file of used language for "Unable to
understand your search query, please rephrase and try again.\n"
6- Update translation
7- Redo the search and notice the string is now translated
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
The ES query builder correctly handles this parameters (passed from
search.pl and opac-search.pl) when it builds the ES query. But the
returned 'query_cgi' that is used later for keeping the search results
query context for building things like facets, just doesn't pass this
values around, leading to a loss of them on the generated links.
This patch makes sure they are passed around.
To test:
1. Apply the regression tests
2. Run:
$ ktd --shell
k$ prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
=> FAIL: Tests fail! These params are not passed around!
3. Set ElasticsearchMARCFormat="Searchable array"
4. Reindex all the things:
k$ koha-elasticsearch --rebuild -d -v kohadev
5. Go to advanced search, using the 'More options' option.
6. Check 'Search entire MARC record' and 'Apply weights to search'
7. Perform a search on 'FULL TEXT'
=> SUCCESS: The URL contains `whole_record=on` and `weighted_fields=on`
8. Hover on any facet and look at the link
=> FAIL: It doesn't contain them!
9. Apply this patch
10. Restart all
k$ restart_all
11. Refresh the search results
12. Repeat 6
=> SUCCESS: Links have the needed parameters!
13. Repeat 2
=> SUCCESS: Tests pass! the parameters are passed around!
14. Sign off :-D
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch restores the previous perl sorting to restore speed. For English language libraries
this will be correct in most instances, and is much more performant.
To test:
1 - Apply patch, update database
2 - Perform some searches, note ordering of facets
3 - Update system preference 'FacetOrder' to 'simple alphabetical'
4 - Peform some searches, note ordering of facets, should be correct unless you have unicode characters
5 - Sign off
NOTE: Benchmarks attached as a comment, you can also enable NYTProf and view results for yourself:
https://wiki.koha-community.org/wiki/Profiling_with_Devel::NYTProf#NYTProf_with_Plack
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
The Koha_Main_OS1 and Koha_Main_OS2 jobs are currently failing on
Jenkins.
```
[NoNodes] ** No nodes are available: [http://es:9200], called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Indexer.pm line 417. # Looks like your test exited with 11 just after 3.
```
We need to workaround Elasticsearch/OpenSearch product check
incompatibility.
This patch suggests to monkey-patch
Search::Elasticsearch::Role::Cxn::process_response to disable the strict
product header check at runtime. This allows the same codebase to talk
to both Elasticsearch 8+ and OpenSearch without forking or patching
upstream.
Test plan:
0. Set KOHA_ELASTICSEARCH=yes in ktd's .env
1. Startup ktd with the compose file for OS1
ktd -f docker-compose.os1.yml up
2. Set SearchEngine=Elasticsearch
3. reindex
ktd --shell
perl misc/search_tools/rebuild_elasticsearch.pl -a -b
4. run the tests
prove -r t/db_dependent/Koha/SearchEngine/Elasticsearch
5. search!
http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=perl
should show results
6. ktd down then repeat using the compose file for OS2 (docker-compose.os2.yml), ES6 and ES7
for ES6 and ES7 you can use the --es6 and --es7 ktd flag.
IF you get a 403 during the reindex you can try to add
cluster.routing.allocation.disk.threshold_enabled: "false"
to the environment section of the compose file.
Or
```
curl -X PUT http://es:9200/_cluster/settings \
-H 'Content-Type: application/json' \
-d '{
"persistent": {
"cluster.blocks.create_index": false
}
}'
```
OS thinks we lack disk space and force all indices into read-only.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
We currently have a sysprefs named "Elasticsearch*" and some name only "ES*". This patch unifies the names to always start with "Elasticsearch".
Test plan:
* Start koha-testing-docker, open the staff interface and go to Administration.
* Enter "Elasticsearch" into the System Preference search box and click "search"
* You will find a bunch of sysprefs
* Now search for "ESPreventAutoTruncate", which should find one match
* Apply the patch
* run (in ktd) the script `updatedatabase`
* You should see output like
DEV atomic update /kohadevbox/koha/installer/data/mysql/atomicupdate/bug_39526_unify_syspref_variable_names_for_elasticsearch.pl [09:35:03]: Bug 39526 - Unify system preference variable names for Elasticsearch
Renamed system preference 'ES..' to 'Elasticsearch..'
* Go again to to sysprefs and search again for "Elasticsearch"
* you should now find "ElasticsearchPreventAutoTruncate"
* Search for "ESPreventAutoTruncate", which should show "No system preferences matched your search for .." (and the list of ALL sysprefs)
Please note that the qa script explains about installer/data/mysql/mandatory/sysprefs.sql, but the last line is not touched by this patch, so not sure why this happens.
Sponsored-by: HKS3
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Use of uninitialized value in string comparison (cmp) at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Search.pm line 536.
I think this is the correct fix. We generate a random av category that
is used in the data, so $label get undef:
521 $label = $authorised_values{$t};
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This follow-up adds two major enhancements to the locale-based facet sorting
implementation:
1. **System Preference Integration**
- Adds FacetSortingLocale system preference for configurable locale selection
- Dynamically detects available system locales using Koha::I18N
- Provides user-friendly dropdown with locale descriptions
- Supports both Elasticsearch and Zebra search engines
2. **Zebra Search Engine Support**
- Extends locale-based sorting to Zebra (in addition to existing Elasticsearch)
- Implements _sort_facets_zebra() function in C4::Search
- Maintains consistent Unicode-aware sorting behavior across search engines
Key Features:
- Dynamic locale detection via Koha::I18N::available_locales()
- Proper module architecture with comprehensive unit tests
- Clean fallback chain: preference → system LC_COLLATE → default
- Admin UI integration in System Preferences → I18N/L10N
- Universal facet sorting for international library systems
Test plan:
1. Apply patch and restart services
2. Install/run database update
3. Navigate to Administration → System Preferences → I18N/L10N
4. Verify "Sort facet names using" dropdown shows available system locales
5. Test facet sorting with different locales and both search engines
6. Run: prove t/Koha/I18N.t t/Koha/SearchEngine/Elasticsearch/Search.t t/db_dependent/Search_FacetSorting.t
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This change uses a configurable locale-based collator to sort
the ES facet names.
Test plan:
0. Apply the patch
1. vi /etc/locale.gen
2. Uncomment the locale you want to generate (e.g. fi_FI.UTF-8 UTF-8)
3. locale-gen
4. vi "/etc/default/koha-common"
5. Add the following to the bottom of the file:
export LC_ALL=fi_FI.UTF-8
6. koha-plack --restart koha-common
7. Setup some test records with authors with accented and unaccented names,
and different cases for the lead letter
e.g. Aa author, Åa author2, aa author, étienne
8. Switch to using Elasticsearch and reindex
koha-elasticsearch -b -v --rebuild kohadev
9. Do a test search
e.g. http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=test
10. Confirm the facet names are sorted in ascending order following
Finnish collation rules
e.g.
aa author
Aa author
étienne
Farley, David
Humble, Jez
Martin, Robert C.
Åa author
NOTE: Any collation and language can be used. Finnish is just
an example of a Latin-based script which has a different
alphabetical ordering than just A-Z
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
With bug 30280 and subsequent patches we attempt to respect the
thesaurus indication when linking with LinkerConsiderThesaurus on.
However, there are still cases when we link wrong.
This happens when thesaurus in bibliographic field 6XX is defined by
subfield $2. In such a case the authority record should have 008/11 = 'z'
and, according to https://www.loc.gov/marc/authority/ad008.html
(cf. comment for 008/11 = 'z': "A MARC code for the conventions used to
formulate the heading *may* be contained in subfield $f"), may or may have
thesaurus declared in 040 $f.
This is why we make a second search attempt in C4::Heading::_search,
but doing so we have to make sure that the resulting authority
record has not a different thesaurus declared in 040 $f. (Note
that 008/11 and 040 $f are indexed with the same search
field: subject-heading-thesaurus. So we search for the
second time in the same search field but with
'notdefined' == 008/11='z' instead of the original
thesaurus taken from subfield $2 of the bibliographic
field 6XX.) This means we have to explicitly check the
content of 040 $f of the resulting authority record.
Test plan:
==========
1. Turn on the LinkerConsiderThesaurus systempreference.
2. Create an authority record (e.g. TOPIC_TERM) with 008/11 = 'z' and
040 $f containing a thesaurus indication (e.g. 'kaba'), and a term
in field 150 $a (e.g. Early music).
3. Edit a bibliographic record: in field 650 put 2nd indicator '7',
the term 'Early music' in $a, and a different thesaurus code in $2
(e.g. 'dbn'). Click the 'Link authorities automatically' button.
Note that 650 'Early music' (dbn) has been linked with and authority
record 'Early music' (kaba). This is wrong.
4. Apply the patch ; restart_all ; reindex ES for authorities with:
koha-elasticsearch --rebuild -d -a kohadev
5. Repeat p. 3. Now the 650 'Early music' (dbn) has not been linked to
an authority record (650 - No matching authority found.)
Signed-off-by: Janusz Kaczmarek <januszop@gmail.com>
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
When building queries for matchign Zebra uses OR to combine searches - Elasticsearch
has been effectively using 'AND' - this patch checks whether and_or has been passed
and follows the variable when present.
To test:
1 - Have an import file of auths with multiple 035
2 - Setup an authority matching rule
Search index: other-control-number
Score: 1000
Tag: 035
Subfields: a
3 - Make sure Koha is using Elasticsearch
4 - Stage file without matching, import file
5 - Stage file again, using your matching rule
6 - Note all records match
7 - Setup Cataloging->Marc modification template
8 - Add an action to delete first 035 field
9 - Run it against the records you just imported
You can use a report:
SELECT authid FROM auth_header WHERE authid > #### (find the lowest new record number in batch you imported)
10 - Change matching rule in batch to none, then back to your rule
11 - No matches!
12 - Apply patch, restart all
13 - Set matchign rule to none, then back to your rule
14 - Matches!
Signed-off-by: Magnus Enger <magnus@libriotech.no>
Works as advertised!
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
to keep the syspref names consistent with the existings values. Also "ES" is not as self-documenting as "Elasticsearch".
No code change, so the test plan stays the same.
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
What this patch does is:
1 - Wraps the existing search code in a a "bool" compound query as a "must". This should not affect
relevancy or results of the existing searches.
2 - Before we clean/truncate terms, loop through the passed in search terms and indexes to build a
new 'should' query, using the 'match' on the specified index/field that is added to the 'bool' query from above.
This means that if a result from the original query is also returned here, that item will be boosted in the result.
For searches on 'keyword' or 'title', or if no index is set, we use 'title-cover' as the most narrow form of title
This query isn't going to help when users enter CCL (i.e. ti:To die for) and it won't boost titles from 505, series, etc
when doing a general search. Nor will it have a detrimental effect, it will only boost field matches
To test:
1 - Add a record with
245 $a novel
2 - Add a record with
245 $a A novel : $b about things / $c by me
3 - Search for: novel
- Above records are returned lower in the list (results #6 and #14 for me)
4 - Search for: a novel
- Above records are returned lower in the list (results #6 and #14 for me)
5 - Apply patch, restart all, enable ESBoostFieldMatch option
6 - Repeat searches, note exact titles are boosted
- For 'novel' record from #1 is #1 result and other is second
- For 'a novel' record form #2 is #1 result and other is third
7 - Disable the new pref and repeat steps 3 and 4 and get results as before patch
8 - Experiment with other searches, turning pref on and off to verify relevant titles are boosted when enabled
9 - Search results when disabled should return as before the patch
Signed-off-by: Matthias Meusburger <matthias.meusburger@biblibre.com>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Added head2 right before sub with missing POD
Signed-off-by: Baptiste Wojtkowski <baptiste.wojtkowski@biblibre.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
To test:
1 - Set search engine to ES
2 - Search the catalog for "*"
3 - Note results
4 - Edit first result
5 - Repeat search, results are reordered
6 - Apply patch
7 - Search for "*"
8 - Highest biblionumbers are returned first
9 - Edit first result, repeat search
10 - Order does not change
11 - Confirm sorting by author, title, etc passes the same trials
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Add a new hook elasticsearch_to_document in order to allow completing
the 'document' to be sent to Elasticsearch. Usage example: (1) add new
search fields, (2) normalize/sanitize existing fields, (3) use id fields
to retrieve info via web services in other repositories (say VIAF), (4)
download file referenced in a field, and index its content.
To test:
1. Apply the unit tests patch, and run it:
prove t/db_dependent/Koha/Plugins/Biblio_and_Items_plugin_hooks.t
=> FAIL: Tests fail! The hook is not implemented so the desired
2. Apply this patch, and then repeat 1:
=> SUCCESS: It works!
Signed-off-by: Shi Yao Wang <shi-yao.wang@inlibro.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds a new option to Elasticsearch to skip facets in build_query and
build_query_compat.
There should be no behavior change, however, queries to ES should be simpler
To test:
1 - Export some record from Koha
2 - Stage the records for import, matching on 999c
3 - Confirm expected matches
4 - Switch batch to use ISBN for matching
5 - Confirm expected matches
6 - Apply patches, restart all
7 - Switch batch to use 999c
8 - Confirm matches
9 - Switch batch to use ISBN
10 - Confirm matches
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
As elsewhere in the code, we should use a state variable for our
'bib_searcher' object to avoid overhead of extra ES connections.
Worth noting - we don't actually use the 'bib_searcher' at all during
record auto linking as we pass skipmetadata
To test:
0 - Apply patch
1 - Enable biblio auto linking
AutoLinkBiblios, CatalogModuleRelink, LinkerRelink
2 - Stage and import a marc file
3 - Confirm record authorities link as expected
4 - Search authorities in the staff interface
5 - Confirm usage counts are shown for results
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This commit is generated using:
% perl misc/devel/tidy.pl
*within* ktd, to get the same version of perltidy than what will be used
by our CI (currently v20230309).
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
After Bug 38416 Elasticsearch indexing explodes with oversized
records, especially with UTF encoded data.
In Koha::SearchEngine::Elasticsearch::marc_records_to_documents a
following snippet has been introduced:
my $usmarc_record = $record->as_usmarc();
my $decoded_usmarc_record = MARC::Record->new_from_usmarc($usmarc_record);
But if $record is oversized (> 99999 bytes), it is OK for MARC::Record
object, but not for $record->as_usmarc. The produced ISO 2709 string
is not correct and hence cannot be properly converted back to
MARC::Record object by new_from_usmarc.
The result in this case can be like:
UTF-8 "\x85" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35.
Since it is done without any eval / try, the whole reindex procedure
(for instance rebuild_elasticsearch.pl) is being randomly interrupted
with no explanation.
Test plan:
==========
Hard to reproduce. But the explanation together with discussion in Bug
38416 (from 2024-12-15) explains and justifies the need of this added
eval.
1. Have a standard KTD installation with Elasticsearch.
2. Use the provided test record - add it to Koha with
./misc/migration_tools/bulkmarcimport.pl -b -file test.xml -m=MARCXML
(have patience).
During load process you should see a message like:
UTF-8 "\xC4" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35.
3. The record should get biblionumber 439. Check in librarian interface with
http://<your_addreess>:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=439
that the record has been imported.
However, you should not be able to make a search for this record.
4. Try to reindex with:
./misc/search_tools/rebuild_elasticsearch.pl -b -bn 439
You should get a message like:
UTF-8 "\xC4" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35.
Again, no search results.
5. Apply the patch ; restart_all.
6. Repeat reindex with:
./misc/search_tools/rebuild_elasticsearch.pl -b -bn 439
There should be no warning now and you should be able to find the record.
Signed-off-by: Magnus Enger <magnus@libriotech.no>
Followed the test plan. Works as advertised.
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch makes the `_process_mappings()` method split the index values
in the event of them being bigger than the allowed 32766 bytes size.
To test:
1. Have KTD running with ES:
$ ktd --proxy --es7 up -d
2. Perform a search
3. Pick the first result for edition
4. Find a cool Wiki page with lots of paragraphs
5. Copy all of the paragraphs and put them on a 500$a field for the record.
6. Repeat 2
=> FAIL: The record is not found
7. Reindex manually:
$ ktd --shell
k$ perl misc/search_tools/rebuild_elasticsearch.pl --biblios --where "biblionumber=3" -v -v
=> FAIL: You get something like:
```
[22229] Committing final records...
One or more ElasticSearch errors occurred when indexing documents at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Indexer.pm line 148.
[22229] There were errors during indexing
Record #3 Document contains at least one immense term in field="note.raw" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[10, 109, 117, 115, 116, 97, 102, 97, 32, 102, 117, 101, 32, 101, 108, 32, 115, 101, 103, 117, 110, 100, 111, 32, 104, 105, 106, 111, 32, 100]...', original message: bytes can be at most 32766 in length; got 32771 (illegal_argument_exception) : max_bytes_length_exceeded_exception (bytes can be at most 32766 in length; got 32771)
[22229] Total 1 records indexed
```
8. Apply this patch
9. Repeat 7
=> SUCCESS: No error!
10. Repeat 2
=> SUCCESS: The record is indexed and can be found!
11. Sign off :-D
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This change failsover to MARCXML from USMARC if there are any
warnings generated by MARC::File::USMARC::decode when trying to
roundtrip the record.
Test plan:
0. Apply the patch
1. Setup your koha-testing-docker to use Elasticsearch
2. Create a new record with 15,000 characters in the 500$a field
3. Index that record
(e.g. perl misc/search_tools/rebuild_elasticsearch.pl --biblios -v -v)
4. Note that a warning saying the following appears:
"Warnings encountered while roundtripping a MARC record to/from USMARC.
Failing over to MARCXML"
5. View the "Elasticsearch record" on the detail page and note that the
marc_format is MARCXML
6. Perform a search for the record (the keyword should be something that
brings up other results too)
7. Note that the record appears correctly in the search results
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
The patch was corrupted, and was rewritten as the original one by
Hammat
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Test plan:
- Enable elasticsearch
- git to labels/label-item-search.pl
- search using after date only
=> check you get items with date-of-acquisition greater than after
date
- search using before date only
=> check you get items with date-of-acquisition less than before
date
- search using after and before date
=> check you get items with date-of-acquisition between after and
before
- Combine this searches with a specified index
- enable zebra and repeat these tests
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
https://bugs.koha-community.org/show_bug.cgi?id=30845
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
And copy fix across to Zebra search as well.
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch ensures the ShowHeadingUse system preference and feature works as expected when using Elasticsearch as the searchengine.
To test, follow the test plan at Bug 29990 Comment 9. Ensure the SearchEngine system preference is set to Elasticsearch.
Sponsored-by: Education Services Australia SCIS
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Some libraries ask for the ability to scan/browse the call numbers index.
Test plan:
==========
1. Have an installation with items with callnumbers (in ktd with standard
test data set: add some callnumbers to the items).
2. Go to Advanced search > More options; choose Call number in the first
drop list and mark Scan indexes. Enter one of the call numbers you
assigned to the items and perform the search. You should get no
results.
3. Apply the pach; restart all; when using ES:
koha-elasticsearch --rebuild -r -d kohadev
4. Repeat p. 2. You should get a list with at least one call number.
The link should lead you to the record(s) with items with the selected
call number.
This should work for ES and for Zebra, but for Zebra you would need to
replace /etc/koha/zebradb/ccl.properties with the repository version; and
you will get one token results on the list, like for other indexes.
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
We can now call metadata_record directly on the Koha::Biblio object.
This aptch updates all modules and controllers to use
Koha::Biblio->metadata_record directly where appropriate. The
exceptions are where we don't require any filtering or the filtering
makes sense to do after the initial fetch.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This change filters the values from the OpacHiddenItems syspref
out of their corresponding search results facets.
Note: This does not mean that all values from hidden items are filtered out of facets.
Rather, it just means that facet data that matches OpacHiddenItems is filtered out. This is
an imperfect fix, but it is a practical fix that has been requested by more than one library.
Test plan:
0. Apply the patch and koha-plack --restart kohadev
1. Go to http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=29
2. Add an item with an item type of "Maps"
3. Go to http://localhost:8081/cgi-bin/koha/admin/preferences.pl?op=search&searchfield=OpacHiddenItems
4. Fill in OpacHiddenItems with the following:
itype: ['MP']
5. Go to http://localhost:8080/cgi-bin/koha/opac-search.pl?idx=&q=test&weight_search=1
6. Note that "Maps" does not appear in the "Item types" facet
7. Go to http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=test
8. Note that "Maps" does appear in the "Item types" facet
9. ***Repeat this test plan using both Elasticsearch and Zebra indexing***
10. prove -v t/Koha/SearchEngine/Search.t
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
With the Bug 36574 we added a new ES search field isbn-all which makes it
possible to find records with the content of the subfield 020 $z
(invalid/cancelled ISBN) in general search (by keyword). But still, when
performing a search with ISBN-search these records will not be found.
Since some libraries deliberately want to find records with only 020 $a
with ISBN-search, a system preference would be needed here to be able to
customize the ISBN search.
The same for ISSNs, with one system preference.
Test plan:
==========
(ISBN)
------
1. Have the ktd with with ES enabled.
2. With standard ktd test data, try to perform a ISBN search from Advanced
search for 9780141930848. You should get no results.
In the general search input field (on the main page) try a search
nb:9780141930848 and/or isbn:9780141930848 - you should get no results.
3. Apply the patch, restart_all.
4. Repeat p. 2 -- you should still get no results.
5. Enable SearchCancelledAndInvalidISBNandISSN syspref.
6. Repeat p. 2 -- now you should get one record (having 9780141930848
in 020 $z).
(ISSN)
------
0. Return to main, delete the bug branch, reset_all == start from fresh.
1. Have the ktd with with ES enabled.
2. With standard ktd test data, try to perform a ISSN search from Advanced
search for 1089-6891. You should get no results.
In the general search input field (on the main page) try a search
ns:1089-6891 and/or issn:1089-6891 - you should get no results.
3. Apply the patch, restart_all.
If you have had to apply also the bug 36727, reindex ES with:
koha-elasticsearch --rebuild -r -d kohadev
(there is a modification of mappings.yaml)
4. Repeat p. 2 -- you should still get no results.
5. Enable SearchCancelledAndInvalidISBNandISSN syspref.
6. Repeat p. 2 -- now you should get one record (having 1089-6891
in 022 $y).
Sponsored-by: Ignatianum University in Cracow
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Tiny adjustment for very special and malicious cases needed in
Koha::SearchEngine::Elasticsearch::QueryBuilder::clean_search_term.
Extended unit tests added following Nick's suggestions in
Bug 33407 discussion.
Test plan
=========
1. Use ktd with Elasticsearch and ktd's test data.
2. Edit the 4th record (The C programming language) and add a semicolon
followed by a space before the slash in 245 $a to have:
The C programming language ; /
3. Make a search for the first ISBD zone taken verbatim from the forth
record:
The C programming language ; / Brian W. Kernighan, Dennis M. Ritchie
There should be no result.
4. Apply the patch, restart plack.
5. Repeat the search. You should get the record.
Sponsored-by: Ignatianum University in Cracow
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Test plan:
With ElasticSearch enabled,
* Perform a search using the default sort order
(i.e. 'relevance').
* Verify that no warnings are generated in
plack-intranet-error.log
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Test plan:
1. Apply patch
2. Rerun and make sure the error "Use of uninitialized value $sub6 in pattern match (m//) at /usr/share/koha/lib/Koha/SearchEngine/Elastricsearch.pm" is no longer present.
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Initial testing done, it works well, needs an additional sign off from an external party
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
The highlight logged-in library in facets feature does not work with
Elasticsearch. This is because facet_title_value contains an additional
" (count)" information and so gives negative results in comparisons made
in facets.inc (line 107).
Test plan:
==========
1. Have a test installation with Elastiscsearch.
2. Set system preference 'DisplayLibraryFacets' to 'both home and holding libraries'
3. Perform a seach that will find items of your logged-in library.
In KTD, search for 'branch:CPL'
4. Check the facet (home and holding libraries) of your logged-in
library is not highlighted
5. Apply the patch ; restart_all.
6. Repeat p. 3. Check the facet (home and holding libraries)--you
should see your logged-in branch highlighted now.
Sponsored-by: Ignatianum University in Cracow
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Like IncludeSeeFromInSearches adds see from heading in search (4xx),
new system preference IncludeSeeAlsoFromInSearches adds see also from heading (5xx).
Test plan:
Test on both Elasticsearch and Zebra:
1)
1.1) Create an authority record with heading, see from heading 4xx and see also from heading 5xx
1.2) Use this authority in a biblio record
2)
2.1) Set IncludeSeeFromInSearches and IncludeSeeAlsoFromInSearches to "Don't include"
2.3) Rebuild search engine
2.4) Perform a search on heading text => you find the biblio record
2.5) Perform a search on see from heading => you do not find the biblio record
2.6) Perform a search on see also from heading => you do not find the biblio record
3)
3.1) Set IncludeSeeFromInSearches and IncludeSeeAlsoFromInSearches to "Include"
3.2) Rebuild search engine
3.3) Perform a search on heading text => you find the biblio record
3.4) Perform a search on see from heading => you find the biblio record
3.5) Perform a search on see also from heading => you find the biblio record
Signed-off-by: Frédéric Demians <f.demians@tamil.fr>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Nick Clemens
Thomas Klausner
David Gustafsson
Janusz Kaczmarek
Shi Yao Wang
Tomás Cohen Arazi
Jonathan Druart
Martin Renvoize
David Cook
Frédéric Demians
Koha Development Team
Olivier V
Alex Arnaud
Aleisha Amohia
Martin Renvoize
Andreas Jonsson
Catalyst Bug Squasher
Kyle M Hall
Fridolin Somers