Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 5e0dbf0adb)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit e16cd2afc5)
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
(cherry picked from commit 9dda7ba2d3)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
The match-heading field is a special field used only by the linker, not accessible
to staff or patrons via the interface. This field is used to store the constructed
'search form' used for matching bib headings to authority fields.
In bug 24269 I attempted to use the mappings defined in the inferface and also inject the search term.
This did not work as too many subfields were indexed on their own and leading to false matches.
In this bug we remove the mappings for this field, and create it ourselves during
the indexing process. The C4::Headings module is still used to generate the correct form,
however, the mappings are set based on the authority types in the system. This gives the user
the ability to add new typoes, but prevents mapping changes from breaking linker functionality
To test:
1 - Start form a sample database with ElasticSearch working
2 - Download via Z39.50 2 authorities, one of which is a narrower heading of the other, e.g.:
Waterworks
Waterworks - Costs
3 - Place a heading for the broader term in a record. e.g. Waterworks
In 650$a, without the cataloguing authority plugin. We don't want
the link created now.
You need syspref BiblioAddsAuthorities => allow
4 - Make sure linker is set to default
5 - Attempt to link the records
misc/link_bibs_to_authorities.pl
6 - Linking fails
7 - Apply patch
8 - refresh index settings (if using a custom file, remove 'match-heading')
You can reset mappings in the UI or run this:
misc/search_tools/rebuild_elasticsearch.pl -v -d -r
9 - Reindex ES
10 - Try to link again
11 - It succeeds!
12 - Run the tests
prove t/db_dependent/Koha/SearchEngine/Elasticsearch.t
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Bug 25273: (follow-up)
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit ce161fda9b)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit 5597b666e4)
Signed-off-by: Aleisha Amohia <aleishaamohia@hotmail.com>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 1a2d7f1eac)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit dbc7f861d4)
Signed-off-by: Aleisha Amohia <aleishaamohia@hotmail.com>
Test plan:
prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
--> should return green
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit d08ce1d940)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit 877cc214f0)
Signed-off-by: Aleisha Amohia <aleishaamohia@hotmail.com>
To test:
0 - Set DisplayLibraryFacets to 'both' and FacetMaxCount to 20
1 - Have more than 10 branches
2 - Have items in each of those branches
3 - Enable ES, set system preference SearchEngine to Elasticsearch
4 - Search for '*'
5 - Expand homebranch/holdingbranch facets
6 - Note you only get 10
7 - Apply patch
8 - Repeat search
9 - Now you get all your facets
Signed-off-by: Michael Springer <mspringer@mylakelibrary.org>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 4dd1672c1b)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit 18215fbc57)
Signed-off-by: Aleisha Amohia <aleishaamohia@hotmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Joy Nelson <joy@bywatersolutions.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Joy Nelson <joy@bywatersolutions.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Joy Nelson <joy@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Joy Nelson <joy@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Joy Nelson <joy@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Joy Nelson <joy@bywatersolutions.com>
Not only should terms from facets/limits be grouped, but order should probably matter
We build a different facet for "Dillinger Girl" or "Girl Dillinger"
The parens are possibly overkill now, but I think it makes it very clear and does not hurt
To test:
1 - Follow original plan
2 - Create a new record with 'Girl Dillinger' as author
3 - Search for 'Dill*'
4 - You get all three records (and maybe others that match)
5 - Limit by 'Girl Dillinger' - you get two records
6 - Same for 'Dillinger Girl'
7 - Apply patch
8 - Limits/facets for 'Dillinger Girl' and 'Girl Dillinger' now match a isngle record
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Joy Nelson <joy@bywatersolutions.com>
Test plan:
- Use Elasticsearch 6 (you'll need Bug 18969),
- create a biblio (#1) with "Dillinger Girl" in author and what you
want in title,
- create another biblio (#2) with the word "girl" in the title and
"Dillinger Escaplan" as author
- reindex
- search * and refine on "Dillinger Girl"
- Ko => Biblio #1 and #2 appear
- Apply this patch,
- search * and refine on "Dillinger Girl"
- Ok => anly biblio #1 appears
- use Elasticsearch 5 again
- check for no search regression
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Joy Nelson <joy@bywatersolutions.com>
We altered the query form passed by the ElasticSearch Query Builder
but neglected to update the test.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Joy Nelson <joy@bywatersolutions.com>
Note: I also remove warnings for undefined operation in this patch, is a trivial fix
To test:
prove -v t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
To test:
1 - Export your authorities via Tools->Export data
2 - Define a record matching rule in Admin->Record matchign rules
Use index: LC-card-number
field: 010$a
3 - Stage the exported records for import and use the rule created above for matching
4 - The process does not complete
5 - Check intranet error logs - exception on unknown marclist
6 - Apply patch
7 - Repeat
8 - Success!
9 - prove -v t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Adds support for using the "scan indexes" action in advanced search by using faceting with a prefix filter. Requires that the field be set as facetable for anything to be found.
Test plan:
1. Apply patch
2. Go to advanced search and click "More options"
3. Select author as the search field, enter a last name and check "Scan indexes"
4. Perform search and observe the result list resembling scan results
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Elasticsearch indexing uses 999$c to store record id by deleting the all field first !
So you can not store anything in field 999, even in UNIMARC and even in authorities records.
Looks like it is quick fix code added to start Elasticsearch use.
This behavior is disturbing and very strange for UNIMARC flavour.
This patch corrects by defining record ids mandatory in Koha::SearchEngine::Elasticsearch::Indexer::update_index().
This ids array is actually always given (except in UT).
I think it is useless to allow adding a record without its id.
Test plan :
1) Use Elasticsearch as SearchEngine
2) Create a subfield 999$z in default framework
3) Create a record with default framework
4) Enter a random string (never used in catalog) like "tototata" in 999$z
5) In Search engine configuration, define search field "subject" for 999$z
6) Rebuild record : misc/search_tools/rebuild_elasticsearch.pl -b -bn <biblionumber> -v
7) Search for the random string => You get a result
8) Optionnaly look at records in ES : <es server>:9200/<es index name>/data/<biblionumber>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Test plan
- Set up a elasticsearch 6 instance to work with Koha,
- you may need to make koha works with ES 6 (see bug 20589),
- make a search and limit it to available items only,
=> no result
- Apply this patch,
- make a search and limit it to available items only,
- you should get some results
- Do the same with ES 5.x
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Thi is a rebase from 14419 to remove an unused variable
Also update tests to expect the new expected result
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Generate a list of fields for the query_string query fields parameter,
with possible boosts, instead of using "_all"-field. Also add "search"
flag in search_marc_to_field table so that certain mappings can be
excluded from searches. Add option to include/exclude fields in
query_string "fields" parameter depending on searching in OPAC or staff
client. Refactor code to remove all other dependencies on "_all"-field.
How to test:
1) Reindex authorities and biblios.
2) Search biblios and try to verify that this works as expected.
3) Search authorities and try to verify that this works as expected.
4) Go to "Search engine configuration"
5) Change some "Boost", "Staff client", and "OPAC" settings and save.
6) Verify that those settings where saved accordingly.
7) Click the "Biblios" or "Authorities" tab and change one or more
"Searchable" settings
8) Verfiy that those settings where saved accordingly.
9) Try to verify that these settings has taken effect by peforming
some biblios and/or authorities searches.
Sponsorded-by: Gothenburg Univesity Library
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Also fixes the mappings.yaml to use correct field name (left over issue from bug 19575), so reset mappings and reindex before testing.
Test plan:
1. Reset mappings and reindex biblios.
2. Check that tests in t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t pass.
3. Try that all of the following year range type work in publication date search and year limit in advanced search:
yyyy
yyyy-yyyy
-yyyy
yyyy-
Signed-off-by: Liz Rea <wizzyrea@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Test plan:
- apply this patch,
- edit admin/searchengine/elasticsearch/mappings.yaml
to add weight for some fields,
- go to admin > search engine configuration,
- reset your mappings (add ?op=reset&i_know_what_i_am_doing=1 to url),
- check that weights you've added are set
Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch removes the constraint of only passing 5 facets to the template unless the list is expanded, in fact, it removes the 'expanded' attribute from Search.pm
Now that all facets are passed to page it adds a 'show more' link at the bottom of lists and allows user to expand or collapse any facet set without reloading page.
Updated tests included.
To test:
1 - Perform an OPAC search that returns more than 5 of any given facet type
2 - Click the "Show more" link on the facets and see that the search is reloaded
3 - Apply patch
4 - Repeat search
5 - Note that you can click "Show more" without reloading page
6 - Test that page load is not greatly affected
7 - Ensure that all facet links function normally
8 - Ensure that facets are the same a prior to patch
9 - Repeat for staff client
10 - Prove t/Search.t
NOTE: This patch makes it much easier to see that there is an existing issue with marking the "active" facet. Ending punctuation seems to confuse the matcher.
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Test plan:
After each of the following steps, verify that the results and query description looks ok.
1. Do a simple keyword query
2. Change sort order
3. Click a facet
4. Do an advanced search with title and author
5. Change sort order
6. Click a facet
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Marjorie <marjorie.barry-vila@collecto.ca>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Test plan:
1. Do an advanced search for
Title = new
AND
Title = york
2. Verify that the results match an advanced search for:
Title = new york
3. Verify that tests in t/db_dependent/Koha/SearchEngine/Elasticsearch still pass
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
This patch lower cases the sort by fields to normalize checking them and adjusts
some existing tests to meet the new expectations.
The regex for splitting terms has been moved into a subroutine so that adjustment was made
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Adjust elastic search mappings to more closely match Zebra equivalents
resolving serveral issues with coded Zebra searches in templates and
sorting of search results in UI. Also make field names in search strings
case insensitive to accept case variations in template links and user input.
Sponsored-by: Gothenburg University Library
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
- Makes token splitting work like in biblio search
- Only adds right hand truncation since that's what Zebra also does
- Only adds truncation if the token is not a phrase and ends in a word character
- Adds tests to existing and new QueryBuilder functions
Test plan:
1. Create an authority record for "Duck, Donald"
2. Try contains type authority searches with the following terms and make sure they find the record:
Duck, Donald
donald duck
don duck
"Duck, Donald"
3. Make sure the following search works but does not find anything:
"Duck, Don"
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Moved db_dependent tests to the proper directory in the hierarchy, fixed tests to check correct results now that the authority query builder works better and added several new tests to cover changed functionality.
Sponsored-by: National Library of Finland
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Rebased-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
1 - Apply previous patches
2 - prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
3 - Should be green
4 - high fives!
Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: David Bourgault <david.bourgault@inlibro.com>
Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This test makes it explicit that the only string producing _all as index on build_authorities_query_compat is 'all'.
To test:
- Apply this patch
- Run:
$ sudo koha-shell kohadev
k$ cd kohaclone
k$ prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
=> FAIL: Test fails because the list of valid values is wrong in Koha.
Note: this list has to be in sync with the templates passing the same values. A followup will be added
to fix a discrepancy found between OPAC and Intranet.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
When we will replace the Zebra code in sub merge, we will call SearchEngine
to pass records and we need a routine to extract a biblionumber from
a search result record. A record from Zebra still must be converted to
MARC::Record. This is no longer needed for a ElasticSearch record.
Test plan:
Run t/db_dependent/Koha/SearchEngine/Search.t
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Marc Véron <veron@veron.ch>
Signed-off-by: Jacek Ablewicz <abl@biblos.pk.edu.pl>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Nick Clemens
Josef Moravec
Jonathan Druart
Alex Arnaud
Martin Renvoize
Ere Maijala
Fridolin Somers
David Gustafsson
Alex Arnaud
root
Tomas Cohen Arazi
Marcel de Rooy