Commit Graph

149 Commits

Author SHA1 Message Date
Nick 2b769cbc32 Bug 23089: Fix sorting on sum and regular fields
As of bug 20589 we no longer analyze sort fields and so we no longer need to append ".phrase"
to our sort in searches.

Additionally, sort fields based on 'sum' should also use sum in building the value to sort on

To test:
0 - Be using ES
1 - Find the most circulated item in your collection
2 - Search for '*'
3 - Sort by popularity DESC
4 - Note that item is not first
5 - Try to sort by anything but relevancy, it fails
6 - Apply patch
7 - Redo searches and sorts
8 - Things should now work as expected

Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
(cherry picked from commit 6a2ba8b6b2)
Signed-off-by: Hayley Mapley <hayleymapley@catalyst.net.nz>
2019-12-18 14:22:38 +13:00
Nick Clemens 340e2da40b Bug 23089: [19.05 and earlier] Use integer field directly for sorting
(cherry picked from commit 55ccf90711)
2019-12-18 09:00:13 +13:00
Fridolin Somers d161374e01 Bug 23004: Missing authtype filter in auth_finder.pl
In cataloguing, the use of tag editor opens authorities finder with a limit on specific authorities type.
This limit is missing with Elasticsearch.

This patch adds in query a "filter" on "term" which is the most performant way because there must be no ranking computed on this part.

Test plan :
1) Use Elasticsearch
2) Create an autority of type author (NP in UNIMARC) with heading "Tolkien"
3) Create an autority of type subject-author (SAUT in UNIMARC) with heading "Tolkien"
4) Create a biblio record
5) Use tag editor on a author field (700 in UNIMARC)
6) Seach for "Tolkien" in $a
without patch : you see the 2 authorities in results
with patch : you see only the correct authority in results (NP in UNIMARC)
7) Check search in authorities-home.pl is still OK

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
(cherry picked from commit 949e36c7b8)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit 1d8b1fafc4)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2019-10-17 00:04:24 +00:00
Ere Maijala bc69881893 Bug 22524: Fix date/time-last-modified search with Elasticsearch
Avoid using slash in the field name since it would need to be escaped. Fix conversion of dtlm and any existing mapping.

Signed-off-by: Liz Rea <wizzyrea@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
(cherry picked from commit 5d27fb71cb)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit ab0de52ab2)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2019-08-31 03:37:55 +00:00
Ere Maijala 22585ec127 Bug 22524: Fix publication date and year search with Elasticsearch
Also fixes the mappings.yaml to use correct field name (left over issue from bug 19575), so reset mappings and reindex before testing.

Test plan:
1. Reset mappings and reindex biblios.
2. Check that tests in t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t pass.
3. Try that all of the following year range type work in publication date search and year limit in advanced search:
yyyy
yyyy-yyyy
-yyyy
yyyy-

Signed-off-by: Liz Rea <wizzyrea@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
(cherry picked from commit 2293e6597c)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit d21a1fb964)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2019-08-31 03:35:36 +00:00
Fridolin Somers 60b4ab8a92 Bug 21534: Always compute wildcards
You get no results when searching with an hyphen + with * in query string (or with preference QueryAutoTruncate) :
ie /cgi-bin/koha/opac-search.pl?q=saints-anges*

Looks like query-string by default does not compute wildcards, see analyze_wildcard in :
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html

Test plan :
1) Use Elasticsearch
2) Create a record with "saints-anges"
3) Search for "saints-anges" => you get results
4) Search for "saints-anges*" => you get results
5) Search for "saints-ang*" => you get results

Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Arthur Bousquet <arthur.bousquet@inlibro.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
(cherry picked from commit 001d433b42)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit a8cc9d5f49)
2019-07-26 06:00:41 +00:00
Julian Maurice 022b696057
Bug 22705: Change default value of cxn_pool to 'Static'
'Static' is the default value of Search::Elasticsearch and for a good
reason : it works in most cases, unlike the 'Sniff' option

To test:
1 - Apply patch
2 - Edit koha-conf.xml
3 - Add '<cxn_pool>Static</cxn_pool>' to the elasticsearch stanza
4 - Restart all the things!
5 - Reindex ES, it works
6 - Set SearchEngine to ES, try searching
7 - It works!

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
(cherry picked from commit 9532a3edea)
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-05-30 20:41:28 +01:00
Ere Maijala e2aa81da2c
Bug 22413: Improve query string and desc creation for Elasticsearch
Test plan:

After each of the following steps, verify that the results and query description looks ok.

1. Do a simple keyword query
2. Change sort order
3. Click a facet
4. Do an advanced search with title and author
5. Change sort order
6. Click a facet

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>

Signed-off-by: Marjorie <marjorie.barry-vila@collecto.ca>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
(cherry picked from commit aba8b130f8)
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-04-26 16:24:15 +01:00
Ere Maijala b1290c0602
Bug 22339: Fix search field mappings of MARC fixed fields
Fixes handling of /0 selector in addition to several fixed field positions. Note that ff7-01-02 is in fact 00-01 in Zebra and that has been replicated here.

Test plan:
1. Before applying a patch, check from Elasticsearch (e.g. localhost:9200/koha_biblios/_search?q=_id:123) what is indexed in Elasticsearch for a record for date-entered-on-file, ff7-00, ff7-01, ff7-02 and lleader.
2. Apply the patch, update database and save the record again.
3. Verify that the contents of the forementioned fields is now correct in Elasticsearch.
4. Verify that tests pass:  prove -v t/Koha/SearchEngine/Elasticsearch.t

Signed-off-by: Björn Nylén <bjorn.nylen@ub.lu.se>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
(cherry picked from commit 0047ede37e)
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-04-26 16:20:48 +01:00
Ere Maijala 16846985e1
Bug 22474: Clone field config before adding fields to it
Test plain:

1. Reindex both authorities and biblios with the -d flag to recreate the index, e.g. "perl misc/search_tools/rebuild_elastic_search.pl -v -d"
2. Check that authorities only contains authority-specific fields in Elasticsearch e.g. by fetching  http://localhost:9200/koha_dev_authorities/_mappings
3. Reindex both without the -d flag and verify it works, e.g. "perl misc/search_tools/rebuild_elastic_search.pl -v"

Signed-off-by: Björn Nylén <bjorn.nylen@ub.lu.se>
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
(cherry picked from commit 118f685dbf)
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-04-25 15:04:38 +01:00
Ere Maijala 8f8bcbbc0d
Bug 22295: Make Elasticsearch query builder group multi-term queries
Test plan:

1. Do an advanced search for
Title = new
AND
Title = york
2. Verify that the results match an advanced search for:
Title = new york
3. Verify that tests in t/db_dependent/Koha/SearchEngine/Elasticsearch still pass

Signed-off-by: Michal Denar <black23@gmail.com>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
(cherry picked from commit e90cb16d79)
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-04-15 12:30:04 +01:00
Ere Maijala 9ef8a3e328
Bug 21974: Make Elasticsearch connection settings configurable
Default settings are appended in get_elasticsearch_params so that there's a central place for them. Defaults can be overridden in koha-conf.xml.

Test plan:

1. Verify that searches still work with the patch applied.
2. Verify that search fails after adding the following under <elasticsearch> in koha-conf.xml:
<cxn_pool>Static</cxn_pool>
3. Verify that search fails after adding the following under <elasticsearch> in koha-conf.xml:
<cxn_pool>FOO</cxn_pool>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
(cherry picked from commit 65f35447e9)
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-04-01 11:42:17 +01:00
Ere Maijala 7d741a6991
Bug 22246: Fix indexing of large fields with Elasticsearch
Deduplicate multivalued fields and make sure sort fields are not excessively long. Also updates default mappings so that sort fields are not created for item fields where it doesn't make sense.

Test plan:
1. Reset ES mappings in administration
2. Check that sort is '0' for local-classification in biblio mappings.
3. Change sort back to '1' for local-classification for the next steps.
4. Create a record with 20 items, each with a 100 character long call number
5. Check that when indexed, the record in ES does not have duplicates in any of the item fields and local-classification__sort is truncated to 255 characters.

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
(cherry picked from commit bd68e2f75b)
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-03-12 08:46:52 +00:00
Ere Maijala 33489f2df1
Bug 22228: Escape lone colons and make sure term is balanced
Escapes unquoted colons that have whitespace on either side. Removed unbalanced quotes.

Test plan:

1. Make sure the test case described in the bug works
2. Make sure tests pass:
  prove t/Koha/SearchEngine/Elasticsearch/QueryBuilder.t

Signed-off-by: Björn Nylén <bjorn.nylen(at)ub.lu.se>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
(cherry picked from commit dd98584c85)
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-03-05 15:08:25 +00:00
David Gustafsson e522e32ea3
Bug 19575: Keep authority-number as alias and fix query
Keep authority-number as alias and change field name
from alias to real field in hard coded Elasticsearch query

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
(cherry picked from commit eeda2afd3a)
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-03-05 14:52:07 +00:00
Nick Clemens dc219b6c10
Bug 19575: Fix an search field
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
(cherry picked from commit ef2a03a092)
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-03-05 14:51:47 +00:00
root ce578a9aca
Bug 19575: Rebase and fix tests
This patch lower cases the sort by fields to normalize checking them and adjusts
some existing tests to meet the new expectations.

The regex for splitting terms has been moved into a subroutine so that adjustment was made

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
(cherry picked from commit c2061adc8b)
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-03-05 14:51:27 +00:00
David Gustafsson 68b129e0c4
Bug 19575: Use canonical field names and resolve aliased fields
Adjust elastic search mappings to more closely match Zebra equivalents
resolving serveral issues with coded Zebra searches in templates and
sorting of search results in UI. Also make field names in search strings
case insensitive to accept case variations in template links and user input.

Sponsored-by: Gothenburg University Library

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
(cherry picked from commit 6d53910f0c)
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-03-05 14:50:59 +00:00
Ere Maijala 66e041fece
Bug 21084: Fix automatic truncation in authority search
- Makes token splitting work like in biblio search
- Only adds right hand truncation since that's what Zebra also does
- Only adds truncation if the token is not a phrase and ends in a word character
- Adds tests to existing and new QueryBuilder functions

Test plan:

1. Create an authority record for "Duck, Donald"
2. Try contains type authority searches with the following terms and make sure they find the record:
Duck, Donald
donald duck
don duck
"Duck, Donald"
3. Make sure the following search works but does not find anything:
"Duck, Don"

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
(cherry picked from commit 0d9a42b492)
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-01-23 12:43:33 +00:00
Martin Renvoize 277f9be04b Bug 19365: Clean up, fix and add new tests.
Moved db_dependent tests to the proper directory in the hierarchy, fixed tests to check correct results now that the authority query builder works better and added several new tests to cover changed functionality.

Sponsored-by: National Library of Finland
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-26 16:11:02 +00:00
Ere Maijala a4c02b53cd Bug 19365: Fix search for duplicate authorities with Elasticsearch
Sponsored-by: National Library of Finland
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-26 16:11:02 +00:00
Ere Maijala be2bdfd511 Bug 19365: Fix several issues with the Elasticsearch code
Also optimize it so it's actually usable.

Test plan:

1. To test it properly you need biblio and authority data. You might get away with enabling AutoCreateAuthorities and BiblioAddsAuthorities so that authorities are created in the process. Another option would be to import authorities first e.g. from LoC.
2. Make sure the authority index has been properly created with "misc/search_tools/rebuild_elastic_search.pl -a -d"
3. Run "misc/link_bibs_to_authorities.pl -v -l" twice and observe the results.

Sponsored-by: National Library of Finland
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-26 16:11:01 +00:00
Nick Clemens 6ef2233093 Bug 20244: Pod fixes
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-16 12:08:39 +00:00
Ere Maijala aece4f62ea Bug 20244: Add alt script indexing and fix sort field indexing
https://bugs.koha-community.org/show_bug.cgi?id=20244

Test plan:

1. Add a record with alternate script fields in 880 (sample attached in the bug).
2. Make sure the the record can be found e.g. with the alternate script title.
3. Add a record with multiple authors.
4. Check that in the index there is only a single author__sort field.

Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-16 11:27:22 +00:00
Ere Maijala 5a0f2821a2 Bug 20244: Improve Elasticsearch ISBN indexing
https://bugs.koha-community.org/show_bug.cgi?id=20244

Test plan:

1. Add a record with an ISBN-10 or ISBN-13 that can be converted to ISBN-10 (e.g. 1-56619-909-3).
2. Verify that the record can be found by searching for "1-56619-909-3", "1566199093", "978-1-56619-909-4" and "9781566199094".

Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-16 11:27:22 +00:00
Martin Renvoize da22f3cc33 Bug 19893: (QA follow-up) Spelling correction in POD
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-16 11:05:00 +00:00
David Gustafsson 2f4b141892 Bug 19893: Increase test coverage
Sponsored-by: Gothenburg University Library
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-16 11:04:59 +00:00
David Gustafsson 6d47d82ac9 Bug 19893: Add code review fixes
Sponsored-by: Gothenburg University Library
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-16 11:04:59 +00:00
David Gustafsson 1d29d0e449 Bug 19893: Add pods, remove syspref, add tests for serialization format
Add missing pods, remove obsolete syspref and add test for serialization format for records exceeding max record size

Sponsored-by: Gothenburg University Library
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-16 11:04:59 +00:00
David Gustafsson d22b6392de Bug 19893: Support for joined subfields in mappings
Sponsored-by: Gothenburg University Library
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-16 11:04:58 +00:00
David Gustafsson 1e1ca322d3 Bug 19893: Restore and fix removed tests
Sponsored-by: Gothenburg University Library
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-16 11:04:58 +00:00
David Gustafsson 53cf97bb2d Bug 19893: Add index status
Add persistent per index "index status" state to provide useful
user feedback when update of Elasticsearch server mappings fails

Sponsored-by: Gothenburg University Library
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-16 11:04:57 +00:00
David Gustafsson fc0abee558 Bug 19893: Remove serialization format setting
Default to base64 encoded binary MARC with MARCXML
fallback if record exceeds maximum size

Sponsored-by: Gothenburg University Library
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-16 11:04:57 +00:00
David Gustafsson 5389ffa38a Bug 19893: Alternative optimized indexing for Elasticsearch
Implement optimized indexing for Elasticsearch

How to test:
1) Time a full elasticsearch re-index without this patch by running the
   rebuild_elastic_search.pl with the -d flag:
   `koha-shell <instance_name> -c "time rebuild_elastic_search.pl -d"`.
2) Apply this patch.
3) Time a full re-index again, it should be about twice at fast (for a
   couple of thousand biblios, with fewer biblios results may be more
   unpredictable).

Sponsored-by: Gothenburg University Library
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-16 11:04:56 +00:00
Nick Clemens 4abe3a4082 Bug 18316: (follow-up) Don't let 'fields' be undef and adjust tests
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-08 02:18:45 +00:00
Alex Arnaud 973bc0aceb Bug 18316: (follow-up) remove use_dis_max
It is true by default on ES 5 and has been removed on ES 6

Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-07 22:17:00 +00:00
Alex Arnaud 9380134a02 Bug 18316: (follow-up) Koha::SearchField::search_marc_maps return a result set - code refactoring for gettings weighted fields - Koha::SearchFields::weighted_fields return a result set
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>

Rebased-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-07 22:00:31 +00:00
Nick Clemens fa853f7d38 Bug 18316: Add weighting/relevancy options to ES query on simple search
The idea is the following: if some search field(s) are weighted in
search engine config page, Koha will query ES on all fields plus those with
the coresponding weights. Else, search is done on the entire record with
no weighting. The advanced search page is unaffected by these changes

Test plan (having Koha working with Elasticsearch):
  - apply this patch
  - have some weights defined for various fields
  - try searches from the search bar and from the advanced search page
  - confirm weighting affects the relevancy (in expected ways)
      e.g.
      1. search for 'a' from advanced search, note results
      2. give 'title' a weight
      3. search for 'a' using the simple search bar
      4. results with 'a' in the title should now be more relevant
  - confirm search results on advanced search page are unaffected

Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>

Rebased-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-07 22:00:30 +00:00
Ere Maijala 23f78a6c2a Bug 20248: Improve Elasticsearch mappings UI and rebuild_elastic_search.pl.
Improvements:
1) Mappings UI now has button that allows one to reset the mappings.
2) Mappings UI displays the items in alphabetical order.
3) Indexing script drops and recreates the index right away, which
helps prevent ES from autocreating a bad index if someone does something
while the first batch of records is being processed.
4) Indexing script has nicer output.

To test:
1) Change mappings.yaml file
2) Reset mappings in UI in mappings.pl
3) Verify the mappings have been changed in UI
4) The field order is alphabetical
5) Rebuild script has clean output
6) Run test t/db_dependent/Koha_Elasticsearch_Indexer.t

Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-08-22 14:47:43 +00:00
Alex Arnaud 96f253d5ed Bug 21032: Fix using facet for searches made on specific index
Test plan:

  - Try a simple search (OPAC or staff) like "title:foo",
  - click on a facet: badaboum!
  - apply this patch,
  - retry

Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2018-07-31 13:11:56 -03:00
Jonathan Druart 4d45bd5755 Bug 19604: Fix perlcritic "Loop iterator is not lexical"
Loop iterator is not lexical at line 318, column 13.  See page 108 of
PBP.  (Severity: 5)

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2018-07-31 10:44:36 -03:00
Alex Arnaud 96e768eb43 Bug 20273: Use compat routines for autocomplete in auth_finder.pl
Test plan:
  - Context: Koha working with elasticsearch,
  - Apply BZ 19582 (that make auth_finder.pl works again),
  - apply this patch,
  - go to cgi-bin/koha/authorities/auth_finder.pl,
  - type at least 3 chararcters in a search field and
    check that the autocomplete returns results

Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-07-19 17:25:16 +00:00
Nick Clemens f983c1e3f7 Bug 19604: Elasticsearch Fixes for build_authorities_query for auth searching
To test:
1 - Do some authority searches in Zebra
2 - Switch to ES and repeat, results will vary and some may fail
3 - Apply patch and dependencies
4 - Reindex ES
5 - Repeat searches, they should suceed and results should be similar to
Zebra
6 - Slight differences are okay, but results should (mostly) meet
expectations

A few notes:
We add a 'normalizer' to ensure we get a single token from the heading
indexes, this makes 'starts with' work as expcted
We switch to 'AND' for fields searched from cataloging editor - this
matches Zebra results
We force the '__sort' fields for sorting - if sorting looks wrong try
reducing the heading field to a single subfield - this will need to be
addressed on a future bug (multiple subfields create an array, ES sorts
        those randomly)

Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-07-19 17:23:54 +00:00
Julian Maurice ea184236a4 Bug 19502: Add POD for max_result_window
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-07-12 13:04:23 +00:00
Julian Maurice 95b1983a25 Bug 19502: Retrieve index.max_result_window from ES
This avoid hardcoding '10000' in two different places and allow users to
adjust this setting.

Also, this patch fixes a bug when the search return less than 10000
results

Test plan:
1. Do a search that returns 10000+ records.
2. Note the warning above the pagination buttons
3. Go to the last page, no error
4. Change the ES setting:
   curl -XPUT http://elasticsearch/koha_master_biblios/_settings -d \
     '{"index": {"max_result_window": 20000}}'
5. Do another search that returns more than 10000 but less than 20000
6. Note that the warning does not show up
7. Go to the last page, still no error

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-07-12 13:04:21 +00:00
Jonathan Druart e0f6450ae9 Bug 20073: Remove perlcritic error
"return" statement with explicit "undef" at line 245, column 5.  See
page 199 of PBP.  (Severity: 5)

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-06-22 14:38:02 +00:00
Ere Maijala 133df5e912 Bug 20073: Move Elasticsearch configs to yaml files and improve the default settings.
Improvements:
1) Index settings moved from code to etc/searchengine/elasticsearch/index_config.yaml. An alternative can be specified in koha-conf.xml.
2) Field settings moved from code to etc/searchengine/elasticsearch/field_config.yaml. An alternative can be specified in koha-conf.xml.
3) mappings.yaml has been moved from admin/searchengine/elasticsearch to etc/searchengine/elasticsearch. An alternative can be specified in koha-conf.xml.
4) Default settings have been improved to remove punctuation from phrases used for sorting etc.
5) State variables are used for storing configuration to avoid parsing it multiple times.
6) A possibility to reset the fields too has been added to the reset operation of mappings administration.
7) mappings.yaml has been moved from admin/searchengine/elasticsearch to etc/searchengine/elasticsearch.
8) An stdno field type has been added for standard identifiers.

To test:
1) Run tests in t/Koha/SearchEngine/Elasticsearch.t
2) Clear tables search_fields and search_marc_map
3) Go to admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
4) Verify that admin/searchengine/elasticsearch/mappings.pl displays the mappings properly, including ISBN and other standard number fields.
5) Index some records using the -d parameter with misc/search_tools/rebuild_elastic_search.pl to recreate the index
6) Verify that you can find the records
7) Put <elasticsearch_index_mappings>non_existent</elasticsearch_index_mappings> to koha-conf.xml
8) Verify that admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1 fails because it can't find non_existent.
9) Copy etc/searchengine/elasticsearch/mappings.yaml to a new location and make elasticsearch_index_mappings setting in koha-conf.xml point to it.
10) Make a change in the new mappings.yaml.
11) Clear table search_fields (mappings reset doesn't do it yet, see bug 20248)
12) Go to admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
13) Verify that the changes you made are now visible in the mappings UI

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Bug 20073: Move Elasticsearch yaml files back to admin directory

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-06-22 14:37:56 +00:00
Jonathan Druart cff62796ea Revert "Bug 20073: Move Elasticsearch configs to yaml files and improve the default settings."
This reverts commit f489d2034b.

This commit breaks the install process when using debian packages.
Reverting as we are very close to the 18.05.00 release

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2018-05-21 11:02:16 -03:00
Jonathan Druart e1a95094e7 Revert "Bug 20073: Remove perlcritic error"
This reverts commit 67359a97f4.

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2018-05-21 11:01:58 -03:00
Jonathan Druart 67359a97f4 Bug 20073: Remove perlcritic error
"return" statement with explicit "undef" at line 245, column 5.  See
page 199 of PBP.  (Severity: 5)

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2018-04-23 15:12:48 -03:00