Commit Graph

63 Commits

Author SHA1 Message Date
Nick 2b769cbc32 Bug 23089: Fix sorting on sum and regular fields
As of bug 20589 we no longer analyze sort fields and so we no longer need to append ".phrase"
to our sort in searches.

Additionally, sort fields based on 'sum' should also use sum in building the value to sort on

To test:
0 - Be using ES
1 - Find the most circulated item in your collection
2 - Search for '*'
3 - Sort by popularity DESC
4 - Note that item is not first
5 - Try to sort by anything but relevancy, it fails
6 - Apply patch
7 - Redo searches and sorts
8 - Things should now work as expected

Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
(cherry picked from commit 6a2ba8b6b2)
Signed-off-by: Hayley Mapley <hayleymapley@catalyst.net.nz>
2019-12-18 14:22:38 +13:00
Nick Clemens 340e2da40b Bug 23089: [19.05 and earlier] Use integer field directly for sorting
(cherry picked from commit 55ccf90711)
2019-12-18 09:00:13 +13:00
Fridolin Somers d161374e01 Bug 23004: Missing authtype filter in auth_finder.pl
In cataloguing, the use of tag editor opens authorities finder with a limit on specific authorities type.
This limit is missing with Elasticsearch.

This patch adds in query a "filter" on "term" which is the most performant way because there must be no ranking computed on this part.

Test plan :
1) Use Elasticsearch
2) Create an autority of type author (NP in UNIMARC) with heading "Tolkien"
3) Create an autority of type subject-author (SAUT in UNIMARC) with heading "Tolkien"
4) Create a biblio record
5) Use tag editor on a author field (700 in UNIMARC)
6) Seach for "Tolkien" in $a
without patch : you see the 2 authorities in results
with patch : you see only the correct authority in results (NP in UNIMARC)
7) Check search in authorities-home.pl is still OK

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
(cherry picked from commit 949e36c7b8)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit 1d8b1fafc4)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2019-10-17 00:04:24 +00:00
Ere Maijala bc69881893 Bug 22524: Fix date/time-last-modified search with Elasticsearch
Avoid using slash in the field name since it would need to be escaped. Fix conversion of dtlm and any existing mapping.

Signed-off-by: Liz Rea <wizzyrea@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
(cherry picked from commit 5d27fb71cb)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit ab0de52ab2)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2019-08-31 03:37:55 +00:00
Ere Maijala 22585ec127 Bug 22524: Fix publication date and year search with Elasticsearch
Also fixes the mappings.yaml to use correct field name (left over issue from bug 19575), so reset mappings and reindex before testing.

Test plan:
1. Reset mappings and reindex biblios.
2. Check that tests in t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t pass.
3. Try that all of the following year range type work in publication date search and year limit in advanced search:
yyyy
yyyy-yyyy
-yyyy
yyyy-

Signed-off-by: Liz Rea <wizzyrea@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
(cherry picked from commit 2293e6597c)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit d21a1fb964)

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2019-08-31 03:35:36 +00:00
Fridolin Somers 60b4ab8a92 Bug 21534: Always compute wildcards
You get no results when searching with an hyphen + with * in query string (or with preference QueryAutoTruncate) :
ie /cgi-bin/koha/opac-search.pl?q=saints-anges*

Looks like query-string by default does not compute wildcards, see analyze_wildcard in :
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html

Test plan :
1) Use Elasticsearch
2) Create a record with "saints-anges"
3) Search for "saints-anges" => you get results
4) Search for "saints-anges*" => you get results
5) Search for "saints-ang*" => you get results

Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Arthur Bousquet <arthur.bousquet@inlibro.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
(cherry picked from commit 001d433b42)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit a8cc9d5f49)
2019-07-26 06:00:41 +00:00
Ere Maijala e2aa81da2c
Bug 22413: Improve query string and desc creation for Elasticsearch
Test plan:

After each of the following steps, verify that the results and query description looks ok.

1. Do a simple keyword query
2. Change sort order
3. Click a facet
4. Do an advanced search with title and author
5. Change sort order
6. Click a facet

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>

Signed-off-by: Marjorie <marjorie.barry-vila@collecto.ca>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
(cherry picked from commit aba8b130f8)
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-04-26 16:24:15 +01:00
Ere Maijala 8f8bcbbc0d
Bug 22295: Make Elasticsearch query builder group multi-term queries
Test plan:

1. Do an advanced search for
Title = new
AND
Title = york
2. Verify that the results match an advanced search for:
Title = new york
3. Verify that tests in t/db_dependent/Koha/SearchEngine/Elasticsearch still pass

Signed-off-by: Michal Denar <black23@gmail.com>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
(cherry picked from commit e90cb16d79)
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-04-15 12:30:04 +01:00
Ere Maijala 33489f2df1
Bug 22228: Escape lone colons and make sure term is balanced
Escapes unquoted colons that have whitespace on either side. Removed unbalanced quotes.

Test plan:

1. Make sure the test case described in the bug works
2. Make sure tests pass:
  prove t/Koha/SearchEngine/Elasticsearch/QueryBuilder.t

Signed-off-by: Björn Nylén <bjorn.nylen(at)ub.lu.se>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
(cherry picked from commit dd98584c85)
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-03-05 15:08:25 +00:00
David Gustafsson e522e32ea3
Bug 19575: Keep authority-number as alias and fix query
Keep authority-number as alias and change field name
from alias to real field in hard coded Elasticsearch query

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
(cherry picked from commit eeda2afd3a)
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-03-05 14:52:07 +00:00
Nick Clemens dc219b6c10
Bug 19575: Fix an search field
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
(cherry picked from commit ef2a03a092)
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-03-05 14:51:47 +00:00
root ce578a9aca
Bug 19575: Rebase and fix tests
This patch lower cases the sort by fields to normalize checking them and adjusts
some existing tests to meet the new expectations.

The regex for splitting terms has been moved into a subroutine so that adjustment was made

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
(cherry picked from commit c2061adc8b)
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-03-05 14:51:27 +00:00
David Gustafsson 68b129e0c4
Bug 19575: Use canonical field names and resolve aliased fields
Adjust elastic search mappings to more closely match Zebra equivalents
resolving serveral issues with coded Zebra searches in templates and
sorting of search results in UI. Also make field names in search strings
case insensitive to accept case variations in template links and user input.

Sponsored-by: Gothenburg University Library

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
(cherry picked from commit 6d53910f0c)
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-03-05 14:50:59 +00:00
Ere Maijala 66e041fece
Bug 21084: Fix automatic truncation in authority search
- Makes token splitting work like in biblio search
- Only adds right hand truncation since that's what Zebra also does
- Only adds truncation if the token is not a phrase and ends in a word character
- Adds tests to existing and new QueryBuilder functions

Test plan:

1. Create an authority record for "Duck, Donald"
2. Try contains type authority searches with the following terms and make sure they find the record:
Duck, Donald
donald duck
don duck
"Duck, Donald"
3. Make sure the following search works but does not find anything:
"Duck, Don"

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
(cherry picked from commit 0d9a42b492)
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-01-23 12:43:33 +00:00
Ere Maijala a4c02b53cd Bug 19365: Fix search for duplicate authorities with Elasticsearch
Sponsored-by: National Library of Finland
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-26 16:11:02 +00:00
Ere Maijala be2bdfd511 Bug 19365: Fix several issues with the Elasticsearch code
Also optimize it so it's actually usable.

Test plan:

1. To test it properly you need biblio and authority data. You might get away with enabling AutoCreateAuthorities and BiblioAddsAuthorities so that authorities are created in the process. Another option would be to import authorities first e.g. from LoC.
2. Make sure the authority index has been properly created with "misc/search_tools/rebuild_elastic_search.pl -a -d"
3. Run "misc/link_bibs_to_authorities.pl -v -l" twice and observe the results.

Sponsored-by: National Library of Finland
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-26 16:11:01 +00:00
Nick Clemens 4abe3a4082 Bug 18316: (follow-up) Don't let 'fields' be undef and adjust tests
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-08 02:18:45 +00:00
Alex Arnaud 973bc0aceb Bug 18316: (follow-up) remove use_dis_max
It is true by default on ES 5 and has been removed on ES 6

Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-07 22:17:00 +00:00
Alex Arnaud 9380134a02 Bug 18316: (follow-up) Koha::SearchField::search_marc_maps return a result set - code refactoring for gettings weighted fields - Koha::SearchFields::weighted_fields return a result set
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>

Rebased-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-07 22:00:31 +00:00
Nick Clemens fa853f7d38 Bug 18316: Add weighting/relevancy options to ES query on simple search
The idea is the following: if some search field(s) are weighted in
search engine config page, Koha will query ES on all fields plus those with
the coresponding weights. Else, search is done on the entire record with
no weighting. The advanced search page is unaffected by these changes

Test plan (having Koha working with Elasticsearch):
  - apply this patch
  - have some weights defined for various fields
  - try searches from the search bar and from the advanced search page
  - confirm weighting affects the relevancy (in expected ways)
      e.g.
      1. search for 'a' from advanced search, note results
      2. give 'title' a weight
      3. search for 'a' using the simple search bar
      4. results with 'a' in the title should now be more relevant
  - confirm search results on advanced search page are unaffected

Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>

Rebased-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-07 22:00:30 +00:00
Alex Arnaud 96f253d5ed Bug 21032: Fix using facet for searches made on specific index
Test plan:

  - Try a simple search (OPAC or staff) like "title:foo",
  - click on a facet: badaboum!
  - apply this patch,
  - retry

Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2018-07-31 13:11:56 -03:00
Jonathan Druart 4d45bd5755 Bug 19604: Fix perlcritic "Loop iterator is not lexical"
Loop iterator is not lexical at line 318, column 13.  See page 108 of
PBP.  (Severity: 5)

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2018-07-31 10:44:36 -03:00
Nick Clemens f983c1e3f7 Bug 19604: Elasticsearch Fixes for build_authorities_query for auth searching
To test:
1 - Do some authority searches in Zebra
2 - Switch to ES and repeat, results will vary and some may fail
3 - Apply patch and dependencies
4 - Reindex ES
5 - Repeat searches, they should suceed and results should be similar to
Zebra
6 - Slight differences are okay, but results should (mostly) meet
expectations

A few notes:
We add a 'normalizer' to ensure we get a single token from the heading
indexes, this makes 'starts with' work as expcted
We switch to 'AND' for fields searched from cataloging editor - this
matches Zebra results
We force the '__sort' fields for sorting - if sorting looks wrong try
reducing the heading field to a single subfield - this will need to be
addressed on a future bug (multiple subfields create an array, ES sorts
        those randomly)

Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-07-19 17:23:54 +00:00
Nick Clemens 8b76ee39d4 Bug 19564: (QA follow-up) Make commented out tests pass
We likely shoudln't pass through an uncoverted sort order for now, but
it does allow us to look ahead to implementing the orders directly so
seems a good option to have.

Either this patch should be used, or the commented out tests should be
removed

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2018-03-28 16:05:10 -03:00
David Gustafsson 8e07761f30 Bug 19564: Fix extraction of sort order from sort condition name
Currently sort order is extracted from sort condition by splitting the
field, instead use regular expression to extract the last part preceded
by underscore.

Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2018-03-28 16:05:10 -03:00
Nick Clemens 49c5acc5b9 Bug 19582: Use compat routines for searching authorities in auth_finder.pl
To test:
1 - Enable Zebra
2 - Perform an auth search
3 - note results
4 - Enable ES
5 - Repeat search, note (likely) diff results
6 - Open a record in cataloging and use the button to launch auth search
7 - Perform same search as above, note results match for eiher engine
selected
8 - NOTE: Disbale sorting for ES search - this will be dealt with in
another report

Signed-off-by: David Bourgault <david.bourgault@inlibro.com>

Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2018-03-27 17:50:37 -03:00
Nick Clemens 834b8c749b Bug 19580: Exclude period as splitting character for autotrunction
To test:
0 - Apply Unit test patch only
1 - prove t/db_dependent/Koha_SearchEngine_Elasticsearch_Search.t
2 - Should fail
3 - Apply this patch
4 - prove t/db_dependent/Koha_SearchEngine_Elasticsearch_Search.t
5 - should pass
6 - search for 'Local-number.raw:"4"' (or a vlid biblionumber)
7 - should get expected result

Signed-off-by: David Bourgault <david.bourgault@inlibro.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2018-01-15 10:40:26 -03:00
Nick Clemens a081ed16c0 Bug 19559: Add '-' to list of characters we don't split search terms on
Test plan assumes
QueryAutoTruncate = automatically
SearchEngine = Elasticsearch

To test:
0 - Apply Unit test patch only
1 - prove t/db_dependent/Koha_SearchEngine_Elasticsearch_Search.t
2 - Should fail
3 - Apply this patch
4 - prove t/db_dependent/Koha_SearchEngine_Elasticsearch_Search.t
5 - should pass
6 - search for 'Local-number:"4"' (or a vlid biblionumber)
7 - should get expected result

Signed-off-by: David Bourgault <david.bourgault@inlibro.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-11-26 12:59:46 -03:00
Nick Clemens 088cdcda5f Bug 16660: Add support for OpacSuppression to Elasticsearch
To test:
1 - Enable suppression
2 - Suppress some records
3 - Apply all the patches
4 - Reindex ES
5 - Search and don't get suppressed records
6 - Disable suppression
7 - Search and get all the records

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-10-27 14:09:02 -03:00
Julian Maurice c991ac6b61 Bug 18374: (QA follow-up) Fix auto truncation for field:"value"
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-10-13 13:57:47 -03:00
Julian Maurice 6426a82014 Bug 18374: (QA follow-up) Simplify _truncate_terms
By using a different split regex, we can simplify a bit the process of
appending '*' to every word of the query

Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-10-13 13:57:47 -03:00
Nick Clemens d099dc13bb Bug 18374: (follow-up) Add support and tests for quoted strings
To test:
1 - prove t/db_dependent/Koha_SearchEngine_Elasticsearch_Search.t
2 - do some searches in staff client and test results

Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-10-13 13:57:33 -03:00
Julian Maurice e540651975 Bug 18374: (QA follow-up) Tidy up code
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-10-13 13:57:33 -03:00
Nick Clemens 4c4d982ae4 Bug 18374: Respect QueryAutoTruncate syspref in Elasticsearch
This patchset adds a subroutine '_truncate_terms' to the ES QueryParser.

If QueryAutoTruncate is enabled this function will be called for any
search to add wildcard '*' to all terms

To test:
1 - Enable Elasticsearch and have some records indexed
2 - Search for partial terms
3 - Note they fail unless '*' is appended
4 - Apply patch, leave QueryAutoTruncate disabled
5 - Note partial term searches still fail
6 - Enable QueryAutoTruncate
7 - Note partial term searches succeed
8 - Do some regular and advanced searches to make sure results are as
expected

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-10-13 13:01:12 -03:00
Nick Clemens cfb52579ec Bug 18624: Run time errors when searching authorities with Elastic search 5.3
It seems we have a syntax mismatch - any vs all

all seems to the be list we expect so updating code that way

To test:
1 - Enabled Elasticsearch
2 - Index some authorities
3 - Perform a 'Search entire record' search
4 - Internal server error (
        Invalid marclist field provided: all at
        /usr/local/koha/Koha/SearchEngine/Elasticsearch/QueryBuilder.pm
        line 433.
        )
5 - Run:
    $ sudo koha-shell kohadev
   k$ cd kohaclone
   k$ prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
=> FAIL: Tests fail because 'any is used'
6 - Apply patch
7 - Search should work
8 - Run:
   k$ prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
=> SUCCESS: Tests pass!
9 Sign off :-D

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Edited the test plan so it mentions the new tests

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-08-15 11:39:46 -03:00
Tomas Cohen Arazi ff41b87302 Bug 18624: Minor refactoring to ease testing
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-08-15 11:39:46 -03:00
Nick Clemens 306250e496 Bug 17169: Add a facet for ccode fields to ElasticSearch
To test:
1 - Apply patch
2 - Backup your db
3 - Drop and create a new db to ensure your mappings are refreshed from
the patch
4 - add some titles with items with collection codes
5 - search and see collection code facets
6 - sign off

Work to be done:
1 - Replace codes with descriptions

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
2017-03-22 19:24:22 +00:00
Tomas Cohen Arazi d115602c98 Bug 18068: ES - Fix location and (home|holding)branch facets
This patch makes the 'Locations' facet work as expected (i.e. having the
same behaviour it has for Zebra: picking the 952$c in MARC21 and 995e
for UNIMARC).

It also adds the code to handle holding and home library settings for
facets and makes the facets show the library name instead of the branch
code.

The mappings are updated so the labels match what facets.inc expect to
work properly.

To test:
- On master, do a search that returns biblios with items having
homebranch set.
=> FAIL: Under the 'Locations' label on the facets you will notice
branchcodes are shown.
- Apply the patch
- Restart memcached and plack (just in case, it was tricky)
- Reset your mappings:
  http://localhost:8081/cgi-bin/koha/admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
- Restart memcached and plack (again, not sure if needed)
- Make sure this mappings are set:
  homebranch => HomeLibrary
  holdingbranch => HoldingLibrary
  (Note: it might not be set due to the place the yaml file is being picked)
- Reindex your records:
  $ sudo koha-shell kohadev
 k$ cd kohaclone
 k$ perl misc/search_tools/rebuild_elastic_search.pl -d -v
- Repeat the initial search
=> SUCCESS: 'Location' contains the right stuff, 'Home libraries' and
'Holding libraries' too.
- Run
 k$ prove t/db_dependent/Koha_SearchEngine_Elasticsearch_Search.t
=> SUCCESS: Tests pass!
- Sign off :-D

Note: play with the 'DisplayLibraryFacets' syspref options.
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-02-17 11:34:41 +00:00
Olli-Antti Kivilahti d1077b5ae6 Bug 17255 - Upgrade Elastic Search code to work with version 2.4+ - rebased wip
-Changed deprecated facets to aggregations
-Fixed boolean datatypes not allowing analyzers to be specified
-Fixed deprecated '_id' to 'es_id'. Now the ES-index has the correct id==biblionumber

ZE TEST PLAN

1. Reset Zebra index since facets are hard coded to dynamic search_marc_mappings.
2. perl misc/search_tools/rebuild_elastic_search.pl
3. Fetch all indexed records and the facet for subject__facet

curl -XGET localhost:9200/koha_biblios/data/_search?pretty -d '{
  "aggregations": {
    "my_agg": {
      "terms": {
        "field": "subject__facet"
      }
    }
  }
}'

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-02-07 17:17:53 +00:00
Kyle M Hall d91e75919a Bug 17132 - Availability search broken when using Elastic
Elastic appears to be indexing onloan as a string, but our code assumes
it is a boolean.

Test Plan:
1) Ensure you are set up using Elastic as your search engine
2) Search only for available items from the advanced search
3) Note you get no results
4) Apply this patch
5) Re-run the search
6) You should now get results!

Signed-off-by: Jennifer Schmidt <jschmidt@switchinc.org>

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2016-11-22 11:26:18 +00:00
Jonathan Druart df3df9cf71 Bug 17372: Standardize Elasticsearch paths
What we currently have:
Koha/ElasticSearch.pm
Koha/ElasticSearch/Indexer.pm
Koha/SearchEngine/Elasticsearch/QueryBuilder.pm
Koha/SearchEngine/Elasticsearch/Search.pm

What we want:
Koha/SearchEngine/Elasticsearch.pm
Koha/SearchEngine/Elasticsearch/Indexer.pm
Koha/SearchEngine/Elasticsearch/QueryBuilder.pm
Koha/SearchEngine/Elasticsearch/Search.pm

Test plan:
  % git grep -i Koha::ElasticSearch
  % git grep ElasticSearch|grep -v Catmandu::Store::ElasticSearch
should not return any result

Do a full reindex and search for records

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
2016-10-11 01:07:03 +00:00
Jesse Weaver 37ea74669c Bug 12748: (QA followup) fix several tiny QA tools warnings
* admin/searchengine/elasticsearch/mappings.yaml:
    * Fix indentation so QA tools believe it's valid YAML
  * C4::AuthoritiesMarc: Remove tab character
  * C4::Search: Fix POD
  * Koha::BiblioUtils: remove "return undef"
  * Koha::Schema::Result::Search*: rename type to _type
  * Koha::SE::ES::Search:
    * Fix spelling/@_[0]
    * Add my to loop variable
    * Remove unused var $error
    * Add use Modern::Perl
    * Fix undefined variables
  * Koha::ES::Indexer: fix POD
  * Koha::SE::ES::QueryBuilder:
    * Don't define variables in conditionals
    * Fix POD
  * Koha::SE::Zebra::Search:
    * Fix debug warns/POD/barewords
    * Koha::SE::Zebra::Search: add use Modern::Perl
  * kohastructure.sql: remove old, unused elasticsearch_mappings table,
    add new tables
  * labels/label-item-search.pl: remove C4::Dates
  * opac/elasticsearch.pl: remove old file

Signed-off-by: Jesse Weaver <jweaver@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
2016-04-26 20:20:13 +00:00
Jonathan Druart 2de8fd9e33 Bug 12478: Take the FacetMaxCount pref into account
The system preference FacetMaxCount should work as expected with ES.

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jesse Weaver <jweaver@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
2016-04-26 20:20:11 +00:00
Jonathan Druart ce5bc33ef3 Bug 12478: Remove empty limit parameter
If no limit are passed, the url will contain '&limit=' anyway.
It is not necessary and can be avoided easily

Test plan:
1/ Search for a term in your catalogue
2/ Hover over a link in the facet area
3/ The link is
cgi-bin/koha/opac-search.pl?idx=kw&q=your_term&limit=&limit=[...]

With this patch, the empty limit parameter does not appear.

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jesse Weaver <jweaver@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
2016-04-26 20:20:11 +00:00
Robin Sheat 8c76e99fca Bug 12478: add location, su-geo, se to facets
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jesse Weaver <jweaver@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
2016-04-26 20:20:09 +00:00
Jonathan Druart 7f04ac955d Bug 12478: Fix encoding issue on facets
Filer on "Ślez, Ts." => Can't escape \x{015A}, try uri_escape_utf8()
instead at
/home/koha/src/Koha/SearchEngine/Elasticsearch/QueryBuilder.pm line 221.

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jesse Weaver <jweaver@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
2016-04-26 20:20:09 +00:00
Robin Sheat 2e79ea0f74 Bug 12478: allow more granular sorting configuration
This allows sorting to be configured within a field. For example, while
many values are included for search on author, sorting should only be
done on the main entry values. This permits that by have a sort value,
which can be true, false, or null. true and null are pretty much the
same, but false means that a field isn't available for sorting on. By
default (null), fields can be sorted on.

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jesse Weaver <jweaver@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
2016-04-26 20:20:08 +00:00
Robin Sheat 349ecc3400 Bug 12478: add the lenient option to the query_string query
It doesn't really help.

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jesse Weaver <jweaver@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
2016-04-26 20:20:07 +00:00
Robin Sheat 374f1e6384 Bug 12478: ES is now updated when records are updated/deleted
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jesse Weaver <jweaver@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
2016-04-26 20:20:07 +00:00
Robin Sheat 1c737eaaf8 Bug 12478: a replacement to the SimpleSearch interface implemented
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jesse Weaver <jweaver@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
2016-04-26 20:20:07 +00:00