Commit Graph

79 Commits

Author SHA1 Message Date
Julian Maurice 3302ad1461
Bug 24902: Join different mc- limits with AND (elasticsearch)
In the advanced search form, you can enable several limits using syspref
AdvancedSearchTypes (namely itemtypes, shelving locations, collections)
When used, the resulting query parts end up being joined with OR, even
if the field is different. That means that if you pick "Book" under
itemtypes tab, and "Fiction" under collection tab, it will search
"itype:BOOK OR ccode:FIC". It should be AND.

For instance, if you select:
    Itemtypes:
        ✓ Book
        ✓ DVD
    Location:
        ✓ Child
        ✓ Adult

it should search:
    itype:(Book OR DVD) AND location:(Child OR Adult)

Test plan:
0. Do not apply the patch yet
1. Enable elasticsearch
2. Set syspref AdvancedSearchTypes = 'itemtypes|loc|ccode'
3. Create a new itemtype and a new authorised value for categories LOC
   and CCODE
4. Create a biblio with the new itemtype, another biblio with the new
   location, another biblio with the new collection, and again another
   biblio with the new itemtype, location and collection
5. Verify that you can find these new biblio records using only the
   "advanced search types" in the advanced search form
6. In the advanced search form, pick all 3 limits (itemtype, location,
   collection) and verify that it returns the 4 records.
7. Apply the patch
8. Repeat step 6, it should now return only the biblio that satisfies
   all criteria
9. Verify that if you select more than one
   {itemtype|location|collection} it still returns results that
   satisfies any selected criteria
10. prove t/Koha/SearchEngine/ElasticSearch/QueryBuilder.t

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2020-03-24 08:00:26 +00:00
Nick Clemens 63641856dd
Bug 23521: (follow-up) Add quotes
Not only should terms from facets/limits be grouped, but order should probably matter
We build a different facet for "Dillinger Girl" or "Girl Dillinger"

The parens are possibly overkill now, but I think it makes it very clear and does not hurt

To test:
1 - Follow original plan
2 - Create a new record with 'Girl Dillinger' as author
3 - Search for 'Dill*'
4 - You get all three records (and maybe others that match)
5 - Limit by 'Girl Dillinger' - you get two records
6 - Same for 'Dillinger Girl'
7 - Apply patch
8 - Limits/facets for 'Dillinger Girl' and 'Girl Dillinger' now match a isngle record

Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2020-03-16 10:24:45 +00:00
Alex Arnaud c5e367c54e
Bug 23521: Put parentheses arround limit terms
Test plan:
  - Use Elasticsearch 6 (you'll need Bug 18969),
  - create a biblio (#1) with "Dillinger Girl" in author and what you
    want in title,
  - create another biblio (#2) with the word "girl" in the title and
    "Dillinger Escaplan" as author
  - reindex
  - search * and refine on "Dillinger Girl"
  - Ko => Biblio #1 and #2 appear
  - Apply this patch,
  - search * and refine on "Dillinger Girl"
  - Ok => anly biblio #1 appears
  - use Elasticsearch 5 again
  - check for no search regression

Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2020-03-16 10:24:34 +00:00
Fridolin Somers fbf326f857
Bug 22426: Map 'loc' to 'location' when building ES queries
A search made on Shelving location tab in advanced search gives 0 result.

Test plan :
Without patch:
    - in advanced search, choose a location in shelving location tab and start search (opac-search.pl?idx=kw&op=and&idx=kw&op=and&idx=kw&do=Search&limit=mc-loc)
    - 0 result
    - check that location exist by doing a simple search and filtering on the    location facet
    - have some results
With patch:
    - in advanced search, choose a location in shelving location tab and start search (opac-search.pl?idx=kw&op=and&idx=kw&op=and&idx=kw&do=Search&limit=mc-loc)
    - found results

Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2020-01-10 16:22:47 +00:00
Nick Clemens bed334ee4d
Bug 23676: Use 'false' for opac suppression
To test:
 1 - Enable ES
 2 - Enable OpacSuppression
 3 - Suppress a bib in staff client
     Edit 942n to be 1
 4 - Search the opac for anything
 5 - Tail the plack logs and note a deprecation warning
 6 - Apply patch
 7 - Restart all the things
 8 - Repeat search on opac
 9 - No error/warn in logs
10 - Record is correctly suppressed

Signed-off-by: Myka Kennedy Stephens <mkstephens@lancasterseminary.edu>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2020-01-10 16:21:22 +00:00
Nick Clemens 9d68838af6
Bug 24128: Add alias for biblionumber search
To test:
1 - In staff or OPAC using ES, search for "biblionumber:3" or any existing biblionumber
2 - No results
3 - Apply patch, restart all the things
4 - Search again
5 - You go to the biblionumber

Signed-off-by:  Bob Bennhoff <bbennhoff@clicweb.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-12-09 14:48:14 +00:00
Nick Clemens 4a36c1ac04
Bug 23719: (follow-up) Add warn when passed invalid search field in marclist
Note: I also remove warnings for undefined operation in this patch, is a trivial fix

To test:
prove -v t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-11-06 17:06:38 +00:00
Nick 33595e1a3e
Bug 23719: Allow searching specific fields for matching authorities in ES
To test:
1 - Export your authorities via Tools->Export data
2 - Define a record matching rule in Admin->Record matchign rules
    Use index: LC-card-number
    field: 010$a
3 - Stage the exported records for import and use the rule created above for matching
4 - The process does not complete
5 - Check intranet error logs - exception on unknown marclist
6 - Apply patch
7 - Repeat
8 - Success!
9 - prove -v t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-11-06 17:06:29 +00:00
Ere Maijala 6e1d759981
Bug 22592: Add index scan emulation to Elasticsearch
Adds support for using the "scan indexes" action in advanced search by using faceting with a prefix filter. Requires that the field be set as facetable for anything to be found.

Test plan:
1. Apply patch
2. Go to advanced search and click "More options"
3. Select author as the search field, enter a last name and check "Scan indexes"
4. Perform search and observe the result list resembling scan results

Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-11-03 07:51:04 +00:00
Nick 6a2ba8b6b2
Bug 23089: Fix sorting on sum and regular fields
As of bug 20589 we no longer analyze sort fields and so we no longer need to append ".phrase"
to our sort in searches.

Additionally, sort fields based on 'sum' should also use sum in building the value to sort on

To test:
0 - Be using ES
1 - Find the most circulated item in your collection
2 - Search for '*'
3 - Sort by popularity DESC
4 - Note that item is not first
5 - Try to sort by anything but relevancy, it fails
6 - Apply patch
7 - Redo searches and sorts
8 - Things should now work as expected

Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-11-03 07:34:01 +00:00
Alex Arnaud 7df185c783
Bug 22874: Use false instead of 0 value for limiting on availability
Test plan

  - Set up a elasticsearch 6 instance to work with Koha,
  - you may need to make koha works with ES 6 (see bug 20589),
  - make a search and limit it to available items only,
  => no result

  - Apply this patch,
  - make a search and limit it to available items only,
  - you should get some results

  - Do the same with ES 5.x

Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-10-28 12:24:34 +00:00
Fridolin Somers a94e863c3e
Bug 22997: Searching gives no results in auth_finder.pl
During cataloguing of an existing biblio, on an heading field,
the use of tag editor fills authorities finder with existing value :
  Search main heading ($a only)
  Search main heading
Default operator beeing 'contains'.

Actually with Elasticsearch those search give no results.

Example with heading :
200
  $a Casaubon
  $b Isaac
  $f 1559-1614

Call to Elasticsearch :
    "query" : {
        "bool" : {
           "must" : [
              {
             "query_string" : {
                  "query" : "Casaubon*",
                  "default_field" : "heading-main",
                   }
              },
              {
             "query_string" : {
                  "query" : "(Isaac*) AND (1559-1614*)",
                  "default_field" : "heading"
                   }
              }
            ]
         }
      },
    "sort" : [
       {
      "heading__sort.phrase" : "asc"
       }
     ]
  }

Patch adds to "query_string" :
  analyze_wildcard : true.

Test plan :
1) Use Elasticsearch
2) Edit an existing biblio record
3) Use tag editor on a heading
4) Click search => You get correct results
5) Check also search in authorities-home.pl

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-10-09 14:29:06 +01:00
David Gustafsson da16f6bc91
Bug 20334: Option for escaping slashes in search queries
Add "QueryRegexEscapeOption" system preference to provide option to escape
Elasticsearch regexp delimiters (/) within queries, or alternativly to
unescape escaped slashes (\/) while escaping unescaped slashes, in
effect making "\/" the new regexp delimiter.

How to test:
1) Run tests in ./t/Koha/SearchEngine/ElasticSearch/QueryBuilder.t
2) All tests should succeed

Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Maksim Sen <maksim.sen@inlibro.com>
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-10-08 14:09:53 +01:00
Nick Clemens 72fcd4c7f2
Bug 20589: Add option for searching entire record if record stored as array
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-10-02 13:29:40 +01:00
David Gustafsson 8d630cd786
Bug 20589: Exclude boolean fields from search fields
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-10-02 13:29:29 +01:00
David Gustafsson 91947546f7
Bug 20589: Add field boosting and use query_string fields parameter
Generate a list of fields for the query_string query fields parameter,
with possible boosts, instead of using "_all"-field. Also add "search"
flag in search_marc_to_field table so that certain mappings can be
excluded from searches. Add option to include/exclude fields in
query_string "fields" parameter depending on searching in OPAC or staff
client. Refactor code to remove all other dependencies on "_all"-field.

How to test:
1) Reindex authorities and biblios.
2) Search biblios and try to verify that this works as expected.
3) Search authorities and try to verify that this works as expected.
4) Go to "Search engine configuration"
5) Change some "Boost", "Staff client", and "OPAC" settings and save.
6) Verify that those settings where saved accordingly.
7) Click the "Biblios" or "Authorities" tab and change one or more
   "Searchable" settings
8) Verfiy that those settings where saved accordingly.
9) Try to verify that these settings has taken effect by peforming
   some biblios and/or authorities searches.

Sponsorded-by: Gothenburg Univesity Library
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-10-02 13:28:57 +01:00
Fridolin Somers 949e36c7b8
Bug 23004: Missing authtype filter in auth_finder.pl
In cataloguing, the use of tag editor opens authorities finder with a limit on specific authorities type.
This limit is missing with Elasticsearch.

This patch adds in query a "filter" on "term" which is the most performant way because there must be no ranking computed on this part.

Test plan :
1) Use Elasticsearch
2) Create an autority of type author (NP in UNIMARC) with heading "Tolkien"
3) Create an autority of type subject-author (SAUT in UNIMARC) with heading "Tolkien"
4) Create a biblio record
5) Use tag editor on a author field (700 in UNIMARC)
6) Seach for "Tolkien" in $a
without patch : you see the 2 authorities in results
with patch : you see only the correct authority in results (NP in UNIMARC)
7) Check search in authorities-home.pl is still OK

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-09-10 09:39:43 +01:00
Ere Maijala 5d27fb71cb
Bug 22524: Fix date/time-last-modified search with Elasticsearch
Avoid using slash in the field name since it would need to be escaped. Fix conversion of dtlm and any existing mapping.

Signed-off-by: Liz Rea <wizzyrea@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-07-26 16:16:34 +01:00
Ere Maijala 2293e6597c
Bug 22524: Fix publication date and year search with Elasticsearch
Also fixes the mappings.yaml to use correct field name (left over issue from bug 19575), so reset mappings and reindex before testing.

Test plan:
1. Reset mappings and reindex biblios.
2. Check that tests in t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t pass.
3. Try that all of the following year range type work in publication date search and year limit in advanced search:
yyyy
yyyy-yyyy
-yyyy
yyyy-

Signed-off-by: Liz Rea <wizzyrea@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-07-26 16:16:11 +01:00
Nick Clemens 14e6375637
Bug 14419: Expanding facets (Show more) performs a new search
This patch removes the constraint of only passing 5 facets to the template unless the list is expanded, in fact, it removes the 'expanded' attribute from Search.pm
Now that all facets are passed to page it adds a 'show more' link at the bottom of lists and allows user to expand or collapse any facet set without reloading page.

Updated tests included.

To test:
1  - Perform an OPAC search that returns more than 5 of any given facet type
2  - Click the "Show more" link on the facets and see that the search is reloaded
3  - Apply patch
4  - Repeat search
5  - Note that you can click "Show more" without reloading page
6  - Test that page load is not greatly affected
7  - Ensure that all facet links function normally
8  - Ensure that facets are the same a prior to patch
9  - Repeat for staff client
10 - Prove t/Search.t

NOTE: This patch makes it much easier to see that there is an existing issue with marking the "active" facet.  Ending punctuation seems to confuse the matcher.

Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-07-04 09:22:02 +01:00
Fridolin Somers 001d433b42
Bug 21534: Always compute wildcards
You get no results when searching with an hyphen + with * in query string (or with preference QueryAutoTruncate) :
ie /cgi-bin/koha/opac-search.pl?q=saints-anges*

Looks like query-string by default does not compute wildcards, see analyze_wildcard in :
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html

Test plan :
1) Use Elasticsearch
2) Create a record with "saints-anges"
3) Search for "saints-anges" => you get results
4) Search for "saints-anges*" => you get results
5) Search for "saints-ang*" => you get results

Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Arthur Bousquet <arthur.bousquet@inlibro.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-06-15 08:00:40 +01:00
Ere Maijala aba8b130f8 Bug 22413: Improve query string and desc creation for Elasticsearch
Test plan:

After each of the following steps, verify that the results and query description looks ok.

1. Do a simple keyword query
2. Change sort order
3. Click a facet
4. Do an advanced search with title and author
5. Change sort order
6. Click a facet

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>

Signed-off-by: Marjorie <marjorie.barry-vila@collecto.ca>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2019-04-25 10:38:17 +00:00
Ere Maijala e90cb16d79 Bug 22295: Make Elasticsearch query builder group multi-term queries
Test plan:

1. Do an advanced search for
Title = new
AND
Title = york
2. Verify that the results match an advanced search for:
Title = new york
3. Verify that tests in t/db_dependent/Koha/SearchEngine/Elasticsearch still pass

Signed-off-by: Michal Denar <black23@gmail.com>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2019-04-10 20:09:10 +00:00
Nick Clemens 59918a3c21 Bug 18213: Add language facets to Elasticsearch
This patch add language as a facet to ES results - it adds
a new template plugin for languages to get the appropriate
description given an iso 639-2 code

To test:
1 - Make sure you have records with differing languages (in the MARC21 008
        field characters 35-37 or UNIMARC 101a)
2 - Apply patch
3 - Reload Elasticsearch settings:
http://localhost:8081/cgi-bin/koha/admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
4 - Reindex your records
5 - Search for a phrase that will return results in several languages
6 - Verify you see factes correctly labelled for 'Language'
7 - Verify the facets work
8 - Verify both opac and staff results
9 - prove t/db_dependent/Languages.t

Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2019-03-28 15:57:32 +00:00
Ere Maijala dd98584c85 Bug 22228: Escape lone colons and make sure term is balanced
Escapes unquoted colons that have whitespace on either side. Removed unbalanced quotes.

Test plan:

1. Make sure the test case described in the bug works
2. Make sure tests pass:
  prove t/Koha/SearchEngine/Elasticsearch/QueryBuilder.t

Signed-off-by: Björn Nylén <bjorn.nylen(at)ub.lu.se>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2019-03-04 18:36:11 +00:00
David Gustafsson eeda2afd3a Bug 19575: Keep authority-number as alias and fix query
Keep authority-number as alias and change field name
from alias to real field in hard coded Elasticsearch query

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2019-03-04 18:25:49 +00:00
Nick Clemens ef2a03a092 Bug 19575: Fix an search field
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2019-03-04 18:25:49 +00:00
root c2061adc8b Bug 19575: Rebase and fix tests
This patch lower cases the sort by fields to normalize checking them and adjusts
some existing tests to meet the new expectations.

The regex for splitting terms has been moved into a subroutine so that adjustment was made

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2019-03-04 18:25:49 +00:00
David Gustafsson 6d53910f0c Bug 19575: Use canonical field names and resolve aliased fields
Adjust elastic search mappings to more closely match Zebra equivalents
resolving serveral issues with coded Zebra searches in templates and
sorting of search results in UI. Also make field names in search strings
case insensitive to accept case variations in template links and user input.

Sponsored-by: Gothenburg University Library

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2019-03-04 18:25:49 +00:00
Ere Maijala 0d9a42b492 Bug 21084: Fix automatic truncation in authority search
- Makes token splitting work like in biblio search
- Only adds right hand truncation since that's what Zebra also does
- Only adds truncation if the token is not a phrase and ends in a word character
- Adds tests to existing and new QueryBuilder functions

Test plan:

1. Create an authority record for "Duck, Donald"
2. Try contains type authority searches with the following terms and make sure they find the record:
Duck, Donald
donald duck
don duck
"Duck, Donald"
3. Make sure the following search works but does not find anything:
"Duck, Don"

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2019-01-18 20:18:31 +00:00
Ere Maijala a4c02b53cd Bug 19365: Fix search for duplicate authorities with Elasticsearch
Sponsored-by: National Library of Finland
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-26 16:11:02 +00:00
Ere Maijala be2bdfd511 Bug 19365: Fix several issues with the Elasticsearch code
Also optimize it so it's actually usable.

Test plan:

1. To test it properly you need biblio and authority data. You might get away with enabling AutoCreateAuthorities and BiblioAddsAuthorities so that authorities are created in the process. Another option would be to import authorities first e.g. from LoC.
2. Make sure the authority index has been properly created with "misc/search_tools/rebuild_elastic_search.pl -a -d"
3. Run "misc/link_bibs_to_authorities.pl -v -l" twice and observe the results.

Sponsored-by: National Library of Finland
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-26 16:11:01 +00:00
Nick Clemens 4abe3a4082 Bug 18316: (follow-up) Don't let 'fields' be undef and adjust tests
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-08 02:18:45 +00:00
Alex Arnaud 973bc0aceb Bug 18316: (follow-up) remove use_dis_max
It is true by default on ES 5 and has been removed on ES 6

Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-07 22:17:00 +00:00
Alex Arnaud 9380134a02 Bug 18316: (follow-up) Koha::SearchField::search_marc_maps return a result set - code refactoring for gettings weighted fields - Koha::SearchFields::weighted_fields return a result set
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>

Rebased-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-07 22:00:31 +00:00
Nick Clemens fa853f7d38 Bug 18316: Add weighting/relevancy options to ES query on simple search
The idea is the following: if some search field(s) are weighted in
search engine config page, Koha will query ES on all fields plus those with
the coresponding weights. Else, search is done on the entire record with
no weighting. The advanced search page is unaffected by these changes

Test plan (having Koha working with Elasticsearch):
  - apply this patch
  - have some weights defined for various fields
  - try searches from the search bar and from the advanced search page
  - confirm weighting affects the relevancy (in expected ways)
      e.g.
      1. search for 'a' from advanced search, note results
      2. give 'title' a weight
      3. search for 'a' using the simple search bar
      4. results with 'a' in the title should now be more relevant
  - confirm search results on advanced search page are unaffected

Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>

Rebased-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-07 22:00:30 +00:00
Alex Arnaud 96f253d5ed Bug 21032: Fix using facet for searches made on specific index
Test plan:

  - Try a simple search (OPAC or staff) like "title:foo",
  - click on a facet: badaboum!
  - apply this patch,
  - retry

Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2018-07-31 13:11:56 -03:00
Jonathan Druart 4d45bd5755 Bug 19604: Fix perlcritic "Loop iterator is not lexical"
Loop iterator is not lexical at line 318, column 13.  See page 108 of
PBP.  (Severity: 5)

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2018-07-31 10:44:36 -03:00
Nick Clemens f983c1e3f7 Bug 19604: Elasticsearch Fixes for build_authorities_query for auth searching
To test:
1 - Do some authority searches in Zebra
2 - Switch to ES and repeat, results will vary and some may fail
3 - Apply patch and dependencies
4 - Reindex ES
5 - Repeat searches, they should suceed and results should be similar to
Zebra
6 - Slight differences are okay, but results should (mostly) meet
expectations

A few notes:
We add a 'normalizer' to ensure we get a single token from the heading
indexes, this makes 'starts with' work as expcted
We switch to 'AND' for fields searched from cataloging editor - this
matches Zebra results
We force the '__sort' fields for sorting - if sorting looks wrong try
reducing the heading field to a single subfield - this will need to be
addressed on a future bug (multiple subfields create an array, ES sorts
        those randomly)

Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-07-19 17:23:54 +00:00
Nick Clemens 8b76ee39d4 Bug 19564: (QA follow-up) Make commented out tests pass
We likely shoudln't pass through an uncoverted sort order for now, but
it does allow us to look ahead to implementing the orders directly so
seems a good option to have.

Either this patch should be used, or the commented out tests should be
removed

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2018-03-28 16:05:10 -03:00
David Gustafsson 8e07761f30 Bug 19564: Fix extraction of sort order from sort condition name
Currently sort order is extracted from sort condition by splitting the
field, instead use regular expression to extract the last part preceded
by underscore.

Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2018-03-28 16:05:10 -03:00
Nick Clemens 49c5acc5b9 Bug 19582: Use compat routines for searching authorities in auth_finder.pl
To test:
1 - Enable Zebra
2 - Perform an auth search
3 - note results
4 - Enable ES
5 - Repeat search, note (likely) diff results
6 - Open a record in cataloging and use the button to launch auth search
7 - Perform same search as above, note results match for eiher engine
selected
8 - NOTE: Disbale sorting for ES search - this will be dealt with in
another report

Signed-off-by: David Bourgault <david.bourgault@inlibro.com>

Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2018-03-27 17:50:37 -03:00
Nick Clemens 834b8c749b Bug 19580: Exclude period as splitting character for autotrunction
To test:
0 - Apply Unit test patch only
1 - prove t/db_dependent/Koha_SearchEngine_Elasticsearch_Search.t
2 - Should fail
3 - Apply this patch
4 - prove t/db_dependent/Koha_SearchEngine_Elasticsearch_Search.t
5 - should pass
6 - search for 'Local-number.raw:"4"' (or a vlid biblionumber)
7 - should get expected result

Signed-off-by: David Bourgault <david.bourgault@inlibro.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2018-01-15 10:40:26 -03:00
Nick Clemens a081ed16c0 Bug 19559: Add '-' to list of characters we don't split search terms on
Test plan assumes
QueryAutoTruncate = automatically
SearchEngine = Elasticsearch

To test:
0 - Apply Unit test patch only
1 - prove t/db_dependent/Koha_SearchEngine_Elasticsearch_Search.t
2 - Should fail
3 - Apply this patch
4 - prove t/db_dependent/Koha_SearchEngine_Elasticsearch_Search.t
5 - should pass
6 - search for 'Local-number:"4"' (or a vlid biblionumber)
7 - should get expected result

Signed-off-by: David Bourgault <david.bourgault@inlibro.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-11-26 12:59:46 -03:00
Nick Clemens 088cdcda5f Bug 16660: Add support for OpacSuppression to Elasticsearch
To test:
1 - Enable suppression
2 - Suppress some records
3 - Apply all the patches
4 - Reindex ES
5 - Search and don't get suppressed records
6 - Disable suppression
7 - Search and get all the records

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-10-27 14:09:02 -03:00
Julian Maurice c991ac6b61 Bug 18374: (QA follow-up) Fix auto truncation for field:"value"
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-10-13 13:57:47 -03:00
Julian Maurice 6426a82014 Bug 18374: (QA follow-up) Simplify _truncate_terms
By using a different split regex, we can simplify a bit the process of
appending '*' to every word of the query

Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-10-13 13:57:47 -03:00
Nick Clemens d099dc13bb Bug 18374: (follow-up) Add support and tests for quoted strings
To test:
1 - prove t/db_dependent/Koha_SearchEngine_Elasticsearch_Search.t
2 - do some searches in staff client and test results

Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-10-13 13:57:33 -03:00
Julian Maurice e540651975 Bug 18374: (QA follow-up) Tidy up code
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-10-13 13:57:33 -03:00
Nick Clemens 4c4d982ae4 Bug 18374: Respect QueryAutoTruncate syspref in Elasticsearch
This patchset adds a subroutine '_truncate_terms' to the ES QueryParser.

If QueryAutoTruncate is enabled this function will be called for any
search to add wildcard '*' to all terms

To test:
1 - Enable Elasticsearch and have some records indexed
2 - Search for partial terms
3 - Note they fail unless '*' is appended
4 - Apply patch, leave QueryAutoTruncate disabled
5 - Note partial term searches still fail
6 - Enable QueryAutoTruncate
7 - Note partial term searches succeed
8 - Do some regular and advanced searches to make sure results are as
expected

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-10-13 13:01:12 -03:00