Starting in Elasticsearch 6.0 _key should be used instead of _term to order
buckets by their term
To test:
1) Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t
2) If you observe an error about types, apply patch for bug 25669
3) Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t
4) Observe if you get a deprecation warning about order key
5) Apply patch
6) Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t
7) Observe no deprecation warning about order key
8) Sign off
Sponsored-by: Lund University Library
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This expands the regex to handle this specific case
To test:
1 - Load record created for last patch
2 - Note analytics error
3 - Apply patch
4 - Restart and reload
5 - No more errorm also no Analytics link
6 - Add a 773$t to a record with title used before:
Digger does it all (not really!)
7 - reload the initial record
8 - See 'Show analytics' link
9 - Click the link
10 - You should end up on the record you added the 773 to
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
With all the work that's gone into improving the internal
_clean_search_term method I feel we should expose it publically as it's
going to be more widely helpful
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
if QueryAutoTruncate enabled we will have any special operators ruined
for example: "test [6 TO 7]" will be converted to "test* [6* TO* 7]"
so no reason to keep ranges when QueryAutoTruncate set to "enabled"
1) enable QueryAutoTruncate at your sysprefs.
2) perform a search using range, for example: "[1999 TO 2020]",
it shouldn't work the way it's supposed to.
3) apply the patch.
4) perform the same search with range, ensure that it works correctly.
Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch ensures that the behavior with
QueryRegexEscapeOptions set to values other than
"Escape" still will works as expected.
It does so by storing the contents of regexes
before escaping special characters and
then restores the contents of regexes back to how
it was before, ensuring that searching with regex is possible.
Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Currently having exclamation sign at the end of the query makes ES
search fail, and when you try to search for a book that has exclamation
sign in the tittle (something like "Words! words") won't show results
correctly as it tries to negate everything that is after exclamation
sign, making it impossible to search for books that have in in the title
This patch escapes exclamation signs if it's at the end of the query or
has a space after it, resolving both of the issues listed above.
To reproduce:
1) with ES enabled, search for the book with title that contains
exclamation sight at the end, like "book!", this search should result
in error.
2) do another search, but this time find/prepare beforehand book with a
title that has exclamation sign with a space after it,
e.g "exclamation! sign", it shouldn't find it as ES treats everything
after that exclamation sign as negation.
2) apply the patch.
3) perform searches from the steep one and two again.
Search from step one should no longer fail, while search from the step
two should find that book.
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch screens square and curly brackets which have no special
language meaning.
To reproduce:
1) using ES, search for the book with title that contains
square and/or curly brackets, like "book [second edition]", which will
result in error.
2) apply the patch.
3) search for that book again, ensure that it works now.
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Currently searches like: "book:", ":book" and "host-item:test:n"
cause internal server errors.
This patch adds additional regexes that remove the colons at the start
and end of the query, and another regex that screens all follow-up
colons that go after the first colon to avoid errors when searching for
"host-item:test:n".
To reproduce:
1) using ES, search for the book with title that contains
semicolon at the start or at the end of the line, separated with spaces,
this should cause internal server error.
2) try doing the same with something like "host-item:test:n", it should
result in error as well.
3) apply the patch.
4) repeat steps 1-2, ensure that it works now.
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch removes the code from the search scripts into QueryBuilder
modules.
To test:
1 - Have a library group defined as a search group for both staff and opac
2 - Search on staff client and opac with that group limit and a single branch limit
3 - Note your results/counts
4 - Note the visuals of the search description
5 - Apply patch
6 - Repeat searches
7 - All should work as before
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds the cni/Control-number-identifier index to enable
searches to use the 003 field.
Test plan
1/ Apply patch
2/ Re-index using updated configurations
3/ Confirm cni:number searches yield the expected results
4/ Signoff
Split-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Pasi Kallinen <pasi.kallinen@koha-suomi.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
In advanced search with Elasticsearch the limit on years range is actually using copydate :
Koha/SearchEngine/Elasticsearch/QueryBuilder.pm in _fix_limit_special_cases() :
if ( $l =~ /^yr,st-numeric,ge=/ ) {
my ( $start, $end ) =
( $l =~ /^yr,st-numeric,ge=(.*) and yr,st-numeric,le=(.*)$/ );
next unless defined($start) && defined($end);
push @new_lim, "copydate:[$start TO $end]";
}
With Zebra it uses date-of-publication and also in Koha/SearchEngine/Elasticsearch/QueryBuilder.pm :
our %index_field_convert = (
(...)
'yr' => 'date-of-publication',
This patch uses %index_field_convert to perform 'yr' limit.
Test plan:
1) Apply patch
2) Use Elasticsearch searchengine
3) Go to advanced search with 'More options'
4) Perform a search with a year limit (value or range)
5) Check results are correct
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
On bug 17591 we discovered that there was something weird going on with
the way we export and use subroutines/modules.
This patch tries to standardize our EXPORT to use EXPORT_OK only.
That way we will need to explicitely define the subroutine we want to
use from a module.
This patch is a squashed version of:
Bug 17600: After export.pl
Bug 17600: After perlimport
Bug 17600: Manual changes
Bug 17600: Other manual changes after second perlimports run
Bug 17600: Fix tests
And a lot of other manual changes.
export.pl is a dirty script that can be found on bug 17600.
"perlimport" is:
git clone https://github.com/oalders/App-perlimports.git
cd App-perlimports/
cpanm --installdeps .
export PERL5LIB="$PERL5LIB:/kohadevbox/koha/App-perlimports/lib"
find . \( -name "*.pl" -o -name "*.pm" \) -exec perl App-perlimports/script/perlimports --inplace-edit --no-preserve-unused --filename {} \;
The ideas of this patch are to:
* use EXPORT_OK instead of EXPORT
* perltidy the EXPORT_OK list
* remove '&' before the subroutine names
* remove some uneeded use statements
* explicitely import the subroutines we need within the controllers or
modules
Note that the private subroutines (starting with _) should not be
exported (and not used from outside of the module except from tests).
EXPORT vs EXPORT_OK (from
https://www.thegeekstuff.com/2010/06/perl-exporter-examples/)
"""
Export allows to export the functions and variables of modules to user’s namespace using the standard import method. This way, we don’t need to create the objects for the modules to access it’s members.
@EXPORT and @EXPORT_OK are the two main variables used during export operation.
@EXPORT contains list of symbols (subroutines and variables) of the module to be exported into the caller namespace.
@EXPORT_OK does export of symbols on demand basis.
"""
If this patch caused a conflict with a patch you wrote prior to its
push:
* Make sure you are not reintroducing a "use" statement that has been
removed
* "$subroutine" is not exported by the C4::$MODULE module
means that you need to add the subroutine to the @EXPORT_OK list
* Bareword "$subroutine" not allowed while "strict subs"
means that you didn't imported the subroutine from the module:
- use $MODULE qw( $subroutine list );
You can also use the fully qualified namespace: C4::$MODULE::$subroutine
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
In Elasticsearch, query for biblios uses lenient=true.
This is also needed for authorities search.
In case a search field is defined as type numeric.
Test plan :
1) Use Elasticsearch searchengine
2) Define a search field 'local-number' as type 'Number'
3) Be sure to us 'local-number' in autorities mapping
4) Rebuild autorities
5) Performe a search for autorities with 'Search entire record' and
'contains' with term '123'
=> Without patch you get error :
[query_shard_exception] Can only use prefix queries on keyword and text fields - not on [local-number] which is of type [integer]
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
If we are searching on kw there is a leading colon at the beginning of
the generated query:
kw:foo becomes :foo
Note that it only happens when there is no other terms before.
Test plan:
0. Don't apply the patch
1. Search for kw:foo
2. Notice the error
Error: Unable to perform your search. Please try again.
and the logs say
Failed to parse query [(:foo*)]
3. Apply the patch
4. Repeat the search and notice that you know get:
"12 result(s) found for 'kw:foo'."
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
In zebra we sort by callnumber using 8007 cn-sort
We should do the same in elasticsearch
To test:
1 - Have Koha using Elasticsearch
2 - Perform a search
3 - Attempt to sort by callnumber
4 - Error in logs: No mapping found for [local-classification__sort]
5 - Apply patch
6 - Restart all
7 - Perform a search and sort by callnumber
8 - Success!
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
It'd be great if the Search Engine Configuration page would display
the various aliases (shortcuts) available : ti for title, sn for local-number, etc.
Patch changes Koha/SearchEngine/Elasticsearch/QueryBuilder.pm to move
hard-coded vars at the beging and adds a method to provide to %index_field_convert via a method.
Test plan :
1) Use Elasticsearch
2) Go to Administration > Search engine configuration (Elasticsearch)
3) Check you see new column 'Aliases' with for example ti for title.
4) Perform a search 'ti:<title>' and check you get results
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
We convert 'keyword' to '' as an index since we want it to search all fields
When we are searching 'as phrase', however, we should not drop the search type
To test:
1 - Enable IntranetCatalogPullDown
2 - Set searchEngine to Elasticsearch
3 - Perform a search for 'Keyword as phrase' for a phrase that does appear in a record
4 - You get the result
5 - Reverse the order of words in the phrase
6 - You still get a result?
7 - Apply patch
8 - Restart all the things
9 - Reversed search does not return record
10 - Correct order and search, correct record returned
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch prepares Koha to officially no longer support Elasticsearch 5.X
It adds a new system preference 'ElasticsearchCrossFields' to allow users to choose whether or not
to enable this feature
It updates the about page to add a deprecation warning if a site is running ES5
To test:
1 - Be running Koha with Elasticsearch 5.X
2 - Attempt to search
Error: Unable to perform your search. Please try again.
3 - Apply patch
4 - Update database
5 - Searching works
6 - Find syspref 'ElasticsearchCrossFields'
7 - Enable it
8 - Searching is now broken
9 - Check the about page
10 - you can now see the Elasticsearch version
11 - The systeminformation tab has a deprectaion warning
12 - Set SearchEngine preference to 'Zebra'
13 - View the about page - no warnings
14 - Test again with ES6 - searching should "work" with either pref setting
15 - There should be no warning on about pages
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds the 'cross_fields' type to our searches:
https://www.elastic.co/guide/en/elasticsearch/reference/6.8/query-dsl-query-string-query.html#query-string-syntax
Without this patch the search terms seem to all require being in the same field when using Elasticsearch 6
To test:
0 - Set QueryAutoTruncate to 'only if * is added'
1 - Find a record with a title and publisher
2 - Search for a word form the title and confirm the record is returned
3 - Search for a work from the title and the publisher's name
4 - The record is not returned
5 - Apply patch
6 - Repeat #3
7 - The record is returned
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Without this patch we get
Use of uninitialized value $3 in concatenation (.) or string at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/QueryBuilder.pm line 943.
This converts the | OR operator to two different regexes so that the
capture group variables will be defined in every case.
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This updates the regex used for removing colons to capture those with space on either side, and remove the colon
while preserving the space
To test:
1 - Have Koha using ES
2 - Search for:
ti:chess AND chess
3 - You should get a result in sample data, otherwise replace 'chess' with a title in your catalogue
4 - Search for:
ti:chess AND kw:chess
5 - No result
6 - Enable DumpTemplateVarsIntranet and DumpSearchQueryTemplate
7 - Repeate search and check page source
8 - search_query has:
title:chess ANDchess
9 - Apply patch
10 - Repeat
11 - Seaerch works!
12 - query is now:
title:chess AND chess
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Bug 24567: (follow-up) Use dollar sign to refer to captures
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
The match-heading field is a special field used only by the linker, not accessible
to staff or patrons via the interface. This field is used to store the constructed
'search form' used for matching bib headings to authority fields.
In bug 24269 I attempted to use the mappings defined in the inferface and also inject the search term.
This did not work as too many subfields were indexed on their own and leading to false matches.
In this bug we remove the mappings for this field, and create it ourselves during
the indexing process. The C4::Headings module is still used to generate the correct form,
however, the mappings are set based on the authority types in the system. This gives the user
the ability to add new typoes, but prevents mapping changes from breaking linker functionality
To test:
1 - Start form a sample database with ElasticSearch working
2 - Download via Z39.50 2 authorities, one of which is a narrower heading of the other, e.g.:
Waterworks
Waterworks - Costs
3 - Place a heading for the broader term in a record. e.g. Waterworks
In 650$a, without the cataloguing authority plugin. We don't want
the link created now.
You need syspref BiblioAddsAuthorities => allow
4 - Make sure linker is set to default
5 - Attempt to link the records
misc/link_bibs_to_authorities.pl
6 - Linking fails
7 - Apply patch
8 - refresh index settings (if using a custom file, remove 'match-heading')
You can reset mappings in the UI or run this:
misc/search_tools/rebuild_elasticsearch.pl -v -d -r
9 - Reindex ES
10 - Try to link again
11 - It succeeds!
12 - Run the tests
prove t/db_dependent/Koha/SearchEngine/Elasticsearch.t
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Bug 25273: (follow-up)
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This takes care of more occurences of staff client and changes it to
staff interface, including in code comments.
To test:
- I think in this case careful code review is what we look for.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
The code in build_query_compat contained a 'my' in the assigning of limits for the search in a conditional
This meant the limits were being set correctly during the conditional, but we blanked when passed to the rest
of the code. The effect was that the searches worked, however, the template params to repeat the search were
incomplete.
Removing the my ensures the same limits are applied during search and on re-sorting
To test:
1 - Enable Elasticsearch
2 - On OPAC perform advanced search, selecting only an itype, ccode, or LOC limit
3 - Attempt to sort results
4 - You are returned to Advanced search
5 - Apply patch
6 - Repeat
7 - It sorts!
Signed-off-by: Lisette Scheer <lisetteslatah@gmail.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
We assume all limits from advanced search to be a phrase and quote them
when doing this we should remove the phrase marker to avoid doulbe quoting
To test:
1 - Have koha using ES
2 - Go to advanced search
3 - Limit by a single itemtype that exists
4 - Get some results
5 - Limit by a different itemtype that exists
6 - Get some results
7 - Limit by both itemtypes
8 - Get only the results for the second itemtype
9 - Enable DumpTemplateVarsIntranet and DumpSearchQueryTemplate
10 - Repeat search
11 - View page source and find 'search_query'
12 - See limit looks like itype:("("BK")" OR "("CR")")
13 - Apply patches
14 - Restart all the things
15 - Repeat search for both itemtypes
16 - Note results now include both types
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
To test:
0 - Set DisplayLibraryFacets to 'both' and FacetMaxCount to 20
1 - Have more than 10 branches
2 - Have items in each of those branches
3 - Enable ES, set system preference SearchEngine to Elasticsearch
4 - Search for '*'
5 - Expand homebranch/holdingbranch facets
6 - Note you only get 10
7 - Apply patch
8 - Repeat search
9 - Now you get all your facets
Signed-off-by: Michael Springer <mspringer@mylakelibrary.org>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
To recreate:
1 - Have Koha using ES5 and Elasticsearch as search engine
2 - Enable DumpTemplateVarsIntranet and DumpSearchQueryTemplate
3 - Do a search in authorities using 'Search entire record' (abduction if using sample db)
4 - Note no results
5 - View the page source and find 'search_query'
6 - Note the uppercased fields
7 - curl 'es:9200/koha_kohadev_authorities/data/417?pretty'
8 - Note all fields lower-cased
9 - Apply patch
10 - Repeat search
11 - It works!
Signed-off-by: Heather Hernandez <heather_hernandez@nps.gov>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
In the advanced search form, you can enable several limits using syspref
AdvancedSearchTypes (namely itemtypes, shelving locations, collections)
When used, the resulting query parts end up being joined with OR, even
if the field is different. That means that if you pick "Book" under
itemtypes tab, and "Fiction" under collection tab, it will search
"itype:BOOK OR ccode:FIC". It should be AND.
For instance, if you select:
Itemtypes:
✓ Book
✓ DVD
Location:
✓ Child
✓ Adult
it should search:
itype:(Book OR DVD) AND location:(Child OR Adult)
Test plan:
0. Do not apply the patch yet
1. Enable elasticsearch
2. Set syspref AdvancedSearchTypes = 'itemtypes|loc|ccode'
3. Create a new itemtype and a new authorised value for categories LOC
and CCODE
4. Create a biblio with the new itemtype, another biblio with the new
location, another biblio with the new collection, and again another
biblio with the new itemtype, location and collection
5. Verify that you can find these new biblio records using only the
"advanced search types" in the advanced search form
6. In the advanced search form, pick all 3 limits (itemtype, location,
collection) and verify that it returns the 4 records.
7. Apply the patch
8. Repeat step 6, it should now return only the biblio that satisfies
all criteria
9. Verify that if you select more than one
{itemtype|location|collection} it still returns results that
satisfies any selected criteria
10. prove t/Koha/SearchEngine/ElasticSearch/QueryBuilder.t
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Not only should terms from facets/limits be grouped, but order should probably matter
We build a different facet for "Dillinger Girl" or "Girl Dillinger"
The parens are possibly overkill now, but I think it makes it very clear and does not hurt
To test:
1 - Follow original plan
2 - Create a new record with 'Girl Dillinger' as author
3 - Search for 'Dill*'
4 - You get all three records (and maybe others that match)
5 - Limit by 'Girl Dillinger' - you get two records
6 - Same for 'Dillinger Girl'
7 - Apply patch
8 - Limits/facets for 'Dillinger Girl' and 'Girl Dillinger' now match a isngle record
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Test plan:
- Use Elasticsearch 6 (you'll need Bug 18969),
- create a biblio (#1) with "Dillinger Girl" in author and what you
want in title,
- create another biblio (#2) with the word "girl" in the title and
"Dillinger Escaplan" as author
- reindex
- search * and refine on "Dillinger Girl"
- Ko => Biblio #1 and #2 appear
- Apply this patch,
- search * and refine on "Dillinger Girl"
- Ok => anly biblio #1 appears
- use Elasticsearch 5 again
- check for no search regression
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
A search made on Shelving location tab in advanced search gives 0 result.
Test plan :
Without patch:
- in advanced search, choose a location in shelving location tab and start search (opac-search.pl?idx=kw&op=and&idx=kw&op=and&idx=kw&do=Search&limit=mc-loc)
- 0 result
- check that location exist by doing a simple search and filtering on the location facet
- have some results
With patch:
- in advanced search, choose a location in shelving location tab and start search (opac-search.pl?idx=kw&op=and&idx=kw&op=and&idx=kw&do=Search&limit=mc-loc)
- found results
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
To test:
1 - Enable ES
2 - Enable OpacSuppression
3 - Suppress a bib in staff client
Edit 942n to be 1
4 - Search the opac for anything
5 - Tail the plack logs and note a deprecation warning
6 - Apply patch
7 - Restart all the things
8 - Repeat search on opac
9 - No error/warn in logs
10 - Record is correctly suppressed
Signed-off-by: Myka Kennedy Stephens <mkstephens@lancasterseminary.edu>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
To test:
1 - In staff or OPAC using ES, search for "biblionumber:3" or any existing biblionumber
2 - No results
3 - Apply patch, restart all the things
4 - Search again
5 - You go to the biblionumber
Signed-off-by: Bob Bennhoff <bbennhoff@clicweb.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Note: I also remove warnings for undefined operation in this patch, is a trivial fix
To test:
prove -v t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
To test:
1 - Export your authorities via Tools->Export data
2 - Define a record matching rule in Admin->Record matchign rules
Use index: LC-card-number
field: 010$a
3 - Stage the exported records for import and use the rule created above for matching
4 - The process does not complete
5 - Check intranet error logs - exception on unknown marclist
6 - Apply patch
7 - Repeat
8 - Success!
9 - prove -v t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Adds support for using the "scan indexes" action in advanced search by using faceting with a prefix filter. Requires that the field be set as facetable for anything to be found.
Test plan:
1. Apply patch
2. Go to advanced search and click "More options"
3. Select author as the search field, enter a last name and check "Scan indexes"
4. Perform search and observe the result list resembling scan results
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
As of bug 20589 we no longer analyze sort fields and so we no longer need to append ".phrase"
to our sort in searches.
Additionally, sort fields based on 'sum' should also use sum in building the value to sort on
To test:
0 - Be using ES
1 - Find the most circulated item in your collection
2 - Search for '*'
3 - Sort by popularity DESC
4 - Note that item is not first
5 - Try to sort by anything but relevancy, it fails
6 - Apply patch
7 - Redo searches and sorts
8 - Things should now work as expected
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Test plan
- Set up a elasticsearch 6 instance to work with Koha,
- you may need to make koha works with ES 6 (see bug 20589),
- make a search and limit it to available items only,
=> no result
- Apply this patch,
- make a search and limit it to available items only,
- you should get some results
- Do the same with ES 5.x
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
During cataloguing of an existing biblio, on an heading field,
the use of tag editor fills authorities finder with existing value :
Search main heading ($a only)
Search main heading
Default operator beeing 'contains'.
Actually with Elasticsearch those search give no results.
Example with heading :
200
$a Casaubon
$b Isaac
$f 1559-1614
Call to Elasticsearch :
"query" : {
"bool" : {
"must" : [
{
"query_string" : {
"query" : "Casaubon*",
"default_field" : "heading-main",
}
},
{
"query_string" : {
"query" : "(Isaac*) AND (1559-1614*)",
"default_field" : "heading"
}
}
]
}
},
"sort" : [
{
"heading__sort.phrase" : "asc"
}
]
}
Patch adds to "query_string" :
analyze_wildcard : true.
Test plan :
1) Use Elasticsearch
2) Edit an existing biblio record
3) Use tag editor on a heading
4) Click search => You get correct results
5) Check also search in authorities-home.pl
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Add "QueryRegexEscapeOption" system preference to provide option to escape
Elasticsearch regexp delimiters (/) within queries, or alternativly to
unescape escaped slashes (\/) while escaping unescaped slashes, in
effect making "\/" the new regexp delimiter.
How to test:
1) Run tests in ./t/Koha/SearchEngine/ElasticSearch/QueryBuilder.t
2) All tests should succeed
Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Maksim Sen <maksim.sen@inlibro.com>
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Generate a list of fields for the query_string query fields parameter,
with possible boosts, instead of using "_all"-field. Also add "search"
flag in search_marc_to_field table so that certain mappings can be
excluded from searches. Add option to include/exclude fields in
query_string "fields" parameter depending on searching in OPAC or staff
client. Refactor code to remove all other dependencies on "_all"-field.
How to test:
1) Reindex authorities and biblios.
2) Search biblios and try to verify that this works as expected.
3) Search authorities and try to verify that this works as expected.
4) Go to "Search engine configuration"
5) Change some "Boost", "Staff client", and "OPAC" settings and save.
6) Verify that those settings where saved accordingly.
7) Click the "Biblios" or "Authorities" tab and change one or more
"Searchable" settings
8) Verfiy that those settings where saved accordingly.
9) Try to verify that these settings has taken effect by peforming
some biblios and/or authorities searches.
Sponsorded-by: Gothenburg Univesity Library
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
In cataloguing, the use of tag editor opens authorities finder with a limit on specific authorities type.
This limit is missing with Elasticsearch.
This patch adds in query a "filter" on "term" which is the most performant way because there must be no ranking computed on this part.
Test plan :
1) Use Elasticsearch
2) Create an autority of type author (NP in UNIMARC) with heading "Tolkien"
3) Create an autority of type subject-author (SAUT in UNIMARC) with heading "Tolkien"
4) Create a biblio record
5) Use tag editor on a author field (700 in UNIMARC)
6) Seach for "Tolkien" in $a
without patch : you see the 2 authorities in results
with patch : you see only the correct authority in results (NP in UNIMARC)
7) Check search in authorities-home.pl is still OK
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Avoid using slash in the field name since it would need to be escaped. Fix conversion of dtlm and any existing mapping.
Signed-off-by: Liz Rea <wizzyrea@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Fridolin Somers
Nick Clemens
Kevin Carnes
Martin Renvoize
Petro Vashchuk
Jonathan Druart
Joonas Kylmälä
Katrin Fischer
Julian Maurice
Alex Arnaud
Ere Maijala
David Gustafsson