This patch uses the filter option from the mappings to add a value_callback
to reord processing for indexing.
Fields defined with 'punctuation' filter will have all punctuation stripped when
conveted to documents.
Tests are updated.
Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
In MARC 21, having title search field defined in mappings.yaml as 245abp
(more than one subfield), the Koha generated title__sort will have not
only $a, but also other subfields cut by ind2 characters, i.e.:
245 04 $a Die Renaissance : $b Architektur, Plastik, Malerei, Illustrationen, Zeichnungen
becomes as title__sort:
Renaissance : itektur, Plastik, Malerei, Illustrationen, Zeichnungen
instead of:
Renaissance : Architektur, Plastik, Malerei, Illustrationen, Zeichnungen
This is because copying an array of pointers in Perl causes you to
point to the same (anonymous) element. Therefore, the intended
modification only for subfield "a"
(Koha/SearchEngine/Elasticsearch.pm, ~line 1157) causes in fact
modification for all the subfields sharing the same mapping
definition and introduction of the nonfiling_characters_indicator to
all subfields 245 when processing 245abp. And this is obviously not
what is intended and results (in some cases) with bad sort order
when sorting by titles.
Test plan
=========
1. Have a standard Koha ES installation. In mappings.yaml
you should have 245abp for title search field.
2. Insert / download / modify a record with 245 $b or $p,
an article at the beginning of the title, and correctly
set 2nd indicator. For instance:
245 04 $a Die Renaissance : $b Architektur, Plastik, Malerei, Illustrationen, Zeichnungen
3. In ES, with Kibana or culr, control the content of
title__sort field for the related document. It will be:
Renaissance : itektur, Plastik, Malerei, Illustrationen, Zeichnungen
4. Apply the patch, full reindex ES (--rebuild -r -d -a -b)
5. In ES, with Kibana or culr, control the content of
title__sort field for the related document. It should
be now correct, i.e.:
Renaissance : Architektur, Plastik, Malerei, Illustrationen, Zeichnungen
Signed-off-by: Nick <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Before this patch we used two indexes for the thesaurus values, we can
simply index both needed fields into a single index and just form the
search correctly.
This patch also ensures we pass the 'thesaurus' vlaue for the heading
directly to the query builder - for zebra it goes through, and for ES
we convert it to the expected code.
This patch also moves the necessary mappings out of the user definable
mappings and hardcodes them. There is precedent for this with
'match-heading', it ensures matching works as expected
To test:
1 - Follow previous test plan in Zebra and ES
Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Previous commit correctly lists Zebra availability test, but was a
bad edit. This fixes the availability test to use itemlost
Additiionally, tests were written before limits were agreed. Adjusted
for the current settings
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
In Zebra the availability limit code states:
'available' is defined as (items.onloan is NULL) and (items.itemlost = 0)
We should make ES follow the same behaviour so that changing the engine does not
change this.
To test:
1 - Find or create several records with two items each
2 - For those records setup items as:
a - one checked out, one available
b - both marked lost
c - both damaged
d - both withdrawn
e - both notforloan
3 - Enable ES and reindex to confirm everything is set
4 - Perform an advanced search to return those records and 'limit to available items'
5 - Confirm you see records b,c,d,e and records say '2 items, none available'
6 - Apply patch
7 - Run `misc/search_tools/rebuild_elasticsearch.pl -d -b`
8 - Perform an advanced search to return those records and 'limit to available items'
9 - Confirm you see records a,b,c,d and al but a say '2 items, none available'
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Add a new boolean ES field named 'available', which is true if at least
one item is available, which means the item is not on loan, not
"notforloan", not withdrawn, not lost and not damaged
A full indexation is required
Test plan:
1. Apply patch and run updatedatabase.pl
2. Run `misc/search_tools/rebuild_elasticsearch.pl -d -b`
3. Make sure you have some biblios whose items are all unavailable, some
biblios whose items are all available, and some biblios with at least
one item available and at least one item unavailable
4. Use the 'available' filter on both opac and intranet and make sure it
works as expected.
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
When defining our sort fields in we defined all as 'numeric'
For other string containing numbers this is likely correct, however,
for callnumbers it is not. e.g. E45 should sort before E7
This patch adds a new 'callnumber' type and deifnes this for cn-sort and
adds to the field maping a sort without numeric set
To test:
0 - Be using ES with Koha
1 - On records with single item, add callnumbers:
VA65 E7 R63 1984
VA65 E7 T35 1990
VA65 E45 R67 1985
2 - Add public note 'shrimp' or something to make them easily searchable as a group
3 - Search for 'shrimp', sort by callnumber
4 - Note E45 comes last, it should come first
5 - Apply patch
6 - Reset ES mappings
7 - Reindex ES
8 - Repeat search
9 - Sorting should be correct when set to callnumber
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Michal Urban <michalurban177@gmail.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch includes the parameter:
include_type_name
on our ES calls
It removes the deprecation warning we are seeing, and should allow using ES7 with no other chnages.
For ES8 we will need to remove the hardcoded type,
I would suggest a second patch, with a new syspref "ElasticsearchLegacyType" set to 'True' by default.
Description can explain that this must be set to false, and data reindexed after upgrading to ES7 and before
upgrading to ES8 - then we can drop this pref when we drop ES7 support
When we drop ES6 support the pref can be set default true for upgrades, default 'false' for new installs
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
and some more...
There are lot of inconsistencies in our ->search calls. We could
simplify some of them, but not in this patch. Here we want to prevent
regressions as much as possible and so don't add unecessary changes.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
I need to add a 'request_timeout' entry to the ES configuration to fix some timeout problems on the sandboxes.
Instead of hard coding this new option it seems preferable to be
flexible and allow different options to be passed from the config file.
Test plan:
Add to the $KOHA_CONF, inside the elasticsearch section
<request_timeout>60</request_timeout>
Rebuild the ES index
koha-elasticsearch --rebuild kohadev
Change the value of the timeout to 1
Rebuild the index
It should fail (with a quite bad error 'Bad response received when
submitting request to Elasticsearch')
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Test plan:
1. Create an authority with at least a 1XX$a and a 4XX$a, for instance:
100 $a Foo
400 $a Bar
2. Create a biblio and add a link to this authority using the
cataloguing plugin
3. Disable syspref IncludeSeeFromInSearches
4. Reindex the biblio. You should be able to find it when searching
'Foo' but not when searching 'Bar'
5. Enable syspref IncludeSeeFromInSearches
6. Reindex the biblio. You should be able to find it when searching
'Foo' and also when searching 'Bar'
without the patch, 'Bar' doesn't yeld results. With it, it does.
7. prove t/db_dependent/Koha/SearchEngine/Elasticsearch.t
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Lucy Vaux-Harvey <lucy.vaux-harvey@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
There is existing error message in the code stating:
"Unable to understand your search query, please rephrase and try again."
which fits perfectly but because it looks for "ParseException" in the
warning output it doesn't show up on this page as it's actually
"parse_exception".
This patch makes that it's also checked if "parse_exception" is present
in the warning output.
Side note:
"ParseException" reaction code was added here:
e0f6c4dc Bug 12478: improve error reporting a bit
Search::Elasticsearch seems propagates clean ES JSON answer,
and in current ES version inside of $@ it contains "parse_exception"
string in dumped JSON answer ("'type' => 'parse_exception'").
Old seeked phrase "ParseException" wasn't reproduced, only in ES logs
("Caused by: org.apache.lucene.queryparser.classic.ParseException:
Cannot parse ..."). Check for both phrases won't complicate future
changes, but this note added for reference and code cleanup if needed.
To reproduce:
1) using ES search for something like "// ^ ! { } [ ] .. , <>" that
will for sure break the syntax of ES.
2) after the search query fails note that the error is
"Unable to perform your search. Please try again."
3) apply the patch
4) search for the same thing again
5) error message should be "Unable to understand your search query,
please rephrase and try again." now.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Andrew Nugged <nugged@gmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
On bug 17591 we discovered that there was something weird going on with
the way we export and use subroutines/modules.
This patch tries to standardize our EXPORT to use EXPORT_OK only.
That way we will need to explicitely define the subroutine we want to
use from a module.
This patch is a squashed version of:
Bug 17600: After export.pl
Bug 17600: After perlimport
Bug 17600: Manual changes
Bug 17600: Other manual changes after second perlimports run
Bug 17600: Fix tests
And a lot of other manual changes.
export.pl is a dirty script that can be found on bug 17600.
"perlimport" is:
git clone https://github.com/oalders/App-perlimports.git
cd App-perlimports/
cpanm --installdeps .
export PERL5LIB="$PERL5LIB:/kohadevbox/koha/App-perlimports/lib"
find . \( -name "*.pl" -o -name "*.pm" \) -exec perl App-perlimports/script/perlimports --inplace-edit --no-preserve-unused --filename {} \;
The ideas of this patch are to:
* use EXPORT_OK instead of EXPORT
* perltidy the EXPORT_OK list
* remove '&' before the subroutine names
* remove some uneeded use statements
* explicitely import the subroutines we need within the controllers or
modules
Note that the private subroutines (starting with _) should not be
exported (and not used from outside of the module except from tests).
EXPORT vs EXPORT_OK (from
https://www.thegeekstuff.com/2010/06/perl-exporter-examples/)
"""
Export allows to export the functions and variables of modules to user’s namespace using the standard import method. This way, we don’t need to create the objects for the modules to access it’s members.
@EXPORT and @EXPORT_OK are the two main variables used during export operation.
@EXPORT contains list of symbols (subroutines and variables) of the module to be exported into the caller namespace.
@EXPORT_OK does export of symbols on demand basis.
"""
If this patch caused a conflict with a patch you wrote prior to its
push:
* Make sure you are not reintroducing a "use" statement that has been
removed
* "$subroutine" is not exported by the C4::$MODULE module
means that you need to add the subroutine to the @EXPORT_OK list
* Bareword "$subroutine" not allowed while "strict subs"
means that you didn't imported the subroutine from the module:
- use $MODULE qw( $subroutine list );
You can also use the fully qualified namespace: C4::$MODULE::$subroutine
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
koha-elastic --rebuild was failing with
Unable to update mappings for index "koha_kohadev_biblios". Reason was: "Could not convert [marc_data.index] to boolean"
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Subdivision authorities are not used for linking, however, they are recognized by
C4::AuthoritiesMarc
While these records are not used for linking, they could provide reference and
should be allowed to exist in the catalog without breaking ES indexing
THis patch simply skips the step of parsing the authorities into the linking form
if the type contains '_SUBD'
To test:
1 - Import a subdivision authority record via Z39 or use the one attached to this bug
2 - perl misc/search_tools/rebuild_elasticsearch.pl -v -d
3 - Authority indexing dies:
Use of uninitialized value $tag in hash element at /usr/share/perl5/MARC/Record.pm line 202.
Use of uninitialized value $tag in regexp compilation at /usr/share/perl5/MARC/Record.pm line 206.
Use of uninitialized value $tag in hash element at /usr/share/perl5/MARC/Record.pm line 207.
Can't call method "tag" on an undefined value at /kohadevbox/koha/C4/Heading.pm line 71.
4 - Apply patches
5 - reindex
6 - Success!
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
To test:
1 - Apply patch
2 - ./installer/data/mysql/updatedatabase.pl
3 - Reset ES mapping: Administration->Search engine configuration , button at bottom of page
4 - 'issues' and 'title' mapping under 'search fields' should be mandatory and not editable
5 - On 'Bibliographic records' tab you should not be able to delete the single entry for issues
6 - You should be able to delete 'title' mappings, however, at the final one you should be stopped by javascript
7 - Bonus: force remove the last mapping from the page using developer tools - attempt to save and should be warned of missing mandatory mapping
Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
For debugging purposes we may wish to see the requests and responses made to
Elasticsearch
To test:
1 - prove -v t/Koha/SearchEngine/Elasticsearch.t
2 - Set <trace_to>Stderr</trace_to> in koha-conf
3 - Restart all
4 - perl misc/search_tools/rebuild_elasticsearch.pl
5 - Note requests are shown
6 - Set
<trace_to>File</trace_to>
<trace_to>/var/log/koha/kohadev/plack-error.log</trace_to>
in koha-conf
7 - Restart all
8 - perl misc/search_tools/rebuild_elasticsearch.pl
9 - Check the plack log and see the ES requests
Signed-off-by: Bob Bennhoff <bbennhoff@clicweb.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Currently if you combine subfields in the marc mappings the subfields are indexed in the order
listed in the mapping.
i.e. 650(avxyz) in mapping
and in record:
650 $aHeading $zGeosubdiv $vFormsubdiv
is indexed as:
Heading Formsubdiv Geosubdiv
We should preserve the order and index as:
Heading Geosubdiv Formsubdiv
We can use built in function in Marc::Field to achieve this
To test:
1 - It is easy to find examples of this using authorities
2 - Find or create a record with subfields order azv
e.g. 150$aActresses$zUnited states$vBiography
3 - Add or have a second authority
e.g. 150$aActresses$vPortraits
4 - Set an authorities mapping for 'Heading' to 150(abgvxyz)
find at:
Administration->Search engine configuration (Elasticsearch)->Authorities tab
5 - Index the records in Elasticsearch
perl misc/search_tools/rebuild_elaticsearch.pl -a -ai 1691 -ai 1692
6 - View the first record in the ES index
curl es:9200/koha_kohadev_authorities/data/1692?pretty
7 - Note 'Heading' field is ordered as in the mapping
8 - Search authorities for 'contains' "act"
9 - Note the records sort incorrectly
10 - Apply patches
11 - perl misc/search_tools/rebuild_elaticsearch.pl -a -ai 1692
12 - curl es:9200/koha_kohadev_authorities/data/1692?pretty
13 - Note the order is now preserved
14 - Search authorities for 'contains' "act"
15 - Note the records sort correctly
Signed-off-by: Heather Hernandez <heather_hernandez@nps.gov>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
JD amended patch: Fix
FAIL spelling
combind ==> combined
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
To test:
1 - Load the sample DB or edit a record (using advanced cataloging editor) to have a blank subfield in a field that is indexed as suggestible
2 - For example 'author' / 100a
100 _ _ ‡a
3 - Index that record into Elasticsearch 5.X:
perl misc/search_tools/rebuild_elasticsearch.pl -v -bn 115 -b -d
4 - Note error 'value must have length > 0'
5 - Edit mappings to set author 100a not suggestible
6 - perl misc/search_tools/rebuild_elasticsearch.pl -v -bn 115 -b -d
7 - Success
8 - Set field to suggestible again
9 - Apply patch
10 - perl misc/search_tools/rebuild_elasticsearch.pl -v -bn 115 -b -d
11 - Success!
Signed-off-by: Bob Bennhoff <bbennhoff@clicweb.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
To test:
1 - Have some records with uncertain dates in the 008
19uu, 195u, etc.
2 - Index them in Elasticsearch
3 - Do a search that will return them
4 - Sort results by publication/copyright date
5 - Note odd results
6 - Apply patch
7 - Reindex
8 - Sorting should be improved
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Add a "year" search field type. Fields with this type will only
retain values that looks like years, so invalid values such as
whitespace or word characters will not be indexed.
This for instance improves the behaviour when sorting by
"date-of-publication". If all values are indexed, records with
junk data instead of valid years will appear first among the search
results, drowning out more relevant hits. If assigning this field
the "year" type these records will instead always appear last,
regarless of sort order.
To test:
1) Have at least two biblios, one with a valid year in 008 (pos 7-10)
and another with an invalid one ("uuuu" for example)
2) Perform a wildcard search (*) and sort results by publication date.
3) The record with invalid year of pulication in 008 should appear first
4) Apply patch and run database updates
5) Reindex ElasticSearch
6) Perform the same search as in 2)
7) The record with the invalid year should now appear last
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
In get_elasticsearch_params we set the conf to static if undefined,
but we never defined it
To test:
1 - Apply unit test patch
2 - prove -v t/Koha/SearchEngine/Elasticsearch.t
3 - It fails
4 - Apply this patch
5 - It succeeds
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
We are guessing authtype code and inserting the heading built accorindg to C4::Heading
If we can't identify the auth type, we can format the heading. There is a record in the koha test
data that is missing the heading field so type cannot be idenfitied
This prevents us from dying n a record where we cannot identify the type.
Note: This code will also be triggered for custom authority types, higlighting
that they won't link because of hardcoded mappings in C4::Heading. We must tackle this
on a new bug
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
The match-heading field is a special field used only by the linker, not accessible
to staff or patrons via the interface. This field is used to store the constructed
'search form' used for matching bib headings to authority fields.
In bug 24269 I attempted to use the mappings defined in the inferface and also inject the search term.
This did not work as too many subfields were indexed on their own and leading to false matches.
In this bug we remove the mappings for this field, and create it ourselves during
the indexing process. The C4::Headings module is still used to generate the correct form,
however, the mappings are set based on the authority types in the system. This gives the user
the ability to add new typoes, but prevents mapping changes from breaking linker functionality
To test:
1 - Start form a sample database with ElasticSearch working
2 - Download via Z39.50 2 authorities, one of which is a narrower heading of the other, e.g.:
Waterworks
Waterworks - Costs
3 - Place a heading for the broader term in a record. e.g. Waterworks
In 650$a, without the cataloguing authority plugin. We don't want
the link created now.
You need syspref BiblioAddsAuthorities => allow
4 - Make sure linker is set to default
5 - Attempt to link the records
misc/link_bibs_to_authorities.pl
6 - Linking fails
7 - Apply patch
8 - refresh index settings (if using a custom file, remove 'match-heading')
You can reset mappings in the UI or run this:
misc/search_tools/rebuild_elasticsearch.pl -v -d -r
9 - Reindex ES
10 - Try to link again
11 - It succeeds!
12 - Run the tests
prove t/db_dependent/Koha/SearchEngine/Elasticsearch.t
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Bug 25273: (follow-up)
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Remove unused Elasticsearch parameters 'key_prefix' and 'request_timeout'.
Refactor so that get_elasticsearch_params only returns parameters used
by Search::Elasticsearch, add class accessor for index_name.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Replace remaining Catmandu dependant code with the Search::Elasticsearch
equivalent.
To test:
1) Apply patch
2) Run tests in t/Koha/SearchEngine/Elasticsearch.t, t/Koha/SearchEngine/ElasticSearch/*
and t/db_dependent/Koha/SearchEngine/*
3) All tests should pass
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Run test:
prove t/db_dependent/Koha/SearchEngine/Elasticsearch/ExportConfig.t
Or:
Run export_elasticsearch_mappings.pl script. See that opac/staff_client fields
are exorted.
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
There was an assumption in the ES code that match-heading mappings will appear in
a specified portion of the mappings array.
Certain mappings setups will not meet this assumption.
We need to move our searching up one level
The key seems to be having a mapping for a complete field, say 150, in both the
match-heading and another field as well as having mappings for ungrouped fields like
150a 150ab etc.
The unit test coverage should be sufficient for testing
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Test plan :
Launching script several times and see that marc field is alway ordered increasing
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Nick Clemens
Janusz Kaczmarek
Julian Maurice
Jonathan Druart
Tomas Cohen Arazi
Petro Vashchuk
David Gustafsson
David Gustafsson
Frédéric Demians
Fridolin Somers
Alex Arnaud