Subdivision authorities are not used for linking, however, they are recognized by
C4::AuthoritiesMarc
While these records are not used for linking, they could provide reference and
should be allowed to exist in the catalog without breaking ES indexing
THis patch simply skips the step of parsing the authorities into the linking form
if the type contains '_SUBD'
To test:
1 - Import a subdivision authority record via Z39 or use the one attached to this bug
2 - perl misc/search_tools/rebuild_elasticsearch.pl -v -d
3 - Authority indexing dies:
Use of uninitialized value $tag in hash element at /usr/share/perl5/MARC/Record.pm line 202.
Use of uninitialized value $tag in regexp compilation at /usr/share/perl5/MARC/Record.pm line 206.
Use of uninitialized value $tag in hash element at /usr/share/perl5/MARC/Record.pm line 207.
Can't call method "tag" on an undefined value at /kohadevbox/koha/C4/Heading.pm line 71.
4 - Apply patches
5 - reindex
6 - Success!
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit d67fc5e010)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit 8512ccb446)
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
(cherry picked from commit 13cdda5ae9)
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
We are guessing authtype code and inserting the heading built accorindg to C4::Heading
If we can't identify the auth type, we can format the heading. There is a record in the koha test
data that is missing the heading field so type cannot be idenfitied
This prevents us from dying n a record where we cannot identify the type.
Note: This code will also be triggered for custom authority types, higlighting
that they won't link because of hardcoded mappings in C4::Heading. We must tackle this
on a new bug
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit 5a87514b2e)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit 401133b5b1)
Signed-off-by: Aleisha Amohia <aleishaamohia@hotmail.com>
The match-heading field is a special field used only by the linker, not accessible
to staff or patrons via the interface. This field is used to store the constructed
'search form' used for matching bib headings to authority fields.
In bug 24269 I attempted to use the mappings defined in the inferface and also inject the search term.
This did not work as too many subfields were indexed on their own and leading to false matches.
In this bug we remove the mappings for this field, and create it ourselves during
the indexing process. The C4::Headings module is still used to generate the correct form,
however, the mappings are set based on the authority types in the system. This gives the user
the ability to add new typoes, but prevents mapping changes from breaking linker functionality
To test:
1 - Start form a sample database with ElasticSearch working
2 - Download via Z39.50 2 authorities, one of which is a narrower heading of the other, e.g.:
Waterworks
Waterworks - Costs
3 - Place a heading for the broader term in a record. e.g. Waterworks
In 650$a, without the cataloguing authority plugin. We don't want
the link created now.
You need syspref BiblioAddsAuthorities => allow
4 - Make sure linker is set to default
5 - Attempt to link the records
misc/link_bibs_to_authorities.pl
6 - Linking fails
7 - Apply patch
8 - refresh index settings (if using a custom file, remove 'match-heading')
You can reset mappings in the UI or run this:
misc/search_tools/rebuild_elasticsearch.pl -v -d -r
9 - Reindex ES
10 - Try to link again
11 - It succeeds!
12 - Run the tests
prove t/db_dependent/Koha/SearchEngine/Elasticsearch.t
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Bug 25273: (follow-up)
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(cherry picked from commit ce161fda9b)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit 5597b666e4)
Signed-off-by: Aleisha Amohia <aleishaamohia@hotmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Joy Nelson <joy@bywatersolutions.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Joy Nelson <joy@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Joy Nelson <joy@bywatersolutions.com>
There was an assumption in the ES code that match-heading mappings will appear in
a specified portion of the mappings array.
Certain mappings setups will not meet this assumption.
We need to move our searching up one level
The key seems to be having a mapping for a complete field, say 150, in both the
match-heading and another field as well as having mappings for ungrouped fields like
150a 150ab etc.
The unit test coverage should be sufficient for testing
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Joy Nelson <joy@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Joy Nelson <joy@bywatersolutions.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Joy Nelson <joy@bywatersolutions.com>
Strip initial characters from search fields in accordance with
nonfiling character indicators.
To test:
1) Apply patch
2) Run tests in t/Koha/SearchEngine/Elasticsearch.t
3) All tests should pass
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Joy Nelson <joy@bywatersolutions.com>
There is no need to specify the `client` param here because
'5_0::Direct' is the default value for Search::Elasticsearch 5.x and it
breaks when Search::Elasticsearch 6.x is installed
Test plan:
1. Apply the patch
2. Make sure you have Search::Elasticsearch 5.x
3. Reload starman
4. Verify that search works
5. Install Search::Elasticsearch 6.x
6. Reload starman
7. Verify that search works
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Joy Nelson <joy@bywatersolutions.com>
To test:
1 - Be using Elasticsearch
2 - Check that field 150 is mapped to 'Match-heading' or add it (the subfields don't matter)
3 - Add a topic term authority record like "150 $aCats$vFiction"
4 - Add a 650 with $aCats and $vFiction and $e depicted to a bibliographic record
5 - Run the linker for the bib
perl misc/link_bibs_to_authorities.pl --bib-limit "biblionumber=89" -v
6 - Confirm the record is not correctly linked to the record
7 - Apply patch
8 - Reindex authorities for ES
perl misc/search_tools/rebuild_elasticsearch.pl -v -d -a
9 - Run the linker and confirm record is correctly linked
perl misc/link_bibs_to_authorities.pl --bib-limit "biblionumber=89" -v
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Joy Nelson <joy@bywatersolutions.com>
When executing
Koha::SearchEngine::Elasticsearch->reset_elasticsearch_mappings()
against populated search engine mappings tables, one gets this
exception:
DBD::mysql::st execute failed: Duplicate entry '388-73' for key
'PRIMARY' [for Statement \"INSERT INTO `search_marc_to_field` ( `facet`,
`search_field_id`, `search_marc_map_id`, `sort`, `suggestible`) VALUES (
?, ?, ?, ?, ? )\" with ParamValues: 0='', 1='73', 2='388', 3=undef,
4=''] at /usr/share/perl5/DBIx/Class/Storage/DBI.pm line
1834.\nDBIx::Class::Storage::DBI::_dbh_execute(): Duplicate entry
'388-73' for key 'PRIMARY' at /home/koha/Koha/Koha/SearchField.pm line
38"
This patch fixes this, by first deleting all entries, then adding the
default ones.
If a subroutine says reset_elasticsearch_mappings(), I expect it to
reset the elasticsearch mappings without errors. When writing tests it
is better to call that one subroutine reset_elasticsearch_mappings
instead of the bunch of code in
admin/searchengine/elasticsearch/mappings.pl?op=reset_confirmed
Also this promotes code reuse.
TEST PLAN:
a. perl -e 'use Koha::SearchEngine::Elasticsearch; Koha::SearchEngine::Elasticsearch->reset_elasticsearch_mappings'
b. Run misc/devel/populate_db.pl on an already populated DB and you eventually
face this problem.
After this patch, step a. works.
This patch is adapted from the original patch made by Olli-Antti
Kivilahti <olli-antti.kivilahti@jns.fi>.
Signed-off-by: Johanna Raisa <johanna.raisa@gmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Joy Nelson <joy@bywatersolutions.com>
As of bug 20589 we no longer analyze sort fields and so we no longer need to append ".phrase"
to our sort in searches.
Additionally, sort fields based on 'sum' should also use sum in building the value to sort on
To test:
0 - Be using ES
1 - Find the most circulated item in your collection
2 - Search for '*'
3 - Sort by popularity DESC
4 - Note that item is not first
5 - Try to sort by anything but relevancy, it fails
6 - Apply patch
7 - Redo searches and sorts
8 - Things should now work as expected
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Elasticsearch indexing uses 999$c to store record id by deleting the all field first !
So you can not store anything in field 999, even in UNIMARC and even in authorities records.
Looks like it is quick fix code added to start Elasticsearch use.
This behavior is disturbing and very strange for UNIMARC flavour.
This patch corrects by defining record ids mandatory in Koha::SearchEngine::Elasticsearch::Indexer::update_index().
This ids array is actually always given (except in UT).
I think it is useless to allow adding a record without its id.
Test plan :
1) Use Elasticsearch as SearchEngine
2) Create a subfield 999$z in default framework
3) Create a record with default framework
4) Enter a random string (never used in catalog) like "tototata" in 999$z
5) In Search engine configuration, define search field "subject" for 999$z
6) Rebuild record : misc/search_tools/rebuild_elasticsearch.pl -b -bn <biblionumber> -v
7) Search for the random string => You get a result
8) Optionnaly look at records in ES : <es server>:9200/<es index name>/data/<biblionumber>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Prevent Elasticsearch from throwing an exception when the mappings
include uppercase subfield identifier.
Though forbidden in MARC21, It doesn't seem to be the case for Unimarc
or ISO 2709 as a whole. It is also possible to define such a subfield
identifier in the Koha MARC Framework and in the Elasticsearch
mappings.
Test plan:
1. add a 995Z subfield identifier in Elasticsearch mappings
(Administration -> Search engine configuration)
2. run $KOHA_PATH/misc/search_tools/rebuild_elasticsearch.pl -v -d -b
3. see it throw: “Invalid MARC field expression: 995Z”
4. apply patch
5. rerun $KOHA_PATH/misc/search_tools/rebuild_elasticsearch.pl -v -d -b
6. sababa
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Prevent Koha::SearchEngine::Elasticsearch from dying when trying to run
Koha::Exceptions::Elasticsearch::MARCFieldExprParseError->throw()
Plan test:
1. Go to Administration -> Searchengine configuration, and add a mapping
for 995Z, whatever the search field (I did it with local-classification),
save
2. run $KOHA_PATH/misc/search_tools/rebuild_elasticsearch.pl -v -d -b
3. see it die asking for Koha::Exceptions::Elasticsearch
4. apply patch
5. rerun $KOHA_PATH/misc/search_tools/rebuild_elasticsearch.pl -v -d -b
6. see it throw: Invalid MARC field expression: 995Z
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Generate a list of fields for the query_string query fields parameter,
with possible boosts, instead of using "_all"-field. Also add "search"
flag in search_marc_to_field table so that certain mappings can be
excluded from searches. Add option to include/exclude fields in
query_string "fields" parameter depending on searching in OPAC or staff
client. Refactor code to remove all other dependencies on "_all"-field.
How to test:
1) Reindex authorities and biblios.
2) Search biblios and try to verify that this works as expected.
3) Search authorities and try to verify that this works as expected.
4) Go to "Search engine configuration"
5) Change some "Boost", "Staff client", and "OPAC" settings and save.
6) Verify that those settings where saved accordingly.
7) Click the "Biblios" or "Authorities" tab and change one or more
"Searchable" settings
8) Verfiy that those settings where saved accordingly.
9) Try to verify that these settings has taken effect by peforming
some biblios and/or authorities searches.
Sponsorded-by: Gothenburg Univesity Library
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Adds preference ElasticsearchMARCFormat that controls whether MARC records are stored as ISO2709/MARCXML or array. Array is searchable by field and also indexes all subfields in the _all field for searching.
Test plan:
1. Test that searching and indexing works with the patch without any changes.
2. Switch to array format and index some records.
3. Check e.g. the 008 field of a record and verify that the record can be found with the contents enclosed in quotes.
4. Check that it's possible to search for a specific field/subfield. Search query: marc_data_array.fields.655.subfields.a:Diaries
5. Check that tests still pass, especially t/Koha/SearchEngine/Elasticsearch.t
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Test plan:
- apply this patch,
- edit admin/searchengine/elasticsearch/mappings.yaml
to add weight for some fields,
- go to admin > search engine configuration,
- reset your mappings (add ?op=reset&i_know_what_i_am_doing=1 to url),
- check that weights you've added are set
Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
'Static' is the default value of Search::Elasticsearch and for a good
reason : it works in most cases, unlike the 'Sniff' option
To test:
1 - Apply patch
2 - Edit koha-conf.xml
3 - Add '<cxn_pool>Static</cxn_pool>' to the elasticsearch stanza
4 - Restart all the things!
5 - Reindex ES, it works
6 - Set SearchEngine to ES, try searching
7 - It works!
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Fixes handling of /0 selector in addition to several fixed field positions. Note that ff7-01-02 is in fact 00-01 in Zebra and that has been replicated here.
Test plan:
1. Before applying a patch, check from Elasticsearch (e.g. localhost:9200/koha_biblios/_search?q=_id:123) what is indexed in Elasticsearch for a record for date-entered-on-file, ff7-00, ff7-01, ff7-02 and lleader.
2. Apply the patch, update database and save the record again.
3. Verify that the contents of the forementioned fields is now correct in Elasticsearch.
4. Verify that tests pass: prove -v t/Koha/SearchEngine/Elasticsearch.t
Signed-off-by: Björn Nylén <bjorn.nylen@ub.lu.se>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Test plain:
1. Reindex both authorities and biblios with the -d flag to recreate the index, e.g. "perl misc/search_tools/rebuild_elastic_search.pl -v -d"
2. Check that authorities only contains authority-specific fields in Elasticsearch e.g. by fetching http://localhost:9200/koha_dev_authorities/_mappings
3. Reindex both without the -d flag and verify it works, e.g. "perl misc/search_tools/rebuild_elastic_search.pl -v"
Signed-off-by: Björn Nylén <bjorn.nylen@ub.lu.se>
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
This patch add language as a facet to ES results - it adds
a new template plugin for languages to get the appropriate
description given an iso 639-2 code
To test:
1 - Make sure you have records with differing languages (in the MARC21 008
field characters 35-37 or UNIMARC 101a)
2 - Apply patch
3 - Reload Elasticsearch settings:
http://localhost:8081/cgi-bin/koha/admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
4 - Reindex your records
5 - Search for a phrase that will return results in several languages
6 - Verify you see factes correctly labelled for 'Language'
7 - Verify the facets work
8 - Verify both opac and staff results
9 - prove t/db_dependent/Languages.t
Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Without this we could not recreate mappings as there are no defaults for
the additional fields - this restores behavoiur from before patches
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
This patch adds a new section 'Facet order' in the Biblio tab of the
'Search engine configuration' admin page of the Elastic mappings.
The idea is to let the librarians define the facet to display and order
them as their needs.
The ergonomic is not perfect and I am open to any suggestions.
Test plan:
Move up and down the field list to order the facets
Hide/show some facets
Rebuild index
At the OPAC and the staff interface you should see the changes on the
search result page.
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Default settings are appended in get_elasticsearch_params so that there's a central place for them. Defaults can be overridden in koha-conf.xml.
Test plan:
1. Verify that searches still work with the patch applied.
2. Verify that search fails after adding the following under <elasticsearch> in koha-conf.xml:
<cxn_pool>Static</cxn_pool>
3. Verify that search fails after adding the following under <elasticsearch> in koha-conf.xml:
<cxn_pool>FOO</cxn_pool>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Deduplicate multivalued fields and make sure sort fields are not excessively long. Also updates default mappings so that sort fields are not created for item fields where it doesn't make sense.
Test plan:
1. Reset ES mappings in administration
2. Check that sort is '0' for local-classification in biblio mappings.
3. Change sort back to '1' for local-classification for the next steps.
4. Create a record with 20 items, each with a 100 character long call number
5. Check that when indexed, the record in ES does not have duplicates in any of the item fields and local-classification__sort is truncated to 255 characters.
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Adjust elastic search mappings to more closely match Zebra equivalents
resolving serveral issues with coded Zebra searches in templates and
sorting of search results in UI. Also make field names in search strings
case insensitive to accept case variations in template links and user input.
Sponsored-by: Gothenburg University Library
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
https://bugs.koha-community.org/show_bug.cgi?id=20244
Test plan:
1. Add a record with alternate script fields in 880 (sample attached in the bug).
2. Make sure the the record can be found e.g. with the alternate script title.
3. Add a record with multiple authors.
4. Check that in the index there is only a single author__sort field.
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
https://bugs.koha-community.org/show_bug.cgi?id=20244
Test plan:
1. Add a record with an ISBN-10 or ISBN-13 that can be converted to ISBN-10 (e.g. 1-56619-909-3).
2. Verify that the record can be found by searching for "1-56619-909-3", "1566199093", "978-1-56619-909-4" and "9781566199094".
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Sponsored-by: Gothenburg University Library
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Add missing pods, remove obsolete syspref and add test for serialization format for records exceeding max record size
Sponsored-by: Gothenburg University Library
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Sponsored-by: Gothenburg University Library
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Default to base64 encoded binary MARC with MARCXML
fallback if record exceeds maximum size
Sponsored-by: Gothenburg University Library
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Implement optimized indexing for Elasticsearch
How to test:
1) Time a full elasticsearch re-index without this patch by running the
rebuild_elastic_search.pl with the -d flag:
`koha-shell <instance_name> -c "time rebuild_elastic_search.pl -d"`.
2) Apply this patch.
3) Time a full re-index again, it should be about twice at fast (for a
couple of thousand biblios, with fewer biblios results may be more
unpredictable).
Sponsored-by: Gothenburg University Library
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
"return" statement with explicit "undef" at line 245, column 5. See
page 199 of PBP. (Severity: 5)
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Improvements:
1) Index settings moved from code to etc/searchengine/elasticsearch/index_config.yaml. An alternative can be specified in koha-conf.xml.
2) Field settings moved from code to etc/searchengine/elasticsearch/field_config.yaml. An alternative can be specified in koha-conf.xml.
3) mappings.yaml has been moved from admin/searchengine/elasticsearch to etc/searchengine/elasticsearch. An alternative can be specified in koha-conf.xml.
4) Default settings have been improved to remove punctuation from phrases used for sorting etc.
5) State variables are used for storing configuration to avoid parsing it multiple times.
6) A possibility to reset the fields too has been added to the reset operation of mappings administration.
7) mappings.yaml has been moved from admin/searchengine/elasticsearch to etc/searchengine/elasticsearch.
8) An stdno field type has been added for standard identifiers.
To test:
1) Run tests in t/Koha/SearchEngine/Elasticsearch.t
2) Clear tables search_fields and search_marc_map
3) Go to admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
4) Verify that admin/searchengine/elasticsearch/mappings.pl displays the mappings properly, including ISBN and other standard number fields.
5) Index some records using the -d parameter with misc/search_tools/rebuild_elastic_search.pl to recreate the index
6) Verify that you can find the records
7) Put <elasticsearch_index_mappings>non_existent</elasticsearch_index_mappings> to koha-conf.xml
8) Verify that admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1 fails because it can't find non_existent.
9) Copy etc/searchengine/elasticsearch/mappings.yaml to a new location and make elasticsearch_index_mappings setting in koha-conf.xml point to it.
10) Make a change in the new mappings.yaml.
11) Clear table search_fields (mappings reset doesn't do it yet, see bug 20248)
12) Go to admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
13) Verify that the changes you made are now visible in the mappings UI
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Bug 20073: Move Elasticsearch yaml files back to admin directory
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
This reverts commit f489d2034b.
This commit breaks the install process when using debian packages.
Reverting as we are very close to the 18.05.00 release
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
"return" statement with explicit "undef" at line 245, column 5. See
page 199 of PBP. (Severity: 5)
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Improvements:
1) Index settings moved from code to etc/searchengine/elasticsearch/index_config.yaml. An alternative can be specified in koha-conf.xml.
2) Field settings moved from code to etc/searchengine/elasticsearch/field_config.yaml. An alternative can be specified in koha-conf.xml.
3) mappings.yaml has been moved from admin/searchengine/elasticsearch to etc/searchengine/elasticsearch. An alternative can be specified in koha-conf.xml.
4) Default settings have been improved to remove punctuation from phrases used for sorting etc.
5) State variables are used for storing configuration to avoid parsing it multiple times.
6) A possibility to reset the fields too has been added to the reset operation of mappings administration.
7) mappings.yaml has been moved from admin/searchengine/elasticsearch to etc/searchengine/elasticsearch.
8) An stdno field type has been added for standard identifiers.
To test:
1) Run tests in t/Koha/SearchEngine/Elasticsearch.t
2) Clear tables search_fields and search_marc_map
3) Go to admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
4) Verify that admin/searchengine/elasticsearch/mappings.pl displays the mappings properly, including ISBN and other standard number fields.
5) Index some records using the -d parameter with misc/search_tools/rebuild_elastic_search.pl to recreate the index
6) Verify that you can find the records
7) Put <elasticsearch_index_mappings>non_existent</elasticsearch_index_mappings> to koha-conf.xml
8) Verify that admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1 fails because it can't find non_existent.
9) Copy etc/searchengine/elasticsearch/mappings.yaml to a new location and make elasticsearch_index_mappings setting in koha-conf.xml point to it.
10) Make a change in the new mappings.yaml.
11) Clear table search_fields (mappings reset doesn't do it yet, see bug 20248)
12) Go to admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
13) Verify that the changes you made are now visible in the mappings UI
Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Nick Clemens
Jonathan Druart
David Gustafsson
Julian Maurice
Joonas Kylmälä
Fridolin Somers
Nicolas Legrand
Ere Maijala
Alex Arnaud
Josef Moravec
Martin Renvoize