Commit Graph

36 Commits

Author SHA1 Message Date
Nick 2b769cbc32 Bug 23089: Fix sorting on sum and regular fields
As of bug 20589 we no longer analyze sort fields and so we no longer need to append ".phrase"
to our sort in searches.

Additionally, sort fields based on 'sum' should also use sum in building the value to sort on

To test:
0 - Be using ES
1 - Find the most circulated item in your collection
2 - Search for '*'
3 - Sort by popularity DESC
4 - Note that item is not first
5 - Try to sort by anything but relevancy, it fails
6 - Apply patch
7 - Redo searches and sorts
8 - Things should now work as expected

Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
(cherry picked from commit 6a2ba8b6b2)
Signed-off-by: Hayley Mapley <hayleymapley@catalyst.net.nz>
2019-12-18 14:22:38 +13:00
Nick Clemens 340e2da40b Bug 23089: [19.05 and earlier] Use integer field directly for sorting
(cherry picked from commit 55ccf90711)
2019-12-18 09:00:13 +13:00
Julian Maurice 022b696057
Bug 22705: Change default value of cxn_pool to 'Static'
'Static' is the default value of Search::Elasticsearch and for a good
reason : it works in most cases, unlike the 'Sniff' option

To test:
1 - Apply patch
2 - Edit koha-conf.xml
3 - Add '<cxn_pool>Static</cxn_pool>' to the elasticsearch stanza
4 - Restart all the things!
5 - Reindex ES, it works
6 - Set SearchEngine to ES, try searching
7 - It works!

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
(cherry picked from commit 9532a3edea)
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-05-30 20:41:28 +01:00
Ere Maijala b1290c0602
Bug 22339: Fix search field mappings of MARC fixed fields
Fixes handling of /0 selector in addition to several fixed field positions. Note that ff7-01-02 is in fact 00-01 in Zebra and that has been replicated here.

Test plan:
1. Before applying a patch, check from Elasticsearch (e.g. localhost:9200/koha_biblios/_search?q=_id:123) what is indexed in Elasticsearch for a record for date-entered-on-file, ff7-00, ff7-01, ff7-02 and lleader.
2. Apply the patch, update database and save the record again.
3. Verify that the contents of the forementioned fields is now correct in Elasticsearch.
4. Verify that tests pass:  prove -v t/Koha/SearchEngine/Elasticsearch.t

Signed-off-by: Björn Nylén <bjorn.nylen@ub.lu.se>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
(cherry picked from commit 0047ede37e)
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-04-26 16:20:48 +01:00
Ere Maijala 16846985e1
Bug 22474: Clone field config before adding fields to it
Test plain:

1. Reindex both authorities and biblios with the -d flag to recreate the index, e.g. "perl misc/search_tools/rebuild_elastic_search.pl -v -d"
2. Check that authorities only contains authority-specific fields in Elasticsearch e.g. by fetching  http://localhost:9200/koha_dev_authorities/_mappings
3. Reindex both without the -d flag and verify it works, e.g. "perl misc/search_tools/rebuild_elastic_search.pl -v"

Signed-off-by: Björn Nylén <bjorn.nylen@ub.lu.se>
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
(cherry picked from commit 118f685dbf)
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-04-25 15:04:38 +01:00
Ere Maijala 9ef8a3e328
Bug 21974: Make Elasticsearch connection settings configurable
Default settings are appended in get_elasticsearch_params so that there's a central place for them. Defaults can be overridden in koha-conf.xml.

Test plan:

1. Verify that searches still work with the patch applied.
2. Verify that search fails after adding the following under <elasticsearch> in koha-conf.xml:
<cxn_pool>Static</cxn_pool>
3. Verify that search fails after adding the following under <elasticsearch> in koha-conf.xml:
<cxn_pool>FOO</cxn_pool>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
(cherry picked from commit 65f35447e9)
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-04-01 11:42:17 +01:00
Ere Maijala 7d741a6991
Bug 22246: Fix indexing of large fields with Elasticsearch
Deduplicate multivalued fields and make sure sort fields are not excessively long. Also updates default mappings so that sort fields are not created for item fields where it doesn't make sense.

Test plan:
1. Reset ES mappings in administration
2. Check that sort is '0' for local-classification in biblio mappings.
3. Change sort back to '1' for local-classification for the next steps.
4. Create a record with 20 items, each with a 100 character long call number
5. Check that when indexed, the record in ES does not have duplicates in any of the item fields and local-classification__sort is truncated to 255 characters.

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
(cherry picked from commit bd68e2f75b)
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-03-12 08:46:52 +00:00
David Gustafsson 68b129e0c4
Bug 19575: Use canonical field names and resolve aliased fields
Adjust elastic search mappings to more closely match Zebra equivalents
resolving serveral issues with coded Zebra searches in templates and
sorting of search results in UI. Also make field names in search strings
case insensitive to accept case variations in template links and user input.

Sponsored-by: Gothenburg University Library

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
(cherry picked from commit 6d53910f0c)
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
2019-03-05 14:50:59 +00:00
Nick Clemens 6ef2233093 Bug 20244: Pod fixes
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-16 12:08:39 +00:00
Ere Maijala aece4f62ea Bug 20244: Add alt script indexing and fix sort field indexing
https://bugs.koha-community.org/show_bug.cgi?id=20244

Test plan:

1. Add a record with alternate script fields in 880 (sample attached in the bug).
2. Make sure the the record can be found e.g. with the alternate script title.
3. Add a record with multiple authors.
4. Check that in the index there is only a single author__sort field.

Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-16 11:27:22 +00:00
Ere Maijala 5a0f2821a2 Bug 20244: Improve Elasticsearch ISBN indexing
https://bugs.koha-community.org/show_bug.cgi?id=20244

Test plan:

1. Add a record with an ISBN-10 or ISBN-13 that can be converted to ISBN-10 (e.g. 1-56619-909-3).
2. Verify that the record can be found by searching for "1-56619-909-3", "1566199093", "978-1-56619-909-4" and "9781566199094".

Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-16 11:27:22 +00:00
Martin Renvoize da22f3cc33 Bug 19893: (QA follow-up) Spelling correction in POD
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-16 11:05:00 +00:00
David Gustafsson 6d47d82ac9 Bug 19893: Add code review fixes
Sponsored-by: Gothenburg University Library
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-16 11:04:59 +00:00
David Gustafsson 1d29d0e449 Bug 19893: Add pods, remove syspref, add tests for serialization format
Add missing pods, remove obsolete syspref and add test for serialization format for records exceeding max record size

Sponsored-by: Gothenburg University Library
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-16 11:04:59 +00:00
David Gustafsson d22b6392de Bug 19893: Support for joined subfields in mappings
Sponsored-by: Gothenburg University Library
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-16 11:04:58 +00:00
David Gustafsson fc0abee558 Bug 19893: Remove serialization format setting
Default to base64 encoded binary MARC with MARCXML
fallback if record exceeds maximum size

Sponsored-by: Gothenburg University Library
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-16 11:04:57 +00:00
David Gustafsson 5389ffa38a Bug 19893: Alternative optimized indexing for Elasticsearch
Implement optimized indexing for Elasticsearch

How to test:
1) Time a full elasticsearch re-index without this patch by running the
   rebuild_elastic_search.pl with the -d flag:
   `koha-shell <instance_name> -c "time rebuild_elastic_search.pl -d"`.
2) Apply this patch.
3) Time a full re-index again, it should be about twice at fast (for a
   couple of thousand biblios, with fewer biblios results may be more
   unpredictable).

Sponsored-by: Gothenburg University Library
Signed-off-by: Ere Maijala <ere.maijala@helsinki.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-11-16 11:04:56 +00:00
Jonathan Druart e0f6450ae9 Bug 20073: Remove perlcritic error
"return" statement with explicit "undef" at line 245, column 5.  See
page 199 of PBP.  (Severity: 5)

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-06-22 14:38:02 +00:00
Ere Maijala 133df5e912 Bug 20073: Move Elasticsearch configs to yaml files and improve the default settings.
Improvements:
1) Index settings moved from code to etc/searchengine/elasticsearch/index_config.yaml. An alternative can be specified in koha-conf.xml.
2) Field settings moved from code to etc/searchengine/elasticsearch/field_config.yaml. An alternative can be specified in koha-conf.xml.
3) mappings.yaml has been moved from admin/searchengine/elasticsearch to etc/searchengine/elasticsearch. An alternative can be specified in koha-conf.xml.
4) Default settings have been improved to remove punctuation from phrases used for sorting etc.
5) State variables are used for storing configuration to avoid parsing it multiple times.
6) A possibility to reset the fields too has been added to the reset operation of mappings administration.
7) mappings.yaml has been moved from admin/searchengine/elasticsearch to etc/searchengine/elasticsearch.
8) An stdno field type has been added for standard identifiers.

To test:
1) Run tests in t/Koha/SearchEngine/Elasticsearch.t
2) Clear tables search_fields and search_marc_map
3) Go to admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
4) Verify that admin/searchengine/elasticsearch/mappings.pl displays the mappings properly, including ISBN and other standard number fields.
5) Index some records using the -d parameter with misc/search_tools/rebuild_elastic_search.pl to recreate the index
6) Verify that you can find the records
7) Put <elasticsearch_index_mappings>non_existent</elasticsearch_index_mappings> to koha-conf.xml
8) Verify that admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1 fails because it can't find non_existent.
9) Copy etc/searchengine/elasticsearch/mappings.yaml to a new location and make elasticsearch_index_mappings setting in koha-conf.xml point to it.
10) Make a change in the new mappings.yaml.
11) Clear table search_fields (mappings reset doesn't do it yet, see bug 20248)
12) Go to admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
13) Verify that the changes you made are now visible in the mappings UI

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Bug 20073: Move Elasticsearch yaml files back to admin directory

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
2018-06-22 14:37:56 +00:00
Jonathan Druart cff62796ea Revert "Bug 20073: Move Elasticsearch configs to yaml files and improve the default settings."
This reverts commit f489d2034b.

This commit breaks the install process when using debian packages.
Reverting as we are very close to the 18.05.00 release

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2018-05-21 11:02:16 -03:00
Jonathan Druart e1a95094e7 Revert "Bug 20073: Remove perlcritic error"
This reverts commit 67359a97f4.

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2018-05-21 11:01:58 -03:00
Jonathan Druart 67359a97f4 Bug 20073: Remove perlcritic error
"return" statement with explicit "undef" at line 245, column 5.  See
page 199 of PBP.  (Severity: 5)

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2018-04-23 15:12:48 -03:00
Ere Maijala f489d2034b Bug 20073: Move Elasticsearch configs to yaml files and improve the default settings.
Improvements:
1) Index settings moved from code to etc/searchengine/elasticsearch/index_config.yaml. An alternative can be specified in koha-conf.xml.
2) Field settings moved from code to etc/searchengine/elasticsearch/field_config.yaml. An alternative can be specified in koha-conf.xml.
3) mappings.yaml has been moved from admin/searchengine/elasticsearch to etc/searchengine/elasticsearch. An alternative can be specified in koha-conf.xml.
4) Default settings have been improved to remove punctuation from phrases used for sorting etc.
5) State variables are used for storing configuration to avoid parsing it multiple times.
6) A possibility to reset the fields too has been added to the reset operation of mappings administration.
7) mappings.yaml has been moved from admin/searchengine/elasticsearch to etc/searchengine/elasticsearch.
8) An stdno field type has been added for standard identifiers.

To test:
1) Run tests in t/Koha/SearchEngine/Elasticsearch.t
2) Clear tables search_fields and search_marc_map
3) Go to admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
4) Verify that admin/searchengine/elasticsearch/mappings.pl displays the mappings properly, including ISBN and other standard number fields.
5) Index some records using the -d parameter with misc/search_tools/rebuild_elastic_search.pl to recreate the index
6) Verify that you can find the records
7) Put <elasticsearch_index_mappings>non_existent</elasticsearch_index_mappings> to koha-conf.xml
8) Verify that admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1 fails because it can't find non_existent.
9) Copy etc/searchengine/elasticsearch/mappings.yaml to a new location and make elasticsearch_index_mappings setting in koha-conf.xml point to it.
10) Make a change in the new mappings.yaml.
11) Clear table search_fields (mappings reset doesn't do it yet, see bug 20248)
12) Go to admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
13) Verify that the changes you made are now visible in the mappings UI

Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2018-04-23 14:22:16 -03:00
Nick Clemens 6dfff20145 Bug 19581: Remove unused comment and line
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2018-04-09 16:15:18 -03:00
Nick Clemens ec0ea67a43 Bug 19581: Remove 'split' option from Catmandu fixes for marc
This option in combination with append seems to add null entries in the indexes
for fields that don't exist. These fields then affect sorting. Removing it solves
the issue

To test:
1 - Have an ES setup
2 - Catalog and index some records
3 - View a record in ES:
    curl 'localhost:9200/koha_kohadev_biblios/data/4?pretty=yes'
4 - Note there are 'null' entries in the indexes, save this result for
comparison
5 - Apply patch
6 - Reindex, amking sure to delete existing indexes
    perl misc/search_tools/rebuild_elastic_search.pl -d
7 - View the record again
8 - Note the null entries are removed
9 - Note indices are otherwise intact

Signed-off-by: David Bourgault <david.bourgault@inlibro.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2018-04-09 16:15:17 -03:00
Tomas Cohen Arazi f57c3b36ea Bug 19542: Add a check for ES configuration health
This patch adds a new statuc function to Koha::SearchEngine::ElasticSearch
which is instended to replace most of get_elasticsearch_params. This function
reads the configuration from C4::Context->config('elasticsearch') and raises
relevant exceptions when mandatory entries are missing.

Its behaviour is covered by tests.

To test:
- Run:
  $ kshell
 k$ prove t/Koha/SearchEngine/Elasticsearch.t
=> SUCCESS: Tests pass!
- Sign off :-D

Signed-off-by: David Bourgault <david.bourgault@inlibro.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-12-11 14:30:42 -03:00
Nick Clemens e1ba8d344d Bug 19563: (follow-up) Restore checking sort variable
This code seems a bit odd, $self->sort_fields()->{$name} is checking if
there is any mapping in ES for a field, so if one marc field in an index
is marked as sortable this will exists. We need to check the individual
marc field to see if it should be added to the sort index here.

If you apply the first patch, reindex, and view a record in es:
curl -XGET 'localhost:9200/koha_kohadev_biblios/data/19/?pretty' | grep
-A 10 author__sort

You will see that 245$c is included in the record without that
additional check

Apply this, reindex again, and that field should not be included

Signed-off-by: David Bourgault <david.bourgault@inlibro.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-12-11 14:30:32 -03:00
David Gustafsson 963eaeab7b Bug 19563: Generation of sort_fields uses incorrect condition
Fix incorrect condition for if clause for generating <field>__sort mappings
for Elasticsearch. Also remove redundant check for same condition when
generating fixer rules.

Test plan:
1. Inspect current mappings for example by viewing: http://<elasticsearch_host>:9200/koha_<koha_instance_name>_biblios/_mapping.
2. If using the default configuraion only "author" has a sort field (author__sort).
4. Appy patch.
5. Reindex using rebuild_elastic_search.pl.
6. All fields except those with sort sort set to "0" should now have sort fields, which in the default configuration is all but "author".

Signed-off-by: David Bourgault <david.bourgault@inlibro.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-12-11 13:59:13 -03:00
Nick Clemens 1404654d65 Bug 18318: Unicode support for Elasticsearch
You must install the icu plugin for elasticsearch
https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-icu.html

Once installed, apply this patch
Reindex your data, deleting the existing indexes
perl /home/vagrant/kohaclone/misc/search_tools/rebuild_elastic_search.pl
-d
Find (or add) some titles with accented characters
Verify that a search for the exact character or the unaccented version
works

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-09-29 12:37:00 -03:00
Nick Clemens 241be8ff1a Bug 18434 - Followup - same changes for sort and facet fields
To test:
1 - Index some stuff with multiple fields defined for sorting
  i.e. Authorites - make heading sortable - default is 110a and 111a for
  heading - a record with 111a empty will make the sort field empty
2 - view the record:
curl http://localhost:9200/koha_kohadev_authorities/data/30?pretty=true
3 - Note the blank field
4 - Apply patch
5 - Reindex
6 - Fields should be correctly populated

Unit tests to follow (once I have the originals working for all)

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-07-06 14:29:02 -03:00
Nick Clemens 9eb88f588b Bug 18434: Elasticsearch indexing broken with newer catmandu version
To test:
1 - Make sure you have latest koha deps, catmandu versions should be:
    libcatmandu-marc-perl   1.09-1~kohadev1
    libcatmandu-perl        1.0304-2~kohadev1
2 - Reindex elastic
3 - Try searching and likely notice odd results
4 - Try:
curl -XGET
'http://localhost:9200/koha_kohadev_biblios/data/792?pretty=true'
with a known biblionumber and notice some null fields
5 - Apply patch
6 - Reindex
7 - Note fields are populated and search works as expected

Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2017-07-06 14:29:02 -03:00
Nick Clemens 70377b3757 Bug 18189 - Elasticsearch sorting broken
To test:
1 - Have ES enabled and biblios and authorities indexed
2 - Perform a catalog search
3 - Try to sort results by author, it will fail
4 - Perform an authority search with sorting enabled, it will fail
5 - Apply patch
6 - Reindex, deleting current indexes
    perl /home/vagrant/kohaclone/misc/search_tools/rebuild_elastic_search.pl
    -d -v
7 - Repeat 3&4, they should succeed

Signed-off-by: Srdjan <srdjan@catalyst.net.nz>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-03-31 14:33:19 +00:00
Blou 77337a2ab5 Bug 17260: updatedatabase.pl fails on invalid entries in ENUM and BOOLEAN columns
The loading of file admin/searchengine/elasticsearch/mappings.yaml
specifies 'type' as empty, which fails with Mysql 5.7+ which is more
stringent.

Also, forcing an empty value into a boolean also fails.

Both issues arise when updatedatabase.pl hit 3.23.00.050.

NOTE: both issues could also be resolved by actually setting values in
      the load file. This doesn't make this solution incorrect, though.

To Test/reproduce:
-1) Happens with Mysql 5.7.4.  Maybe earlier, but certainly at
    that point. Use a Xenial kohadevbox to more easily test.
0) Find a database on 3.22 or earlier, save it.
    a) place
           sql_mode=IGNORE_SPACE,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
       into the [mysqld] section of /etc/mysql/mysql.conf.d/mysqld.cnf
    b) restart you mysql server
    c) drop the db and recreate it
    d) checkout the 3.22.x branch
    e) do a web install
    f) remove the added sql_mode
    g) restart the mysql server
1) Set your code base to master
2) run updatedatabase.pl
3) See the errors on 3.23.00.050
4) Apply the patch
5) Reload the 3.22 db.
   a) repeat steps 0(a)-0(g)
   b) don't forget the caching issues
6) succeed with updatedatabase.pl
7) drop the db and recreate it
8) run the web installer
9) notice no issues either.
10) run koha qa test tools

NOTE: This bug only solved the upgrade portion.
      I added the kohastructure.sql change as well.
      I confirmed that all the code changes were
      triggered with this test plan.

Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Mehdi Hamidi <mehdi.hamidi@inlibro.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-03-03 16:43:00 +00:00
Olli-Antti Kivilahti 4f178fad53 Bug 17255 - Upgrade Elastic Search code to work with version 5.1
Builds on top of commit:
    Bug 17255 - Upgrade Elastic Search code to work with version 2.4+ - rebased wip

-Fix data type 'string' to 'keyword' and 'text'
-index: not_analyzed deprecated, replaced with type: keyword which is equivalent
-store: yes was deprecated, use store: true

TODO: Installer bindings to both the debian package install and the raw developer install.
A taster in Buug 17851

ZE TEST PLAN

0. Remove existing ES and reinstall ES 5.1
   apt-get purge elasticsearch
   Follow instruction here:
   https://www.elastic.co/guide/en/elasticsearch/reference/current/deb.html

1. Reset Elasticsearch index since facets are hard coded to dynamic search_marc_mappings.
1a. perl -e 'use Koha::SearchEngine::Elasticsearch; Koha::SearchEngine::Elasticsearch->reset_elasticsearch_mappings();'
1b. If you get trouble, simply DELETE FROM [search_fields|search_marc_to_fields|search_marc_mapping];
    and retry 1a.
1c. Destroy elasticsearch index
    curl -XDELETE localhost:9200/koha_biblios
    so it can be recreated

2. Recreate the index:
   perl misc/search_tools/rebuild_elastic_search.pl
2a. Add something to index if your koha.biblio-table is empty

3. Fetch all indexed records and the facet for subject__facet

curl -XGET localhost:9200/koha_biblios/data/_search?pretty -d '{
  "aggregations": {
    "my_agg": {
      "terms": {
        "field": "subject__facet"
      }
    }
  }
}'

4. Run the included tests:
   perl t/db_dependent/Koha_Elasticsearch_Indexer.t
   perl t/db_dependent/Koha_Elasticsearch.t
   perl t/db_dependent/Koha_SearchEngine_Elasticsearch_Search.t

Have fun with your new ES 5.1 cluster!

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-02-07 17:17:54 +00:00
Olli-Antti Kivilahti d1077b5ae6 Bug 17255 - Upgrade Elastic Search code to work with version 2.4+ - rebased wip
-Changed deprecated facets to aggregations
-Fixed boolean datatypes not allowing analyzers to be specified
-Fixed deprecated '_id' to 'es_id'. Now the ES-index has the correct id==biblionumber

ZE TEST PLAN

1. Reset Zebra index since facets are hard coded to dynamic search_marc_mappings.
2. perl misc/search_tools/rebuild_elastic_search.pl
3. Fetch all indexed records and the facet for subject__facet

curl -XGET localhost:9200/koha_biblios/data/_search?pretty -d '{
  "aggregations": {
    "my_agg": {
      "terms": {
        "field": "subject__facet"
      }
    }
  }
}'

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
2017-02-07 17:17:53 +00:00
Jonathan Druart df3df9cf71 Bug 17372: Standardize Elasticsearch paths
What we currently have:
Koha/ElasticSearch.pm
Koha/ElasticSearch/Indexer.pm
Koha/SearchEngine/Elasticsearch/QueryBuilder.pm
Koha/SearchEngine/Elasticsearch/Search.pm

What we want:
Koha/SearchEngine/Elasticsearch.pm
Koha/SearchEngine/Elasticsearch/Indexer.pm
Koha/SearchEngine/Elasticsearch/QueryBuilder.pm
Koha/SearchEngine/Elasticsearch/Search.pm

Test plan:
  % git grep -i Koha::ElasticSearch
  % git grep ElasticSearch|grep -v Catmandu::Store::ElasticSearch
should not return any result

Do a full reindex and search for records

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
2016-10-11 01:07:03 +00:00