Whne performing batch operations we can send a large numebr of records for reindexing at once.
Currently this can create requetss that are too large for Elasticsearch to process. We need
to break these requests into chunks/
This patch adds a chunk_size configuration to the elasticsearch stanza in koha-conf.xml
If blank we default to 5000.
To test:
0 - Have Koha using Elasticsearch
1 - Create and download a report of all barcodes:
SELECT barcode FROM items
2 - Batch modify these items
3 - Note a single ESindexing job is created
4 - Create and download a report of all authority ids:
SELECT auth_header.authid FROM auth_header
5 - Setup a marc modification template, and batch modify all the authorities
6 - Again note a single ES backgorund job is created
7 - Apply patch
8 - Repeat the modifications above - you still get a single job
9 - Edit koha-conf.xml and add <chunk_size>250</chunk_size> to elasticsearch stanza
10 - Repeat modifications - you now get several background ES jobs
11 - prove -v t/db_dependent/Koha/SearchEngine/Elasticsearch/Indexer.t
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
(cherry picked from commit 9951e230e4)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This patch uses the filter option from the mappings to add a value_callback
to reord processing for indexing.
Fields defined with 'punctuation' filter will have all punctuation stripped when
conveted to documents.
Tests are updated.
Signed-off-by: Danielle M Elder <danielle.elder@law.utexas.edu>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
[EDIT] Removed skip_record_index => 1 from automatic_renewals.pl. See BZ.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Mentored-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
A unit test has been added to show the pass-through of skip_record_index from AddRenewal() to the Item->store() method.
Test plan:
1) In the shell run prove -vv t/db_dependent/Koha/SearchEngine/Indexer.t
2) Observe Subtest 3: Test AddRenewal indexer call
3) All four tests should pass
Mentored-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch alters the code in the ES QueryBuilder. Reflecting
how things are handled in build_query_compat we clean the query,
but not the limits. In Zebra we simply recursivly call buildQuery,
but the ES query structure
This patchset adds a new ability to save searches on the staff client, and display them in the results
page on staff or opac as a new filter.
New filters can be added from the results page after a search, and there is an admin page for updating
deleting and renaming filters
There is a new permission to control management of these filters
New filters can be added that are not displayed along with facets, this allows for building custom links
using these filters to keep URLs shorter
Due to bug 30528 testing in ES is recommended
To test:
1 - Apply patches and update database and restart all
2 - Enable new system preference 'SavedSearchFilters'
3 - As superlibrarian perform a search in staff client, something broad like 'a'
4 - Note new 'Save search as filter' link on results page
5 - Click it, save search as new filter, check 'Staff client' visibility
6 - Perform another search
7 - Note the filter now appears above facets
8 - Click to it filter results
9 - Note results are limited by the new filter, and it is checked in the facets
10 - Confirm click the [x] removes the filter
11 - Go to administration->search filters
12 - Confirm the filter appears
13 - Edit and mark as OPAC visible
14 - Test OPAC to ensure it shows and can be applied/removed
15 - Copy URL with filter applied
16 - In adminsitration mark filter as not visible on staff or opac
17 - Confirm link above still works
18 - Create a new staff with catalogue and search filters permission
19 - Ensure they can access/save filters
20 - Remove filter permission and ensure they cannot
21 - Disable system preference
22 - Confirm links to search filters page are removed from admin home and admin sidebar
23 - Confirm filters do not appear on results and cannot be created
24 - Enable pref
25 - Create a filter
26 - From search filters page, click 'Edit search'
27 - Confirm you are taken to advanced search page letting you know which filter you are editing
28 - Confirm you can change searhc options and save
29 - Confirm you can perform the search from this page
Sponsored-by: Sponsored by: Round Rock Public Library [https://www.roundrocktexas.gov/departments/library/]
Signed-off-by: Michal Urban <michalurban177@gmail.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Sponsored-by: Sponsored by: Round Rock Public Library [https://www.roundrocktexas.gov/departments/library/]
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Sponsored-by: Lund University Library, Sweden
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Previous commit correctly lists Zebra availability test, but was a
bad edit. This fixes the availability test to use itemlost
Additiionally, tests were written before limits were agreed. Adjusted
for the current settings
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch adds two new sysprefs:
AutoLinkBiblios
AllowManualAuthorityEditing
Both inherit the setting from BiblioAddsAuhtorities which is removed
To test:
1 -Apply patches
2 - Update database
3 - Confirm old setting is transferred to new option
4 - Confirm you can edit authoriteis manually if AllowManualAuthorityEditing set to allow
5 - Confirm you cannot edit authorities manually if AllowManualAuthorityEditing set to don't allow
6 - Confirm a new bib is linked when AutoLinkBiblios is enabled (set AllowManualAuthorityEditing to add unlinked heading)
7 - Confirm new bib notlinked when AutoLinkBiblios is disabled
8 - Confim new bib not linked when AutoLinkBiblios is enabled, but heading doesn't match an authority and AutoCreateAuthorities is disabled
9 - Confim new bib linked to new authority when AutoLinkBiblios is enabled, but heading doesn't match an authority and AutoCreateAuthorities is enabled
Signed-off-by: AFHDubCoLib <andrewfh@dubcolib.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
To test:
1 - Enable ES and index
2 - Set expandedSearchOption to 'show'
3 - Advanced search on staff and opac using Year field on staff and Publication date range on OPAC for:
2005
2005-
-2005
<2005
>2005
2005-2010
4 - Confirm ranges don't work on staff side, but do on opac
5 - Apply patch
6 - Repeat searches - they work on both staff and opac
7 - prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Includes:
Bug 29697: (follow-up) Use flag embed_items
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch adds unit tests, as well as changing existing test to use a mock
and read the data as passed in tests, rather than relying on what exists in
the DB
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
When a query with "OR" is combined with a limit in Elasticsearch, the precedence is not preserved and the results are not correct.
To test:
1) Set SearchEngine to Elasticsearch
2) Index records in Elasticsearch
3) Do an advanced search
4) Select More options
5) Enter a value for the first Keyword (e.g. Novels)
6) Change "and" before the second Keyword to "or"
7) Enter another value for the second Keyword (e.g. Prose)
8) Limit the search (e.g. Item type Books)
9) Do the search
10) Observe that records with the first keyword are not in the results
11) Apply the patch
12) Repeat the search
13) Observe that results with both keywords are in the results
14) Sign off
Sponsored-by: Lund University Library
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Bulk doesn't support include_type_name, update requirement to
Search::Elasticseach@6.80, and remove data type name from tests.
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch adds quoting when handling branchcodes in searching in order
to prevent errors when branchcodes are reserved words in ES
To test:
0 - Be using Elasticsearch with Koha
1 - Add a new branch code:OR name:Orly
2 - Add an item to this branch
3 - Use advanced search to limit search to only Orly
4 - Oh really bad, the search fails!
5 - Apply patch
6 - Repeat search
7 - Oh really good, the search succeeds
8 - prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
We changed from 'null' being meainingful to using a tinyint and
explicitly requiring '0'. This patch simply updates the unit test to
reflect that change.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Can't use an undefined value as an ARRAY reference at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Indexer.pm line 121.
Caught that when working on follow-up bug reports.
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Starting in Elasticsearch 6.0 _key should be used instead of _term to order
buckets by their term
To test:
1) Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t
2) If you observe an error about types, apply patch for bug 25669
3) Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t
4) Observe if you get a deprecation warning about order key
5) Apply patch
6) Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t
7) Observe no deprecation warning about order key
8) Sign off
Sponsored-by: Lund University Library
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
I need to add a 'request_timeout' entry to the ES configuration to fix some timeout problems on the sandboxes.
Instead of hard coding this new option it seems preferable to be
flexible and allow different options to be passed from the config file.
Test plan:
Add to the $KOHA_CONF, inside the elasticsearch section
<request_timeout>60</request_timeout>
Rebuild the ES index
koha-elasticsearch --rebuild kohadev
Change the value of the timeout to 1
Rebuild the index
It should fail (with a quite bad error 'Bad response received when
submitting request to Elasticsearch')
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Fluid is not a valid value for cxn_pool - valid values are:
Sniff,Static,NoPing
AddAuthority call calls index - the indexing call was trying to access
the ES server with mocked values and dying when they were invalid
This patch uses a correct value for cxn_pool and inserts directly into DB
to avoid indexing
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Test plan:
1. Create an authority with at least a 1XX$a and a 4XX$a, for instance:
100 $a Foo
400 $a Bar
2. Create a biblio and add a link to this authority using the
cataloguing plugin
3. Disable syspref IncludeSeeFromInSearches
4. Reindex the biblio. You should be able to find it when searching
'Foo' but not when searching 'Bar'
5. Enable syspref IncludeSeeFromInSearches
6. Reindex the biblio. You should be able to find it when searching
'Foo' and also when searching 'Bar'
without the patch, 'Bar' doesn't yeld results. With it, it does.
7. prove t/db_dependent/Koha/SearchEngine/Elasticsearch.t
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Lucy Vaux-Harvey <lucy.vaux-harvey@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Note: tests are very similar between ES and Zebra, however, ES requires the uppercase
OR and doesn't use '=' in the same way. I feel having test coverage in each module is fair and more future
proof in case of changes to search engine
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds tests and handling for calling move_to_biblio on a
Koha::Items set that contains items from more than one source biblio.
Test plan
1/ Inspect the changes to t/db_dependent/Koha/SearchEngine/Indexer.t
2/ Run t/db_dependent/Koha/SearchEngine/Indexer.t and confirm it passes
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch moves the Koha::Biblio->adopt_items_from_biblio method to the
Koha::Items set class and updates all calls from
Biblio2->adopt_items_from_biblio(Biblio1) to Biblio->items->move_to_biblio(Biblio2)
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
We need to update the search index record for the old biblio where the
item was moved from to keep the item info in search index up-to-date.
To test:
1) $ prove t/db_dependent/Koha/SearchEngine/Indexer.t
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
In our test setup we mock the index_records() to produce warnings like
this:
Koha::Item at t/db_dependent/Koha/SearchEngine/Indexer.t line
93.
By wrapping all our item creations to warnings_are{} we can silence them.
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
- Tests for adopt_items_from_biblio
- Tests for the relationship between items and acquisition orders
- Tests for indexer calls in adopt_items_from_biblio
Signed-off-by: Michal Denar <black23@gmail.com>
Rebased-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch allows merging of records with many items without the web server timing out.
Test plan:
Without the patch:
- Create 2 records (one with e.g. 1000 items).
- Do a cataloguing search that displays both records, select them and click "Merge selected".
- Choose the record with many items as the one to be eliminated.
- Start the merging.
- After a while the web server should give you a timeout error (the merging process may still continue)
With the patch:
- Do the same as above
- This time verify that the records are merged without timeout
- Create a new biblio with an item
- Add with the item:
* acquisition order
* hold (reserve)
- Merge the biblio to another one
- Verify that the item and its related data was moved
- Verify that tests pass:
prove -v t/db_dependent/Koha/Biblio.t
prove -v t/db_dependent/Koha/Item.t
prove -v t/db_dependent/Koha/SearchEngine/Indexer.t
Signed-off-by: Michal Denar <black23@gmail.com>
Rebased-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
On bug 17591 we discovered that there was something weird going on with
the way we export and use subroutines/modules.
This patch tries to standardize our EXPORT to use EXPORT_OK only.
That way we will need to explicitely define the subroutine we want to
use from a module.
This patch is a squashed version of:
Bug 17600: After export.pl
Bug 17600: After perlimport
Bug 17600: Manual changes
Bug 17600: Other manual changes after second perlimports run
Bug 17600: Fix tests
And a lot of other manual changes.
export.pl is a dirty script that can be found on bug 17600.
"perlimport" is:
git clone https://github.com/oalders/App-perlimports.git
cd App-perlimports/
cpanm --installdeps .
export PERL5LIB="$PERL5LIB:/kohadevbox/koha/App-perlimports/lib"
find . \( -name "*.pl" -o -name "*.pm" \) -exec perl App-perlimports/script/perlimports --inplace-edit --no-preserve-unused --filename {} \;
The ideas of this patch are to:
* use EXPORT_OK instead of EXPORT
* perltidy the EXPORT_OK list
* remove '&' before the subroutine names
* remove some uneeded use statements
* explicitely import the subroutines we need within the controllers or
modules
Note that the private subroutines (starting with _) should not be
exported (and not used from outside of the module except from tests).
EXPORT vs EXPORT_OK (from
https://www.thegeekstuff.com/2010/06/perl-exporter-examples/)
"""
Export allows to export the functions and variables of modules to user’s namespace using the standard import method. This way, we don’t need to create the objects for the modules to access it’s members.
@EXPORT and @EXPORT_OK are the two main variables used during export operation.
@EXPORT contains list of symbols (subroutines and variables) of the module to be exported into the caller namespace.
@EXPORT_OK does export of symbols on demand basis.
"""
If this patch caused a conflict with a patch you wrote prior to its
push:
* Make sure you are not reintroducing a "use" statement that has been
removed
* "$subroutine" is not exported by the C4::$MODULE module
means that you need to add the subroutine to the @EXPORT_OK list
* Bareword "$subroutine" not allowed while "strict subs"
means that you didn't imported the subroutine from the module:
- use $MODULE qw( $subroutine list );
You can also use the fully qualified namespace: C4::$MODULE::$subroutine
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Run t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Add a unit test to check that skip_record_index is adhered to when
passed to LostItem with the force_mark_returned option enabled.
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
With bug 27281 we removed a call to reset an items holdingbranch back to
the 'frombranch' of the transfer. This is correct, as with bug 23129
(and friends) we ensure that the holdingbranch was not changed at
transfer request time and instead would only be updated at the reciept
of transfer. As such, it's safe to remove the update here as it would be
just setting the same branch as is already present.
However, when removing this ModItem call, I neglected to realise that
also remoes the requirement for passing around the defered indexing bit
as we no longer need to trigger a re-index as there's not any change to
the item.
This patch therefore cleans up the POD for LostItem, removes the no
longer references 'params' parameter and drops the corresponding index
test.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
The tests were failing because we weren't actually changing the item, so store was doing nothing.
We need to transfer 'from' a new branch so that the holdingbranch is changed
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Nick Clemens
Matt Blenkinsop
Tomas Cohen Arazi
Jonathan Druart
Frank Hansen
Julian Maurice
Kevin Carnes
Martin Renvoize
Joonas Kylmälä
Ere Maijala
Fridolin Somers