This updates the regex used for removing colons to capture those with space on either side, and remove the colon
while preserving the space
To test:
1 - Have Koha using ES
2 - Search for:
ti:chess AND chess
3 - You should get a result in sample data, otherwise replace 'chess' with a title in your catalogue
4 - Search for:
ti:chess AND kw:chess
5 - No result
6 - Enable DumpTemplateVarsIntranet and DumpSearchQueryTemplate
7 - Repeate search and check page source
8 - search_query has:
title:chess ANDchess
9 - Apply patch
10 - Repeat
11 - Seaerch works!
12 - query is now:
title:chess AND chess
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Bug 24567: (follow-up) Use dollar sign to refer to captures
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch introduces a tiny class for encapsulating execution messages in
Koha::Object derived classes.
To test:
1. Apply this patch
2. Run;
$ kshell
k$ prove t/Koha/Object/Message.t
=> SUCCESS: Tests pass!
3. Sign off :-D
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch completely rewrites the Koha::Email class, inheriting from
Email::Stuffer. The latter suits well the use by Email::Sender, which is
to replace Mail::Sendmail on this patchset.
To test:
1. Apply this patch
2. Run:
$ kshell
k$ prove t/Koha/Email.t
=> SUCCESS: Tests pass!
3. Verify all conditional codepaths are covered
4. Sign off :-D
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
.pm must not have -x
.t must have -x
.pl must have -x
Test plan:
Apply only the first patch, run the tests and confirm that the failures
make sense
Apply this patch and confirm that the test now returns green
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
The match-heading field is a special field used only by the linker, not accessible
to staff or patrons via the interface. This field is used to store the constructed
'search form' used for matching bib headings to authority fields.
In bug 24269 I attempted to use the mappings defined in the inferface and also inject the search term.
This did not work as too many subfields were indexed on their own and leading to false matches.
In this bug we remove the mappings for this field, and create it ourselves during
the indexing process. The C4::Headings module is still used to generate the correct form,
however, the mappings are set based on the authority types in the system. This gives the user
the ability to add new typoes, but prevents mapping changes from breaking linker functionality
To test:
1 - Start form a sample database with ElasticSearch working
2 - Download via Z39.50 2 authorities, one of which is a narrower heading of the other, e.g.:
Waterworks
Waterworks - Costs
3 - Place a heading for the broader term in a record. e.g. Waterworks
In 650$a, without the cataloguing authority plugin. We don't want
the link created now.
You need syspref BiblioAddsAuthorities => allow
4 - Make sure linker is set to default
5 - Attempt to link the records
misc/link_bibs_to_authorities.pl
6 - Linking fails
7 - Apply patch
8 - refresh index settings (if using a custom file, remove 'match-heading')
You can reset mappings in the UI or run this:
misc/search_tools/rebuild_elasticsearch.pl -v -d -r
9 - Reindex ES
10 - Try to link again
11 - It succeeds!
12 - Run the tests
prove t/db_dependent/Koha/SearchEngine/Elasticsearch.t
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Bug 25273: (follow-up)
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
In our test data we have OPACBaseURL that is filled with http://kohadev.myDNSname.org:8080,
but the CGI referer in local_referer is lowercase
We should compare it without case sensitivity
Test plan:
- Use koha-testing-docker
- Notice the value of OPACBaseURL
=> http://kohadev.myDNSname.org:8080
- Use the "reports a problem" feature at the OPAC
- Notice that the "Problem found on page" is correctly filled with the
referer
Signed-off-by: Kelly McElligott <kelly@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch introduces tests for the per_page=-1 handling use case. From
now on per_page=-1 means 'all resources'.
On writing this I noticed that we always paginate results no matter
what, but there was a weird condition under which on pagination headers
were sent back to the API consumer. This is highlighted in the precedent
patch, which is not the -1 situation this one tries to tackle.
Both pagination and searching are broken with per_page=-1, which is a
standard, and we actually didn't explicitly set a way to request all
resources.
To verify this:
1. Apply the previous tests patch and this one
2. Run:
$ kshell
k$ prove t/Koha/REST/Plugin/Pagination.t \
t/db_dependent/Koha/REST/Plugin/Objects.t
=> FAIL: Things are damn broken
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch removes the <client> ip address in the X-Forwarded-For
header from being tested against koha_trusted_proxies.
Without this patch, REMOTE_ADDR will be set to null, if the <client>
ip address matches against koha_trusted_proxies.
To Test:
1. Run the unit test t/Koha/Middleware/RealIP.t
Signed-off-by: Didier Gautheron <didier.gautheron@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Remove unused Elasticsearch parameters 'key_prefix' and 'request_timeout'.
Refactor so that get_elasticsearch_params only returns parameters used
by Search::Elasticsearch, add class accessor for index_name.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This simple patch introduces a test for the lock waiting scenario. It
replicates the previous tests, but calls a script that passes the wait
=> 1 parameter to ->lock_exec.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch simplay alters the data we use for the tests, doing so causes them to fail
To test:
1 - Apply only this patch
2 - prove -v t/Koha/SearchEngine/Elasticsearch.t
3 - It fails!
4 - Apply next patch
5 - It passes!
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Strip initial characters from search fields in accordance with
nonfiling character indicators.
To test:
1) Apply patch
2) Run tests in t/Koha/SearchEngine/Elasticsearch.t
3) All tests should pass
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
In the advanced search form, you can enable several limits using syspref
AdvancedSearchTypes (namely itemtypes, shelving locations, collections)
When used, the resulting query parts end up being joined with OR, even
if the field is different. That means that if you pick "Book" under
itemtypes tab, and "Fiction" under collection tab, it will search
"itype:BOOK OR ccode:FIC". It should be AND.
For instance, if you select:
Itemtypes:
✓ Book
✓ DVD
Location:
✓ Child
✓ Adult
it should search:
itype:(Book OR DVD) AND location:(Child OR Adult)
Test plan:
0. Do not apply the patch yet
1. Enable elasticsearch
2. Set syspref AdvancedSearchTypes = 'itemtypes|loc|ccode'
3. Create a new itemtype and a new authorised value for categories LOC
and CCODE
4. Create a biblio with the new itemtype, another biblio with the new
location, another biblio with the new collection, and again another
biblio with the new itemtype, location and collection
5. Verify that you can find these new biblio records using only the
"advanced search types" in the advanced search form
6. In the advanced search form, pick all 3 limits (itemtype, location,
collection) and verify that it returns the 4 records.
7. Apply the patch
8. Repeat step 6, it should now return only the biblio that satisfies
all criteria
9. Verify that if you select more than one
{itemtype|location|collection} it still returns results that
satisfies any selected criteria
10. prove t/Koha/SearchEngine/ElasticSearch/QueryBuilder.t
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Remove a few last references to branchprinter, mostly from tests.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch adds the possibility to object.search helper, to also filter by prefetched columns.
In order to dynamically add filter parameters, they must be coded as json and placed in the body of the request, coded as string in 'q' query parameter or as string in 'x-koha-query' header.
The coded json, is in fact dbix syntax.
To test:
1. apply this patch
2. prove t/Koha/REST/Plugin/Query.t t/db_dependent/Koha/REST/Plugin/Objects.t
3. Sign off
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Test plan:
0. Apply patch
1. Set up Koha on stretch (e.g. koha/koha-testing:master)
2. perl t/Koha/Middlware/RealIP.t
3. Observe the following:
Subtest: IPv6 support
ok 1 - Warn on IPv6 koha_trusted_proxies
ok 2 - Unable to parse IPv6 address for trusted proxy, so ignore the X-Forwarded-For header
1..2
ok 13 - IPv6 support
4. Set up Koha on buster (e.g. koha/koha-testing:master-buster)
5. perl t/Koha/Middlware/RealIP.t
6. Observe the following:
Subtest: IPv6 support
ok 1 - Trust proxy (2001:db8:1234:5678:abcd🔢abcd:1234) using IPv6 CIDR notation, so use the X-Forwarded-For header for the remote address
1..1
ok 13 - IPv6 support
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Bug 9978 should have fixed them all, but some were missing.
We want all the license statements part of Koha to be identical, and
using the GPLv3 statement.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch makes the Koha::Object(s) derived classes expose information
about prefetch-able relations. This is then used by a new helper to
generate the prefetch information for the DBIC query.
To test:
1. Apply this patch
2. Run:
$ kshell
k$ prove t/db_dependent/Koha/Object* \
t/db_dependent/Koha/REST/Plugin/Objects.t \
t/Koha/REST/Plugin/Query.t
=> SUCCESS: Tests pass!
3. Sign off :-D
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch adds some unit tests for Koha::Middleware::RealIP
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
To test:
1 - Be using Elasticsearch
2 - Check that field 150 is mapped to 'Match-heading' or add it (the subfields don't matter)
3 - Add a topic term authority record like "150 $aCats$vFiction"
4 - Add a 650 with $aCats and $vFiction and $e depicted to a bibliographic record
5 - Run the linker for the bib
perl misc/link_bibs_to_authorities.pl --bib-limit "biblionumber=89" -v
6 - Confirm the record is not correctly linked to the record
7 - Apply patch
8 - Reindex authorities for ES
perl misc/search_tools/rebuild_elasticsearch.pl -v -d -a
9 - Run the linker and confirm record is correctly linked
perl misc/link_bibs_to_authorities.pl --bib-limit "biblionumber=89" -v
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch adds a + syntax to specify on the x-koha-embed header that we
want the count of a relation in the response. For example:
GET /patrons/1
x-koha-embed: checkouts+count
Would return a JSON representation of a Koha::Patron, with a new
attribute added: checkouts_count, which will be the result of calling
$patron->checkouts->count.
This is all done automatically in to_api. This patch makes parsing the
x-koha-embed header build the right structure for passing to the to_api
method.
To test:
1. Apply this patch
2. Run:
$ kshell
k$ prove t/Koha/REST/Plugin/Query.t
=> SUCCESS: Tests pass!
3. Sign off :-D
Sponsored-by: ByWater Solutions
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch separates query parameters from path parameters, and uses exact matching for the later.
To test:
1. Apply this patch
2. prove t/Koha/REST/Plugin/Query.t t/db_dependent/Koha/REST/Plugin/Objects.t
SUCCESS => tests ok
3. Sign off
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
To test:
1. apply this patch
2. prove t/Koha/REST/Plugin/Query.t
Test should fail at this point.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch introduces a helper for handling x-koha-embed headers on API
requests. It reads the embed definitions and adds them to the stash for
later use (either manually on the controllers, or in the objects.search
helper.
x-koha-embed needs to be defined as a list on the OpenAPI spec.
It throws an exception when invalid combinations are found.
To test:
1. Apply this patch
2. Run:
$ kshell
k$ prove t/Koha/REST/Plugin/Query.t
=> SUCCESS: Tests pass!
3.Sign off :-D
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch makes dbic_merge_sorting accept a result set as parameter and
solves a FIXME in _build_order_atom.
To test:
1. Apply this patch
2. Run:
$ kshell
k$ prove t/Koha/REST/Plugin/Query.t
=> SUCCESS: Tests pass!
3. Sign off :-D
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch adds a to_model parameter to dbic_merge_sorting so it is
passed when used (for example from objects.search). The to_model param
is passed along to the _build_order_atom method where it is finally
used.
In the process I wrote tests that reflected some problems in the current
code:
- Mojolicious automatically returns a scalar if a query parameter only
happens once on a request. The code expected an arrayref in every case.
- There's a design issue that forced me to use some hacky code in
_build_order_atom.
The first issue is dealth with, by using Scalar::Util::reftype as the
Perl docs recommend.
The second issue, I don't plan to clean it here, as there's ongoing work
on a Koha::Objects->search_from_api method that will obsolete this code
most probably (see bug 23893 for a better picture of where the mappings
will be living soon).
To test:
1. Apply this patch
2. Run:
$ kshell
k$ prove t/Koha/REST/Plugin/Query.t
=> SUCCESS: Tests pass!!
3. Sign off :-D
Sponsored-by: ByWater Solutions
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This minimal class encapsulates the tabs to be passed around to the
templates, so error checking on missing bits is done in a single place.
It throws exceptions on errors
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
marc_records_to_documents is now an arrayref of hashes, not an arrayref of arrays
_sanitise_records has been removed, we don't need those tests anymore
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch moves the Elasticsearch/Indexer.t to ElasticSearch/Indexer.t
and thus removes the case directory name collision introduced.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch moved the tests created in Elasticsearch/QueryBuilder.t into
ElasticSearch/QueryBuilder.t and thus removes the case directory name
collision introduced.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Add "QueryRegexEscapeOption" system preference to provide option to escape
Elasticsearch regexp delimiters (/) within queries, or alternativly to
unescape escaped slashes (\/) while escaping unescaped slashes, in
effect making "\/" the new regexp delimiter.
How to test:
1) Run tests in ./t/Koha/SearchEngine/ElasticSearch/QueryBuilder.t
2) All tests should succeed
Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Maksim Sen <maksim.sen@inlibro.com>
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Nick Clemens
Tomas Cohen Arazi
Jonathan Druart
David Cook
David Gustafsson
Julian Maurice
Agustin Moyano
Martin Renvoize
Ere Maijala