This patch adds a central FTP/SFTP management UI, with an associated
database object and class, plus the beginnings of an API. We can use
this code as a building block for adding FTP/SFTP connections to
other parts of the system, as well as to improve the usability of the
EDI transports. It is also now possible to test these connections
from within the Koha UI.
To test:-
You will need access to an FTP or SFTP server
Contact me if you need a disposable FTP/SFTP server
a) Go to Koha Administration
b) Notice no FTP/SFTP referenced on this page
c) APPLY PATCH, reset_all
d) Create new user, let's say sftpuser
e) Set permissions for new user to include:
1) catalogue
2) parameters -> manage_sftp_servers
f) Log in with this new user, go back to Koha Administration
1) Notice the FTP/SFTP servers option
g) Undo step e2) for the sftpuser via the koha admin user
1) Refresh the Koha Administration page as the sftpuser
2) Notice how FTP/SFTP servers has disappeared
h) Back to koha admin, go to Koha Administration -> FTP/SFTP Servers -> New
i) Complete form depending on the FTP or SFTP server you have access to, Submit
1) Notice you are prompted to confirm before saving
j) Select Edit, observe your details are the same, and Submit again
k) Repeat step J one additional time
l) Select Test, observe all tests passing
m) Select Delete, confirm deletion, and observe server disappear
n) Optionally, run prove on new tests
1) t/Koha/Auth/Permissions.t
2) t/db_dependent/Koha/SFTP/Server.t
3) t/db_dependent/api/v1/sftp_servers.t
o) SIGN-OFF
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Test plan:
Check if git_install is set correctly for your system
in koha-conf.xml. Adjust otherwise.
cp debian/scripts/koha-functions.sh /usr/share/koha/bin/
cp debian/scripts/koha-* /usr/sbin/
Run koha-shell your_clone. Does that works like expected?
Run t/Koha/Config.t.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Aude Charillon <aude.charillon@openfifth.co.uk>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch tweaks the `Koha::Exceptions::Plugin` class by adding an
extra parameter to the plugin install and upgrade related exceptions so
they carry the actuall error (if any).
* Koha::Exceptions::Plugin::InstallDied: install_error
* Koha::Exceptions::Plugin::UpgradeDied: upgrade_error
I didn't pick plain 'error' because it is a reserved word for
`Exception::Class` which makes it stringify using its value only.
To test:
1. Apply this patch
2. Run:
$ ktd --shell
k$ prove t/Koha/Exceptions.t
=> SUCCESS: Tests pass!
3. Sign off :-D
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Signed-off-by: Laura_Escamilla <laura.escamilla@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch adds a new xt test (xt/use-test-no-warnings.t) to make sure
all our Perl test files will use Test::NoWarnings.
It also adds Test::NoWarnings to some files that were missing it and
that didn't need any other changes.
However we still have 7 non-trivial tests that we will consider
exception for now, until they are fixed.
Test plan:
prove xt/use-test-no-warnings.t
should return green.
And all other tests modified by this patch (the qa script will do that
for you)
You can also test the 2 selenium tests:
1. DROP and CREATE the DB
DROP DATABASE koha_kohadev;
CREATE DATABASE koha_kohadev;
restart_all
2. KOHA_TESTING=1 prove t/db_dependent/selenium/00-onboarding.t
3. Repeat 1
2. KOHA_TESTING=1 prove t/db_dependent/selenium/01-installation.t
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Matt Blenkinsop <matt.blenkinsop@openfifth.co.uk>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch adds a skippable unit test for the fi_FI locale to prove
that the patches work well for other locales that don't use
the default Unicode collation. It will run on koha-testing-docker,
but will be skipped on systems lacking it.
This patch also fleshes out the existing unit tests a bit more
to prove that the patches also improves the sorting of names
with diacritics generally. That is, the default Unicode sort
is better than the existing stringwise/bytewise sort even for
English and French locales.
Test plan:
0. prove -v t/Koha/SearchEngine/Elasticsearch/Search.t \
t/db_dependent/Search_FacetSorting.t
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This follow-up adds two major enhancements to the locale-based facet sorting
implementation:
1. **System Preference Integration**
- Adds FacetSortingLocale system preference for configurable locale selection
- Dynamically detects available system locales using Koha::I18N
- Provides user-friendly dropdown with locale descriptions
- Supports both Elasticsearch and Zebra search engines
2. **Zebra Search Engine Support**
- Extends locale-based sorting to Zebra (in addition to existing Elasticsearch)
- Implements _sort_facets_zebra() function in C4::Search
- Maintains consistent Unicode-aware sorting behavior across search engines
Key Features:
- Dynamic locale detection via Koha::I18N::available_locales()
- Proper module architecture with comprehensive unit tests
- Clean fallback chain: preference → system LC_COLLATE → default
- Admin UI integration in System Preferences → I18N/L10N
- Universal facet sorting for international library systems
Test plan:
1. Apply patch and restart services
2. Install/run database update
3. Navigate to Administration → System Preferences → I18N/L10N
4. Verify "Sort facet names using" dropdown shows available system locales
5. Test facet sorting with different locales and both search engines
6. Run: prove t/Koha/I18N.t t/Koha/SearchEngine/Elasticsearch/Search.t t/db_dependent/Search_FacetSorting.t
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This change uses a configurable locale-based collator to sort
the ES facet names.
Test plan:
0. Apply the patch
1. vi /etc/locale.gen
2. Uncomment the locale you want to generate (e.g. fi_FI.UTF-8 UTF-8)
3. locale-gen
4. vi "/etc/default/koha-common"
5. Add the following to the bottom of the file:
export LC_ALL=fi_FI.UTF-8
6. koha-plack --restart koha-common
7. Setup some test records with authors with accented and unaccented names,
and different cases for the lead letter
e.g. Aa author, Åa author2, aa author, étienne
8. Switch to using Elasticsearch and reindex
koha-elasticsearch -b -v --rebuild kohadev
9. Do a test search
e.g. http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=test
10. Confirm the facet names are sorted in ascending order following
Finnish collation rules
e.g.
aa author
Aa author
étienne
Farley, David
Humble, Jez
Martin, Robert C.
Åa author
NOTE: Any collation and language can be used. Finnish is just
an example of a Latin-based script which has a different
alphabetical ordering than just A-Z
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Not caught by the script
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This has been generated.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This change adds a Template::Toolkit filter which is invoked
via " | scrub_html type => 'note' ".
Test plan:
0. Apply the patch
1. prove t/Koha/Plugins/HtmlScrubber.t
Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This commit is generated using:
% perl misc/devel/tidy.pl
*within* ktd, to get the same version of perltidy than what will be used
by our CI (currently v20230309).
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
To test:
prove -v t/Koha/Installer/Output.t
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
prove t/Koha/SearchEngine/Elasticsearch/Search.t
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This change adds a "safe_url" filter which takes a text input and
returns a Perl URL object which stringifies to a safe URL.
This change is only needed in the OPAC as the staff interface
handles the item URL display using Javascript not Template Toolkit.
0. Apply patch and koha-plack --restart kohadev
1. Create an item for a record using the following URL
https://koha-community.org?url=https%3A%2F%2Fkoha-community.org
2. Go to the OPAC for that record and verify that the URL is
not double-escaped
3. Create a malicious payload (talk to QA/security team for this if necessary)
4. Note that the malicious payload is escaped
5. prove t/Koha/Plugins/SafeURL.t
6. Celebrate!
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This change filters the values from the OpacHiddenItems syspref
out of their corresponding search results facets.
Note: This does not mean that all values from hidden items are filtered out of facets.
Rather, it just means that facet data that matches OpacHiddenItems is filtered out. This is
an imperfect fix, but it is a practical fix that has been requested by more than one library.
Test plan:
0. Apply the patch and koha-plack --restart kohadev
1. Go to http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=29
2. Add an item with an item type of "Maps"
3. Go to http://localhost:8081/cgi-bin/koha/admin/preferences.pl?op=search&searchfield=OpacHiddenItems
4. Fill in OpacHiddenItems with the following:
itype: ['MP']
5. Go to http://localhost:8080/cgi-bin/koha/opac-search.pl?idx=&q=test&weight_search=1
6. Note that "Maps" does not appear in the "Item types" facet
7. Go to http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=test
8. Note that "Maps" does appear in the "Item types" facet
9. ***Repeat this test plan using both Elasticsearch and Zebra indexing***
10. prove -v t/Koha/SearchEngine/Search.t
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
prove t/Koha/REST/Plugin/Query.t
prove t/db_dependent/Koha/Objects/Mixin/ExtendedAttributes.t
Co-authored-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds two new system preferences, use public lists, and create public
lists. Use public lists determines if a librarian is permitted to see public
lists, whilst create public lists determines if a librarian can create new public
lists.
It also fixes erroneously allowing staff to add items to existing lists, by
honouring the existing edit_public_list_contents
To test:
a) notice the new my lists link on the account pulldown
1) ensure it goes to the lists module
b) create new public list, add at least one item, make it editable by everyone
1) note the name of the list
c) create a new patron with full access to the staff client minus lists permissions
d) log in as the newly created patron
e) notice the lists button is missing from the staff client mainpage
f) set any of the lists permissions except create public lists, use public lists, edit public list contents
g) notice how the lists button is no longer missing from the staff client mainpage
h) go to the lists module
1) notice that public lists are now missing from the datatable
i) click add list
1) notice that the public drop-down is now a fixed label set to private
j) create list and confirm it is not public
k) turn use public lists permission on
l) return to the lists module
1) notice that the datatable now shows private and public lists tabs
m) turn create public lists permission on
n) repeat steps h-i
1) notice that the public drop-down is now visible again
o) create list and confirm it is public
p) go to the list you created in step b
q) notice that add items button, and remove selected button, is missing
r) turn edit public list contents permission on
s) repeat steps p-q
1) notice that add items button is now visible
t) click add items
u) enter an item barcode or biblio number, click save
1) notice that the items are now added to the list
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
There are a number of libraries that would like for their staff to be able to submit (and view existing) purchase suggestions from the borrower record, but not give the staff the ability to edit/manage/delete purchase suggestions.
Test Plan:
1) Apply this patch
2) Run restart all the things!
3) Run updatedatabase
4) Verify anyone with the suggestions manage permissions now has the create and delete permissions as well
5) Verify anyone without the suggestions manage permission has not recieved the new permissions
6) Enable only the create permission for a librarian
7) Verify that librarian can only create new suggestions ( not manage or delete )
8) Enable only the manage permission for a librarian
9) Verify that librarian can only manage existing suggestions ( not create or delete )
10) Enable only the delete permission for a librarian
11) Verify that librarian can only delete suggestions ( not create or manage )
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Sponsored-by: Cuyahoga County Public Library
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch moves the header set into Koha::Email->create and passes the
template_id in a few additional locations to ensure we catch more cases.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds geosearch to Koha (using Elasticsearch 7). ElasticSearch
search_mappings get new types to store lat/lon, which can be indexed
from MARC 034$s and 034$t. There is a small change to the DB to allow a
new value in search_field.type ENUM.
The QueryBuilder is extended to allow for building advanced
ElasticSearch Querys (eg geo_distance) that cannot be represented in a
simple string query. The UI for searching (including showing the results
on a OSM/Leaflet map) is implemented in a separate plugin
(https://github.com/HKS3/HKS3GeoSearch)
Test Plan:
* make sure you're running ElasticSearch 7
(eg via `curl http://es:9200?pretty | grep number`)
* apply patch
* got to a Framework, check Editor for 034$s and 034$t and save
* got to some books (in the correct framework) and enter some lat and lon into 034$s and 034$t (for example lat=48.216, lon=16.395)
* Run the elasticsearch indexer, maybe limited on the books you edited (-bn 123 -bn 456):
misc/search_tools/rebuild_elasticsearch.pl -b -v
* You can check if the indexing worked by inspecting the document in elasticsearch:
* get the biblionumber (eg 123)
* curl http://es:9200/koha_kohadev_biblios/_doc/123?pretty | grep -A5 geolocation
* You should get back a JSON fragment containing the lat/lon you stored
* You can query elasticsearch directly:
* Run the following curl command, but adapt the value for lat/lng and/or the distance (in meters)
* curl -X GET "http://es:9200/koha_kohadev_biblios/_search?pretty" -H 'Content-Type: application/json' -d '{"query": {"bool":{"must":{"match_all":{}},"filter":{"geo_distance":{"distance":100000,"geolocation":{"lat":48.2,"lon":16.4}}}}}}'
* To run the search via Koha, you need to either install and use https://github.com/HKS3/HKS3GeoSearch or create a handcrafted query string:
* handcrafted query string:
* /cgi-bin/koha/opac-search.pl?advsearch=1&idx=geolocation&q=lat:48.25+lng:18.35+distance:100km&do=Search
* HKS3GeoSearch
* install the plugin and enable it
* got to OPAC / Advanced Search
* There is a new input box "Geographic Search" where you can enter lat/long/radius
* On the search result page a map is shown with pins for each found biblioitem
Sponsored-by: ZAMG - Zentralanstalt für Meterologie und Geodynamik, Austria - https://www.zamg.ac.at/
Sponsored-by: Geosphere - https://www.geosphere.at/
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Additional finetuning:
- Fix update and remove fixed fixme
- Update test count as well
- fix last small issues raised in Comment 23
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Permissions.t was failing when running a jenkins build to another branch with this bug applied. Permissions.t was missing the permission for record sources and is now added in this patch
Test plan:
1) prove t/Koha/Auth/Permissions.t
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
prove t/Koha/REST/Plugin/Query.t
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
NB in t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t,
I had to modify four existing test by changing Local-number:123456 to
Personal-name:donald -- since Local-number should never be
auto truncated according to the submitted patch.
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Koha::Object depends on Koha::DateUtils, which depends on C4::Context,
which depends on Koha::Config::SysPref, which depends on... Koha::Object
Apart from the circular dependency, the dependency on C4::Context alone
is problematic as it loads a bunch of modules that are not needed at all
in Koha::Object (YAML::XS and ZOOM for instance).
As Koha::Object is used as a base for a lot of modules, we should take
care to only load the minimum required.
This patch moves some date parsing code to specific modules:
- Koha::DateTime::Format::RFC3339
- Koha::DateTime::Format::SQL
and it uses them in Koha::Object and Koha::DateUtils where it is
possible.
Test plan:
1. Do not apply the patch yet and run the following command:
`perl -cw Koha/Object.pm`
It should print several warnings about redefined subroutines, meaning
there is a circular dependency.
2. Apply the patch
3. Run `perl -cw Koha/Object.pm`. It should only say:
"Koha/Object.pm syntax OK"
4. Run the following command:
prove \
t/DateUtils.t \
t/Koha/DateTime/Format/RFC3339.t \
t/db_dependent/Koha/Object.t
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
<key content="value"></key>
was wrongly parsed as
{ key => 'value' }
whereas it should be
{ key => { content => 'value' } }
The 'content' attribute is used in shibboleth config
Test plan:
1 Run `prove t/Koha/Config.t`
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
and inherit from it
The new usage is now:
1. With a Koha::Biblio object
my $extractor = Koha::Biblio::Metadata::Extractor->new({biblio => $biblio});
$extractor->get_normalized_upc;
or
2. With a MARC::Record
my $extractor = Koha::Biblio::Metadata::Extractor->new({metadata=> $biblio->metadata->record});
$extractor->get_normalized_upc;
Note that there are "Inconsistent hierarchy during C3 merge of class"
warnings raised by the QA script. We could remove them by replacing the
'use' by 'require' in Koha::Biblio::Metadata::Extractor::MARC (in ->new)
but that's suboptimal.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
We are not directly linked with the other Koha::Metadata* packages.
Better to isolate this under Koha::Biblio::Metadata (at least for now).
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Tomás Cohen Arazi
Lucas Gass
Jake Deery
Marcel de Rooy
Jonathan Druart
Tomás Cohen Arazi
David Cook
Martin Renvoize
Koha Development Team
Emily Lamancusa
Matt Blenkinsop
Aleisha Amohia
Pedro Amorim
Martin Renvoize
Kyle M Hall
Julian Maurice
Katrin Fischer
Mark Hofstetter
Janusz Kaczmarek