Cronjob cleanup_database.pl usage is wrong :
Usage: $0 [-h|--help] [--confirm] (...) [--bg-days DAYS [--bg-type TYPE] ]
Those args has been renamed jobs-days and jobs-type.
Also some args are missing.
There are too many args, I propose to remove them from there, one need to read the full help.
Test plan:
Run command line `misc/cronjobs/cleanup_database.pl -h`
You see first line:
Usage: misc/cronjobs/cleanup_database.pl
-h --help prints this help message, and exits, ignoring all
...
Signed-off-by: Anke Bruns <anke.bruns@gwdg.de>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
If the holds reminder cronjob would send a reminder on a holiday,when holidays are taken into account, it also will send it on the subsequent holidays and the first open day.
This bugfix adds a check to the cron when --holidays is set to skip sending the notices for that library if today is a holiday.
To Test:
Before applying:
1. Create a hold for a patron (I used patron 23529000035676 and 39999000000252)
2. Ensure your patron has an email address set and the hold reminder email selected.
3. Set the hold to waiting by checking it and confirming the hold.
4. In Tools->Calendar add a holiday to yesterday and a seperate holiday to today.
5. On the command line (the date should be set to 5 days ago for my commands to work and any dates adjusted accordingly)
koha-mysql kohadev
update reserves set waitingdate="2025-09-25"
exit
perl misc/cronjobs/holds/holds_reminder.pl -days 3 -t -v -holidays --date 2025-09-29
6. Observe that 1 notice would have sent to your patron
7. On the command line
perl misc/cronjobs/holds/holds_reminder.pl -days 3 -t -v -holidays
8. Observe that 1 notice would have sent to your patron
9. On the command line
perl misc/cronjobs/holds/holds_reminder.pl -days 3 -t -v -holidays --date 2025-10-01
10. Observe that 1 notice would have sent to your patron
11. Apply patches
12. Repeat the last command from step 5 and step 7
13. Observe that no notice should be sent for your patron
14. Repeat step 7
15. Observe that a notice should be sent for your patron.
16. Run
perl misc/cronjobs/holds/holds_reminder.pl -days 5 -t -v
17. Confirm the notice should be sent for your patron
18. Sign off
Signed-off-by: Katie Bliss <KEBliss@dmpl.org>
Bug 35700: (QA follow-up) fix POD and $today
This patch replaces the inncorrectly removed POD note and replaces $today with the more correct $date_to_run for checking if we should be skipping sending the message on the day.
Amended-by: Jonathan Druart
Squashed and adjusted commit title
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Executing marc_ordering_process.pl will fail to run with the following errors:
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 21: =head1: command not found
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 23: marc_ordering_process.pl: command not found
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 25: =head1: command not found
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 27: --confirm]: command not found
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 27: --verbose]: command not found
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 27: ./marc_ordering_process.pl: No such file or directory
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 29: or,: command not found
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 31: 0: command not found
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 33: =head1: command not found
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 35: This: command not found
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 36: If: command not found
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 38: =head1: command not found
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 40: =over: command not found
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 42: syntax error near unexpected token `newline'
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 42: `=item B<-v|--verbose>'
This is due to an inadvertent newline before the shebang of the script.
Test Plan:
1) Run ./misc/cronjobs/marc_ordering_process.pl
2) Note the error
3) Apply this patch
4) Run ./misc/cronjobs/marc_ordering_process.pl
5) No error!
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
The es_indexer_daemon.pl was missing handling for the case where
Koha::BackgroundJob->connect() returns undef without throwing
an exception (which happens when connection fails silently).
This adds the missing warning for that case, matching the behavior
in background_jobs_worker.pl.
Test plan:
1. Set JobsNotificationMethod to 'STOMP'
2. Configure invalid STOMP settings that cause connect() to return undef
3. Run es_indexer_daemon.pl - should show connection warning
4. Verify jobs are still processed via database polling
Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
The background job workers are showing STOMP/RabbitMQ connection error
warnings even when the JobsNotificationMethod system preference was set
to 'polling', which doesn't require a message broker connection.
This patch adds checks for the JobsNotificationMethod preference before
displaying connection warnings, so warnings only appear when STOMP is
actually expected to be available.
Changes:
- misc/workers/background_jobs_worker.pl: Check syspref before warning
- misc/workers/es_indexer_daemon.pl: Check syspref before warning
Test plan:
1. Set JobsNotificationMethod to 'polling'
2. Run background_jobs_worker.pl - should not show STOMP warnings
3. Run es_indexer_daemon.pl - should not show STOMP warnings
4. Set JobsNotificationMethod to 'STOMP' without RabbitMQ running
5. Run workers - should show appropriate STOMP connection warnings
6. Sign off :-D
Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This change checks for HTTP(S) URLs and skips the shell when using
system() so that shell metacharacters aren't interpreted.
Test plan:
0. Apply the patch
1. Insert the following into the Babeltheque_url_update system preference:
$(whoami > /tmp/rce.log)
2. koha-shell kohadev
3. perl misc/cronjobs/social_data/update_social_data.pl
4. Note the script says the following:
$(whoami > /tmp/rce.log) is not a HTTP URL. Aborting.
5. Note also that /tmp/rce.log is not created
6. Change Babeltheque_url_update syspref to something like the following:
http://localhost:8080
7. perl misc/cronjobs/social_data/update_social_data.pl
8. Note that the wget succeeds but the bunzip2 fails (as expected) since
we haven't fetched a bzipped file
9. Try to create a payload that passes the HTTP URL test with shell
metacharacters and note that they're not interpetted and instead
just used as part of a URL string.
Signed-off-by: Jake Deery <jake.deery@openfifth.co.uk>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch fixes an identified RCE in the update_social_data.pl
social_data, which when executed, allows an attacker to run arbitrary
code on the system's shell.
TO TEST:
a) In the Babeltheque_url_update syspref, wrap any bash in $(), and then
run the update_social_data.pl script.
*) For example, $(whoami >
/tmp/rce.log) will output to a file the username the perl script is
running as.
APPLY PATCH
b) Run the update_social_data.pl script again. The string is now
meta-escaped, and the RCE no longer works.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Since this patch updates the script to use Koha::Items->search instead of a direct database query
--where options that refer to the items table will be broken and result in an error like:
execute failed: Unknown column 'items.withdrawn' in 'where clause'
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch adds a new option to the delete_items cronjob
When --del_bibs is set, bib records will be deleted
if the last item is deleted.
The following plan assumes default ktd data.
To test:
1. Apply patch
2. Check the holdings for bib #115 and see 3 items
http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=115
2. Run the cron with the following flags
perl ./misc/cronjobs/delete_items.pl --where "biblionumber=115 and homebranch ='MPL'" --verbose --del_bibs --commit
3. Notice in the output the count of current items on the bib record:
Where statement: where biblionumber=115 and homebranch='MPL'
Deleting empty bib records!
Deleting item '237' (of 3)
Deleting item '238' (of 2)
4. Confirm the two Midway items were deleted from bib #115, and the bib still exists with one item
4. Run the cron again but change the the --where flag to delete all the items from bib #115
perl ./misc/cronjobs/delete_items.pl --where "biblionumber=115" --verbose --del_bibs --commit
5. Notice in the output the bib is deleted after the last item is deleted
Where statement: where biblionumber=115
Deleting empty bib records!
Deleting item '239' (of 1)
Last item deleted - deleting bib '115'
8. Confirm the bib #115 was deleted
9. Add some more items and test with some other --where parameters
10. Test with and without --commit and --del_bibs, test the --help flag
Sponsored-by: CLAMS
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
I have added the options --include and --exclude to the install_plugins.pl script. When running the script, you can now specify which plugins you want to install (--include) and which ones you do not want to install (--exclude).
--include installs only the plugins from the specified classes.
--exclude installs all plugins except those from the specified classes.
It's possible to specify the package to install a specific plugin.
--include and --exclude cannot be used simultaneously.
Usage example:
./misc/devel/install_plugins --include <package::pluginClass> --include <anotherPluginClass>"
TEST PLAN
1) Apply the patch
2) Add at least three plugins to the plugin directory ([Koha]/plugins/Koha/Plugin/)
3) Empty the "plugin_data" and "plugin_methods" tables (SQL command: "delete from plugin_data; delete from plugin_methods;")
4) In a terminal, execute this command "./misc/devel/install_plugins.pl --include <package::pluginClass> --include <anotherPluginClass>" (Replace <package::pluginClass> and <anotherPluginClass> with the class and package of your plugins)
5) In your Koha intranet, go to "Koha administration > Manage plugins"
6) Verify that only the specified plugins are installed
7) Repeat step 3
8) In a terminal, execute this command from your Koha project"./misc/devel/install_plugins.pl --exclude <package::pluginClass> --exclude <anotherPluginClass>"
9) Verify that all plugins except the specified ones are installed
10) Run the tests from the Plugins.t file ([Koha]/t/db_dependent/Koha/Plugins/Plugins.t)
11) Ensure that all tests are successful
Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Hammat Wele <hammat.wele@inlibro.com>
Signed-off-by: Amaury GAU <amaurygau@gmail.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
From https://css-tricks.com/snippets/css/make-pre-text-wrap/
/* Browser specific (not valid) styles to make preformatted text wrap */
pre {
white-space: pre-wrap; /* css-3 */
white-space: -moz-pre-wrap; /* Mozilla, since 1999 */
white-space: -pre-wrap; /* Opera 4-6 */
white-space: -o-pre-wrap; /* Opera 7 */
word-wrap: break-word; /* Internet Explorer 5.5+ */
}
... I think it's now safe to remove the backward compatibility versions and simply keep pre-wrap.
Test plan (from bug 921)0:
* place a hold on a title with a very very long title or author
* mark it as "waiting for pickup" by doing a checkin
* go to command line interface, and run
* misc/cronjobs/gather_print_notices.pl /tmp
* misc/cronjobs/printoverdues.sh /tmp
* if your title/author is long enough, it's splitted on 2 lines
with this patch (it is not before this patch)
It will also remove the warnings from t/db_dependent/Patron/Borrower_Discharge.t
WARNING: Ignored `white-space: -moz-pre-wrap` at 6:14, invalid value.
WARNING: Ignored `white-space: -o-pre-wrap` at 7:14, invalid value.
WARNING: Ignored `word-wrap: break-work` at 8:14, invalid value.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
With this patch we adjust what's needed to make misc/devel/tidy.pl run
identically as before
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch introduces a new Koha::Devel namespace. It clearly shows
that its purpose is for development and will not contain any business
logic.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
For creating this patch I looked at all parameters defined in the
handler mapping, removed duplicated ones and tracked down which ones
really were parameters in `GetOptions`. This included tracking down
variables that were later assigned other values and things like that.
The mapping is used to display the right parameter name if missing a
mandatory one.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Bug 31143 created a script in cronjob and then moved it to maintenance. However, it is still referred as in cronjob in help strings. This patch updates the hint.
No test needed I guess. Only changing a string
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
In order to remove false positives with the QA script we should replace some tabs with spaces.
There are more but I wanted to make sure no regression would be
introduced
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch corrects a few spelling mistakes and missing grammer and adds
a little more detail to the '--date', '--email' and '--frombranch'
options.
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Although we're reverting 32216, we still want to be able to force print notices as per the original intent.
This patch adds that into queue_notices, rather than in the cronjob
Co-authored-by: Jake Deery <jake.deery@openfifth.co.uk>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This reverts commit 3040f46d5e.
Bug 30300 adds a patron messaging preference for the expiration notice and abstracts this logic into queue_notice
Bug 32216 removes this call to enable print notice functionality but this would be handled already by queue_notice based on the patron's messaging preferences
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
The patches on this bug add the ability for staff members to set the patron expiry notice as mandatory at the category level. When set, the cronjob will automatically send an expiry notice regardless of existing messaging preferences. If it is turned off, the old messaging preferences will still be in place and actioned
Test plan:
1) Apply patch and run updatedatabase
2) In system preferences set MembershipExpiryDaysNotice to 5
3) Choose a patron and edit their expiry date to today's date, ensuring they have a primary email set
4) Create a MEMBERSHIP_EXPIRY notice (content of the notice is irrelevant, just needs to exist)
5) Run perl misc/cronjobs/membership_expiry.pl -c -v -before 10 -after 10
6) Check the patron's messages, there should be no expiry notice queued
7) Set the patron's category 'enforce expiry notice' field to Yes
8) Repeat step 5, there should be a notice queued
9) Set that patron's primary contact method to primary email and run step 5 again
10) There should be a notice queued for an email
11) Set the category level enforce expiry notice back to No
12) Choose a second patron and set an email address, expiry date to today and set the patron expiry messaging preference for that patron to 'email'
13) Repeat step 5 - this time only one notice should be queued on the second patron, no new notice should queue for the first
14) Re-set the category level option to Yes
15) Repeat step 5 - this time both patron's should have notices queued
16) Unset the category level option again, the behaviour should be the same as step 13
17) Set the category level option back to Yes and check the patron messaign preferences menus in both the OPAC and Staff client - the patron expiry option should not be visible while the cateogry level enforcement is set to Yes
Signed-off-by: Hanna Leiker <tech.hanna@salinapublic.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
->connect does not raise an exception, so we didn't reached the catch
block.
Adjusting the code but I am not really happy with it, but more changes
will derail the original bug
Now we log:
Cannot connect to broker (Access refused for user 'guest') at /kohadevbox/koha/Koha/BackgroundJob.pm line 96.
Cannot connect to the message broker, the jobs will be processed anyway at /kohadevbox/koha/misc/workers/background_jobs_worker.pl line 100.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This change adds error handling for ERROR frames to background_jobs_worker.pl.
A previous patch added it to es_indexer_daemon.pl which is a different
background jobs worker.
To test ERROR handling for background_jobs_worker.pl:
1. Change your RabbitMQ connection details in koha-conf.xml or the hard-coded ones in Koha/BackgroundJob.pm
to include an invalid password. For example, if you don't have a message_broker already defined, use the following:
<message_broker>
<password>bad</password>
</message_broker>
2. sudo koha-worker --restart --quiet kohadev
3. View the errors in the logs:
tail -f /var/log/koha/kohadev/worker-output.log
tail -f /var/log/koha/kohadev/worker-error.log
To test MESSAGE handling for background_jobs_worker.pl:
1. Ensure RabbitMQ connection details are correct
2. Go to Batch item modification and add the following barcodes:
39999000001310
39999000004571
3. Add a z - Public Note of the following and click Save:
THEBACKGROUNDWORKERWORKED
4. Search the catalogue using THEBACKGROUNDWORKERWORKED in the top
catalogue search
5. Note that you get 2 results in your search
6. Look in the logs just to make sure there are no new messages there
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This change causes es_indexer_daemon.pl to die on ERROR frames instead
of spinning in an infinite loop. It also adds a content-type STOMP frame header
to help consumers know what data type is in the MESSAGE frame body.
To test ERROR handling for es_indexer_daemon.pl:
1. Switch to using Elasticsearch
2. Change your RabbitMQ connection details in koha-conf.xml or the hard-coded ones in Koha/BackgroundJob.pm
to include an invalid password. For example, if you don't have a message_broker already defined, use the following:
<message_broker>
<password>bad</password>
</message_broker>
3. sudo koha-es-indexer --restart kohadev
4. View the errors in the logs:
tail -f /var/log/koha/kohadev/es-indexer-output.log
tail -f /var/log/koha/kohadev/es-indexer-error.log
To test MESSAGE handling for es_indexer_daemon.pl:
1. Switch to using Elasticsearch (and ensure RabbitMQ connection details are correct)
2. Change the title of a bib record to something new and unique
3. Try to search for the bib using the new title
4. Note you can find it
5. Look in the logs just to make sure there are no messages there
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Bug 27486 renamed system preference "delimiter" into "CSVDelimiter"
This patch just replaces the commented code with a TODO text
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
The cronjob debar_patrons_with_fines.pl is meant to add a restriction based on a fines threshold.
The problem is that the script adds a MANUAL restriction, which makes no sense. MANUAL restrictions are meant to be manually created, never automatic as this script does.
In addition, they are repeatable, not unique. Every time this script runs, it will add an additional restriction.
There is already a restriction type meant for this type of use case: SUSPENSION
We should change this script to create SUSPENSION restrictions instead of MANUAL restrictions.
In summary, this script should use the unique restriction type SUSPENSION to prevent multiple additional restrictions from being created on each run.
Test Plan:
1) Set up a patron with fines
2) Run misc/cronjobs/debar_patrons_with_fines.pl -c -m "Test message"
3) Note the restriction is created
4) Run misc/cronjobs/debar_patrons_with_fines.pl -c -m "Test message" again
5) Note another restriction is created
6) Delete the restrictions
7) Apply this patch
8) Run misc/cronjobs/debar_patrons_with_fines.pl -c -m "Test message"
9) Note the restriction is created
10) Run misc/cronjobs/debar_patrons_with_fines.pl -c -m "Test message" again
11) Note another restriction is *not* created!
Signed-off-by: Magnus Enger <magnus@libriotech.no>
Signed-off-by: Emmi Takkinen <emmi.takkinen@koha-suomi.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds a new test to the data inconsistencies script to report where
MARC records contain item fields.
These may not cause issues in Koha, however, in discovery harvests they have eben found to create errors
To test:
1 - Follow test plan on bug 39412 to generate records with item tags
2 - perl misc/maintenance/search_for_data_inconsistencies.pl
3 - Note new section and hint on how to fix
4 - perl misc/maintenance/touch_all_biblios.pl
5 - repeat 2
6 - Errors cleared
Signed-off-by: Magnus Enger <magnus@libriotech.no>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch makes the indexer poll the DB in configurable batches. Pretty
much like Rabbit does.
The only change is that it will now pass a LIMIT <batch_size> to the
query.
To test:
0. Have KTD started with ES (skip if you have an instance running):
$ ktd --es7 up -d
1. Stop the indexer and rabbit:
$ ktd --shell
k$ sudo koha-es-indexer --stop kohadev
k$ sudo service rabbitmq-server stop
2. On a separate terminal, watch the queue:
k$ watch -n 0.5 'echo "SELECT COUNT(*) FROM background_jobs WHERE status=\"new\"" | koha-mysql kohadev'
=> SUCCESS: It should be empty
3. Add a bunch of things to index (this works with default KTD data):
k$ for i in $(seq 1 438; seq 1 438) ; do echo "INSERT INTO background_jobs (status,queue,data) VALUES ('new','elastic_index','{\"record_server\":\"biblioserver\",\"record_ids\":[$i]}')" | koha-mysql kohadev ; done
4. Start the ES indexer:
k$ sudo koha-es-indexer --start kohadev
5. Look at the queue size
=> FAIL: It shrinks too fast [1]
6. Apply this patch
7. Restart the indexer
k$ perl misc/workers/es_indexer_daemon.pl &
8. Repeat 3
9. Look at the queue size:
=> SUCCESS: It shrinks 'slowly'
10. Sign off :-D
[1] This is a problem in certain scenarios, not in all of them. This is
the best I come up with in order to test the behavior change. Anyone can
set a new size of their taste.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
If using ES, the script will collect the newly created records and biblionumbers in atwo arrays (@search_engine_record, @search_engine_record_ids) and pass that to the indexer when $commitnum records are read. The arrays are cleaned afterwards.
BUT: The records and biblionumbers were always stored, even if we're not using Elasticsearch. And as the cleaning of those arrays only happens if we are in fact using ES, all the data is added but never cleaned.
This patch fixes this by only storing record and biblionumber if we have an $indexer (i.e. if we're using Elasticsearch).
I have no idea though how/if the records are indexed when using Zebra.
After applying this patch and importing a large enough file without using Elasticsearch, the RAM usage also stays more or less constant.
@Jan, please verify!
Signed-off-by: Jan Kissig <bibliothek@th-wildau.de>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
So, I reviewed the code, which has a bunch of issues:
* there's a call to `next RECORD`, but no `RECORD` loop label.
* $record_number is incremented twice
* and the calls to txn_begin is moved from the outside the loop into each record, starting one transaction per record. Not sure how mysql handles this, but postgres would not like it.
This patch fixes those issues:
After I fixed the first two issues, I ran the patched script on a file containing 75 authorities (from GND), and not one was added to the DB, presuambly because nested transactions don't work in mysql?
So I also moved the BEGIN transaction outside of the main loop and started a new one after each commit, and added a final commit after to loop.
Signed-off-by: Jan Kissig <bibliothek@th-wildau.de>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Include checking for invalid marc data in main loop to avoid
loading all marc data into memory at once.
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
connexion is a product, added to ignore file
tablee -> tables
a few other small corrections
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Add the option "--without_restriction_type" so that delete_patrons.pl
does not delete patrons with given restriction types.
To test:
1) Run
misc/cronjobs/delete_patrons.pl --not_borrowed_since 2010-01-01
and choose a borrowernumber that would be deleted.
2) In your browser go to
/cgi-bin/koha/members/moremember.pl?borrowernumber=<borrowernumber>
and add a manual restriction to this patron.
3) Apply the patch.
4) Run
misc/cronjobs/delete_patrons.pl --not_borrowed_since 2010-01-01 \
--without_restriction_type MANUAL
5) Check that the patron with the manual restriction would not be
deleted.
Sponsored-by: Karlsruhe Institute of Technology (KIT)
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch changes the script to use 'Koha::PseudonymizedTransaction->new_from_statistic' directly
rather than calling 'Koha::Statistic->pseudonymize()' which generates a background job.
Additionally, some of the checks from 'pseudonymnize' and the background job are moved into the search
so we don't fetch any to pseudonymize that should not be.
To test:
1 - Generate some entries in statistics (checkin, checkout, renew, etc)
2 - Enable pseudonymization (see bug 28911 if you need bcrypt settings)
3 - perl misc/maintenance/pseudonymize_statistics.pl -v -c
4 - Statistics pseudonymized
5 - Check the background jobs table, note jobs were generated
6 - Apply patch
7 - perl misc/maintenance/pseudonymize_statistics.pl -v -c
8 - You get same count as before, and are warned of existing pseudonymized transactions
9 - enter 'Y' to continue
10 - Check the pseudonymized_transacations table
11 - Confirm same rows were generated twice
12 - Check the background jobs
13 - Confirm there were no new jobs generated (for pseudonymization)
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds a new option switch `--fix` (or `-f`) to make the script
try to fix the problems it found.
Bonus: `--help` option switch added.
To test:
1. Apply this patch
2. Run:
$ ktd --shell
k$ perl -MKoha::SearchEngine -MKoha::SearchEngine::Indexer -e 'my $i = Koha::SearchEngine::Indexer->new({index => $Koha::SearchEngine::BIBLIOS_INDEX}); $i->delete_index([1]);'
=> SUCCESS: Bibliographic record #deleted from ES index
3. Run:
k$ koha-mysql kohadev
> DELETE FROM biblio WHERE biblionumber=2 OR biblionumber=3;
=> SUCCESS: Bibliographic records 2 and 3 deleted directly from the
database, no reindex triggered by Koha.
4. Run:
k$ perl misc/maintenance/compare_es_to_db.pl
=> SUCCESS: The results from (2) and (3) are explained. [1]
5. Run:
k$ perl misc/maintenance/compare_es_to_db.pl --fix
=> SUCCESS: It says it is gonna fix things
6. Repeat 4
=> SUCCESS: The problems we generated are no longer reported! They've
been fixed!
7. Sign off :-D
[1] Note KTD ships some broken records on purpose so devs need to deal
with a not-perfect DB everyday, and catch misses in the code.
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
In ktd context:
if host's node_modules ktd has been generated prior to bug 38644, the template-toolkit's
prettier plugin is not there and prettier will empty the file.
It seems to be a prettier bug, and the following behaviour is confusing:
From the host:
% yarn run prettier --write koha-tmpl/intranet-tmpl/prog/en/modules/test.tt
yarn run v1.22.22
$ prettier koha-tmpl/intranet-tmpl/prog/en/modules/test.tt
[error] Cannot find module '@koha-community/prettier-plugin-template-toolkit'
[error] Require stack:
[error] - /home/jonathan/workspace/koha/node_modules/prettier/index.js
[error] - /home/jonathan/workspace/koha/node_modules/prettier/cli.js
[error] - /home/jonathan/workspace/koha/node_modules/prettier/bin-prettier.js
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
=> The file is not modified
Within ktd:
$ yarn run prettier --write koha-tmpl/intranet-tmpl/prog/en/modules/test.tt
yarn run v1.22.22
$ prettier --write koha-tmpl/intranet-tmpl/prog/en/modules/test.tt
koha-tmpl/intranet-tmpl/prog/en/modules/test.tt 6ms
Done in 0.39s.
=> No error, and the file is emptied!
It seems that the plugin is found by prettier but not used (?)
Test plan:
1. Follow this from step 1 to 14
https://gitlab.com/koha-community/koha-testing-docker/-/issues/483
2. Apply patch
3. misc/devel/tidy.pl --no-write koha-tmpl/intranet-tmpl/prog/en/modules/intranet-main.tt
It should print the new error message
4. Make a change to a .tt file, commit
5. There will be a message about and issue with prettier. But it will
still commit. It's expect.
6. Inspect and see that the commit has the changes
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Emily Lamancusa <emily.lamancusa@montgomerycountymd.gov>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds the same logging as other cronjobs have
To test:
1 - Enable system preference CronjobLog
2 - perl misc/cronjobs/writeoff_debts.pl -v -c --added-after 2025-01-01
3 - Check the logs - no entries
4 - Apply patch
5 - Repeat 2
6 - Check the logs, two entries - one for the start and one for the end
Signed-off-by: Magnus Enger <magnus@libriotech.no>
Tested on top of bug 39236. Works as advertised.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
From bug 22435 - we need to use 'CREATE' as the account_offset type
From bug 31095 - we need to 'use Koha::Patrons;'
To test:
1 - Add a fine to a patron account
2 - perl misc/cronjobs/writeoff_debts.pl -v -c --added-after 2025-01-01
3 - Cronjob dies
4 - Apply patch
5 - Rerun
6 - Job succeeds!
7 - Confirm patron's fine is written off
8 - Sign off
Signed-off-by: Magnus Enger <magnus@libriotech.no>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Fridolin Somers
Lisette Scheer
Kyle M Hall
Tomás Cohen Arazi
David Cook
Jake Deery
Brendan Lawlor
Pedro Amorim
Emily-Rose Francoeur
Jonathan Druart
Baptiste Wojtkowski
Martin Renvoize
Katrin Fischer
Matt Blenkinsop
Matt Blenkinsop
Nick Clemens
Thomas Klausner
David Gustafsson
Raphael Straub
Emily Lamancusa