Introduces a new background job class for patron imports. enqueue() reads
the uploaded file content into the job_args and uses line count as job_size
for progress tracking. process() reconstitutes the file handle from the
stored content, delegates to Koha::Patrons::Import with step_callback wired
to $self->step, handles patron list creation, stores the import results, and
clears file_content from the stored data to free space.
Also registers patron_import in Koha::BackgroundJob::core_types_to_classes
so _derived_class can resolve the job correctly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Pedro Amorim <pedro.amorim@openfifth.co.uk>
It fixes the following warnings:
Can't use an undefined value as a HASH reference at /kohadevbox/koha/Koha/BackgroundJob.pm line 128
I started to add mock_userenv to all our tests that enqueue but finally
found a test in t/db_dependent/Koha/BackgroundJob.t that expect an
userenv not set
102 is( $job->borrowernumber, undef, 'No userenv, borrowernumber undef' );
So falling back to {} seems safe for now.
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
C4::Context->userenv returns a direct reference to the singleton
userenv hash. enqueue() was mutating this reference — deleting
session_id and adding an interface key — which corrupted the
session state for the remainder of the request.
See bug 40943 comment 32 for a detailed explanation of the issue.
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
remaning ocurrences in test files seem okay, this covers the others
Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
We should not log/leak session_id!
Failed test 'Context set from userenv + interface'
at t/db_dependent/Koha/BackgroundJob.t line 132.
Structures begin differing at:
$got->{session_id} = undef
$expected->{session_id} = Does not exist
Looks like you failed 1 test of 10.
Failed test 'enqueue() tests'
at t/db_dependent/Koha/BackgroundJob.t line 135.
Failed test 'Userenv set from job context on process'
at t/db_dependent/Koha/BackgroundJob.t line 240.
Structures begin differing at:
$got->{session_id} = undef
$expected->{session_id} = Does not exist
Looks like you failed 1 test of 5.
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
We now use background jobs triggered by Transport->store to update the
connection status.
Status is now a json blob with more details, so we update the api
definition to reflect that.
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch suggests to deal with connection issues at
Koha::BackgroundJob->connect level.
It incorrectly returned a Net::Stomp object when the connection failed.
With this patch, if the CONNECT does not return a CONNECTED frame then
Koha::BackgroundJob->connect returns nothing and the caller is not
fooled.
It also fixes the about page, we now have:
"Using SQL polling (Fallback, Error connecting to RabbitMQ)"
instead of
"Using RabbitMQ"
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This change causes es_indexer_daemon.pl to die on ERROR frames instead
of spinning in an infinite loop. It also adds a content-type STOMP frame header
to help consumers know what data type is in the MESSAGE frame body.
To test ERROR handling for es_indexer_daemon.pl:
1. Switch to using Elasticsearch
2. Change your RabbitMQ connection details in koha-conf.xml or the hard-coded ones in Koha/BackgroundJob.pm
to include an invalid password. For example, if you don't have a message_broker already defined, use the following:
<message_broker>
<password>bad</password>
</message_broker>
3. sudo koha-es-indexer --restart kohadev
4. View the errors in the logs:
tail -f /var/log/koha/kohadev/es-indexer-output.log
tail -f /var/log/koha/kohadev/es-indexer-error.log
To test MESSAGE handling for es_indexer_daemon.pl:
1. Switch to using Elasticsearch (and ensure RabbitMQ connection details are correct)
2. Change the title of a bib record to something new and unique
3. Try to search for the bib using the new title
4. Note you can find it
5. Look in the logs just to make sure there are no messages there
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Bug 35655 needed to mock the config, but instead introduced new env variables.
Test plan:
prove t/db_dependent/Koha/BackgroundJob.t
should still return green
Bonus 1: no more warning
t/db_dependent/Koha/BackgroundJob.t .. 1/6 Cannot connect to broker Failed to connect: Error connecting to not_localhost:99999: Invalid argument at /usr/share/perl5/Net/Stomp.pm line 27.; giving up at /usr/share/perl5/Net/Stomp.pm line 27.
Bonus 2: tests are passing even if you have 'message_broker' in the
config
To test this, add the following entry to $KOHA_CONf then prove again
<message_broker>
<hostname>localhost</hostname>
<port>61613</port>
<username>guest</username>
<password>guest</password>
<vhost></vhost>
</message_broker>
=> Without this patch the tests fail because the config will overwrite
the env vars.
Signed-off-by: Magnus Enger <magnus@libriotech.no>
Warning goes away and prove works even with the message_broker config.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This commit is generated using:
% perl misc/devel/tidy.pl
*within* ktd, to get the same version of perltidy than what will be used
by our CI (currently v20230309).
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Add the ability to specify STOMP connection info via environment
variables and utilize those in unit tests.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch makes the background jobs (jobs from now on) connect step not
try to connect to the STOMP broker, unless explicitly told to do so.
This is achieved by using a newly introduced system preference
`JobsNotificationMethod` which falls back to STOMP if undefined.
The "about" page now has 3 states for
+ Message broker: Using SQL polling
+ Message broker: Using SQL polling (Fallback, Error connecting to RabbitMQ)
+ Message broker: Using RabbitMQ
The goal for this patch is to stop flooding the logs on very busy sites
when Rabbit is not being used. This shouldn't predate any efforts on
making the STOMP based jobs brokering mechanism solid and usable.
To test:
1. On clean `main`, run:
$ ktd --shell
2. Open the logs on a separate terminal:
$ ktd --shell
k$ tail -f /var/log/koha/kohadev/*.log
3. Stage some MARC for import
=> SUCCESS: No warnings about STOMP
4. Import the staged records
=> SUCCESS: No warnings about STOMP
5. Stop rabbitmq:
k$ sudo service rabbitmq-server stop
6. Repeat 2-4
=> FAIL: The logs say it couldn't connect to STOMP
7. Apply this patches and run:
k$ restart_all
8a. See the "about" page, should read:
Message broker: Using SQL polling (Fallback, Error connecting to RabbitMQ)
8b. Repeat 2-4
should work with one warning in the
9. Start rabbit:
k$ sudo service rabbitmq-server start
k$ restart_all (not really necessary)
10a. See the "about" page, should read:
Message broker: Using RabbitMQ
10b. Repeat 2-4
=> SUCCESS: No warnings about STOMP
11. Change the `JobsNotificationMethod` system preference to *polling*
and stop rabbit:
k$ sudo service rabbitmq-server stop
k$ restart_all
12a. See the "about" page, should read:
Message broker: Using SQL polling
12b. Repeat 2-4
=> SUCCESS: No warnings about STOMP, not even on startup
13. Sign off :-D
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch adds a call to get_enabled_plugins before processing background jobs to ensure
that all plugin hooks are loaded and cached
To test:
1 - Install a plugin that adds new objects e.g. the Contracts plugin
https://github.com/bywatersolutions/fs-koha-plugin-contracts
or the Koha Advent plugin:
https://gitlab.com/koha-community/koha-advent/koha-plugin-fancyplugin
2 - Restart all
3 - Tail all your logs
4 - Stage and import a file containing items
5 - Note in the logs
DBI Exception: DBD::mysql::st execute failed: Lock wait timeout exceeded; try restarting transaction
6 - Apply this patch
7 - Restart all
8 - Stage and import again
9 - Success!
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
TODO - better review C4::Auth's changes. Are all the removal of
_new_userenv correct/enough?
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch adds a background job that will import a KBART file
Sponsored-by: UKHSA
Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Test plan:
Run background job tests. Especially the changed one.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Magnus Enger <magnus@libriotech.no>
All test pass when running this:
prove t/db_dependent/Koha/BackgroundJobs*
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds a new background job for pseduonymization
To test:
1 - Download the benchmarking script
2 - Ensure your koha has bcrypt_settings in koha-conf.xml
See bug 28911
3 - perl perf_check_pseudo.pl
4 - Note the slowdown after pseudonymization enabled
5 - Apply patches, restart all
6 - perl perf_check_pseudo.pl
7 - Note improvement
8 - Enable pseudonymization in sytem preferences
9 - Perform some checkouts and returns
10 - Verify the background jobs complete successfully
11 - Verify the pseudonymized_transactions table is updated correctly
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This commit is a squash of the following:
SUSHI harvesting process in the data providers class:
* Builds the URL query and requests the SUSHI service endpoint
* Parses the JSON response and builds the csv COUNTER file and adds it to counter_files table
Usage statistics data processing:
* When a counter_files entry is stored, CounterFile.pm will:
* Parse the csv COUNTER file and
* Add a usage_titles entry for each unique title in the COUNTER file
* Add the title's respective erm_usage_mus (monthly usage) entries, repeating for each metric_type
* Add the title's respective erm_usage_yus (yearly usage) entries, repeating for each metric_type
Harvesting cronjob;
'Run now':
* API endpoint to start the harvesting process of a data provider
* Button in the data providers list to run the harvesting process for each data provider upon clicked
ERM SUSHI: Background job
Job progress is updated to total amount of usage titles after retrieving
the response from SUSHI;
Job warning and success messages are added accordingly
Redundant duplicate titles will not be added
Redundant duplicate monthly and yearly usage statistics will not be added
Data provider harvest background job harvests once per report_type
Enqueue one background job for each report_type in the usage data provider
Update the way we measure progress in the background job.
It now uses the COUNTER report body rows instead of SUSHI response results.
We're now incrementing and showing the number of skipped mus, skipped
yus, added mus and added yus
There's a bug in the way we calculate yus
Updates to background job progress bar - Depends on 34468
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Test plan:
NOTE: It is very hard to add a Koha unit test for adding this
single header when communicating to RabbitMQ via Stomp plugin.
When we would mock the send, we are only testing if perl can
pass a hashref to a subroutine ;)
Do NOT yet apply this patch.
Make sure that RabbitMQ runs.
Stop the koha-worker for long_tasks:
koha-worker --stop --queue long_tasks myclone
Stage a MARC file.
Check queues with rabbitmqctl list_queues.
Look for: koha_myclone-long_tasks 1 (at least 1)
Stop rabbitmq (something like /etc/init.d/rabbitmq-server stop)
Start rabbitmq (/etc/init.d/rabbitmq-server start)
Check queue again with with rabbitmqctl list_queues.
Look for: koha_myclone-long_tasks 0
Your messages have been gone.
Now apply this patch.
Reiterate the former steps. But note that you will still see
a non-empty queue in the last step:
koha_myclone-long_tasks 1 (at least 1)
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch simply ensures we return the job id even if we couldn't send
the job to rabbit.
To test:
1 - In KTD: sudo service rabbitmq-server stop
2 - sudo koha-worker --stop kohadev; sudo koha-worker --queue long_tasks --stop kohadev
3 - In staff client enqueue a batch item modification - note the link to view the enqueued job has no id in the url and takes you to the main background jobs viewer
4 - Run t/db_dependent/Koha/BackgroundJob.t and notice that it fails.
5 - Apply patch
6 - Restart all
7 - Enqueue a new batch item modification
8 - Confirm the link works and has the correct id
9 - Run t/db_dependent/Koha/BackgroundJob.t; it should pass now.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Regression of bug 28413, bug 30410 added a "use Koha::Plugins" statement at the top of Koha::BackgroundJob, and so all Koha module are loaded by the worker on startup.
See bug 28413 for more info
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This is meant as a quick fix. It should be enhanced later on by
letting Net::Stomp do the UTF8 conversion.
Test plan:
Run batch mod items. Add a Unicode char in the notes.
Check if job runs and results are correct.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Add the ability to create new titles and attach them to a package.
The MARC to KBART2 mapping is the following (based on
https://github.com/adambuttrick/marc_to_kbart/blob/master/convert.py):
publication_title = biblio.title
print_identifier = 020$a||020$z||022$a||022$y
online_identifier = 020$a||020$z||022$a||022$y
date_first_issue_online = 866$a (before '-')
date_last_issue_online = 866$a (after '-')
num_first_vol_online = 863$a (before '-')
num_last_vol_online = 863$a (after '-')
num_first_issue_online = ?
num_last_issue_online = ?
title_url = 856$u
first_author = biblio.first_author
embargo_info = ?
coverage_depth = title_url ? 'fulltext' : 'print'
notes = $852$z
publisher_name = 260$b
publication_type = ?
date_monograph_published_print = ?
date_monograph_published_online = ?
monograph_volume = ?
monograph_edition = ?
first_editor = ?
parent_publication_title_id = ?
preceeding_publication_title_id = ?
access_type = ?
Note that title is not created (and so the resource) if a title from
this package already has a link to this bibliographic record.
Is that correct, or should we create another resource?
Should the import screen also have "start date" and "end date" to set for the
resource?
QA note: Ideally we would like to fetch the list from the REST API but the routes
are not there yet.
Signed-off-by: Jonathan Field <jonathan.field@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
[1] Correct: BackgrounJob
[2] If should filter out not current jobs
=> Had a hard time reading that one until I replaced if by it.
=> Decided to rephrase it in a more positive way.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch makes the following changes to the 'background_jobs' API:
* We now call them 'jobs'
* Removed deprecated query parameter definitions
* Added only_current query parameter
* Controller gets adapted to use $rs->filter_by_current when
only_current is passed
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Making the disabling utf8 flag explicit instead of depending on
the default of the CPAN module.
Incorporating the change in background_jobs_worker too.
Test plan:
See next patches when we look at unit tests.
Restart koha-worker.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Our database connections have been set up so that they will
automatically convert perl encoded strings to utf8 encoded strings
when sending data to database tables and decode the utf8 encoded
strings from the tables back to internal perl strings. As we can see
from a call in Koha/BackgroundJob.pm we are encoding the perl internal
string to utf8 and then decoding it back to perl internal string:
my $data_dump = decode_json encode_utf8 $self->data;
We can skip this unnecessary encode<->decode step (as the database
object has done the decoding for us) by simply calling the
JSON->new->decode() method which doesn't perform any string decoding.
Furthermore, the original code was buggy and didn't always remember to
encode the unencoded strings, in Koha::BackgroundJob::process we can
see
my $context = decode_json($self->context);
is missing the encode step. Now after this change encoding before
decoding is not necessary as we are using the methods from the JSON
module that do not perform any transcoding.
Note to those concerned whether the old data in the database is
compatible with this new code:
Luckily our database connection object seems to be smart and
didn't utf8 encode the utf8 returned data from the old encode_json()
calls (probably checks the utf8 flag for the string (Encode::is_utf8($str))).
To test whether this fixes the original bug reported of not being able
to schedule background jobs with koha user having non-ASCII letters in
their surname:
1) Change your staff users surname/lastname to "ääää"
2) Log out and back in.
3) Go to a biblio record detail page and click "Select all" in the
items table
4) Click Delete selected items and proceed with the deletion
5) Notice the batch item deletion job has failed status
6) Apply patch and repeat but this time the deletion job should finish.
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch makes the code skip calculating the plugin-related task
codes => class mapping if plugins are entirely disbaled.
To test:
1. Apply the regression tests
2. Run:
$ kshell
k$ prove t/db_dependent/Koha/Plugins/BackgroundJob.t
=> FAIL: Tests explode with: Can't call method "GetPlugins" on an
undefined value at... Koha/BackgroundJob.pm line 424.
3. Apply this patch
4. Repeat 2
=> SUCCESS: Tests pass!
5. Sign off :-D
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch records the current context to the background_jobs table at
enqueue time and then uses that record to set the context at process
time.
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Added test for empty string or undefined. We could still consider
evaling the json operations.
Test plan:
Run t/db_dependent/Koha/BackgroundJob.t
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This patch adds a background job that takes care of updating the holds
queue rows for a given list of biblios.
It is designed to be used for real-time batch updating the holds queue
when required. Tasks would be added by the relevant methods under the
right use cases.
Sponsored-by: Montgomery County Public Libraries
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This patch adds high-level methods to handle the JSON payload
(de)serialization as needed. This allows devs implementing background
jobs to abstract themselves from the internals of the jobs handling.
To test:
1. Apply this patch
2. Run:
$ kshell
k$ prove t/db_dependent/Koha/BackgroundJob.t
=> SUCCESS: Tests pass!
3. Sign off :-D
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch adds some helper methods to ease writing background jobs, and
also making it more consistent/solid.
To test:
1. Apply this patch
2. Run:
$ kshell
k$ prove t/db_dependent/Koha/BackgroundJob.t
=> SUCCESS: Tests pass. The methods have the expected behaviour.
3. Sign off :-D
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch adds a background job submodule, UpdateElasticIndex, to deal
with async ES index update (not the deletion).
Using NYTProf (on a checkin):
Without
618ms, executing 35676 statements and 26355 subroutine calls in 266 source files and 83 string evals.
With
521ms, executing 13282 statements and 7979 subroutine calls in 195 source files and 26 string evals.
However there are some problems with this patch:
1. We don't want *all* the index update to be in the background_jobs
tabtle (we could add a filter on the list view)
2. We don't track the "progress" of the job as we are sending all the
records to Elastic. It is okish in my opinion but it must be noted.
Signed-off-by: Arthur Suzuki <arthur.suzuki@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This patch implements a mechanism for plugins to tell Koha they
implement background jobs, by returning a mapping
```
task => class
```
* _task_ is a string that will be used when queueing new tasks, to let
the workers know which class to instantiate to launch the job (a.k.a.
process).
* _class_ is a string for a class name.
For this to work, plugins need to include the 'namespace' attribute in
their metadata. If they don't, they will be skipped when listing the
valid _background jobs_.
After this, high-level methods for letting plugins (easily) enqueue
their tasks will be provided.
To test:
1. Apply this patch
2. Run:
$ kshell
k$ prove t/db_dependent/Koha/Plugins/BackgroundJob.t
=> SUCCESS: Tests pass!
3. Sign off :-D
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This patch adds a new column background_jobs.queue, which default to
'default'
By default, new jobs are enqueued into this default queue, and the
background job worker will subscribe to the default queue unless told
otherwise by the --job-queue option
The new job UpdateElasticIndex is automatically enqueued in another
queue named 'index'
So you can have 'default' worker with
misc/background_jobs_worker.pl
and a dedicated indexing worker with
misc/background_jobs_worker.pl --queue index
This is to address bug 27344 comment #15
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This patch makes ->enqueue set undef if no userenv is provided. The CLI
scripts will usually use Koha::Script and have this set to undef as
well, but on the tests, there's usually no userenv set unless we need to
test it. And as we are adding triggers everywhere, that will schedule
background jobs, it means we would end up mocking userenv or the
->enqueue method everywhere and it will get messy.
This patch just make it set undef.
To test:
1. Apply the regression tests patch
2. Run:
$ kshell
k$ prove t/db_dependent/Koha/BackgroundJob.t
=> FAIL: Wow, it explodes badly. Same as any test that happens to
trigger a background job enqueuing.
3. Apply this patch
4. Repeat 2
=> SUCCESS: Great, tests pass!
5. Sign off :-D
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Arthur Suzuki <arthur.suzuki@biblibre.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
As we are sending the job to the rabbitmq before in the transaction, the
worker can receive the job to process before the transaction committed.
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Pedro Amorim
Jonathan Druart
Nick Clemens
Kyle M Hall
Tomas Cohen Arazi
Martin Renvoize
David Cook
Tadeusz „tadzik” Sośnierz
Koha Development Team
Matt Blenkinsop
Marcel de Rooy
Pedro Amorim
Joonas Kylmälä
Julian Maurice