This patch implements a mechanism for plugins to tell Koha they
implement background jobs, by returning a mapping
```
task => class
```
* _task_ is a string that will be used when queueing new tasks, to let
the workers know which class to instantiate to launch the job (a.k.a.
process).
* _class_ is a string for a class name.
For this to work, plugins need to include the 'namespace' attribute in
their metadata. If they don't, they will be skipped when listing the
valid _background jobs_.
After this, high-level methods for letting plugins (easily) enqueue
their tasks will be provided.
To test:
1. Apply this patch
2. Run:
$ kshell
k$ prove t/db_dependent/Koha/Plugins/BackgroundJob.t
=> SUCCESS: Tests pass!
3. Sign off :-D
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This patch adds a new column background_jobs.queue, which default to
'default'
By default, new jobs are enqueued into this default queue, and the
background job worker will subscribe to the default queue unless told
otherwise by the --job-queue option
The new job UpdateElasticIndex is automatically enqueued in another
queue named 'index'
So you can have 'default' worker with
misc/background_jobs_worker.pl
and a dedicated indexing worker with
misc/background_jobs_worker.pl --queue index
This is to address bug 27344 comment #15
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This patch makes ->enqueue set undef if no userenv is provided. The CLI
scripts will usually use Koha::Script and have this set to undef as
well, but on the tests, there's usually no userenv set unless we need to
test it. And as we are adding triggers everywhere, that will schedule
background jobs, it means we would end up mocking userenv or the
->enqueue method everywhere and it will get messy.
This patch just make it set undef.
To test:
1. Apply the regression tests patch
2. Run:
$ kshell
k$ prove t/db_dependent/Koha/BackgroundJob.t
=> FAIL: Wow, it explodes badly. Same as any test that happens to
trigger a background job enqueuing.
3. Apply this patch
4. Repeat 2
=> SUCCESS: Great, tests pass!
5. Sign off :-D
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Arthur Suzuki <arthur.suzuki@biblibre.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
As we are sending the job to the rabbitmq before in the transaction, the
worker can receive the job to process before the transaction committed.
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Now $self is actually an instance of the job class, there's no need to
have the job_id parameter passed, or the have the ->process method
re-fetch the object from the database.
This patch cleans things up.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This patch makes _derived_class rely on _new_from_dbic to generate a new
object with the right class, but based on the same DB row. Not an empty
one as it was before. This way we can remove some biolerplate that is
required now when writing background job classes.
To test:
1. Apply the regression tests
2. Run:
$ kshell
k$ prove t/db_dependent/Koha/BackgroundJob.t
=> FAIL: Boo, tests fail
3. Apply this patch
4. Repeat 2
=> SUCCESS: Tests pass!
5. Sign off :-D
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
It's loading the modules in RAM then forks.
Then do not need it to have them loaded in the parent process (the worker/daemon).
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Here we go!
Disclaimer: this patch is huge and does many things, but splitting it in
several chunks would be time consuming and painful to rebase. However it
adds many tests and isolate/refactor code to make it way more reusable.
This patchset will make the "batch item modification" and "batch item
deletion" features use the task queue (reminder: Since bug 28158, and so
21.05.00, we do no longer use the old "background job" functionality and
the user does not get any info about the progress of the job).
More than that, more of the code to build an item form and a list of
items is now isolated in module (.pm) and include files (.inc)
We are reusing the changes made by bug 27526 that simplifies the way we
edit/create items (no more unecessary serialization Koha > MARC > MARCXML
> XML > HTML)
New module:
* Koha::BackgroundJob::BatchDeleteItem
Subclass for process item deletion in batch
* Koha::BackgroundJob::BatchUpdateItem
Subclass for process item modification in batch
* Koha::Item::Attributes
We needed an object to represent item's attributes that are not
mapped with a koha field (aka "more subfields xml")
This module will help us to create the marcxml from a hashref and the
reverse.
* Koha::UI::Form::Builder::Item
The code that was used to build the add/edit item form is
centralised in this module. In conjunction with the
subfields_for_item BLOCK (from html_helpers.inc) it will be really
easy to reuse this code in other places where the item form is used
(acquisition and serials modules)
* Koha::UI::Table::Builder::Items
Same as previously for the table. We are now using this table from 3
different places (batch item mod, batch item del, backgroung job
detail view) and the code is only in one place.
To use with items_table_batchmod BLOCK (still from html_helpers.inc)
This patch is fixing some bugs about repeatable subfields and regex. A UI
change will reflect the limitation: if you want to apply a regex on a
subfield you cannot add several subfields for the same subfield code.
Test plan:
Prepare the ground:
- Make sure you are always using a bibliographic/item record using the framework
you are modifying!
- Add some subfields for items that are not mapped with a koha field
(note that you can use 'é' for more fun, don't try more funny
characters)
- Make some subfields (mapped and not mapped with a kohafield)
repeatable
- Add default values to some of your subfields
There are 4 main screens to test:
1. Add/edit item form
The behaviour should be the same before and after this patch.
See test plan from bug 27526.
Those 2 prefs must be tested:
* SubfieldsToAllowForRestrictedEditing
* SubfieldsToUseWhenPrefill
2. Batch modification
a. Fill some values, play with repeatable and regex.
Note that the behaviour in master was buggy, only the first value was modified by the regex:
* With subfield = "a | b"
1 value added with "new"
=> "new | b"
* With subfield = "a | b"
2 new fields "new1","new2"
=> "new2 | b"
Important note: For repeatable subfields, a regex will apply on the subfields in
the "concatenated form". To apply the regex on all the different subfields of a given
subfield code you must use the "g" modifier.
This could be improved later, but keep in mind that it's not a regression or behaviour
change.
b. Play with the "Populate fields with default values from default framework" checkbox
c. Use this tool to modify items and play with the different sysprefs that
interfer with it:
* NewItemsDefaultLocation
* SubfieldsToAllowForRestrictedBatchmod
* MaxItemsToDisplayForBatchMod
* MaxItemsToProcessForBatchMod
3. Batch deletion
a. Batch delete some items
b. Check items out and try to delete them
c. Use the "Delete records if no items remain" checkbox to delete
bibliographic records without remaining items.
d. Play with the following sysprefs and confirm that it works as
expected:
* MaxItemsToDisplayForBatchDel
e. Stress the tool: Go to the confirmation screen with items that can be
deleted, don't request the job to be processed right away, but check the
item out before.
4. Background job detail view
You must have seen it already if you are curious and tested the above.
When a new modification or deletion batch is requested, the confirmation
screen will tell you that the job has enqueued. A link to the progress
of the job can be followed.
On this screen you will be able to see the result of the job once it's
fully processed.
QA notes:
* There are some FIXME's that are not blocker in my opinion. Feel free to
discuss them if you have suggestions.
* Do we still need MaxItemsToProcessForBatchMod?
* Prior to this patchset we had a "Return to the cataloging module" link
if we went from the cataloguing module and that the biblio was deleted.
We cannot longer know if the biblio will be deleted but we could display
a "Go to the cataloging module" link on the "job has been enqueued"
screen regardless from where we were coming from.
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch allows staff patrons to cancel multiple holds in bulk.
To test:
1. Apply this patch
2. restart_all
3. In cataloge go to a book and place many holds
CHECK => Holds table shows a column of checkboxes
4. Play with checkboxes (have some fun ;-P)
CHECK => When you manually check all checkboxes, the checkbox in the
header also gets checked.
=> When you uncheck one of the checkboxes, the one in the header also gets unchecked.
=> If no checkbox is checked and you check the one in the header,
all checkboxes get checked.
=> If there are some checkboxes that are checked and others are
not, when you click on the checkbox in the header all checkboxes get
unchecked.
=> If all checkboxes are checked, when you uncheck the one in the
header, all checkboxes get unchecked.
=> Every time you play with checkboxes, the number in the button
"Cancel selected" changes.
5. Check some of the checkboxes and click on cancel selected.
SUCCESS => A background job gets fired to cancel all selected holds.
=> A message should appear with a link to the job.
6. Wait a few seconds and click on the link
SUCCESS => A message appears with the report of the execution of the
background job.
7. Grab a patron and search to hold
8. Select multiple biblios and click on "place hold for <patron>"
CHECK => After holds are confirmed, multiple holds table are shown.. one for
each record. Checkboxes work exactly the same as before, but scoped
for each individual table. Checkboxes from one table will not affect
checkboxes from other tables.
9. Repeat steps 4 to 6.
10. Check In some of the items so the get in Waiting state.
11. Update expirationdate os some of those holds and set it to
ReservesMaxPickUpDelay + 1 days earlier
NOTE => ReservesMaxPickUpDelay = 7 days by default, so sql syntax to update would be
=> update reserves set expirationdate = date_sub(expirationdate, interval 8 day) where reserve_id in (...)
12. Repeat steps 4 to 6 but in waitingreserves.pl, in both tabs.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Bug 23678: (QA follow-up) Add missing template filter
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Bug 23678: (QA follow-up) Add missing filters
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Bug 23678: (QA follow-up) Use correct indentation
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
JD amended patch: also Koha/BackgroundJob/BatchCancelHold.pm
JD Amended patch: Full rebase and adjustements made on top of bug 26080.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch changes the mapping so it is more readable, and also allows
adding things there more easily, like allowing to add code => class
mappings from plugins, when time comes.
To test:
1. Just verify things still work
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(Patch extracted from bug 28445 to make it reusable for bug 23678)
We already had the need for that, when bibliographic records are
modified in batch we wanted to add a "Add to list" feature, and so pass
a list of lists/virtual shelves to the template.
Here (in 28445) we will want to pass the infos of the items that have been modified
to display a table.
Test plan:
0. Create at least one list (virtual shelf)
1. batch update biblios
2. Go to the job detail
3. Notice that dropdown list to add the record to a list
=> No regression found!
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Same as the first patch, for authorities
Test plan:
Delete authority records using the batch record deletion tool
Confirm that the job is now delegated to the task queue and that
everything else is working as before
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch takes advantage of the task queue to delegate the batch
delete biblios tool.
Test plan:
Delete bibliographic records using the batch record deletion tool
Confirm that the job is now delegated to the task queue and that
everything else is working as before
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
On bug 17591 we discovered that there was something weird going on with
the way we export and use subroutines/modules.
This patch tries to standardize our EXPORT to use EXPORT_OK only.
That way we will need to explicitely define the subroutine we want to
use from a module.
This patch is a squashed version of:
Bug 17600: After export.pl
Bug 17600: After perlimport
Bug 17600: Manual changes
Bug 17600: Other manual changes after second perlimports run
Bug 17600: Fix tests
And a lot of other manual changes.
export.pl is a dirty script that can be found on bug 17600.
"perlimport" is:
git clone https://github.com/oalders/App-perlimports.git
cd App-perlimports/
cpanm --installdeps .
export PERL5LIB="$PERL5LIB:/kohadevbox/koha/App-perlimports/lib"
find . \( -name "*.pl" -o -name "*.pm" \) -exec perl App-perlimports/script/perlimports --inplace-edit --no-preserve-unused --filename {} \;
The ideas of this patch are to:
* use EXPORT_OK instead of EXPORT
* perltidy the EXPORT_OK list
* remove '&' before the subroutine names
* remove some uneeded use statements
* explicitely import the subroutines we need within the controllers or
modules
Note that the private subroutines (starting with _) should not be
exported (and not used from outside of the module except from tests).
EXPORT vs EXPORT_OK (from
https://www.thegeekstuff.com/2010/06/perl-exporter-examples/)
"""
Export allows to export the functions and variables of modules to user’s namespace using the standard import method. This way, we don’t need to create the objects for the modules to access it’s members.
@EXPORT and @EXPORT_OK are the two main variables used during export operation.
@EXPORT contains list of symbols (subroutines and variables) of the module to be exported into the caller namespace.
@EXPORT_OK does export of symbols on demand basis.
"""
If this patch caused a conflict with a patch you wrote prior to its
push:
* Make sure you are not reintroducing a "use" statement that has been
removed
* "$subroutine" is not exported by the C4::$MODULE module
means that you need to add the subroutine to the @EXPORT_OK list
* Bareword "$subroutine" not allowed while "strict subs"
means that you didn't imported the subroutine from the module:
- use $MODULE qw( $subroutine list );
You can also use the fully qualified namespace: C4::$MODULE::$subroutine
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
The change was not advertised and it needs to be done on a separate bug
report (+ test coverage)
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
The connection details for the message broker should be configurable.
This patch adds configuration options to koha-conf.xml. If they
are not specified, then default connection details will be used.
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Otherwise the code in the template is not handled correctly:
112 [% FOR m IN job.messages %]
m will become each key/val of the message hash:
{"type":"success","biblionumber":"1","code":"biblio_modified"}
And not the whole hash
Test plan:
Modify one biblio using the batch record modification tool
Modify two biblios using the batch record modification tool
Go to the detail view of each job and confirm that with this patch the
display is correct
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Also remove "authnotrequired => 0,"
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This cannot be:
/queue/$namespace/$job_type
I got: '/koha_kohadev/batch_biblio_record_modification' is not a valid queue destination
So keeping the dash in $namespace-$job_type
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Bug 22417: Add a note about the existence of the DB row
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Bug 22417: (follow-up) Fix the batch authority tool
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Kyle Hall
Tomas Cohen Arazi
Julian Maurice
Jonathan Druart
Agustin Moyano
David Cook