Commit Graph

97 Commits

Author SHA1 Message Date
Marcel de Rooy 9af1b34508 Bug 35277: (QA follow-up) POD corrections
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
(cherry picked from commit 4d3c838093)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
2024-02-09 09:08:05 +01:00
Kyle M Hall 14f7caffee Bug 35277: (QA follow-up) Fix POD
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
(cherry picked from commit cf84410ce5)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
2024-02-09 09:08:05 +01:00
Nick Clemens 3d8fd974a9 Bug 35277: Pseudonymize in a background job
This patch adds a new background job for pseduonymization

To test:
 1 - Download the benchmarking script
 2 - Ensure your koha has bcrypt_settings in koha-conf.xml
     See bug 28911
 3 - perl perf_check_pseudo.pl
 4 - Note the slowdown after pseudonymization enabled
 5 - Apply patches, restart all
 6 - perl perf_check_pseudo.pl
 7 - Note improvement
 8 - Enable pseudonymization in sytem preferences
 9 - Perform some checkouts and returns
10 - Verify the background jobs complete successfully
11 - Verify the pseudonymized_transactions table is updated correctly

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
2024-02-09 09:08:05 +01:00
Marcel de Rooy 73577f17ad Bug 35843: Correct invalid exception
Test plan:
Run background job tests. Especially the changed one.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Magnus Enger <magnus@libriotech.no>
All test pass when running this:
prove t/db_dependent/Koha/BackgroundJobs*
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
(cherry picked from commit 291268f318)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
2024-01-31 13:51:08 +01:00
Kyle M Hall 8d58f0231c Bug 35438: Remove skip_intermediate_commit parameter
It no longer does anything.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
(cherry picked from commit 067034f9e0)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
2024-01-17 10:30:13 +01:00
Pedro Amorim d55ea648ac
Bug 34587: Improve SUSHI COUNTER error handling
This is doing a few things, checking for sushi errors:
1) Abort background job and provide error message if SUSHI returned
   {Severity} in response, this means SUSHI error, docs here:
https://app.swaggerhub.com/apis/COUNTER/counter-sushi_5_0_api/5.0.2#/SUSHI_error_model

2) Abort background job and provide error message if SUSHI provided
   proper response, but contains Exceptions in Report_Header

3) Abort background job and provide error message if SUSHI returned 0
   report items (e.g. everything is fine but there are no items for the
   harvest period provided

4) Abort background job and provide error message if COUNTER file
   created is larger than allowed by the database.

5) Provide error message if manual upload file size exceeds the max
   allowed by the database

Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-10-31 16:46:25 -03:00
Pedro Amorim e629e37bd7
Bug 34587: Fix total added objects in job report
Remove obsolete total_records code, this was a leftover from when this code was in UsageDataProvider.pm
Add a new job_callback 'added_usage_objects' to handle total objects added (titles, platforms, databases, items)
Update job report include file

Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-10-31 16:46:24 -03:00
Pedro Amorim ca8ea02fda
Bug 34587: Move everything into a ERM/EUsage subfolder
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-10-31 16:46:18 -03:00
Pedro Amorim cf87082682
Bug 34587: File upload - Background job
ErmSushiHarvester background job will now be initiated by either
enqueue_counter_file_processing_job or by enqueue_sushi_harvest_jobs
from Koha/ERM/UsageDataProvider.pm, the former if triggered by a manual
file upload, the latter if by the 'run now' button or by the cron script.

This commit also includes some rewording/refactoring, namely:
- COUNTER file validation now happens in the API, before enqueuing the job.
- Removal of no longer used POST /erm/counter_files endpoint
- Koha/ERM/UsageDataProvider.pm:
-- run method is now enqueue_sushi_harvest_jobs
-- new enqueue_counter_file_processing_job method
-- harvest method is now harvest_sushi
-- new set_background_job_callbacks method to set the background job callbacks
- REST/V1/ERM/UsageDataProviders
-- run method is now process_SUSHI_response
-- new process_COUNTER_file endpoint

Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-10-31 16:46:15 -03:00
Pedro Amorim 16592d267a
Bug 34587: Accept date inputs upon confirming 'run now' button
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-10-31 16:46:13 -03:00
Pedro Amorim 2951edc697
Bug 34587: SUSHI COUNTER Harvester
This commit is a squash of the following:

SUSHI harvesting process in the data providers class:
* Builds the URL query and requests the SUSHI service endpoint
* Parses the JSON response and builds the csv COUNTER file and adds it to counter_files table

Usage statistics data processing:
* When a counter_files entry is stored, CounterFile.pm will:
* Parse the csv COUNTER file and
* Add a usage_titles entry for each unique title in the COUNTER file
* Add the title's respective erm_usage_mus (monthly usage) entries, repeating for each metric_type
* Add the title's respective erm_usage_yus (yearly usage) entries, repeating for each metric_type

Harvesting cronjob;

'Run now':
* API endpoint to start the harvesting process of a data provider
* Button in the data providers list to run the harvesting process for each data provider upon clicked

ERM SUSHI: Background job

Job progress is updated to total amount of usage titles after retrieving
the response from SUSHI;
Job warning and success messages are added accordingly
Redundant duplicate titles will not be added
Redundant duplicate monthly and yearly usage statistics will not be added
Data provider harvest background job harvests once per report_type

Enqueue one background job for each report_type in the usage data provider

Update the way we measure progress in the background job.
It now uses the COUNTER report body rows instead of SUSHI response results.
We're now incrementing and showing the number of skipped mus, skipped
yus, added mus and added yus
There's a bug in the way we calculate yus
Updates to background job progress bar - Depends on 34468

Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-10-31 16:45:58 -03:00
Matt Blenkinsop 7d2ce611f7
Bug 34789: Update usage throughout Koha modules and Vue components
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-09-18 12:06:18 -03:00
Matt Blenkinsop 38fc85ade9
Bug 33755: Set profile_id and add missing import
This patch adds a missed parameter (profile_id) in stage-marc-import.pl and also adds a missing import in the background job for staging marc files. This means that the profile used when importing can now be shown in the import batches table and also in batch details

Test plan:
1) Stage a record for import and make sure to save the import profile you use
2) Import a record using that profile
3) Navigate to Cataloging > Manage staged MARC records
4) In the Profile column, no value will be shown for the staged record
5) Click on the batch, the profile will be missing in this screen as well
6) Apply patch
7) Restart_all
8) Repeat steps 2-5, the profile name should be visible in both places

Signed-off-by: Sam Lau <samalau@gmail.com>
Signed-off-by: Emmanuel Bétemps <e.betemps@gmail.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-08-15 12:16:59 +03:00
Marcel de Rooy f7b1c85f73
Bug 33576: (QA follow-up) Polish comment, typo
No test plan.

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-05-09 10:58:06 -03:00
Jonathan Druart 17be09b903
Bug 32782: Koha to KBart mapping for UNIMARC
Used when creating eholdings from Koha items.

Test plan:
Add some items to a list, then use this list to create eholdings
/cgi-bin/koha/erm/eholdings/local/titles/import

Signed-off-by: Laurence Rault <laurence.rault@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-05-05 09:13:19 -03:00
Tomas Cohen Arazi 8fcc61e82e
Bug 33486: POD fix
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-04-13 11:47:51 -03:00
Nick Clemens 42c1059d1d
Bug 33486: Remove Koha::BackgroundJob::UpdateElasticIndex->process
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-04-13 11:47:51 -03:00
Jonathan Druart 71bf715596
Bug 32781: Prevent import from list to fail if package does not exist
This is theorical (caught when writting tests) but we need to deal with
it.

Can't call method "resources" on an undefined value at /kohadevbox/koha/Koha/BackgroundJob/CreateEHoldingsFromBiblios.pm line 94.

Test plan:
  prove t/db_dependent/Koha/BackgroundJob/CreateEHoldingsFromBiblios.t
must return green

Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-03-13 15:56:41 -03:00
Jonathan Druart 7076e4f087
Bug 30943: Adjust CreateEHoldingsFromBiblios
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-03-13 15:52:24 -03:00
Tomas Cohen Arazi 632a43362f
Bug 30943: Simplify background jobs code using helpers
This patch makes the current background jobs classes use the added
helpers in order to simplify them.

No behavior change is expected.

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Rebased-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-03-13 15:52:19 -03:00
Nick Clemens aba2453ad6
Bug 32594: Add a dedicated Elasticsearch biblio indexing background queue
Currently we generate large numebrs if single record reindex for circulation and other
actions. It can take a long time to process these as we need to load the ES settings for each.

This patch updates the Elasticsearch background jobs to throw records into a new queue
that can be processed by it's own worker and adds a dedicated worker that batches the jobs
every 1 second.

To test:
1 - Apply patches, set SearchEngine system preference to 'Elasticsearch'
2 - perl misc/search_tools/es_indexer_daemon.pl
3 - Leave the running in terminal and perform actions in staff interface:
    - Checking out a bib
    - Returning a bib
    - Editing a single bib
    - Editing a single item
    - Batch editing bibs
    - Batch editing items
4 - Confirm for each action that records are updated in search/search results
5 - Stop the script
6 - set SearchEngine system preference to 'Zebra'
7 - perl misc/search_tools/es_indexer_daemon.pl
8 - Script dies as Elasticsearch not enabled

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Emily Lamancusa <emily.lamancusa@montgomerycountymd.gov>

Bug 32594: (follow-up) Adjust logging per bug 32612

JD amended patch: tidy! There were tabs here...

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-02-24 17:52:20 -03:00
Nick Clemens 78e144a9a0
Bug 32019: Add option to mark items returned via batch modification
Ther are options that allow avoiding marking an issue returned when se tot lost.
Libraries do this to tie a lost item to a patron clearly.

Eventually, after the patron has paid, or after some time, the library may wish
to delete the item. When doing so, they do not wish to check the item in and affect any statuses
or fines. This is also true for withdrawn items

It would be helpful to have a way to mark issues returned in bulk, when they have not been marked
returned in the past

To test:
 1 - Apply patches
 2 - Restart all
 3 - Set 'MarkLostItemsAsReturned' system preferences to 'None'
 4 - Check out some items that have replacement fees set
 5 - Mark them lost
 6 - Check out more items, mark them withdrawn
 7 - Set system preference 'BlockReturnOfWithdrawnItems' to block
 8 - Enter item barcodes into batch modification
 9 - Leave 'Mark items returned as blank
10 - Verify nothing was checked in
11 - Repeat but set 'Mark items as returned' to 'No'
12 - Verify nothing was checked in
13 - Repeat, but set 'Mark items as returned' to 'Yes'
14 - Verify items are returned
15 - Verify no fines have been adjusted
16 - Verify withdrawn items are sitll withdrawn
17 - Verify you can now delete the items

Signed-off-by: Andrew Fuerste-Henry <andrewfh@dubcolib.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-02-20 09:54:20 -03:00
Nick Clemens e5f251a709
Bug 15869: Change framework on overlay
This patch allows for selection of framework to use when overlaying
records - by default it is set to keep the initial framework

To test:
 1 - Create some records using one framework
 2 - Export the records
 3 - Edit the records to add fields not in original framework
 4 - Stage records using a rule that will find matches
 5 - Import
 6 - Note records contain new fields on display, but they are lost on edit
 7 - Apply patch
 8 - Stage records again
 9 - Select a framework that contains the new fields on import
10 - Import records
11 - Note records now use selected framework and are displayed/edited
correctly

Signed-off-by: Andrew Fuerste-Henry <andrewfh@dubcolib.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-02-07 10:32:22 -03:00
Jonathan Druart d8022865ca Bug 32779: ERM - Fix import from list
There is a typo, '$' is missing, which caused the following error:
Use of inherited AUTOLOAD for non-method Koha::BackgroundJob::CreateEHoldingsFromBiblios::fix_coverage() is no longer allowed at /kohadevbox/koha/Koha/BackgroundJob/CreateEHoldingsFromBiblios.pm line 126

Test plan:
Create a list, add an item
Turn on ERM
Create a package
Go to /cgi-bin/koha/erm/eholdings/local/titles
Click "Import from list"
Select the package, then your list, and import
It should succeed

Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-02-02 11:59:26 -03:00
Tomas Cohen Arazi 3dcdd6db78
Bug 32394: (follow-up) Add param for Koha::BackgroundJob::StageMARCForImport->enqueue
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-12-16 09:00:45 -03:00
Jonathan Druart 9810fc5e0e
Bug 32394: Restore long tasks queue
We are sending "queue" as parameter, but "job_queue" is retrieved in
Koha::BackgroundJob->enqueue...

Test plan:
Read code and confirm the above.
More fun for test plan will be found on bug 32393

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-12-14 14:32:27 -03:00
Jonathan Druart 94c67c439e
Bug 32030: Create eHolding titles from a list
Add the ability to create new titles and attach them to a package.
The MARC to KBART2 mapping is the following (based on
https://github.com/adambuttrick/marc_to_kbart/blob/master/convert.py):

publication_title = biblio.title
print_identifier  = 020$a||020$z||022$a||022$y
online_identifier = 020$a||020$z||022$a||022$y
date_first_issue_online = 866$a (before '-')
date_last_issue_online  = 866$a (after '-')
num_first_vol_online    = 863$a (before '-')
num_last_vol_online     = 863$a (after '-')
num_first_issue_online  = ?
num_last_issue_online   = ?
title_url = 856$u
first_author = biblio.first_author
embargo_info = ?
coverage_depth = title_url ? 'fulltext' : 'print'
notes = $852$z
publisher_name = 260$b
publication_type = ?
date_monograph_published_print = ?
date_monograph_published_online = ?
monograph_volume = ?
monograph_edition = ?
first_editor = ?
parent_publication_title_id = ?
preceeding_publication_title_id = ?
access_type = ?

Note that title is not created (and so the resource) if a title from
this package already has a link to this bibliographic record.
Is that correct, or should we create another resource?

Should the import screen also have "start date" and "end date" to set for the
resource?

QA note: Ideally we would like to fetch the list from the REST API but the routes
are not there yet.

Signed-off-by: Jonathan Field <jonathan.field@ptfs-europe.com>

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-11-08 09:44:20 -03:00
Tomas Cohen Arazi d3798e778e
Bug 27421: MARC staging/import are long-running tasks
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-09-19 15:33:56 -03:00
Marcel de Rooy 5b6d1e99a2
Bug 27421: (QA follow-up) Polishing the backgroundjob modules
StageMARCForImport:
- Rollback in catch
- Setting progress, size or status after BatchStageMarcRecords
  in both try and catch block

ImportCommitBatch:
- Move setting size back to enqueue moment
- Rollback in catch
- Setting progress, size or status after BatchStageMarcRecords
  in both try and catch block

ImportRevertBatch:
- Move setting size back to enqueue moment
- Adding transaction/rollback to module since import routine
  does not support it. Could be moved later.
- Setting progress, size or status after BatchStageMarcRecords
  in both try and catch block

Test plan:
Run t/db_dependent/Koha/BackgroundJobs/StageMARCForImport.t
Test staging file
Bonus: Put a die statement in BatchStageMarcRecords.

Test importing batch
Bonus: Include some records with an invalid library code; this will
trigger an FK exception. (Reduce the progress from 50 to 1. If your
first record would be fine, check if it is NOT imported when the job
fails.)

Test reverting batch.
Bonus: Put a die in BatchRevertRecords.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-09-19 15:14:59 -03:00
Marcel de Rooy bdb107bcec
Bug 27421: (QA follow-up) BatchCommitImportRecords needs param for skipping commits
When you submit a background jobs, and it fails, you do not expect
partial results in the database.
Note that when the Background feature would support a partially
completed status, things might change again.

Note that the >0 test was superfluous if you check for ^\d+$.

Test plan:
Run t/db_dependent/Koha/BackgroundJobs/StageMARCForImport.t

Note: This serves to verify that it still runs as expected.
The test plan of the following patch covers the new param.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-09-19 15:14:59 -03:00
Marcel de Rooy 52dc337ced
Bug 27421: (QA follow-up) Similar changes for revert commit
Test plan:
Look at job results when reverting imported batch.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-09-19 15:14:58 -03:00
Marcel de Rooy 7b355ee54b
Bug 27421: (QA follow-up) Include manage URL and item counts for import commit
Test plan:
Run an stage and import.
Check import commit job.
Click to managed batch.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-09-19 15:14:58 -03:00
Jonathan Druart a25feb05c9
Bug 27421: Fix job's size for import
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-09-19 15:14:57 -03:00
Jonathan Druart 9470cf8c1b
Bug 27421: Commit and revert
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-09-19 15:14:56 -03:00
Jonathan Druart 8497ed67b7
Bug 27421: Use Background job for staging MARC records for import
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-09-19 15:14:55 -03:00
Marcel de Rooy 202db89f83
Bug 31351: (QA follow-up) Use $self->json in Background modules
Making the disabling utf8 flag explicit instead of depending on
the default of the CPAN module.

Incorporating the change in background_jobs_worker too.

Test plan:
See next patches when we look at unit tests.
Restart koha-worker.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-09-07 13:59:03 -07:00
Joonas Kylmälä d95805c6aa
Bug 31351: Koha::BackgroundJob: Let database connection object handle utf8 transcoding
Our database connections have been set up so that they will
automatically convert perl encoded strings to utf8 encoded strings
when sending data to database tables and decode the utf8 encoded
strings from the tables back to internal perl strings. As we can see
from a call in Koha/BackgroundJob.pm we are encoding the perl internal
string to utf8 and then decoding it back to perl internal string:

my $data_dump = decode_json encode_utf8 $self->data;

We can skip this unnecessary encode<->decode step (as the database
object has done the decoding for us) by simply calling the
JSON->new->decode() method which doesn't perform any string decoding.

Furthermore, the original code was buggy and didn't always remember to
encode the unencoded strings, in Koha::BackgroundJob::process we can
see

my $context = decode_json($self->context);

is missing the encode step. Now after this change encoding before
decoding is not necessary as we are using the methods from the JSON
module that do not perform any transcoding.

Note to those concerned whether the old data in the database is
compatible with this new code:
 Luckily our database connection object seems to be smart and
 didn't utf8 encode the utf8 returned data from the old encode_json()
 calls (probably checks the utf8 flag for the string (Encode::is_utf8($str))).

To test whether this fixes the original bug reported of not being able
to schedule background jobs with koha user having non-ASCII letters in
their surname:
 1) Change your staff users surname/lastname to "ääää"
 2) Log out and back in.
 3) Go to a biblio record detail page and click "Select all" in the
 items table
 4) Click Delete selected items and proceed with the deletion
 5) Notice the batch item deletion job has failed status
 6) Apply patch and repeat but this time the deletion job should finish.

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-09-07 13:58:58 -07:00
Tomas Cohen Arazi 12bc623ae7
Bug 31245: Do not try to report results if task not started
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Andrew <andrewfh@dubcolib.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-08-16 09:09:17 -03:00
Jonathan Druart 70d61d80fb
Bug 29697: Replace GetMarcBiblio occurrences with $biblio->metadata->record
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

JD Amended patch:
-# FIXME Special case here

-    print "Biblio not found\n,";
+    print "Biblio not found\n";

- my $biblio = Koha::Biblio->find($hostbiblionumber);
+ my $biblio = Koha::Biblios->find($hostbiblionumber);

Rebased-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-07-22 15:24:11 -03:00
Jonathan Druart 85dc90c151
Bug 30974: Set size for indexing jobs to 1
job's size is the number of records to index, but when finished the progress is always set to 1.

Wich leads to: finished 1/42

Test plan:
Don't apply this patch
Use the batch item modification tool to update more than 1 item
Notice that the new indexing job has a size of the number of items you
modified
When finished, the job detail shows "Finished. 1/X"
Apply this patch, repeat
Notice that now the progress shows 0/1 before the job is done, then 1/1
when finished

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-06-16 09:41:32 -03:00
Tomas Cohen Arazi af1f8d34db Bug 30728: Only trigger real-time holds queue update if enabled
This patch makes the places in which Koha enqueues holds queue for
real time updates verify the feature is enabled.

To test:
1. Apply this patches up to the unit tests
2. Run:
   $ updatedatabase
   $ kshell
  k$ git diff origin/master --name-only | grep -e '\.t$' | xargs prove
=> FAIL: tests fail, the code doesn't care about the syspref
3. Apply this patch
4. Repeat 2
=> SUCCESS: Tests pass!
5. Be happy!
6. Sign off :-D

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
2022-05-12 22:17:46 -10:00
Tomas Cohen Arazi a3ac2a20dd Bug 30727: Avoid holds queue updates multiple times on BatchDeleteBiblio
This patch makes the background job only enqueue holds queue updates
once per biblio, when appropriate.

To test:
1. Apply the regression tests patch
2. Run:
   $ kshell
  k$ prove t/db_dependent/Koha/BackgroundJobs/BatchDeleteBiblio.t
=> FAIL: It enqueues 3 times for a bib!
3. Apply this patch
4. Repeat 2
=> SUCCESS: It only enqueues once!
5. Sign off :-D

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
2022-05-12 22:17:45 -10:00
Nick Clemens a3649905df Bug 30710: Build holds queue once per biblio when batch deleting items
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
2022-05-10 23:09:09 -10:00
Nick Clemens 5d50e73652 Bug 30467: (QA follow-up) Fix copy paste error to update updated_biblionumbers
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
2022-05-06 10:33:09 -10:00
Martin Renvoize 62277d7096 Bug 30467: (QA follow-up) Restore indexing for item only deletion
This patch restores indexing for item only bulk deletions.

JD amended patch:
* remove trailing whitespaces
* Move if inside first condition

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
2022-05-06 10:33:09 -10:00
Jonathan Druart 1c7fd5a588 Bug 30467: Don't send the delete request for each item
We are sending the request when all the items have been deleted.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
2022-05-06 10:33:08 -10:00
Jonathan Druart 734a02cf82 Bug 30465: Make BatchUpdateBiblio update the index in one request
When using the batch record modification tool to modify several
bibliographic records, we don't want to send one index request per
biblio, we want to index them all on the fly after the records have been
modified.
Otherwise we will end up with one task per record, and records will be
indexed in background.

Test plan:
Use the batch mod tool to modify bibliographic records and confirm the
behaviour is correct.

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
2022-05-05 11:17:37 -10:00
Jonathan Druart c04782e1cc Bug 30464: Make BatchUpdateAuthority update the index in one request
Same as bug 30465 for authorities

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
2022-05-05 11:17:36 -10:00
Jonathan Druart 471850e495 Bug 30460: Make BatchDeleteBiblio update the index in one request
No need to send 1 indexation request per item + 1 per biblio.

Test plan:
Confirm that the batch biblio deletion tool still works correctly

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
2022-05-05 11:17:36 -10:00
Jonathan Druart 9dc00051c7 Bug 30459: Make BatchDeleteAuthority update the index in one request
Same as bug 30460 for authorities

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
2022-05-05 11:17:36 -10:00