Commit Graph

120 Commits

Author SHA1 Message Date
Lucas Gass 845171e4aa
Bug 23010: Report on count of items not modified via batchMod
Signed-off-by: Emily Lamancusa <emily.lamancusa@montgomerycountymd.gov>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit ccf4bc0626)
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
2025-09-16 14:02:05 +02:00
Tomas Cohen Arazi 8b494a3789
Bug 40579: Fix CSV formula injection vulnerabilities
This patch addresses CSV formula injection vulnerabilities by ensuring
all Text::CSV instances use formula protection.

Changes made:
* Add formula => 'empty' to all Text::CSV constructors in:
  - Koha::Patrons::Import
  - Koha::BackgroundJob::ImportKBARTFile
  - Koha::ERM::EUsage::CounterFile

The 'empty' formula mode causes potentially dangerous formulas (starting
with =, +, -, @) to be treated as empty strings rather than executed,
preventing CSV injection attacks while maintaining data integrity.

Security impact:
* Prevents execution of malicious formulas in CSV imports
* Protects against data exfiltration via CSV formula injection
* Maintains backward compatibility for legitimate CSV data

To test:
1. Apply the previous patch (author test)
2. Run:
   $ ktd --shell
  k$ prove xt/author/Text_CSV_Various.t
=> FAIL: Should show 3 files without formula protection
3. Apply this patch
4. Repeat 2
=> SUCCESS: All tests should now pass
5. Try importing CSV with formula content (=cmd|"/c calc",test,data)
=> SUCCESS: Formula content is rejected/emptied, not executed
6. Import normal CSV data
=> SUCCESS: Normal data imports correctly
7. Sign off :-D

Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
2025-08-26 21:05:14 +02:00
Matt Blenkinsop e766667c91
Bug 37115: Batch delete linked serials
Test plan:
- Add a new subscription to a serial record (with the option 'create an item record' selected)
- Receive several issues and create the linked items
- Return to the bibliographic record
- Select several items
- Click to 'Delete selected items'
- Tick the box 'delete serial issues linked to items' and confirm
- The items should be deleted correctly
- Back to subscription : the issues linked to items should no longer exist

Signed-off-by: Boubacar OUATTARA <boubacar.ouattara@biblibre.com>
Signed-off-by: Nathalie <nathalie.savarit@univ-amu.fr>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit 69b1d07d76)
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
2025-08-22 09:48:39 +02:00
Pedro Amorim 54205bd6ba
Bug 39652: Rename new_from_statistic
This new method name better reflects the fact that it now creates - and stores - the PseudonymizedTransaction object

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-04-24 19:53:17 +02:00
Pedro Amorim f23632a352
Bug 39652: store PseudonymizedTransaction before adding related entries
Test plan:
Run the tests before and after applying this patch:
perl t/db_dependent/Koha/Pseudonymization.t

They should fail before this patch and pass after applying patch.

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-04-24 19:53:16 +02:00
Martin Renvoize 914022d30d
Bug 26869: Use the default framework
The unit tests highlighted that without passing an frameworkcode to the
MarcSubfieldStructures->find call we throw errors as we can't identify
the correct subfield structure rules to apply.

This patch adds the default frameworkcode, empty string, to the lookup.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-03-17 11:59:50 +01:00
Pedro Amorim bf19d27f5b
Bug 26869: Add MARC modification template mapping validation
Follow the same test plan as above, but attempt to add an item without having at least homebranch, holdingbranch and type subfields.
Notice you now get an error with an appropriate message and the item is not added.

Signed-off by: Chris Rowlands <chris.rowlands6@nhs.net>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-03-17 11:59:49 +01:00
Pedro Amorim c225641cca
Bug 26869: Add items from MARC in BatchUpdateBiblio
Test plan, clean k-t-d:

Before test, pick a record and confirm it has no items, for this test we're using biblio 79:
<staff_url>/cgi-bin/koha/catalogue/detail.pl?biblionumber=79

1) Add a new MARC modification template, visit:
<staff_url>/cgi-bin/koha/tools/marc_modification_templates.pl
2) Click 'New template', enter a name and click 'Submit'
3) Click 'New action'
4) Pick 'Add new' and enter '952' as field and 'a' as subfield, enter 'CPL' as value
5) Click 'Add action'
6) Do a batch record modification, visit:
<staff_url>/cgi-bin/koha/tools/batch_record_modification.pl
7) Select the tab 'Enter a list of record numbers' and enter '79'
8) On the template, select the template previously created. Click 'continue'
9) Click 'Modify selected records'. Click 'View detail of the enqueued job'. Notice success message is shown with no errors
10) Check the biblio, visit:
<staff_url>/cgi-bin/koha/catalogue/detail.pl?biblionumber=79
11) Notice it now has an item with homebranch 'CPL'

test with 952 but with other subfields, without subfield 'a' -> confirm 'FK constraint' error is shown, no modification made
test with more than one occurrence of 952$a -> confirm 'FK constraint' error is shown, no modification made
test with 952$a too long e.g. 'aaaaaaaa' -> confirm 'too long' error is shown
test with no 952 mapping -> confirm things work as intended
test with 952 mapping plus other tags -> confirm things work as intended

prove koha/t/db_dependent/Koha/BackgroundJobs/BatchUpdateBiblio.t
prove koha/t/db_dependent/Koha/BackgroundJob/BatchUpdateBiblio.t

Signed-off by: Chris Rowlands <chris.rowlands6@nhs.net>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-03-17 11:59:48 +01:00
Koha Development Team d659526b5a
Bug 38664: Tidy the whole codebase
This commit is generated using:
  % perl misc/devel/tidy.pl
*within* ktd, to get the same version of perltidy than what will be used
by our CI (currently v20230309).

Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-02-11 14:58:24 +01:00
Pedro Amorim 0231911c98
Bug 38782: Remove concat in background process
The ud_provider->{report_type} is only defined on ->harvest_sushi, meaning it'll never be defined here.
We don't need this anyway. The filename should be the filename of the file being manually uploaded. We only need this concat logic for COUNTER files being generated by Koha following a SUSHI harvest, and that works.
This also gets rid of a warning when running the tests

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-12-27 15:19:07 +01:00
Matt Blenkinsop af22e7fad7
Bug 34355: (QA follow-up) Make record staging code DRY
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-11-11 17:43:18 +01:00
Hinemoea Viault 929e906187
Bug 37403: Wrong progress quantity in job details
In fact the system adds a previous value to the progress. As if the system keeps a previous value and adds the correct count of records in addition.
This patch removes the previously stored value.

To test:
1. Go to Cataloging > Stage records for import
2. Choose a file with bibliographic records and click Upload file
3. In "Look for existing records in catalog?" choose a record matching rule (e.g. ISBN)
4. Click Stage for import
5. Click View detail of the enqueued job
   --> Progress show 2/1
6. Apply the patch
7. Repeat steps 1 to 4
8. Click View detail of the enqueued job
   --> Progress show 1/1

Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>

qa script:
Commit title does not start with 'Bug XXXXX: ' - d1bebb34d7

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Fixed patch subject line
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-10-18 14:26:07 +02:00
Matt Blenkinsop 04a038f6b0
Bug 36935: Add a 'die' message to file parsing
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-07-02 10:20:35 +02:00
Matt Blenkinsop 7c0ed91c80
Bug 36935: Remove warning if file parsing is successful
This patch removes the logged warning if a file parse is successful.

Test plan:
1) prove t/db_dependent/Koha/BackgroundJob/ImportKBARTFile.t
2) The output will have a logged message - 'EOF - End of data in parsing input stream'
3) Apply patch
4) Repeat step 1
5) The warning will be gone

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-07-02 10:20:34 +02:00
Matt Blenkinsop af6a0b78b7
Bug 34788: (follow-up): Fix translations
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-05-10 17:46:15 +02:00
Matt Blenkinsop d2820f57d6
Bug 34788: (QA follow-up)
This patch fixes a few things:
1) The blue dialog box now clears when navigating away from the page
2) The background job now uses skip_record_index to avoid queuing indexing jobs for every new biblio and instead queues one job at the end
3) Large files that get chunked now successfully create linked biblios if requested
4) Title matching rules have been expanded to check the package ID so that we can have duplicate titles in different packages
5) A link to the package is now included on the job report page

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-05-10 15:19:09 +02:00
Matt Blenkinsop e8763a720c
Bug 34788: Allow import of file with additional columns
This patch allows a file with additional columns to be imported. When the file is submitted, the system will enqueue the background job and send back information to the UI that certain columns have been ignored. The data will stil l import as normal but only the standard KBART columns will be parsed and imported

Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-05-10 15:19:09 +02:00
Matt Blenkinsop d31f453186
Bug 34788: Make biblio creation optional
This patch rebases in the changes from bug 36618 to make biblio creation optional

Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-05-10 15:19:08 +02:00
Matt Blenkinsop 3814437003
Bug 34788: Allow for import of CSV files
This commit allows CSV files to be imported alongside TSV files.
It also adds some performance improvements relating to the max_allowed_packet and the matching of titles, as well as some small bugfixes and unit test changes

Test plan:
1) Enable the ERM module
2) Navigate to E-resource management > eHoldings > Local > Packages
3) Create at least one package
4) Navigate to E-resource management > eHoldings > Local > Titles
5) There should be a button for "Import from KBART file"
6) Click this button
7) Select the package that you created from the dropdown and then choose your KBART file using the "Choose file" button. I have attached some example files to the bug but feel free to use your own if you have them.
8) Click Submit
9) If your file is a valid file, a background job will be queued, if not then a warning will display showing what is incorrect in your file
10) To test the file format warning, edit your file and add a random column heading into the file e.g. test_column. When you upload it, the warning should show that an invalid column "test_column" has been detected
11) Click on the background job. (If you have uploaded a very large file, the system will chunk the file into smaller pieces and create multiple background jobs)
12) It should display a progress bar followed by a report and any error messages
13) Navigate to E-resource management > eHoldings > Local > Titles and you should see your new titles.
14) Run the unit test: prove t/db_dependent/Koha/BackgroundJob/ImportKBARTFile.t

Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-05-10 15:19:08 +02:00
Matt Blenkinsop 698c1e930b
Bug 34788: Add a relationship to a package for the imported file
Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-05-10 15:19:07 +02:00
Matt Blenkinsop 1320dc2cd2
Bug 34788: Add a background job to import a KBART file
This patch adds a background job that will import a KBART file

Sponsored-by: UKHSA
Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-05-10 15:19:06 +02:00
Marcel de Rooy c317bb34c0
Bug 35979: (follow-up) Add check in ->enqueue
Foundation for removing same check elsewhere (later on).

Test plan:
Run t/db_dependent/Koha/BackgroundJob/BatchUpdateBiblioHoldsQueue.t

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

PA amended: Fix bug # in commit message

Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-04-26 08:52:54 +02:00
Marcel de Rooy 98a500d120
Bug 27893: Optionally skip biblio with open orders in batch delete
If the user unchecks the skip checkbox (on by default), we must
make sure that order cancellation is done in background job.

Test plan:
Pick two biblios. One has linked open orders, the other not.
Go to batch delete records. Select 'Enter list of numbers'.
Enter both biblio numbers and check that only one is used on the
follow-up form.
Run the deletion without the skip open orders and verify
that linked order line is cancelled.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-04-19 18:06:32 +02:00
Marcel de Rooy 291268f318
Bug 35843: Correct invalid exception
Test plan:
Run background job tests. Especially the changed one.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Magnus Enger <magnus@libriotech.no>
All test pass when running this:
prove t/db_dependent/Koha/BackgroundJobs*
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-01-26 15:13:47 +01:00
Kyle M Hall 067034f9e0
Bug 35438: Remove skip_intermediate_commit parameter
It no longer does anything.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-01-16 12:06:08 +01:00
Marcel de Rooy 4d3c838093
Bug 35277: (QA follow-up) POD corrections
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-01-16 12:06:07 +01:00
Kyle M Hall cf84410ce5
Bug 35277: (QA follow-up) Fix POD
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-01-16 12:06:06 +01:00
Nick Clemens 1ae58daca8
Bug 35277: Pseudonymize in a background job
This patch adds a new background job for pseduonymization

To test:
 1 - Download the benchmarking script
 2 - Ensure your koha has bcrypt_settings in koha-conf.xml
     See bug 28911
 3 - perl perf_check_pseudo.pl
 4 - Note the slowdown after pseudonymization enabled
 5 - Apply patches, restart all
 6 - perl perf_check_pseudo.pl
 7 - Note improvement
 8 - Enable pseudonymization in sytem preferences
 9 - Perform some checkouts and returns
10 - Verify the background jobs complete successfully
11 - Verify the pseudonymized_transactions table is updated correctly

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2024-01-16 12:06:04 +01:00
Pedro Amorim d55ea648ac
Bug 34587: Improve SUSHI COUNTER error handling
This is doing a few things, checking for sushi errors:
1) Abort background job and provide error message if SUSHI returned
   {Severity} in response, this means SUSHI error, docs here:
https://app.swaggerhub.com/apis/COUNTER/counter-sushi_5_0_api/5.0.2#/SUSHI_error_model

2) Abort background job and provide error message if SUSHI provided
   proper response, but contains Exceptions in Report_Header

3) Abort background job and provide error message if SUSHI returned 0
   report items (e.g. everything is fine but there are no items for the
   harvest period provided

4) Abort background job and provide error message if COUNTER file
   created is larger than allowed by the database.

5) Provide error message if manual upload file size exceeds the max
   allowed by the database

Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-10-31 16:46:25 -03:00
Pedro Amorim e629e37bd7
Bug 34587: Fix total added objects in job report
Remove obsolete total_records code, this was a leftover from when this code was in UsageDataProvider.pm
Add a new job_callback 'added_usage_objects' to handle total objects added (titles, platforms, databases, items)
Update job report include file

Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-10-31 16:46:24 -03:00
Pedro Amorim ca8ea02fda
Bug 34587: Move everything into a ERM/EUsage subfolder
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-10-31 16:46:18 -03:00
Pedro Amorim cf87082682
Bug 34587: File upload - Background job
ErmSushiHarvester background job will now be initiated by either
enqueue_counter_file_processing_job or by enqueue_sushi_harvest_jobs
from Koha/ERM/UsageDataProvider.pm, the former if triggered by a manual
file upload, the latter if by the 'run now' button or by the cron script.

This commit also includes some rewording/refactoring, namely:
- COUNTER file validation now happens in the API, before enqueuing the job.
- Removal of no longer used POST /erm/counter_files endpoint
- Koha/ERM/UsageDataProvider.pm:
-- run method is now enqueue_sushi_harvest_jobs
-- new enqueue_counter_file_processing_job method
-- harvest method is now harvest_sushi
-- new set_background_job_callbacks method to set the background job callbacks
- REST/V1/ERM/UsageDataProviders
-- run method is now process_SUSHI_response
-- new process_COUNTER_file endpoint

Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-10-31 16:46:15 -03:00
Pedro Amorim 16592d267a
Bug 34587: Accept date inputs upon confirming 'run now' button
Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-10-31 16:46:13 -03:00
Pedro Amorim 2951edc697
Bug 34587: SUSHI COUNTER Harvester
This commit is a squash of the following:

SUSHI harvesting process in the data providers class:
* Builds the URL query and requests the SUSHI service endpoint
* Parses the JSON response and builds the csv COUNTER file and adds it to counter_files table

Usage statistics data processing:
* When a counter_files entry is stored, CounterFile.pm will:
* Parse the csv COUNTER file and
* Add a usage_titles entry for each unique title in the COUNTER file
* Add the title's respective erm_usage_mus (monthly usage) entries, repeating for each metric_type
* Add the title's respective erm_usage_yus (yearly usage) entries, repeating for each metric_type

Harvesting cronjob;

'Run now':
* API endpoint to start the harvesting process of a data provider
* Button in the data providers list to run the harvesting process for each data provider upon clicked

ERM SUSHI: Background job

Job progress is updated to total amount of usage titles after retrieving
the response from SUSHI;
Job warning and success messages are added accordingly
Redundant duplicate titles will not be added
Redundant duplicate monthly and yearly usage statistics will not be added
Data provider harvest background job harvests once per report_type

Enqueue one background job for each report_type in the usage data provider

Update the way we measure progress in the background job.
It now uses the COUNTER report body rows instead of SUSHI response results.
We're now incrementing and showing the number of skipped mus, skipped
yus, added mus and added yus
There's a bug in the way we calculate yus
Updates to background job progress bar - Depends on 34468

Signed-off-by: Jessica Zairo <jzairo@bywatersolutions.com>
Signed-off-by: Michaela Sieber <michaela.sieber@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-10-31 16:45:58 -03:00
Matt Blenkinsop 7d2ce611f7
Bug 34789: Update usage throughout Koha modules and Vue components
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-09-18 12:06:18 -03:00
Matt Blenkinsop 38fc85ade9
Bug 33755: Set profile_id and add missing import
This patch adds a missed parameter (profile_id) in stage-marc-import.pl and also adds a missing import in the background job for staging marc files. This means that the profile used when importing can now be shown in the import batches table and also in batch details

Test plan:
1) Stage a record for import and make sure to save the import profile you use
2) Import a record using that profile
3) Navigate to Cataloging > Manage staged MARC records
4) In the Profile column, no value will be shown for the staged record
5) Click on the batch, the profile will be missing in this screen as well
6) Apply patch
7) Restart_all
8) Repeat steps 2-5, the profile name should be visible in both places

Signed-off-by: Sam Lau <samalau@gmail.com>
Signed-off-by: Emmanuel Bétemps <e.betemps@gmail.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-08-15 12:16:59 +03:00
Marcel de Rooy f7b1c85f73
Bug 33576: (QA follow-up) Polish comment, typo
No test plan.

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-05-09 10:58:06 -03:00
Jonathan Druart 17be09b903
Bug 32782: Koha to KBart mapping for UNIMARC
Used when creating eholdings from Koha items.

Test plan:
Add some items to a list, then use this list to create eholdings
/cgi-bin/koha/erm/eholdings/local/titles/import

Signed-off-by: Laurence Rault <laurence.rault@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-05-05 09:13:19 -03:00
Tomas Cohen Arazi 8fcc61e82e
Bug 33486: POD fix
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-04-13 11:47:51 -03:00
Nick Clemens 42c1059d1d
Bug 33486: Remove Koha::BackgroundJob::UpdateElasticIndex->process
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-04-13 11:47:51 -03:00
Jonathan Druart 71bf715596
Bug 32781: Prevent import from list to fail if package does not exist
This is theorical (caught when writting tests) but we need to deal with
it.

Can't call method "resources" on an undefined value at /kohadevbox/koha/Koha/BackgroundJob/CreateEHoldingsFromBiblios.pm line 94.

Test plan:
  prove t/db_dependent/Koha/BackgroundJob/CreateEHoldingsFromBiblios.t
must return green

Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-03-13 15:56:41 -03:00
Jonathan Druart 7076e4f087
Bug 30943: Adjust CreateEHoldingsFromBiblios
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-03-13 15:52:24 -03:00
Tomas Cohen Arazi 632a43362f
Bug 30943: Simplify background jobs code using helpers
This patch makes the current background jobs classes use the added
helpers in order to simplify them.

No behavior change is expected.

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Rebased-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-03-13 15:52:19 -03:00
Nick Clemens aba2453ad6
Bug 32594: Add a dedicated Elasticsearch biblio indexing background queue
Currently we generate large numebrs if single record reindex for circulation and other
actions. It can take a long time to process these as we need to load the ES settings for each.

This patch updates the Elasticsearch background jobs to throw records into a new queue
that can be processed by it's own worker and adds a dedicated worker that batches the jobs
every 1 second.

To test:
1 - Apply patches, set SearchEngine system preference to 'Elasticsearch'
2 - perl misc/search_tools/es_indexer_daemon.pl
3 - Leave the running in terminal and perform actions in staff interface:
    - Checking out a bib
    - Returning a bib
    - Editing a single bib
    - Editing a single item
    - Batch editing bibs
    - Batch editing items
4 - Confirm for each action that records are updated in search/search results
5 - Stop the script
6 - set SearchEngine system preference to 'Zebra'
7 - perl misc/search_tools/es_indexer_daemon.pl
8 - Script dies as Elasticsearch not enabled

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Emily Lamancusa <emily.lamancusa@montgomerycountymd.gov>

Bug 32594: (follow-up) Adjust logging per bug 32612

JD amended patch: tidy! There were tabs here...

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-02-24 17:52:20 -03:00
Nick Clemens 78e144a9a0
Bug 32019: Add option to mark items returned via batch modification
Ther are options that allow avoiding marking an issue returned when se tot lost.
Libraries do this to tie a lost item to a patron clearly.

Eventually, after the patron has paid, or after some time, the library may wish
to delete the item. When doing so, they do not wish to check the item in and affect any statuses
or fines. This is also true for withdrawn items

It would be helpful to have a way to mark issues returned in bulk, when they have not been marked
returned in the past

To test:
 1 - Apply patches
 2 - Restart all
 3 - Set 'MarkLostItemsAsReturned' system preferences to 'None'
 4 - Check out some items that have replacement fees set
 5 - Mark them lost
 6 - Check out more items, mark them withdrawn
 7 - Set system preference 'BlockReturnOfWithdrawnItems' to block
 8 - Enter item barcodes into batch modification
 9 - Leave 'Mark items returned as blank
10 - Verify nothing was checked in
11 - Repeat but set 'Mark items as returned' to 'No'
12 - Verify nothing was checked in
13 - Repeat, but set 'Mark items as returned' to 'Yes'
14 - Verify items are returned
15 - Verify no fines have been adjusted
16 - Verify withdrawn items are sitll withdrawn
17 - Verify you can now delete the items

Signed-off-by: Andrew Fuerste-Henry <andrewfh@dubcolib.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-02-20 09:54:20 -03:00
Nick Clemens e5f251a709
Bug 15869: Change framework on overlay
This patch allows for selection of framework to use when overlaying
records - by default it is set to keep the initial framework

To test:
 1 - Create some records using one framework
 2 - Export the records
 3 - Edit the records to add fields not in original framework
 4 - Stage records using a rule that will find matches
 5 - Import
 6 - Note records contain new fields on display, but they are lost on edit
 7 - Apply patch
 8 - Stage records again
 9 - Select a framework that contains the new fields on import
10 - Import records
11 - Note records now use selected framework and are displayed/edited
correctly

Signed-off-by: Andrew Fuerste-Henry <andrewfh@dubcolib.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-02-07 10:32:22 -03:00
Jonathan Druart d8022865ca Bug 32779: ERM - Fix import from list
There is a typo, '$' is missing, which caused the following error:
Use of inherited AUTOLOAD for non-method Koha::BackgroundJob::CreateEHoldingsFromBiblios::fix_coverage() is no longer allowed at /kohadevbox/koha/Koha/BackgroundJob/CreateEHoldingsFromBiblios.pm line 126

Test plan:
Create a list, add an item
Turn on ERM
Create a package
Go to /cgi-bin/koha/erm/eholdings/local/titles
Click "Import from list"
Select the package, then your list, and import
It should succeed

Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2023-02-02 11:59:26 -03:00
Tomas Cohen Arazi 3dcdd6db78
Bug 32394: (follow-up) Add param for Koha::BackgroundJob::StageMARCForImport->enqueue
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-12-16 09:00:45 -03:00
Jonathan Druart 9810fc5e0e
Bug 32394: Restore long tasks queue
We are sending "queue" as parameter, but "job_queue" is retrieved in
Koha::BackgroundJob->enqueue...

Test plan:
Read code and confirm the above.
More fun for test plan will be found on bug 32393

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-12-14 14:32:27 -03:00
Jonathan Druart 94c67c439e
Bug 32030: Create eHolding titles from a list
Add the ability to create new titles and attach them to a package.
The MARC to KBART2 mapping is the following (based on
https://github.com/adambuttrick/marc_to_kbart/blob/master/convert.py):

publication_title = biblio.title
print_identifier  = 020$a||020$z||022$a||022$y
online_identifier = 020$a||020$z||022$a||022$y
date_first_issue_online = 866$a (before '-')
date_last_issue_online  = 866$a (after '-')
num_first_vol_online    = 863$a (before '-')
num_last_vol_online     = 863$a (after '-')
num_first_issue_online  = ?
num_last_issue_online   = ?
title_url = 856$u
first_author = biblio.first_author
embargo_info = ?
coverage_depth = title_url ? 'fulltext' : 'print'
notes = $852$z
publisher_name = 260$b
publication_type = ?
date_monograph_published_print = ?
date_monograph_published_online = ?
monograph_volume = ?
monograph_edition = ?
first_editor = ?
parent_publication_title_id = ?
preceeding_publication_title_id = ?
access_type = ?

Note that title is not created (and so the resource) if a title from
this package already has a link to this bibliographic record.
Is that correct, or should we create another resource?

Should the import screen also have "start date" and "end date" to set for the
resource?

QA note: Ideally we would like to fetch the list from the REST API but the routes
are not there yet.

Signed-off-by: Jonathan Field <jonathan.field@ptfs-europe.com>

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
2022-11-08 09:44:20 -03:00