This patch addresses CSV formula injection vulnerabilities by ensuring
all Text::CSV instances use formula protection.
Changes made:
* Add formula => 'empty' to all Text::CSV constructors in:
- Koha::Patrons::Import
- Koha::BackgroundJob::ImportKBARTFile
- Koha::ERM::EUsage::CounterFile
The 'empty' formula mode causes potentially dangerous formulas (starting
with =, +, -, @) to be treated as empty strings rather than executed,
preventing CSV injection attacks while maintaining data integrity.
Security impact:
* Prevents execution of malicious formulas in CSV imports
* Protects against data exfiltration via CSV formula injection
* Maintains backward compatibility for legitimate CSV data
To test:
1. Apply the previous patch (author test)
2. Run:
$ ktd --shell
k$ prove xt/author/Text_CSV_Various.t
=> FAIL: Should show 3 files without formula protection
3. Apply this patch
4. Repeat 2
=> SUCCESS: All tests should now pass
5. Try importing CSV with formula content (=cmd|"/c calc",test,data)
=> SUCCESS: Formula content is rejected/emptied, not executed
6. Import normal CSV data
=> SUCCESS: Normal data imports correctly
7. Sign off :-D
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Jesse Maseto <jesse@bywatersolutions.com>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
(cherry picked from commit 04a038f6b0)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch removes the logged warning if a file parse is successful.
Test plan:
1) prove t/db_dependent/Koha/BackgroundJob/ImportKBARTFile.t
2) The output will have a logged message - 'EOF - End of data in parsing input stream'
3) Apply patch
4) Repeat step 1
5) The warning will be gone
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
(cherry picked from commit 7c0ed91c80)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch fixes a few things:
1) The blue dialog box now clears when navigating away from the page
2) The background job now uses skip_record_index to avoid queuing indexing jobs for every new biblio and instead queues one job at the end
3) Large files that get chunked now successfully create linked biblios if requested
4) Title matching rules have been expanded to check the package ID so that we can have duplicate titles in different packages
5) A link to the package is now included on the job report page
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch allows a file with additional columns to be imported. When the file is submitted, the system will enqueue the background job and send back information to the UI that certain columns have been ignored. The data will stil l import as normal but only the standard KBART columns will be parsed and imported
Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch rebases in the changes from bug 36618 to make biblio creation optional
Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This commit allows CSV files to be imported alongside TSV files.
It also adds some performance improvements relating to the max_allowed_packet and the matching of titles, as well as some small bugfixes and unit test changes
Test plan:
1) Enable the ERM module
2) Navigate to E-resource management > eHoldings > Local > Packages
3) Create at least one package
4) Navigate to E-resource management > eHoldings > Local > Titles
5) There should be a button for "Import from KBART file"
6) Click this button
7) Select the package that you created from the dropdown and then choose your KBART file using the "Choose file" button. I have attached some example files to the bug but feel free to use your own if you have them.
8) Click Submit
9) If your file is a valid file, a background job will be queued, if not then a warning will display showing what is incorrect in your file
10) To test the file format warning, edit your file and add a random column heading into the file e.g. test_column. When you upload it, the warning should show that an invalid column "test_column" has been detected
11) Click on the background job. (If you have uploaded a very large file, the system will chunk the file into smaller pieces and create multiple background jobs)
12) It should display a progress bar followed by a report and any error messages
13) Navigate to E-resource management > eHoldings > Local > Titles and you should see your new titles.
14) Run the unit test: prove t/db_dependent/Koha/BackgroundJob/ImportKBARTFile.t
Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds a background job that will import a KBART file
Sponsored-by: UKHSA
Signed-off-by: Clemens Tubach <clemens.tubach@kit.edu>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Tomas Cohen Arazi
Matt Blenkinsop