The system is vulnerable to Formula Injection attacks as the data
stored within the database and exported as CSV/Excel is not being
sanitized or validated against implanted formula payloads
This patch modifies all uses of Text::CSV and derived classes to pass
the "formula" parameter with value of "empty" which replaces formulas
by empty string.
Test Plan:
1) Apply this patch
2) For guided_reports.pl, attempt to export CSV where you've set a column to a formula somehow
( such as "=1+3" )
3) Export that CSV file
4) Note the formula has not been exported
5) Repeat this plan for the remaining scripts that export CSV files
where users can define the outputted data
Signed-off-by: Magnus Enger <magnus@libriotech.no>
Fixed two conflicts. I have tested that this works as advertised on:
- Reports (Download > Comma separated text (.csv)) [Text::CSV::Encoded]
- Circulation > Overdues > Download file of all overdues [Text::CSV_XS]
- misc/export_borrowers.pl [Text::CSV]
This covers all modules used, and both GUI and command line.
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
[EDIT] Change none to empty in the commit message ! None is the default,
doing nothing. Empty clears the formulas.
Signed-off-by: wainuiwitikapark <wainuiwitikapark@catalyst.net.nz>
(cherry picked from commit 96152682bf)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This makes sure that we don't change the line breaking for either
HTML nor plain text print notices.
For plain text, the <pre> is used to keep the line breaks as defined
in the notices template.
For HTML we require on the line breaks added by block elements in the
HTML notice, not adding any additional ones like before.
This gives the full control of the formatting to the notice editor.
To test:
* Make sure your patron doesn't have an email address
* Make sure to check the hold filled notice in messaging preferences
* Place a old on any item
* Checkin the item
* Confirm hold
* Verify notices tab shows the Hold (print) notice
* ./misc/cronjobs/gather_print_notices.pl ./ --html
* Verify the generated file has the HTML output wrapped in pre.
* Reformat the HOLD print notice to be HTML by setting the checkbox.
Mmake sure to add some empty lines for (they should not print later)
and some <br>, <p>, <div> - They should be reflected in the notice later.
* Repeat test.
* All should be well :)
Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
(cherry picked from commit 4c15ad658d)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit e0eb511d59)
Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
This patch makes the trivial change of making the linker script rely on
LinkerRelink instead of CatalogModuleRelink, which is confusing and
incorrect (according to both sysprefs descriptions).
To test:
1. Verify that relinking is now tied to the right syspref
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
(cherry picked from commit 4310e13a64)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit 234e00b1d6)
Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
According to the timestamp option for bibs record type I added the
timestamp option for authority records.
Timestamp is already present in database on field "modification_time"
Test Plan :
1 - Be sure to have authority record type for easiest test create one
2 - Execute script export_records.pl in your koha/misc directory and
choose a date (example yesterday if you just created an authority
right now).(see export_records.pl -h for help)
3 - Timestamp option has no effect on authority record type
4 - Execute script again but choose the date of tomorrow for example
5 - Same result
6 - Apply this patch
7 - Play again steps 2 and 4
8 - On step 2 you will see only your authorities created today (because
script show you authority has changes since the date you choose in
option) and for step 4 you must see an empty file.
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
(cherry picked from commit 24ee60cb62)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit e5d014de8c)
Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
Also move earlier print to warn for consistency.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
(cherry picked from commit 3439185bb0)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit c3fda28fc5)
Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
When fetching the record we need to eval in case of exception
To test:
1 - Apply first patch
2 - perl misc/link_bibs_to_authorities.pl
3 - Note it dies on parser error at record 369
4 - Apply this patch
5 - perl misc/link_bibs_to_authorities.pl
6 - The process completes - with warning thrown on record 369
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
(cherry picked from commit 706641e747)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit ed2bb9534a)
Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
This patch simplay adds an eval and a warning to link_bibs_to_authorities to catch any errors while searching and continue linking.
To test:
1 - edit /etc/kohia/sites/kohadev/koha-conf.xml and add to elasticsearch stanza
<request_timeout>0</request_timeout>
2 - perl misc/link_bibs_to_authorities.pl
3 - It dies immediately
4 - Apply patch
5 - perl misc/link_bibs_to_authorities.pl
6 - Now it tries every record, throwing warnings along the way, but completes
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
(cherry picked from commit 22c0be31c7)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit 07cf6fccc0)
Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
To test:
1. In KTD use grep -rn --exclude=*.po "datexpiry" *
2. Notice that the four locations below are showing the typo:
1. git add koha-tmpl/intranet-tmpl/prog/en/modules/tools/import_borrowers.tt
2. git add misc/release_notes/release_notes_18_05_00.html
3. git add misc/release_notes/release_notes_18_05_00.md
4. git add t/db_dependent/Koha/Patrons/Import.t
3. Apply patch. Repeat step 1.
4. Notice that no results show. All instances of the typo have been fixed to dateexpiry.
5. Sign off and have a great day :)
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
(cherry picked from commit d742f29d77)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit 9b87ae61ad)
Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
This change explicitly ignores SIGPIPE signals in the background jobs
worker.
Daemons like Starman ignore SIGPIPE so it makes sense to explicitly set this.
Differences in the inner workings of MySQL vs MariaDB client libraries have yielded
different behaviours in automatic reconnections and potentially SIGPIPE handling,
so this helps to make the overall behaviour more consistent.
Test plan:
0. Apply patch and run "restart_all"
1. Go to http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=29
2. Click "Save" > "MARCXML"
3. Go to http://localhost:8081/cgi-bin/koha/tools/stage-marc-import.pl
4. Click "Choose file", choose the MARCXML file, click "Upload file"
5. Click "Stage for import"
6. Note the job is marked as "100% Finished"
7. In a separate window run "docker restart koha-db-1"
8. Repeat steps 3-5 for uploading file and running stage for import
9. Note that the job is marked as "100% Finished" as you'd expect
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
[EDIT] Added comment on the SIG PIPE line.
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
(cherry picked from commit 2988c54971)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit b57dfb84bc)
Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
The before_send_messages plugin hook currently sends all the non-smtp related options with the exception of the --where option. This should be added.
Test Plan:
1) Install the latest version of the kitchen sink plugin (
https://github.com/bywatersolutions/dev-koha-plugin-kitchen-sink/releases
)
2) Apply this patch
3) Run process_message_queue.pl with a --where option
4) Run the command: ./misc/cronjobs/process_message_queue.pl --where "letter_code='test'"
5) Note the output looks like:
Plugin hook before_send_message called with the params: $VAR1 = {
'verbose' => 0,
'where' => 'letter_code=\'test\'',
'letter_code' => [],
'type' => [],
'limit' => undef
};
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
(cherry picked from commit 1508e81505)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit 247b9da172)
Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
This patch does so by default. If you do not want them sorted,
export CANONICAL-0 on the command line.
Test plan:
Copy a staff PO file from misc/translator to test.po
Now run: time misc/translator/po2json test.po > json1
And do: time misc/translator/po2json test.po > json2
Run: diff json1 json2; #They should be the same.
Now: export CANONICAL=0
And run: time misc/translator/po2json test.po > json3
And again: time misc/translator/po2json test.po > json4
And run: diff json3 json4; # Lots of changes
Remove the created cruft. And signoff :)
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
(cherry picked from commit a5fc346403)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit 82d4eff906)
Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
This is a kind of copy-and-paste variant of background worker.
We probably could do better than having two scripts here ;)
Test plan:
See former test plan. Apply it to ES indexing.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
[EDIT] Removed queue from query at MQ side. Discussed on IRC.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
(cherry picked from commit 709dabb7e8)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit 17560ba443)
Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
MQ mode: The worker has seen that MQ is running and consumes messages
instead of polling db only.
Test plan:
Stop worker for long tasks.
Make sure that Rabbit MQ runs.
Stage a file. (This adds a long task.)
Goto staff view of jobs and cancel this job.
Check if job is still in MQ with rabbitmqctl list_queues.
Now start worker for long tasks.
Check if job is gone in MQ with rabbitmqctl list_queues.
And check logfile for the adjusted warning like:
[WARN] Job 5 not found, or has wrong status/queue main:: /usr/share/koha/misc/workers/background_jobs_worker.pl (134)
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
[EDIT] Removed queue from query at MQ side. Discussed on IRC.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
(cherry picked from commit f1edd8fe77)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit 20b0687b74)
Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
We have our own "quote_po" sub that is stating that Locale::PO::quote
is buggy because it does not deal with quoting new lines correctly.
However it seems that it is fixed now.
Ideally we could use Locale::PO::quote everywhere, but it does not
escape tab characters:
$string =~ s/\\(?!t)/\\\\/g; # \t is a tab
This means the following:
msgid "Tabulation (\\t)"
msgstr "Tabulation (\\t)"
become:
-msgid "Tabulation (\t)"
-msgstr "Tabulation (\t)"
And we are seeing the following on Weblate:
https://snipboard.io/BjQmDC.jpg
Note that Locale::PO has not been updated since 2014...
The real problem behind this is that we have 2 methods to quote strings.
At first glance it seems that Locale::PO::quote was not used before, but
with the introduction of the koha-i18n project we will have scripts that
will use Locale::PO->save_file_fromarray, which uses Locale::PO->quote
=> Those scripts will be used on the translation server for post
processing (security reason, marking potential XSS strings as fuzzy).
Test plan:
0. Do not apply the patch
1. gulp po:update --lang LANG # Replace LANG with your favorite language
code
2. git commit -a -m"init PO files"
3. Apply this patch
4. Repeate 1.
5. git diff
=> The change is about the "Tabulation" and "New line" strings from
tools/csv-profiles.tt
6. Translate them (replace the \t and \n with %s) and remove the fuzzy
flag
7. install the template: cd misc/translator && perl translate install
LANG
8. Enable the language, use it and go to the "Nouveau profil CSV" view
=> Notice that the \t and \n are correctly displayed.
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
The PO files are not sorted when we update them which leads to
unnecessary changes that are commited: hard to see differences and
make git index grow superfluously.
Test plan:
0. Do not apply this patch
1. gulp po:update --lang es-ES
2. git commit -a -m"First PO update"
3. Run again the gulp update command
4. git diff
=> You have a lot of changes generated here, the po:update is not
idempotent.
5. Apply this patch
6. Run the gulp update command
7 git commit -a -m"PO update after 34959"
8. Run the gulp update command
9. git diff
=> No changes are generated
Note that this patch will all the entries by files, and per line
numbers.
It fixes a bug in some condition, where we add information/context about
the string. For instance search for "For the first occurrence" in the
file. Prior to this patch this was not correct, we didn't add info about
the first occurrence (but whichever in the list).
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Nothing special here, this sub is not used and we can remove it.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
(cherry picked from commit bda428399a)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit b9ad842dee)
Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
This patch makes the pre-formatting of notice templates conditional on the content type.
Test plan:
1) Navigate to tools > Notices and slips
2) Edit the notice with the code of HOLD
3) Under print, paste the following
<p>Please collect within 10 days from the date of this letter and don't forget to bring your library card with you.</p>
<p>Be aware that opening hours vary at different library branches so if in doubt double check the times before you visit.</p>
<p>Should you no longer require this item please contact us so we can make it available for another customer.</p>
<p>Thank you</p>
4) Check the box that says HTML message and save the notice
5) In Patrons, select a patron and in Patron messaging preferences, check the box in the Email column for Hold filled
6) Find a record in the catalogue
7) Copy the barcode for one of the items
8) Place a hold on the item for the patron you selected
9) Navigate to Circulation > Check-in
10) Paste the barcode in the Check-in box and click Check-in
11) In the pop-up box, click Confirm hold
12) In the command line, run perl misc/cronjobs/gather_print_notices.pl --letter_code=HOLD --html /tmp
13) cd /tmp
14) There should be a file in there called 'notices_HOLD-date.html
15) cat this file
16) The html you pasted in the template will be wrapped in a <pre> tag
17) Apply patch and restart_all
18) rm the notices file that was created
19) cd /kohadevbox/koha
20) Select a new record from the catalogue and repeat steps 7-15 for this record, using the same patron
21) This time, the html file should not contain the <pre> tag
22) Sign off!
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
(cherry picked from commit d947ae3c1a)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit 5cff31dd78)
Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
Currently --expiration-from-today has the same definition as --update-expiration. The description has now been updated to reflect the fact that it will set the date from today, and not the patron's enrollment date
Test plan:
Look at the patch file and confirm that the POD now mentions today's date rather than the patron's enrollment date
Signed-off-by: Laura Escamilla <laura.escamilla@bywatersolutions.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
(cherry picked from commit 91dd8dd16f)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit 636c601995)
Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
For consistency with other options I opted to have the fee acknowledged
parameter to expect a string and as it's an optional parameter I've
dropped the default value of 'N' too.
Test plan
1) Prior to this patch
1a) Attempt a checkout without passing -fa/--fee_acknowledged flag
`./misc/sip_cli_emulator.pl -a localhost -p 6001 -su term1 -sp term1 -l CPL --patron 23529000035676 --item 39999000007756 -m checkout`
The final field of the SIP request will be '|BON', the default
1b) Attempt a checkout passing -fa/--fee_acknowledged flag
`./misc/sip_cli_emulator.pl -a localhost -p 6001 -su term1 -sp term1 -l CPL --fee-acknowledged --patron 23529000035676 --item 39999000007756 -m checkout`
The final field of the SIP request will still be '|BON', failure
1c) Attempt a checkout passing a string for fee_acknoewledeged flag
`./misc/sip_cli_emulator.pl -a localhost -p 6001 -su term1 -sp term1 -l CPL --fee-acknowledged Y --patron 23529000035676 --item 39999000007756 -m checkout`
The final field of the SIP request will still be '|BON', failure
2) Apply the patch
2a) Attempt a checkout without passing -fa/--fee_acknowledged flag
`./misc/sip_cli_emulator.pl -a localhost -p 6001 -su term1 -sp term1 -l CPL --patron 23529000035676 --item 39999000007756 -m checkout`
The optional `|BO` element should not be present
2b) Attempt a checkout passing -fa/--fee_acknowledged flag
`./misc/sip_cli_emulator.pl -a localhost -p 6001 -su term1 -sp term1 -l CPL --fee-acknowledged N --patron 23529000035676 --item 39999000007756 -m checkout`
The final field of the SIP request will now be '|BON', success
2c) Attempt a checkout passing a string for fee_acknoewledeged flag
`./misc/sip_cli_emulator.pl -a localhost -p 6001 -su term1 -sp term1 -l CPL --fee-acknowledged Y --patron 23529000035676 --item 39999000007756 -m checkout`
The final field of the SIP request will now be '|BOY', success
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
(cherry picked from commit 0e9603bb49)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit 7c07d7b6a6)
Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
(cherry picked from commit a707d10d80)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit 99ac8d1ecb)
Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
misc/cronjobs/holds/holds_reminder.pl help says one can use arge -t for --triggered
But not implemented in code.
Should be like in misc/cronjobs/overdue_notices.pl :
't|triggered' => \$triggered,
Test plan :
Play with misc/cronjobs/holds/holds_reminder.pl with arg -t and
--trigger
Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
(cherry picked from commit f4f4feb50b)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit d019a68079)
Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
Paul Derscheid
JesseM
Lucas Gass
Kyle M Hall
Frédéric Demians
Katrin Fischer
Tomas Cohen Arazi
Fridolin Somers
Pedro Amorim
Koha translators
Katrin Fischer
Thibaud Guillot
Marcel de Rooy
Nick Clemens
Laura Escamilla
David Cook
Jonathan Druart
Matt Blenkinsop
Martin Renvoize
