Commit Graph

3282 Commits

Author SHA1 Message Date
David Cook 90f173ac6a
Bug 38365: Add context to Koha::Devel::Files in add_csp_nonces.pl
The add_csp_nonces.pl file should use the exceptions in
Koha::Devel::Files.

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2026-03-16 11:57:41 +01:00
David Cook e011f2f6e1
Bug 38365: Make tidy.pl compatible with script elements with attributes
tidy.pl will still tidy whitespace around script elements that have
attributes like "nonce"

Signed-off-by: Lari Taskula <lari.taskula@hypernova.fi>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2026-03-16 11:57:38 +01:00
Jonathan Druart bba61e6bc9
Bug 38365: Use Koha::Devel::Files->ls_tt_files in add_csp_nonces.pl
Signed-off-by: Lari Taskula <lari.taskula@hypernova.fi>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2026-03-16 11:57:35 +01:00
David Cook f80451023e
Bug 38365: Add <style to add_csp_nonces.pl to add nonce to style elements
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Lari Taskula <lari.taskula@hypernova.fi>
Signed-off-by: David Cook <dcook@prosentient.com.au>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2026-03-16 11:57:16 +01:00
Martin Renvoize e8d11cc102
Bug 38365: Add script to insert CSP nonces into templates
Add misc/devel/add_csp_nonces.pl which automatically adds CSP nonce
attributes to inline <script> tags in Koha templates.

The script:
- Finds all .tt and .inc files in koha-tmpl/
- Adds nonce="[% Koha.CSPNonce | $raw %]" to inline scripts
- Skips external scripts (those with src= attribute)
- Skips scripts that already have nonce= attribute
- Skips third-party library files in lib/ and vendor/

Usage:
  perl misc/devel/add_csp_nonces.pl          # Dry run
  perl misc/devel/add_csp_nonces.pl --apply  # Apply changes

This script enables easy backporting of CSP nonce support to older
Koha versions by running it against their template directories.

Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Lari Taskula <lari.taskula@hypernova.fi>
Signed-off-by: David Cook <dcook@prosentient.com.au>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
2026-03-16 11:57:14 +01:00
Nick Clemens 036d92e654
Bug 41316: Use patrons homelibrary for selecting overdues that should be sent
Currently the patron homelibrary option for OverdueNoticeFrom will select overdues for a branches triggers
based on the issuing branch

The other two options select the trigger/overdues based on the selection

This patch alters the selection of overdues to match only for the patrons branch

To test:
1 - Apply 41315 and follow the plan there
2 - Apply this bug and run the overdues script, assuming CPL is branch A:
    perl misc/cronjobs/overdue_notices.pl --triggered --library CPL --test --nomail -v -v
3 - No overdues should be triggered
4 - Run the script for one of the patrons home libraries, in my case MPL
    perl misc/cronjobs/overdue_notices.pl --triggered --library MPL --test --nomail -v -v
5 - Note only overdues for patrons from that branch are triggered

Signed-off-by: Ben Daeuber <bdaeuber@fargolibrary.org>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2026-03-05 08:33:54 -07:00
Kyle M Hall 5779eec49b
Bug 39749: RestrictPatronsWithFailedNotices should not trigger for DUPLICATE_MESSAGE failures
Koha is built to prevent sending duplicate notices. Notices with static language ( e.g. "You have one or more holds ready to pick up" ) will easily trigger this filter.

RestrictPatronsWithFailedNotices should ignore these notices because they do not indicate any problem with contacting the patron.

Test Plan:
1) Enable RestrictPatronsWithFailedNotices
2) Enqueue to identical notices for a patron
3) Run process message queue to send one and fail the other as a
   duplicate ( alternately, just modify them in the database )
4) Run restrict_patrons_with_failed_notices.pl
5) Note the restriction
6) Remove the restriction
7) Apply this patch
8) Run restrict_patrons_with_failed_notices.pl
9) Note that no restriction was created!

Signed-off-by: Marion Durand <marion.durand@biblibre.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2026-03-05 08:33:52 -07:00
Lucas Gass fde40177fa
Bug 23260: Update cron to exit if AnonymousPatron is not set
Signed-off-by: Peter Moore <peter@mrcpl.org>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2026-03-05 08:33:48 -07:00
Andrew Fuerste Henry f0d4240ee2
Bug 23260: (QA follow-up) Prevent use of cronjob when AnonymousPatron not set
To test:
1 - configure StoreLastBorrower but not AnonymousPatron
2 - perl misc/cronjobs/anonymize_last_borrowers.pl -c
3 - the cron tells 'Preference 'AnonymousPatron' must be enabled to anonymize item's last borrower'
4 - put a value in AnonymousPatron
5 - repeat 2
6 - cron runs good

Signed-off-by: Peter Moore <peter@mrcpl.org>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2026-03-05 08:33:47 -07:00
Lucas Gass 7eda8d23fa
Bug 23260: Add a confirm flag
To test the confirm flag:
1. Run perl misc/cronjobs/batch_anonymise.pl --days 1
2. Notice the warning about needing a confirm flag
3. Run perl misc/cronjobs/batch_anonymise.pl --days 1 --confirm
4. The script runs

Signed-off-by: Peter Moore <peter@mrcpl.org>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2026-03-05 08:33:46 -07:00
Nick Clemens ecfd85ebb8
Bug 23260: Add cronjob anonymize_last_borrowers.pl
This patch adds the feature to anonymize patrons from items_last_borrower table. This code is run from batch_anonymize script, which is triggered from a new cron job anonymize_last_borrowers.pl

To test:
 1) apply this patch and the previous ones
 2) perl installer/data/mysql/updatedatabase.pl
 3) set StoreLastBorrower preference to Allow.
 4) set AnonymousPatron preference to a valid patron id
 5) Create a Check out followed by a Check in.
    CHECK => a row should appear in items_last_borrower table with the borrower and the item number.
 6) In mysql, update created_on of items_last_borrower and returndate of old_issues to two days earlier.
 7) perl misc/cronjobs/anonymize_last_borrowers.pl
    CHECK => borrower number in items_last_borrower is not anonymized yet.
 8) Repeat step 5 and 6 but instead of two days earlier, set the new entry in items_last_borrower to three days earlier.
 9) set AnonymizeLastBorrower to 'Anonymize' and AnonymizeLastBorrowerDays to 2 days.
10) perl misc/cronjobs/anonymize_last_borrowers.pl
    SUCCESS => borrower number in items_last_borrower changed to AnonymousPatron id
11) repeat step 8
12) set AnonymousPatron preference to 0
13) perl misc/cronjobs/anonymize_last_borrowers.pl
SUCCESS => borrower number in items_last_borrower changed to null
14) prove t/db_dependent/Koha/Patrons.t

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Peter Moore <peter@mrcpl.org>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2026-03-05 08:33:46 -07:00
Laura_Escamilla 82ff1e700b
Bug 35612: Recorded branch context for OVERDUE, LOST, and PROCESSING fees
This patch ensures that the accountlines.branchcode field correctly records
which branch's circulation rules were applied when creating OVERDUE, LOST,
and PROCESSING fee debits. Previously, this field was often NULL or set
to the logged-in user's branch, making it difficult to track which branch's
policies governed the fee calculation.

Changes made:

1. New helper method: Koha::Checkout->branch_for_fee_context()
   - Centralizes branch resolution logic based on system preferences
   - For LOST/PROCESSING: honors LostChargesControl preference
   - For OVERDUE: honors CircControl preference
   - Respects HomeOrHoldingBranch when choosing item's branch
   - Supports both Koha objects and plain hashrefs
   - Returns branchcode or undef if context unavailable
2. C4::Accounts::chargelostitem()
   - Now accepts optional $opts hashref parameter
   - Accepts interface, library_id, and issue_id in options
   - Determines appropriate branch using branch_for_fee_context
   - Passes resolved branch to add_debit as library_id
   - Issue_id lookup handles bundled items correctly
3. C4::Circulation::LostItem()
   - Calls branch_for_fee_context before chargelostitem
   - Passes interface context ('cron' for cronjob, otherwise current)
   - Passes resolved branch and issue_id to chargelostitem
4. C4::Overdues::UpdateFine()
   - Calls branch_for_fee_context to determine OVERDUE branch
   - Passes result as library_id to add_debit
   - Resolves long-standing FIXME about branch selection
5. misc/cronjobs/longoverdue.pl
   - Determines branch context before marking items lost
   - Passes library_id to LostItem when charging fees

The branch resolution logic respects these system preferences:
- CircControl: PatronLibrary | PickupLibrary | ItemHomeLibrary
- LostChargesControl: PatronLibrary | PickupLibrary | ItemHomeLibrary
- HomeOrHoldingBranch: homebranch | holdingbranch

Test plan:
1. Create three test libraries (e.g., FPL, CPL, PVL)
2. Create a test patron with home library PVL
3. Create an item with:
   - homebranch = FPL
   - holdingbranch = CPL
4. Set system preferences:
   - FinesMode: Calculate and charge
   - Create circulation rule that charges overdue fines
5. Test OVERDUE fees with CircControl:
   a. Set CircControl = PatronLibrary, HomeOrHoldingBranch = homebranch
   b. Check out item to patron at CPL, backdate to be overdue
   c. Run: perl /usr/share/koha/bin/cronjobs/fines.pl
   d. Check accountlines: SELECT accountlines_id, branchcode, debit_type_code
      FROM accountlines ORDER BY accountlines_id DESC LIMIT 5;
   e. Verify branchcode = PVL (patron's home library)
   f. Delete the fine, repeat with CircControl = PickupLibrary
   g. Verify branchcode = CPL (checkout library)
   h. Delete the fine, repeat with CircControl = ItemHomeLibrary
   i. Verify branchcode = FPL (item's home library)
   j. Change HomeOrHoldingBranch = holdingbranch, repeat
   k. Verify branchcode = CPL (item's holding library)

6. Test LOST fees with LostChargesControl:
   a. Set WhenLostChargeReplacementFee = Charge
   b. Set item replacement price > 0
   c. Ensure item is checked out to patron
   d. Set LostChargesControl = PatronLibrary
   e. Mark item as lost in staff interface
   f. Check accountlines for LOST debit
   g. Verify branchcode = PVL (patron's home library)
   h. Return item, re-checkout, repeat with LostChargesControl = PickupLibrary
   i. Verify branchcode = CPL (checkout library)
   j. Repeat with LostChargesControl = ItemHomeLibrary + homebranch
   k. Verify branchcode = FPL (item's home)
   l. Repeat with LostChargesControl = ItemHomeLibrary + holdingbranch
   m. Verify branchcode = CPL (item's holding)
7. Test longoverdue cronjob:
   a. Re-checkout item with due date triggering longoverdue
   b. Run: perl /usr/share/koha/bin/cronjobs/longoverdue.pl --lost 30=1 --charge 1 --confirm --verbose
   c. Verify LOST debit created with correct branchcode per LostChargesControl
8. Test edge cases:
   - WhenLostChargeReplacementFee = Don't charge → no LOST debit
   - Replacement price = 0 → no LOST debit
   - FinesMode = Don't calculate → no OVERDUE debits

Signed-off-by: Trevor Diamond <trevor.diamond@mainlib.org>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2026-02-27 11:38:49 -07:00
Aleisha Amohia c72fc81b76
Bug 38549: Provide surname for create_superlibrarian.pl
Interestingly this script already had a $surname variable, just never accepted a parameter for it.

This enhancement enables the user to supply a surname parameter when creating a superlibrarian user via the commandline create_superlibrarian.pl script. Koha patrons MUST have a name, so if no surname is provided, the userid will be used for the surname instead.

To test:

1. Apply the patch and restart services
2. Attempt to create a superlibrarian with no surname provided.
  In KTD: misc/devel/create_superlibrarian.pl --userid test --cardnumber test --password XXXX --branchcode CPL --categorycode S
3. Search for your patron in Koha. Confirm the surname was automatically set to the userid 'test'
4. Attempt to create a superlibrarian, this time provide a surname that is different from the userid so easy to tell apart.
  In KTD: misc/devel/create_superlibrarian.pl --userid test1 --cardnumber test1 --password XXXX --branchcode CPL --categorycode S --surname testsurname
5. Search for your patron in Koha. Confirm the surname was set correctly using the given surname.

Sponsored-by: Catalyst IT
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2026-02-27 11:38:43 -07:00
Nick Clemens d7028939da
Bug 41393: Set the correct reply to address for advance notices
This patch additionally fetches the reply to address for advance notices

To test:
1 - Have some issues that will be due soon for a patron(s)
    Checkout with the due date set to tomorrow
2 - Verify the patron(s) have an email defined, verify the issuing branch has a reply to set
3 - Verify the patron(s) have messaging preference set to receive Advance notices by email 1 day in advance
4 - perl misc/cronjobs/advance_notices.pl -v -c
5 - Check the message queue
    SELECT * FROM message_queue WHERE borrowernumber={##}
6 - Note no rpely to is set
7 - Apply patch
8 - perl misc/cronjobs/advance_notices.pl -v -c
9 - Check the message queue
10 - Confirm newer message has the reply to populated
11 - Ensure the issued item has a different homebranch with a reply to address se
12 - perl misc/cronjobs/advance_notices.pl -v -c --frombranch item-homebranch
13 - Confirm message has correct reply to
14 - Set patron to want digest in messaging preferences
15 - Generate notices
16 - Confirm digets has the right reply to

Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Tidied inline
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2026-02-26 09:04:11 -07:00
Marcel de Rooy b8742964db
Bug 28528: (QA follow-up) Remove "Koha 3" reference
Thats quite old :)

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2026-02-26 09:04:10 -07:00
Roman Dolny 8ec1b421bf
Bug 28528: bulkmarcimport delete option doesn't delete biblio_metadata
When using bulkmarcimport with the -d switch should not only delete
biblio, biblioitems and items, but also biblio_metadata.

I haven't checked if a FK takes care of the data, but if it does,
we should still also reset the AUTO_INCREMENT:...

Test plan:
==========
1. Check (and notice) the number of records in the biblio (bCount) and
   biblio_metadata (bmCount) tables. Check the AUTO_INCREMENT value
   for these tables (bAI, bmAI).
2. Import bibliographic records with delete, turn off foreign key checks
   and non-existent filename:
   misc/migration_tools/bulkmarcimport.pl -b -d -fk -file=non-exist-file
==> "Could not open..." is OK, file doesn't exist.
3. Repeat p. 1 and compare with previously noticed:
   bCount => 0 (OK)
   bmCount => unchanged (NOT OK)
   bAI => 1 (OK)
   bmAI => unchanged (NOT OK)
4. Apply the patch.
5. Repeat p. 2-3:
   bCount => 0 (OK)
   bmCount => 0 (changed, OK)
   bAI => 1 (OK)
   bmAI => 1 (changed, OK)

Sponsored-by: Ignatianum University in Cracow
Signed-off-by: Janusz Kaczmarek <januszop@gmail.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2026-02-26 09:04:09 -07:00
Marcel de Rooy 47d88a6429
Bug 41097: (QA follow-up) Clarify POD
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2026-02-25 07:37:04 -07:00
Nick Clemens 7a5fa4cd50
Bug 41097: Don't use original record twice
This patch prevents us from passing the same record twice into the choose subs
to avoid a situation where the original record is not chosen as the one to keep
and we try to merge/delete the same record twice

To test:
1 - Run the script on a test system that you can reset
    perl misc/maintenance/dedup_authorities.pl -v -v --confirm -a="PERSO_NAME"
2 - Note duplication of entries in the output like:
    Merging 1660,1662 into 1660.
    Updated 0 biblios
    Deleting 1662
    Merge done.
3 - Note how many records are merged
4 - Run again to confirm nothing is merged, i.e. the script removed the dupes
5 - Reset your DB
6 - Apply patch
7 - Run the script again
8 - Note entries no longer have duplicated ids:
    Merging 1662 into 1660.
9 - Note the same number of records are merged
10 - Run again and confirm nothing is merged
11 - Sign off!

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2026-02-25 07:37:04 -07:00
Martin Renvoize 32d277687d
Bug 39802: (QA follow-up) Typo fixes
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2026-02-06 15:17:39 -07:00
Nick Clemens f5377250f5
Bug 39802: Report when no branches will be selected and exit
This highlights the issue mentioned in the last patch, reporting that
nothign will be done and exiting the script

To test:
1 - perl misc/cronjobs/longoverdue.pl  -l 5=1 --library CPL
2 - Confirm script exits with warning that nothing has been done

Sponsored-by: MAIN Library Alliance
Signed-off-by: Trevor Diamond <trevor.diamond@mainlib.org>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2026-02-06 15:17:37 -07:00
Nick Clemens 95789607e2
Bug 39802: Use new preference in longoverdue cron
The test plan here is largely confirming that nothing has changed.

Using CircControl here has an existing problem - when set to 'PickupLibrary'
the check in this cron looks for items from the userenv branch - the cron doesn't
have a userenv branch, it is undef, so for any library limiting by branch and
with CircControl set to 'PickupLibrary' / 'the library you are logged in at' the cron will do nothing

Sponsored-by: MAIN Library Alliance
Signed-off-by: Trevor Diamond <trevor.diamond@mainlib.org>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2026-02-06 15:17:37 -07:00
Jonathan Druart ebfbd0da26
Bug 40947: Remove warning "Aborted connection" from t/db_dependent/www/search_utf8.t
We sent a SIGKILL to the daemon which does not let it cleanup

Updating to SIGTERM (15) and handle the dbh disconnect gracefully in the
daemon.

Also handle SIGINT (ctrl+c) only because I think it can be useful (?)

Test plan:
  prove t/db_dependent/www/search_utf8.t

=> Without this patch there are 3 warnings in the DB container log
db-1         | 2025-10-03 12:30:57 362 [Warning] Aborted connection 42 to db: 'koha_kohadev' user: 'koha_kohadev' host: '172.19.0.6' (Got an error reading communication packets)
=> With this patch applied there are none.

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2026-02-06 15:17:12 -07:00
Jonathan Druart 849f67988f
Bug 41076: Adjust our perl tidiness for v20250105
Generated using
  `perl misc/devel/tidy.pl --perl`

Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2026-02-05 09:03:51 -07:00
Nick Clemens b034685fe8
Bug 41315: Don't change which branch is used for getting the transports
See bug 41316 - the script uses the triggers of the issuing branch when sending
by patron library - we shouldn't change this here, but on the next bug

Signed-off-by: Ben Daeuber <bdaeuber@fargolibrary.org>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2026-01-28 14:24:04 -07:00
Nick Clemens 37eda84469
Bug 41315: (follow-up) Rename statements for clarity
Signed-off-by: Ben Daeuber <bdaeuber@fargolibrary.org>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2026-01-28 14:24:04 -07:00
Nick Clemens ce848197e5
Bug 41315: Remove unecessary conditional
We are in a loop here:
foreach my $branchcode (@branches) {

branchcode must be defined or something has gone seriously wrong with PERL

Signed-off-by: Ben Daeuber <bdaeuber@fargolibrary.org>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2026-01-28 14:24:03 -07:00
Nick Clemens 914c8be2c0
Bug 41315: Change only notice branchcode when OverDueNoticeFrom is set to patron-homelibrary
This patch adds a new variable $notice_branchcode and uses this when getting the patron's
homelibrary to ensure we don't change the branch we are using.

To test:
 1 - At branch A - checkout three items, 1 day overdue to 3 patrons of the same category from different branches (not A)
 2 - Set the system preference OverdueNoticeFrom to 'patron home library'
 3 - Browse to More->Tools->Overdue notice/status triggers
 4 - Set overdue triggers for default library and the category used above to:
     Delay: 1
     Letter: Overdue notice (ODUE)
 5 - From the command line run the overdue notices:
     perl misc/cronjobs/overdue_notices.pl --triggered --library CPL --test --nomail -v -v
 6 - Note that we begin pass 1 with CPL and then use a different branch for passes 2 and 3
 7 - Apply patch
 8 - Repeat
 9 - Confirm branch is consistent
10 - Run without branch param:
     perl misc/cronjobs/overdue_notices.pl --triggered --library CPL --test --nomail -v -v
11 - Confirm branches are consistent

Signed-off-by: Ben Daeuber <bdaeuber@fargolibrary.org>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2026-01-28 14:24:03 -07:00
Martin Renvoize a71c3a5173
Bug 33308: (QA follow-up) Use simplified File::Transport API from bug 40811
This follow-up updates runreport.pl to use the newer simplified
File::Transport API introduced by bug 40811, which provides:

- Automatic connection management (no explicit connect/disconnect)
- Simplified directory handling via options hash
- Cleaner, more maintainable code

The upload functionality now uses a single upload_file() call with
a path option instead of requiring separate connect() and
change_directory() calls.

Also fixes minor inconsistencies in error messages (missing colons
and escaped quotes).

Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2026-01-21 15:28:18 -07:00
Jake Deery 875b35145f
Bug 33308: Add S/FTP upload support to runreport.pl
This patch adds S/FTP upload support to runreport.pl by adding additional command-line flags that will, when utilised, deposit a report onto the configured S/FTP Server.

== TO TEST ==
a)  Goto /cgi-bin/koha/admin/file_transports.pl?op=add_form
b)  Configure an S/FTP Server
    *)  If one needs an S/FTP Server to utilise, contact me directly
c)  Configure an SQL report under Reports > Create from SQL
    *)  If you're stuck, you can use `SELECT * FROM items` to select every item in k.t.d to report on
== APPLY PATCH ==
d)  Run `./misc/cronjobs/runreport.pl` without arguments
    *)  Notice `--sftp-dest-id` and `--filename`
    *)  `--sftp-dest-id` specifies the ID of the configured server to use, which can be garnered from the URL when editing the server (look for the file_transport_id parameter)
    *)  `--filename` optionally specifies what the file should be called when uploaded, and can be customised with bash variables if necessary
e)  Run `./misc/cronjons/runreport.pl --sftp-dest-id 1 1`
    *)  Notice the script outputs nothing, indicating success
    *)  On your S/FTP server, notice a report now named report(id)-(date).csv
    *)  Notice that the outputted file contains a CSV-formatted report
f)  Run `./misc/cronjobs/runreport.pl --format html --sftp-dest-id 1 1`
    *)  Notice the script outputs nothing, indicating success
    *)  On your S/FTP server, notice a report now named report(id)-(date).html
    *)  Notice that the outputted file contains an HTML-formatted report
g)  Run `./misc/cronjons/runreport.pl --filename custom.csv --sftp-dest-id 1 1`
    *)  Notice the script outputs nothing, indicating success
    *)  On your S/FTP server, notice a report now named custom.csv
    *)  Notice that the outputted file contains a CSV-formatted report
h)  Run `./misc/cronjons/runreport.pl --format html --filename custom.somehtml --sftp-dest-id 1 1`
    *)  Notice the script outputs nothing, indicating success
    *)  On your S/FTP server, notice a report now named custom.somehtml
    *)  Notice that the outputted file contains an HTML-formatted report
== SIGN OFF ==

Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2026-01-21 15:28:18 -07:00
Jonathan Druart a4c0746370
Bug 40777: Move all biblio tests to the new package
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2026-01-21 07:44:15 -07:00
Jonathan Druart b88dd24139
Bug 40777: Introduce Koha::Database::DataInconsistency
test plan:
1. Export a record with an item from the staff interface as MARCXML (Save > MARCXML).
2. Edit the file and remove the 952$a (home library) and 952$b (current library).
3. Delete the item and record in the staff interface.
4. Import the exported record and item from the MARCXML file:
   - Cataloging > Import > Stage for import
     . Settings > Format: MARCXML
   - Stage for import
   - View batch link on the job details page (assuming successfully staged)
     . New record framework: change to Books, Booklets, Workbooks
5. Turn on the browser developer tools and view the console.
6. Access the details page for the new record.
7. Note that you get:
   - A "Processing" dialog box on screen
   - A browser pop-up window/dialog box with:
       Something went wrong when loading the table.
       500: Internal server error
   - Click OK
   - The console shows:
      . Failed to load resource: the server responded with a status of 500 (Internal Server Error)
      . Got 500 (Internal Server Error)
      . DataTables warning: table id=holdings_table - Ajax error. For more information about this error, please see https://datatables.net/tn/7
8. Run misc/maintenance/search_for_data_inconsistencies.pl, you get these result (the itemnumber may be different):
    == Not defined items.homebranch and/or items.holdingbranch ==
    	* Item with itemnumber=973 does not have home and holding library defined
    => Edit these items and set valid homebranch and/or holdingbranch
    (↑↑ don't do it, it's part of the message content)
9. Apply the patch.
10. Restart everything.
11. Refresh the details screen for the imported record.
12. You now get the same as step 7, but with added "Have a look at the "Audit" button in the toolbar" text in the browser pop-up dialog box:
    - A "Processing" dialog box on screen
    - A browser pop-up window/dialog box with:
        Something went wrong when loading the table.
        500: Internal server error
        Have a look at the "Audit" button in the toolbar
    - Click OK
    - The console shows:
      . GET http[URL for the record....] 500 (Internal Server Error)
      . Got 500 (Internal Server Error)
      . DataTables warning: table id=holdings_table - Ajax error. For more information about this error, please see https://datatables.net/tn/7
13. Click on the Audit button.
14. You get a yellow box at the top of the record with:
      Errors found
      Item with itemnumber=973 does not have home and holding library defined
    (Note that there is still the "Processing" dialog box, you can edit the item by select the "Items" option in the sidebar menu and then edit the item details - the home and current library files will be auto-populated with the current library, Centerville with KTD)
15. Repeat step 8, you get the same results.
16. The tests pass: prove t/db_dependent/Koha/Database/DataInconsistency.t
17. Check the details page for another record that doesn't have any data issues - note the 'Audit' toolbar item, and if clicked should show a blue box at the top of the record with "No data inconsistency found."

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2026-01-21 07:44:13 -07:00
Rudolf Byker a762e68e0d
Bug 37538: Improve documentation for the OCLC import daemon script
This updates the documentation for the
misc/bin/connexion_import_daemon.pl --help option.

Test plan:
1. Run ./misc/bin/connexion_import_daemon.pl --help
2. Read the output and check for errors, whether grammatical,
   spelling, or factual.

Sponsored-by: Reformational Study Centre <www.refstudycentre.com>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2026-01-12 10:53:30 -07:00
David Nind ef2f7524c0
Bug 41494: Rename "Koha administration" module name to "Administration"
This changes "Koha administration" to "Administration" on the
staff interface home page and the administration module home page.

Everywhere else in the staff interface, such as browser page titles
and breadcrumbs, administration is used.

This improves consistency, and is how it is commonly referred to.

It also change some other occurences in the code base, such
as code comments.

Test plan:
1. Note the inconsistencies in how we name and refer to the
   administration module:
   1.1 Staff interface home page: Koha administration
   1.2 Administration module home page: Koha administration, but note
       that the breadcrumb and browser page title is "Administration".
   1.3 Breadcrumbs and browser page titles for any of the administration
       areas are just "Administration".
   1.4 More > Administration.
2. Apply the patch.
3. Note that on the staff interface home page and the administration
   home page are now labelled "Administration".

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2026-01-06 07:45:37 -07:00
Lucas Gass 28e5c1d1f6
25.11.00: Add release notes
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-12-05 13:31:40 -07:00
Katrin Fischer 59d07337de
Bug 41008: Fix bulkmarimport.pl -a -d
After bug 35659 that added the OAI harvester to Koha the -d switch
for deleting authorities before importing with bulkmarcimport.pl
was broken and resulted in a FK error.

Switching from truncate to DELETE FROM like we already did for items,
biblio etc. fixes the issue.

To test:
- Make sure you have some authorities in your database
- Export some into a file for importing
- Run bulkmarcimport -a -d --file <filename>
- Verify you get the a DB FK constraint error.
- Apply patch
- Repeat steps, verify the script now works as expected again
  * Existing authorities are deleted
  * File is imported

Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Jan Kissig <bibliothek@th-wildau.de>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-12-01 08:34:03 -07:00
Jonathan Druart beac9eae8f
Bug 40680: Move EXPORT_OK right after Modern::Perl
So it is done before the other use statements.

Also uniformize the pattern, must now be:
```
use Modern::Perl;
use base 'Exporter';
BEGIN {
    our @EXPORT_OK = qw(subroutine list);
}
```

Rebased-by: Victor Grousset/tuxayo <victor@tuxayo.net>
  Bug38936 exported redirect_if_opac_suppressed
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-12-01 08:33:56 -07:00
Martin Renvoize d90333e911
Bug 39532: Use new FINES restriction type to avoid conflict with SUSPENSION
This patch introduces a new unique restriction type 'FINES' specifically
for the debar_patrons_with_fines.pl script to prevent duplicate
restrictions on repeated runs.

The original fix used 'SUSPENSION' but this conflicted with the
fine-in-days feature which uses SUSPENSION for time-limited restrictions
when overdue items are returned. Running the script would overwrite
those time-limited restrictions with indefinite ones.

Changes:
- Add new 'FINES' system restriction type to patron_restriction_types.yml
  for new installations
- Create database update to add 'FINES' type to existing installations
- Update debar_patrons_with_fines.pl to use 'FINES' instead of 'SUSPENSION'
- Update AddUniqueDebarment and DelUniqueDebarment POD documentation

This keeps SUSPENSION exclusively for the fine-in-days feature while
preventing duplicate restrictions from the fines script.

Test plan:
1. Run database update
2. Verify new FINES restriction type exists in restriction_types table
3. Run debar_patrons_with_fines.pl multiple times
4. Confirm only one FINES restriction per patron (no duplicates)
5. Verify SUSPENSION restrictions from fine-in-days are not affected
6. Run tests: prove t/db_dependent/Patron/Borrower_Debarments.t

Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-11-21 08:41:51 -07:00
Kyle M Hall fdb7c9b405
Bug 39532: Script debar_patrons_with_fines.pl creates repeat restrictions, uses wrong restriction type
The cronjob debar_patrons_with_fines.pl is meant to add a restriction based on a fines threshold.

The problem is that the script adds a MANUAL restriction, which makes no sense. MANUAL restrictions are meant to be manually created, never automatic as this script does.
In addition, they are repeatable, not unique. Every time this script runs, it will add an additional restriction.

There is already a restriction type meant for this type of use case: SUSPENSION
We should change this script to create SUSPENSION restrictions instead of MANUAL restrictions.

In summary, this script should use the unique restriction type SUSPENSION to prevent multiple additional restrictions from being created on each run.

Test Plan:
1) Set up a patron with fines
2) Run misc/cronjobs/debar_patrons_with_fines.pl -c -m "Test message"
3) Note the restriction is created
4) Run misc/cronjobs/debar_patrons_with_fines.pl -c -m "Test message" again
5) Note another restriction is created
6) Delete the restrictions
7) Apply this patch
8) Run misc/cronjobs/debar_patrons_with_fines.pl -c -m "Test message"
9) Note the restriction is created
10) Run misc/cronjobs/debar_patrons_with_fines.pl -c -m "Test message" again
11) Note another restriction is *not* created!

Signed-off-by: Magnus Enger <magnus@libriotech.no>
Signed-off-by: Emmi Takkinen <emmi.takkinen@koha-suomi.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-11-21 08:41:50 -07:00
Aleisha Amohia a7bd032aff
Bug 38306: Only print EnhancedMessagingPreferences warning if run with verbose
This enhancement will amend the automatic_renewals.pl script to only print a warning about the EnhancedMessagingPreferences system preference being disabled if the script was run with verbose.

To test:

1. Disable EnhancedMessagingPreferences
2. Run misc/cronjobs/automatic_renewals.pl
3. Notice the warning is printed
4. Apply the patch and restart services
5. Run misc/cronjobs/automatic_renewals.pl and notice the warning is not printed
6. Run misc/cronjobs/automatic_renewals.pl --verbose and notice the warning is printed
7. Enable EnhancedMessagingPreferences
8. Run misc/cronjobs/automatic_renewals.pl and notice the warning is not printed

Sponsored-by: Catalyst IT
Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-11-03 14:13:44 -07:00
Martin Renvoize 7e576e58f8
Bug 38115: (follow-up) Use File::Basename to extract filename for remote upload
When --filename contains a path (e.g., /tmp/records.mrc), the script
was uploading to the remote server with the full path, creating
unwanted directory structures (e.g., uploads/tmp/records.mrc).

This patch uses File::Basename::fileparse() to extract just the
filename portion for the remote upload, while still using the full
path for local file operations.

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-11-03 14:13:32 -07:00
Martin Renvoize e7307ddc05
Bug 38115: (follow-up) Improve error handling and connection management
Enhance the export_records.pl script with better error handling
and connection lifecycle management:

1. Connection Management:
   - Always call disconnect() after file transport operations
   - Ensure connections are cleaned up even on errors

2. Error Handling:
   - Replace unusual die+pod2usage with clear error messages
   - Add file existence validation before upload attempts
   - Provide detailed error messages with context
   - Consistent error message formatting

3. User Experience:
   - Add validation for --delete_local_after_run without --destination_server_id
   - Provide informative success messages to STDERR
   - Better error messages with specific failure reasons

4. Robustness:
   - Check if output file was created before upload
   - Check if file exists before attempting deletion
   - Graceful handling of missing upload directories

These improvements make the script more reliable and user-friendly
while maintaining all existing functionality.

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-11-03 14:13:32 -07:00
Tomás Cohen Arazi 631a14ec2e
Bug 38115: (follow-up) Update terminology from SFTP/FTP to file transport
This follow-up patch updates the terminology in export_records.pl to use
generic 'file transport' language instead of specific 'FTP/SFTP' references,
aligning with the recent renaming of the underlying classes to File::Transports.

Changes:
- Replace $sftp_server variable with $file_transport for clarity
- Update error messages to use 'file transport server' terminology
- Fix incorrect error message for report_id validation
- Update POD documentation to reference 'file transport servers'
- Maintain consistency with the renamed admin interface and API

This ensures the script terminology matches the current class structure
and admin interface naming conventions.

Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-11-03 14:13:31 -07:00
Jake Deery 1ee76c6a9f
Bug 38115: Add destination_server_id to export_records
This patch adds two new flags to the export_records.pl script.
* --destination_server_id=ID
    This flag will, when passed, upload the file generated by this script to
    the destination server, as configured in the FTP/SFTP Servers settings.
* --delete_local_after_run
    This flag will, when passed, cause the script to unlink (delete) the
    generated marc or CSV file found on the local disk. This is useful for folk
    who want to upload the file, but not necessarily keep a local copy, filling
    up the server's storage medium

TO TEST:

a)  APPLY PATCH, reset_all
b)  Create valid FTP/SFTP Server in Koha settings, ensuring that the server
    you create is valid and passes testing
c)  Run ./misc/export_records.pl with no agruments
    1)  Notice how koha.mrc file is generated
d)  Run ./misc/export_records.pl --destination_server_id=1
    1)  Notice how koha.mrc file is generated
    2)  On your FTP/SFTP Server, notice how koha.mrc is now present in the
        upload directory
e)  Run ./misc/export_records.pl --delete_local_after_run
    1)  Notice how koha.mrc file is NOT generated locally
f)  Run ./misc/export_records.pl --destination_server_id=1 --delete_local_after_run
    1)  Notice how koha.mrc file is NOT generated locally
    2)  On your FTP/SFTP Server, notice how koha.mrc is still present in the
        upload directory AND the file age is fresh

Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-11-03 14:13:31 -07:00
Marcel de Rooy 4a267d0e9b
Bug 41024: Unify spelling of BorrowersLog, keep plural form
The singular form is wrong and only exists in textual context.
This is a result of git grep | xargs sed.

Test plan:
Read the patch.
git grep BorrowerLog

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Aleisha Amohia <aleisha@catalyst.net.nz>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-10-31 15:26:10 -06:00
Nick Clemens deb11ee1c8
Bug 40722: Add a log when resetting mappings during elastic rebuild
The command currently logs when deleting indexes and indexing records, it would be helpful
to have a log when resetting the mappings, and to log which mapping file has been used.

To test:
1 - perl misc/search_tools/rebuild_elasticsearch.pl -r
2 - No log of reset
3 - Apply patch
4 - perl misc/search_tools/rebuild_elasticsearch.pl -r
5 - Note reset is logged

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-10-15 16:12:07 -06:00
Lucas Gass 83ac70172b
Bug 40545: (follow-up) Correct GNU link
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-10-15 16:12:02 -06:00
Tomas Cohen Arazi 78b659c1a9
Bug 40545: Add utility script for resetting 2FA settings
This adds a command-line utility script that allows administrators to reset
a patron's two-factor authentication settings when they lose access to their
authenticator device.

The script uses the new Koha::Patron methods and provides:
* Parameter validation (mutually exclusive userid/cardnumber/patron_id)
* User-friendly error messages and feedback
* Verification that patron has 2FA enabled before attempting reset

To test:
1. Enable TwoFactorAuthentication system preference
2. Set up 2FA for a test patron
3. Run:
   $ ktd --shell
  k$ perl misc/admin/reset_2fa.pl --userid <userid>
=> SUCCESS: Script resets 2FA and shows confirmation message
4. Verify patron can log in with password only
5. Test error cases:
   - No parameters: shows usage message
   - Multiple parameters: shows mutual exclusivity error
   - Non-existent user: shows user not found error
   - User without 2FA: shows informational message
=> SUCCESS: All scenarios work as expected
6. Sign off :-D

Sponsored-by: ByWater Solutions
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-10-15 16:12:01 -06:00
Nick Clemens 4593392539
Bug 39636: Update curl link
With the move to the latest ES versions we got rid of 'data' type - we need to use '_doc'

To test:
1 - Delete a record from the DB directly
    DELETE FROM biblio WHERE biblionumber=45;
2 - perl misc/maintenance/compare_es_to_db.pl
3 - Try the curl command - it doesn't work
4 - Apply this patch
5 - perl misc/maintenance/compare_es_to_db.pl
6 - Try the curl command - it now fetches the record

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-10-14 14:52:00 -06:00
Nick Clemens ad9648528c
Bug 39636: Add option to specify which indexes to check
This patch adds --biblios and --authorities options to the script to allow
checking only a single index

To test:
0 - Apply patch
1 - perl misc/maintenance/compare_es_to_db.pl
2 - biblios and authorities are checked
3 - perl misc/maintenance/compare_es_to_db.pl -b
4 - Only biblios are checked
5 - perl misc/maintenance/compare_es_to_db.pl -a
6 - Only authorities are checked
7 - perl misc/maintenance/compare_es_to_db.pl -a -b
8 - Both indexes are checked

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-10-14 14:52:00 -06:00
Fridolin Somers 3aefbaeaf6
Bug 40785: Cronjob cleanup_database.pl remove args in usage first line
Cronjob cleanup_database.pl usage is wrong :
Usage: $0 [-h|--help] [--confirm] (...) [--bg-days DAYS [--bg-type TYPE] ]
Those args has been renamed jobs-days and jobs-type.
Also some args are missing.

There are too many args, I propose to remove them from there, one need to read the full help.

Test plan:
Run command line `misc/cronjobs/cleanup_database.pl -h`
You see first line:
Usage: misc/cronjobs/cleanup_database.pl

   -h --help          prints this help message, and exits, ignoring all
...

Signed-off-by: Anke Bruns <anke.bruns@gwdg.de>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-10-09 14:58:38 -06:00
Lisette Scheer b895eb9b7d
Bug 35700: Holds reminder cronjob - Fix --triggered switch if the day to send notice hits concurrent holidays
If the holds reminder cronjob would send a reminder on a holiday,when holidays are taken into account, it also will send it on the subsequent holidays and the first open day.
This bugfix adds a check to the cron when --holidays is set to skip sending the notices for that library if today is a holiday.

To Test:
Before applying:
1. Create a hold for a patron (I used patron 23529000035676 and 39999000000252)
2. Ensure your patron has an email address set and the hold reminder email selected.
3. Set the hold to waiting by checking it and confirming the hold.
4. In Tools->Calendar add a holiday to yesterday and a seperate holiday to today.
5. On the command line (the date should be set to 5 days ago for my commands to work and any dates adjusted accordingly)
koha-mysql kohadev
update reserves set waitingdate="2025-09-25"
exit
perl misc/cronjobs/holds/holds_reminder.pl -days 3 -t -v -holidays --date 2025-09-29
6. Observe that 1 notice would have sent to your patron
7. On the command line
perl misc/cronjobs/holds/holds_reminder.pl -days 3 -t -v -holidays
8. Observe that 1 notice would have sent to your patron
9. On the command line
perl misc/cronjobs/holds/holds_reminder.pl -days 3 -t -v -holidays --date 2025-10-01
10. Observe that 1 notice would have sent to your patron

11. Apply patches
12. Repeat the last command from step 5 and step 7
13. Observe that no notice should be sent for your patron
14. Repeat step 7
15. Observe that a notice should be sent for your patron.
16. Run
perl misc/cronjobs/holds/holds_reminder.pl -days 5 -t -v
17. Confirm the notice should be sent for your patron
18. Sign off

Signed-off-by: Katie Bliss <KEBliss@dmpl.org>

Bug 35700: (QA follow-up) fix POD and $today

This patch replaces the inncorrectly removed POD note and replaces $today with the more correct $date_to_run for checking if we should be skipping sending the message on the day.

Amended-by: Jonathan Druart
Squashed and adjusted commit title

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-10-09 14:58:38 -06:00
Kyle M Hall a067423f3f
Bug 40953: marc_ordering_process.pl broken due to accidental newline
Executing marc_ordering_process.pl will fail to run with the following errors:

/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 21: =head1: command not found
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 23: marc_ordering_process.pl: command not found
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 25: =head1: command not found
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 27: --confirm]: command not found
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 27: --verbose]: command not found
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 27: ./marc_ordering_process.pl: No such file or directory
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 29: or,: command not found
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 31: 0: command not found
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 33: =head1: command not found
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 35: This: command not found
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 36: If: command not found
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 38: =head1: command not found
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 40: =over: command not found
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 42: syntax error near unexpected token `newline'
/usr/share/koha/bin/cronjobs/marc_ordering_process.pl: line 42: `=item B<-v|--verbose>'

This is due to an inadvertent newline before the shebang of the script.

Test Plan:
1) Run ./misc/cronjobs/marc_ordering_process.pl
2) Note the error
3) Apply this patch
4) Run ./misc/cronjobs/marc_ordering_process.pl
5) No error!

Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-10-06 09:15:23 -06:00
Tomás Cohen Arazi 9ca0e183b5
Bug 40820: (follow-up) es_indexer_daemon.pl missed on 34070
The es_indexer_daemon.pl was missing handling for the case where
Koha::BackgroundJob->connect() returns undef without throwing
an exception (which happens when connection fails silently).

This adds the missing warning for that case, matching the behavior
in background_jobs_worker.pl.

Test plan:
1. Set JobsNotificationMethod to 'STOMP'
2. Configure invalid STOMP settings that cause connect() to return undef
3. Run es_indexer_daemon.pl - should show connection warning
4. Verify jobs are still processed via database polling

Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-09-26 10:36:58 -06:00
Tomás Cohen Arazi 4ff1f4ca9c
Bug 40820: Fix STOMP connection warnings when JobsNotificationMethod is 'polling'
The background job workers are showing STOMP/RabbitMQ connection error
warnings even when the JobsNotificationMethod system preference was set
to 'polling', which doesn't require a message broker connection.

This patch adds checks for the JobsNotificationMethod preference before
displaying connection warnings, so warnings only appear when STOMP is
actually expected to be available.

Changes:
- misc/workers/background_jobs_worker.pl: Check syspref before warning
- misc/workers/es_indexer_daemon.pl: Check syspref before warning

Test plan:
1. Set JobsNotificationMethod to 'polling'
2. Run background_jobs_worker.pl - should not show STOMP warnings
3. Run es_indexer_daemon.pl - should not show STOMP warnings
4. Set JobsNotificationMethod to 'STOMP' without RabbitMQ running
5. Run workers - should show appropriate STOMP connection warnings
6. Sign off :-D

Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-09-26 10:36:57 -06:00
David Cook f199a09668
Bug 40748: Check for HTTP URLs and skip shell when using system()
This change checks for HTTP(S) URLs and skips the shell when using
system() so that shell metacharacters aren't interpreted.

Test plan:
0. Apply the patch
1. Insert the following into the Babeltheque_url_update system preference:
$(whoami > /tmp/rce.log)
2. koha-shell kohadev
3. perl misc/cronjobs/social_data/update_social_data.pl
4. Note the script says the following:
$(whoami > /tmp/rce.log) is not a HTTP URL. Aborting.
5. Note also that /tmp/rce.log is not created
6. Change Babeltheque_url_update syspref to something like the following:
http://localhost:8080
7. perl misc/cronjobs/social_data/update_social_data.pl
8. Note that the wget succeeds but the bunzip2 fails (as expected) since
we haven't fetched a bzipped file
9. Try to create a payload that passes the HTTP URL test with shell
metacharacters and note that they're not interpetted and instead
just used as part of a URL string.

Signed-off-by: Jake Deery <jake.deery@openfifth.co.uk>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-09-26 09:10:20 -06:00
Jake Deery 0d89cb1f01
Bug 40748: Fix RCE in update_social_data.pl
This patch fixes an identified RCE in the update_social_data.pl
social_data, which when executed, allows an attacker to run arbitrary
code on the system's shell.

TO TEST:

a)  In the Babeltheque_url_update syspref, wrap any bash in $(), and then
run the update_social_data.pl script.
    *)  For example, $(whoami >
/tmp/rce.log) will output to a file the username the perl script is
running as.
APPLY PATCH
b)  Run the update_social_data.pl script again. The string is now
meta-escaped, and the RCE no longer works.

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-09-26 09:10:20 -06:00
Tomás Cohen Arazi 6b478e0da9
Bug 40163: (follow-up) Fix wrong GNU links and update test
Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-09-24 09:25:04 -06:00
Tomas Cohen Arazi 1873d97c07
Bug 40163: Fix several http addresses in the code docs
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-09-23 16:21:29 -06:00
Brendan Lawlor 2de91aeacd
Bug 35654: (follow-up) Update example script to be compatible with change to using Koha::Items
Since this patch updates the script to use Koha::Items->search instead of a direct database query
--where options that refer to the items table will be broken and result in an error like:
execute failed: Unknown column 'items.withdrawn' in 'where clause'

Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-08-29 10:43:28 -06:00
Brendan Lawlor 80b04f7a90
Bug 35654: Add option to delete_items.pl cronjob to delete empty bibs
This patch adds a new option to the delete_items cronjob

When --del_bibs is set, bib records will be deleted
if the last item is deleted.

The following plan assumes default ktd data.

To test:
1. Apply patch
2. Check the holdings for bib #115 and see 3 items
  http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=115
2. Run the cron with the following flags
  perl ./misc/cronjobs/delete_items.pl --where "biblionumber=115 and homebranch ='MPL'" --verbose --del_bibs --commit
3. Notice in the output the count of current items on the bib record:
  Where statement:  where biblionumber=115 and homebranch='MPL'
  Deleting empty bib records!
  Deleting item '237' (of 3)
  Deleting item '238' (of 2)
4. Confirm the two Midway items were deleted from bib #115, and the bib still exists with one item
4. Run the cron again but change the the --where flag to delete all the items from bib #115
  perl ./misc/cronjobs/delete_items.pl --where "biblionumber=115" --verbose --del_bibs --commit
5. Notice in the output the bib is deleted after the last item is deleted
  Where statement:  where biblionumber=115
  Deleting empty bib records!
  Deleting item '239' (of 1)
  Last item deleted - deleting bib '115'
8. Confirm the bib #115 was deleted
9. Add some more items and test with some other --where parameters
10. Test with and without --commit and --del_bibs, test the --help flag

Sponsored-by: CLAMS
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-08-29 10:43:28 -06:00
Pedro Amorim 77b1fe53ef
Bug 34978: (QA follow-up) Fix rebase issues
- Rebased tests patch (62->51)
- tidy
- spelling ( paramater -> parameter )

Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Hammat Wele <hammat.wele@inlibro.com>
Signed-off-by: Amaury GAU <amaurygau@gmail.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-08-14 16:05:56 -06:00
Emily-Rose Francoeur dd60490292
Bug 34978: Add --include and --exclude options to install_plugins.pl
I have added the options --include and --exclude to the install_plugins.pl script. When running the script, you can now specify which plugins you want to install (--include) and which ones you do not want to install (--exclude).

--include installs only the plugins from the specified classes.
--exclude installs all plugins except those from the specified classes.
It's possible to specify the package to install a specific plugin.
--include and --exclude cannot be used simultaneously.

Usage example:
./misc/devel/install_plugins --include <package::pluginClass> --include <anotherPluginClass>"

TEST PLAN
1) Apply the patch
2) Add at least three plugins to the plugin directory ([Koha]/plugins/Koha/Plugin/)
3) Empty the "plugin_data" and "plugin_methods" tables (SQL command: "delete from plugin_data; delete from plugin_methods;")
4) In a terminal, execute this command "./misc/devel/install_plugins.pl --include <package::pluginClass> --include <anotherPluginClass>" (Replace <package::pluginClass> and <anotherPluginClass> with the class and package of your plugins)
5) In your Koha intranet, go to "Koha administration > Manage plugins"
6) Verify that only the specified plugins are installed
7) Repeat step 3
8) In a terminal, execute this command from your Koha project"./misc/devel/install_plugins.pl --exclude <package::pluginClass> --exclude <anotherPluginClass>"
9) Verify that all plugins except the specified ones are installed
10) Run the tests from the Plugins.t file ([Koha]/t/db_dependent/Koha/Plugins/Plugins.t)
11) Ensure that all tests are successful

Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Hammat Wele <hammat.wele@inlibro.com>
Signed-off-by: Amaury GAU <amaurygau@gmail.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-08-14 16:05:56 -06:00
Jonathan Druart 03cf21fd92
Bug 40407: Remove legacy "pre-wrap" versions
From https://css-tricks.com/snippets/css/make-pre-text-wrap/

/* Browser specific (not valid) styles to make preformatted text wrap */

pre {
 white-space: pre-wrap;       /* css-3 */
 white-space: -moz-pre-wrap;  /* Mozilla, since 1999 */
 white-space: -pre-wrap;      /* Opera 4-6 */
 white-space: -o-pre-wrap;    /* Opera 7 */
 word-wrap: break-word;       /* Internet Explorer 5.5+ */
}

... I think it's now safe to remove the backward compatibility versions and simply keep pre-wrap.

Test plan (from bug 921)0:
  * place a hold on a title with a very very long title or author
  * mark it as "waiting for pickup" by doing a checkin
  * go to command line interface, and run
  * misc/cronjobs/gather_print_notices.pl /tmp
  * misc/cronjobs/printoverdues.sh /tmp
  * if your title/author is long enough, it's splitted on 2 lines
    with this patch (it is not before this patch)

It will also remove the warnings from t/db_dependent/Patron/Borrower_Discharge.t
WARNING: Ignored `white-space: -moz-pre-wrap` at 6:14, invalid value.
WARNING: Ignored `white-space: -o-pre-wrap` at 7:14, invalid value.
WARNING: Ignored `word-wrap: break-work` at 8:14, invalid value.

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-07-21 14:01:32 -06:00
Jonathan Druart e7e42c5e88
Bug 39876: Add context for tidy
With this patch we adjust what's needed to make misc/devel/tidy.pl run
identically as before

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-07-08 15:41:59 -06:00
Jonathan Druart 1d4071298c
Bug 39876: Move `git ls-files` to Koha::Devel::Files
This patch introduces a new Koha::Devel namespace. It clearly shows
that its purpose is for development and will not contain any business
logic.

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-07-08 15:41:58 -06:00
Tomas Cohen Arazi 8f90ac9d4e
Bug 40144: Get rid of `sip_cli_emulator.pl` warnings
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-06-24 15:22:55 -06:00
Tomas Cohen Arazi 46cc88acc3
Bug 23883: sip_cli_emulator.pl should display real parameter names
For creating this patch I looked at all parameters defined in the
handler mapping, removed duplicated ones and tracked down which ones
really were parameters in `GetOptions`. This included tracking down
variables that were later assigned other values and things like that.

The mapping is used to display the right parameter name if missing a
mandatory one.

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-06-24 15:22:54 -06:00
Baptiste Wojtkowski 34886a9b5b
Bug 40109: Fix path for fix_invalid_dates.pl in hint from search_for_data_inconsistencies.pl
Bug 31143 created a script in cronjob and then moved it to maintenance. However, it is still referred as in cronjob in help strings. This patch updates the hint.

No test needed I guess. Only changing a string

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-06-24 15:22:54 -06:00
Jonathan Druart 492e6814a1
Bug 39834: Replace some tabs with spaces
In order to remove false positives with the QA script we should replace some tabs with spaces.

There are more but I wanted to make sure no regression would be
introduced

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-06-02 15:26:17 -06:00
Martin Renvoize f6fd9f2e90
Bug 39887: Improve documentation for overdue_notices.pl
This patch corrects a few spelling mistakes and missing grammer and adds
a little more detail to the '--date', '--email' and '--frombranch'
options.

Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
2025-06-02 14:55:36 -06:00
Katrin Fischer ad4dfd097d
25.05.05: Add release notes
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-05-26 18:45:50 +02:00
Matt Blenkinsop 9a57c7c8f8
Bug 30301: Allow for forcing of print notices
Although we're reverting 32216, we still want to be able to force print notices as per the original intent.
This patch adds that into queue_notices, rather than in the cronjob

Co-authored-by: Jake Deery <jake.deery@openfifth.co.uk>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-05-19 18:54:32 +02:00
Matt Blenkinsop 9b5da196c4
Bug 30301: Restore check for mandatory notice
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-05-19 18:54:32 +02:00
Matt Blenkinsop c54a14a43f
Bug 30301: Revert "Bug 32216: Add print notices to membership_expiry.pl"
This reverts commit 3040f46d5e.

Bug 30300 adds a patron messaging preference for the expiration notice and abstracts this logic into queue_notice
Bug 32216 removes this call to enable print notice functionality but this would be handled already by queue_notice based on the patron's messaging preferences

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-05-19 18:54:31 +02:00
Matt Blenkinsop db9f3126ef
Bug 30301: Update cronjob to check for mandatory notice
The patches on this bug add the ability for staff members to set the patron expiry notice as mandatory at the category level. When set, the cronjob will automatically send an expiry notice regardless of existing messaging preferences. If it is turned off, the old messaging preferences will still be in place and actioned

Test plan:
1) Apply patch and run updatedatabase
2) In system preferences set MembershipExpiryDaysNotice to 5
3) Choose a patron and edit their expiry date to today's date, ensuring they have a primary email set
4) Create a MEMBERSHIP_EXPIRY notice (content of the notice is irrelevant, just needs to exist)
5) Run perl misc/cronjobs/membership_expiry.pl -c -v -before 10 -after 10
6) Check the patron's messages, there should be no expiry notice queued
7) Set the patron's category 'enforce expiry notice' field to Yes
8) Repeat step 5, there should be a notice queued
9) Set that patron's primary contact method to primary email and run step 5 again
10) There should be a notice queued for an email
11) Set the category level enforce expiry notice back to No
12) Choose a second patron and set an email address, expiry date to today and set the patron expiry messaging preference for that patron to 'email'
13) Repeat step 5 - this time only one notice should be queued on the second patron, no new notice should queue for the first
14) Re-set the category level option to Yes
15) Repeat step 5 - this time both patron's should have notices queued
16) Unset the category level option again, the behaviour should be the same as step 13
17) Set the category level option back to Yes and check the patron messaign preferences menus in both the OPAC and Staff client - the patron expiry option should not be visible while the cateogry level enforcement is set to Yes

Signed-off-by: Hanna Leiker <tech.hanna@salinapublic.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-05-19 18:54:30 +02:00
Jonathan Druart 8dfb19a45a
Bug 34070: Display the 'jobs will be processed anyway' message from worker
->connect does not raise an exception, so we didn't reached the catch
block.

Adjusting the code but I am not really happy with it, but more changes
will derail the original bug

Now we log:
Cannot connect to broker (Access refused for user 'guest') at /kohadevbox/koha/Koha/BackgroundJob.pm line 96.
Cannot connect to the message broker, the jobs will be processed anyway at /kohadevbox/koha/misc/workers/background_jobs_worker.pl line 100.

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-05-19 18:54:26 +02:00
David Cook 4dc4a232c4
Bug 34070: Add error handling for ERROR frames for background worker
This change adds error handling for ERROR frames to background_jobs_worker.pl.
A previous patch added it to es_indexer_daemon.pl which is a different
background jobs worker.

To test ERROR handling for background_jobs_worker.pl:
1. Change your RabbitMQ connection details in koha-conf.xml or the hard-coded ones in Koha/BackgroundJob.pm
to include an invalid password. For example, if you don't have a message_broker already defined, use the following:
<message_broker>
 <password>bad</password>
</message_broker>
2. sudo koha-worker --restart --quiet kohadev
3. View the errors in the logs:
tail -f /var/log/koha/kohadev/worker-output.log
tail -f /var/log/koha/kohadev/worker-error.log

To test MESSAGE handling for background_jobs_worker.pl:
1. Ensure RabbitMQ connection details are correct
2. Go to Batch item modification and add the following barcodes:
39999000001310
39999000004571
3. Add a z - Public Note of the following and click Save:
THEBACKGROUNDWORKERWORKED
4. Search the catalogue using THEBACKGROUNDWORKERWORKED in the top
catalogue search
5. Note that you get 2 results in your search
6. Look in the logs just to make sure there are no new messages there

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-05-19 18:54:25 +02:00
David Cook 2c1314ead6
Bug 34070: Add error handling for ERROR frames
This change causes es_indexer_daemon.pl to die on ERROR frames instead
of spinning in an infinite loop. It also adds a content-type STOMP frame header
to help consumers know what data type is in the MESSAGE frame body.

To test ERROR handling for es_indexer_daemon.pl:
1. Switch to using Elasticsearch
2. Change your RabbitMQ connection details in koha-conf.xml or the hard-coded ones in Koha/BackgroundJob.pm
to include an invalid password. For example, if you don't have a message_broker already defined, use the following:
<message_broker>
 <password>bad</password>
</message_broker>
3. sudo koha-es-indexer --restart kohadev
4. View the errors in the logs:
tail -f /var/log/koha/kohadev/es-indexer-output.log
tail -f /var/log/koha/kohadev/es-indexer-error.log

To test MESSAGE handling for es_indexer_daemon.pl:
1. Switch to using Elasticsearch (and ensure RabbitMQ connection details are correct)
2. Change the title of a bib record to something new and unique
3. Try to search for the bib using the new title
4. Note you can find it
5. Look in the logs just to make sure there are no messages there

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-05-19 18:54:24 +02:00
Fridolin Somers 269ad710b2
Bug 39733: Remove system preference delimiter mention in misc/cronjobs/staticfines.pl
Bug 27486 renamed system preference "delimiter" into "CSVDelimiter"

This patch just replaces the commented code with a TODO text

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-05-15 08:34:17 +02:00
Katrin Fischer 202d0e1fcc
Revert "Bug 39532: Script debar_patrons_with_fines.pl creates repeat restrictions, uses wrong restriction type"
This reverts commit 94286343c7.

Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-05-13 18:50:47 +02:00
Kyle M Hall 94286343c7
Bug 39532: Script debar_patrons_with_fines.pl creates repeat restrictions, uses wrong restriction type
The cronjob debar_patrons_with_fines.pl is meant to add a restriction based on a fines threshold.

The problem is that the script adds a MANUAL restriction, which makes no sense. MANUAL restrictions are meant to be manually created, never automatic as this script does.
In addition, they are repeatable, not unique. Every time this script runs, it will add an additional restriction.

There is already a restriction type meant for this type of use case: SUSPENSION
We should change this script to create SUSPENSION restrictions instead of MANUAL restrictions.

In summary, this script should use the unique restriction type SUSPENSION to prevent multiple additional restrictions from being created on each run.

Test Plan:
1) Set up a patron with fines
2) Run misc/cronjobs/debar_patrons_with_fines.pl -c -m "Test message"
3) Note the restriction is created
4) Run misc/cronjobs/debar_patrons_with_fines.pl -c -m "Test message" again
5) Note another restriction is created
6) Delete the restrictions
7) Apply this patch
8) Run misc/cronjobs/debar_patrons_with_fines.pl -c -m "Test message"
9) Note the restriction is created
10) Run misc/cronjobs/debar_patrons_with_fines.pl -c -m "Test message" again
11) Note another restriction is *not* created!

Signed-off-by: Magnus Enger <magnus@libriotech.no>
Signed-off-by: Emmi Takkinen <emmi.takkinen@koha-suomi.fi>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-05-13 18:47:02 +02:00
Nick Clemens f2d40e1c69
Bug 39413: Add a check for item fields in MARC records to data inconsistencies script
This patch adds a new test to the data inconsistencies script to report where
MARC records contain item fields.

These may not cause issues in Koha, however, in discovery harvests they have eben found to create errors

To test:
1 - Follow test plan on bug 39412 to generate records with item tags
2 - perl misc/maintenance/search_for_data_inconsistencies.pl
3 - Note new section and hint on how to fix
4 - perl misc/maintenance/touch_all_biblios.pl
5 - repeat 2
6 - Errors cleared

Signed-off-by: Magnus Enger <magnus@libriotech.no>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-05-12 18:31:21 +02:00
Tomas Cohen Arazi 20b600d450
Bug 39694: Make `es_indexer_daemon.pl` use batch_size to limit DB poll size
This patch makes the indexer poll the DB in configurable batches. Pretty
much like Rabbit does.

The only change is that it will now pass a LIMIT <batch_size> to the
query.

To test:
0. Have KTD started with ES (skip if you have an instance running):
   $ ktd --es7 up -d
1. Stop the indexer and rabbit:
   $ ktd --shell
  k$ sudo koha-es-indexer --stop kohadev
  k$ sudo service rabbitmq-server stop
2. On a separate terminal, watch the queue:
  k$ watch -n 0.5 'echo "SELECT COUNT(*) FROM background_jobs WHERE status=\"new\"" | koha-mysql kohadev'
=> SUCCESS: It should be empty
3. Add a bunch of things to index (this works with default KTD data):
  k$ for i in $(seq 1 438; seq 1 438) ; do echo "INSERT INTO background_jobs (status,queue,data) VALUES ('new','elastic_index','{\"record_server\":\"biblioserver\",\"record_ids\":[$i]}')" | koha-mysql kohadev ; done
4. Start the ES indexer:
  k$ sudo koha-es-indexer --start kohadev
5. Look at the queue size
=> FAIL: It shrinks too fast [1]
6. Apply this patch
7. Restart the indexer
  k$ perl misc/workers/es_indexer_daemon.pl &
8. Repeat 3
9. Look at the queue size:
=> SUCCESS: It shrinks 'slowly'
10. Sign off :-D

[1] This is a problem in certain scenarios, not in all of them. This is
the best I come up with in order to test the behavior change. Anyone can
set a new size of their taste.

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-05-12 18:31:13 +02:00
Thomas Klausner 132593c8a8
Bug 37020: (follow-up): Remove memleak when not using Elasticsearch
If using ES, the script will collect the newly created records and biblionumbers in atwo arrays (@search_engine_record, @search_engine_record_ids) and pass that to the indexer when $commitnum records are read. The arrays are cleaned afterwards.

BUT: The records and biblionumbers were always stored, even if we're not using Elasticsearch. And as the cleaning of those arrays only happens if we are in fact using ES, all the data is added but never cleaned.

This patch fixes this by only storing record and biblionumber if we have an $indexer (i.e. if we're using Elasticsearch).

I have no idea though how/if the records are indexed when using Zebra.

After applying this patch and importing a large enough file without using Elasticsearch, the RAM usage also stays more or less constant.

@Jan, please verify!

Signed-off-by: Jan Kissig <bibliothek@th-wildau.de>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-05-12 18:31:12 +02:00
Thomas Klausner 01d36c5e8b
Bug 37020: (follow-up) Fix txn handling, loop label and record_number counter
So, I reviewed the code, which has a bunch of issues:

* there's a call to `next RECORD`, but no `RECORD` loop label.
* $record_number is incremented twice
* and the calls to txn_begin is moved from the outside the loop into each record, starting one transaction per record. Not sure how mysql handles this, but postgres would not like it.

This patch fixes those issues:

After I fixed the first two issues, I ran the patched script on a file containing 75 authorities (from GND), and not one was added to the DB, presuambly because nested transactions don't work in mysql?

So I also moved the BEGIN transaction outside of the main loop and started a new one after each commit, and added a final commit after to loop.

Signed-off-by: Jan Kissig <bibliothek@th-wildau.de>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-05-12 18:31:11 +02:00
David Gustafsson ba0979a4c9
Bug 37020: bulkmarcimport gets killed after update to 24.05. when inserting large files
Include checking for invalid marc data in main loop to avoid
loading all marc data into memory at once.

Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-05-12 18:31:11 +02:00
Nick Clemens b805f5ff99
Bug 39325: (QA follow-up) Adjust some issues
connexion is a product, added to ignore file

tablee -> tables

a few other small corrections

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-04-28 18:51:30 +02:00
Jonathan Druart 69b8341a8c
Bug 39325: Manual fixes
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-04-28 18:51:23 +02:00
Jonathan Druart 1d844a7d2f
Bug 39325: Automatic fix
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-04-28 18:51:22 +02:00
Raphael Straub d1d9b24979
Bug 37418: delete_patrons.pl: Exclude patrons with restrictions
Add the option "--without_restriction_type" so that delete_patrons.pl
does not delete patrons with given restriction types.

To test:
1) Run
   misc/cronjobs/delete_patrons.pl --not_borrowed_since 2010-01-01
   and choose a borrowernumber that would be deleted.
2) In your browser go to
   /cgi-bin/koha/members/moremember.pl?borrowernumber=<borrowernumber>
   and add a manual restriction to this patron.
3) Apply the patch.
4) Run
   misc/cronjobs/delete_patrons.pl --not_borrowed_since 2010-01-01 \
       --without_restriction_type MANUAL
5) Check that the patron with the manual restriction would not be
   deleted.

Sponsored-by: Karlsruhe Institute of Technology (KIT)
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-04-25 08:46:53 +02:00
Nick Clemens e744f001ae
Bug 39652: (QA follow-up) Update misc/maintenance/pseudonymize_statistics.pl
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-04-24 19:53:17 +02:00
Nick Clemens 342c1d5993
Bug 39301: pseudonymize_statistics.pl script generates too many background jobs
This patch changes the script to use 'Koha::PseudonymizedTransaction->new_from_statistic' directly
rather than calling 'Koha::Statistic->pseudonymize()' which generates a background job.

Additionally, some of the checks from 'pseudonymnize' and the background job are moved into the search
so we don't fetch any to pseudonymize that should not be.

To test:
 1 - Generate some entries in statistics (checkin, checkout, renew, etc)
 2 - Enable pseudonymization (see bug 28911 if you need bcrypt settings)
 3 - perl misc/maintenance/pseudonymize_statistics.pl -v -c
 4 - Statistics pseudonymized
 5 - Check the background jobs table, note jobs were generated
 6 - Apply patch
 7 - perl misc/maintenance/pseudonymize_statistics.pl -v -c
 8 - You get same count as before, and are warned of existing pseudonymized transactions
 9 - enter 'Y' to continue
10 - Check the pseudonymized_transacations table
11 - Confirm same rows were generated twice
12 - Check the background jobs
13 - Confirm there were no new jobs generated (for pseudonymization)

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-04-22 08:46:35 +02:00
Jonathan Druart 7e160cf6a2
Bug 39319: Fix POD for misc/migration_tools/koha-svc.pl
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-04-14 08:51:20 +02:00
Tomas Cohen Arazi 6881b674e6
Bug 36365: (follow-up) Indexing should be done async and in chunks
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-04-14 08:51:17 +02:00
Tomas Cohen Arazi 3669cb0eed
Bug 36365: Add a way for compare_es_to_db.pl to fix problems
This patch adds a new option switch `--fix` (or `-f`) to make the script
try to fix the problems it found.

Bonus: `--help` option switch added.

To test:
1. Apply this patch
2. Run:
   $ ktd --shell
  k$ perl -MKoha::SearchEngine -MKoha::SearchEngine::Indexer -e 'my $i = Koha::SearchEngine::Indexer->new({index => $Koha::SearchEngine::BIBLIOS_INDEX}); $i->delete_index([1]);'
=> SUCCESS: Bibliographic record #deleted from ES index
3. Run:
  k$ koha-mysql kohadev
   > DELETE FROM biblio WHERE biblionumber=2 OR biblionumber=3;
=> SUCCESS: Bibliographic records 2 and 3 deleted directly from the
database, no reindex triggered by Koha.
4. Run:
  k$ perl misc/maintenance/compare_es_to_db.pl
=> SUCCESS: The results from (2) and (3) are explained. [1]
5. Run:
  k$ perl misc/maintenance/compare_es_to_db.pl --fix
=> SUCCESS: It says it is gonna fix things
6. Repeat 4
=> SUCCESS: The problems we generated are no longer reported! They've
been fixed!
7. Sign off :-D

[1] Note KTD ships some broken records on purpose so devs need to deal
with a not-perfect DB everyday, and catch misses in the code.

Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-04-14 08:51:16 +02:00
Emily Lamancusa 8ae546b31b
Bug 39353: (QA follow-up) Clarify error message
Signed-off-by: Emily Lamancusa <emily.lamancusa@montgomerycountymd.gov>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-03-19 17:34:54 +01:00
Jonathan Druart 488d0042e6
Bug 39353: Prevent tidy.pl to empty template files
In ktd context:
if host's node_modules ktd has been generated prior to bug 38644, the template-toolkit's
prettier plugin is not there and prettier will empty the file.

It seems to be a prettier bug, and the following behaviour is confusing:

From the host:
% yarn run prettier --write koha-tmpl/intranet-tmpl/prog/en/modules/test.tt
yarn run v1.22.22
$ prettier koha-tmpl/intranet-tmpl/prog/en/modules/test.tt
[error] Cannot find module '@koha-community/prettier-plugin-template-toolkit'
[error] Require stack:
[error] - /home/jonathan/workspace/koha/node_modules/prettier/index.js
[error] - /home/jonathan/workspace/koha/node_modules/prettier/cli.js
[error] - /home/jonathan/workspace/koha/node_modules/prettier/bin-prettier.js
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

=> The file is not modified

Within ktd:
$ yarn run prettier --write koha-tmpl/intranet-tmpl/prog/en/modules/test.tt
yarn run v1.22.22
$ prettier --write koha-tmpl/intranet-tmpl/prog/en/modules/test.tt
koha-tmpl/intranet-tmpl/prog/en/modules/test.tt 6ms
Done in 0.39s.

=> No error, and the file is emptied!

It seems that the plugin is found by prettier but not used (?)

Test plan:
1. Follow this from step 1 to 14
   https://gitlab.com/koha-community/koha-testing-docker/-/issues/483
2. Apply patch
3. misc/devel/tidy.pl --no-write koha-tmpl/intranet-tmpl/prog/en/modules/intranet-main.tt
   It should print the new error message
4. Make a change to a .tt file, commit
5. There will be a message about and issue with prettier. But it will
   still commit. It's expect.
6. Inspect and see that the commit has the changes

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Emily Lamancusa <emily.lamancusa@montgomerycountymd.gov>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-03-19 17:34:54 +01:00
Nick Clemens de9c1a8ff2
Bug 37920: Add cronjob loggging to writeoff_debts.pl
This patch adds the same logging as other cronjobs have

To test:
1 - Enable system preference CronjobLog
2 - perl misc/cronjobs/writeoff_debts.pl -v -c --added-after 2025-01-01
3 - Check the logs - no entries
4 - Apply patch
5 - Repeat 2
6 - Check the logs, two entries - one for the start and one for the end

Signed-off-by: Magnus Enger <magnus@libriotech.no>
Tested on top of bug 39236. Works as advertised.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-03-17 08:21:24 +01:00
Nick Clemens 93e2a0953d
Bug 39236: writeoff_debts.pl needs adjustments
From bug 22435 - we need to use 'CREATE' as the account_offset type
From bug 31095 - we need to 'use Koha::Patrons;'

To test:
1 - Add a fine to a patron account
2 - perl misc/cronjobs/writeoff_debts.pl -v -c --added-after 2025-01-01
3 - Cronjob dies
4 - Apply patch
5 - Rerun
6 - Job succeeds!
7 - Confirm patron's fine is written off
8 - Sign off

Signed-off-by: Magnus Enger <magnus@libriotech.no>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-03-17 08:21:24 +01:00
Marcel de Rooy 19c905501c
Bug 38408: (QA follow-up) Rename --forks, add usage
Test plan:
Run script with -h

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
2025-03-17 08:21:17 +01:00