To test:
1 - Perform some transactions
2 - Enable Pseudonymization
3 - perl misc/maintenance/pseudonymize_statistics.pl -v
4 - Confirm test run an nothing changed
5 - perl misc/maintenance/pseudonymize_statistics.pl -v -c
6 - Confirm statistics are pseudonymized
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Amend: tidied the changes [tcohen]
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds a hint to the end of the script to notify the end user
that they may need to run the build_holds_queue cronjob if they are
using RealTimeHoldsQueue.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
If RealTimeHoldsQueue is on, touch_all_biblios triggers a update_holds_queue_for_biblios background job for each affected record. This will result in a as many background jobs being queued up as records! It makes far more sense for this script to not do that which gives the administrator the option for running the holds queue builder if the changes would affect holdability, or to not run it at all.
Test Plan:
1) Run touch_all_biblios.pl
3) Note a update_holds_queue_for_biblios background job is queued for each record touched
4) Apply this patch
5) Merge touch_all_biblios.pl again
6) Note that no update_holds_queue_for_biblios jobs were queued
Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
Signed-off-by: Emily Lamancusa <emily.lamancusa@montgomerycountymd.gov>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
Signed-off-by: Philip Orr <philip.orr@lmscloud.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
Signed-off-by: Philip Orr <philip.orr@lmscloud.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
To test:
1. APPLY PATCH and restart all
2. Run the update_localuse_from_statistics.pl script without a confirm flag, nothing happens.
3. Run the script and add the --confirm flag, stuff happens.
Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Emily Lamancusa <emily.lamancusa@montgomerycountymd.gov>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Andrew Fuerste-Henry <andrewfh@dubcolib.org>
Signed-off-by: Andrew Fuerste Henry <andrewfh@dubcolib.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This script allow to deduplicate authorities automatically.
Script is in misc/maintenance/
It works this way:
1) authorities are fetched from the database. You can limit fetched
results by authtypecode, or directly by specifying WHERE clause
2) for each authority:
2.1) build a Zebra query using the 'search_form' for the heading
2.2) run the query, retrieve the results
2.3) among duplicates, choose the one we want to keep (use
--choose-method option).
2.5) use C4::Authorities::merge to merge authorities
3) delete the merged authorities
Use --help for more informations on options
To be done:
1 - Move to module and cover with tests
2 - Add option to only merge unused authorities
3 - Expand 'ppn' option to be 'control-number' option and allow specifying field
4 - More?
1 & 2 I will attempt - 3 & 4 may be future enhancements
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Introducing $orders->filter_by_obsolete and $orders->cancel.
Test plan:
Run t/db_dependent/Koha/Acquisition/Orders.t
Create basket with few orders. Remove some biblio records.
Run acq_cancel_obsolete_orders.pl -c
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: AFHDubCoLib <andrewfh@dubcolib.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This script takes a date parameter in SQL format and pseudonymizes all statistics
found before this date.
Only values that can be found will be added, i.e. no deleted patron or item info
will be present.
Additionally - the values stored will be the current values from patrons and items, so
some info will be approximate, much as it is when joining from the statistics table for reporting.
To test:
1 - Perform some issues/returns/renewals/on-site checkouts
2 - Make sure Pseudonymization is disabled
3 - perl misc/maintenance/pseudonymize_statistics.pl
4 - Script ends and reports that preference is not active
5 - Enable the pref, and choose some borrower and item fields
NOTE: See bug 28911 if you need a bcrypt key for your koha-conf.xml
6 - perl misc/maintenance/pseudonymize_statistics.pl
7 - sudo koha-mysql kohadev
8 - SELECT * FROM pseudonymized_transactions
9 - Confirm data is correctly stored
10 - DELETE FROM pseudonymized_transactions;
11 - UPDATE statistics SET datetime = '2023-01-01 00:00:00';
12 - perl misc/maintenance/pseudonymize_statistics.pl -b "2022-12-31 23:59:59";
13 - SELECT * FROM pseudonymized_transactions;
14 - Confirm no entries were made
15 - Select different options in Pseudonmyization prefs, including borrower attributes
This wil require defining an attribute that can be kept for pseudonymization
16 - Confirm options are correctly pseudonymized
Signed-off-by: AFHDubCoLib <andrewfh@dubcolib.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Test plan:
1) Select a patron A
2) Change all the message prefs away from defaults.
3) Run misc/maintenance/borrowers-force-messaging-defaults.pl --doit --message-name Item_due
Verify that patron A prefs changed only for 'Item due'
Signed-off-by: Sam Lau <samalau@gmail.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Like filter on category Bug 19454, it would be helpful to filter on a specified library for misc/maintenance/borrowers-force-messaging-defaults.pl
Test plan:
1) Select two patrons A and B in say libraries L1 and L2.
2) Change the msg prefs for A and B away from defaults.
3) Run misc/maintenance/borrowers-force-messaging-defaults.pl -doit -library L1
Verify that patron A changed and patron B did not.
4) Run with both category and library filters
Signed-off-by: Sam Lau <samalau@gmail.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This change removes the no_batch_alters option.
When working with real world databases, you really want to use batch
alters to group together changes. Otherwise, the output becomes
too overwhelming/unwieldy.
On real world databases, I'm noticing big discrepancies, although
I think most of the time this is a difference in the collation
at the column level.
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This script can take a connected database handle and compare that database
schema against kohastructure.sql to see what changes the database would
need in order to match kohastructure.sql
NOTE: It uses SQL::Translation::Diff, which is installed with DBIx::Class.
WARNING: The diff doesn't seem to compare comments, so that difference
won't appear in the output. If we wanted, we could easily enhance the
audit_database.pl script to also compare comments.
WARNING: The output is a proposed series of SQL commands. While they
are useful to review, they won't always duplicate the changes done
by updatedatabase.pl, so it's important to carefully analyze the output.
The key purpose of this audit script is to just highlight the differences
between the two.
Test plan:
0. Apply patch
1. vi ./installer/data/mysql/kohastructure.sql
2. Comment out some columns, change NULL status, or whatever you like
3. perl misc/maintenance/audit_database.pl \
--filename /kohadevbox/koha/installer/data/mysql/kohastructure.sql
4. Note that the output includes SQL commands to change the database
to match the new kohastructure.sql
5a. Try using koha-foreach and note that the database name appears above
the database comparison
5b. koha-foreach "perl misc/maintenance/audit_database.pl \
--filename /kohadevbox/koha/installer/data/mysql/kohastructure.sql"
Signed-off-by: Emmi Takkinen <emmi.takkinen@koha-suomi.fi>
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Bug 26311 added patron invalid age in search_for_data_inconsistencies.pl
But this is not relevant for expired patrons, especially in child categories.
Check should skip expired patrons
1) Set a non-expired patron with invalid age
2) Run misc/maintenance/search_for_data_inconsistencies.pl
=> Check you see the patron
3) Edit patron to be expired
4) Run misc/maintenance/search_for_data_inconsistencies.pl
=> Check you do not see the patron
Signed-off-by: Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
[EDIT] If we do not want to test valid age for expired patrons, you should
obviously test expired first. Reversing the order in the test.
Putting it in the search criteria would be nicer but leads to more
complicated code.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch corrects a typo in the output of
search_for_data_inconsistencies.pl when a bibliographic record has no
title.
The patch also replaces biblio to bibliographic record in the same
sentence for terminology consistency.
To test:
- Have a bibliographic record without a title
- Run misc/maintenance/search_for_data_inconsistencies.pl
- Read output, make sure the sentence is correct
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch reformats the params passed to scroll_helper as defined here:
https://metacpan.org/pod/Search::Elasticsearch::Client::7_0::Scroll
To test:
1 - perl misc/maintenance/compare_es_to_db.pl
2 - It dies:
[Param] ** Unknown param (scroll_in_qs) in (search) request. , called from sub Search::Elasticsearch::Client::7_0::Direct::scroll_helper at misc/maintenance/compare_es_to_db.pl line 55.
3 - Apply patch
4 - Repeat
5 - It succeeds!
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Since bug 29486, misc/maintenance/search_for_data_inconsistencies.pl search for biblio.biblionumber in MARC record with $record->subfield().
This fails when field is a control field (< 10).
Idem for biblioitems.biblioitemnumber
Test plan :
1.0) On a UNIMARC database (biblio.biblionumber is on 001)
1.1) Run misc/maintenance/search_for_data_inconsistencies.pl
=> Without patch you get error : Control fields (generally, just tags below 010) do not have subfields, use data()
=> With patch no error
2.0) On a MARC21 database (biblio.biblionumber is on 999c)
2.1) Run misc/maintenance/search_for_data_inconsistencies.pl
=> Check you see no error
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Essentially only adds a warn, some cosmetic changes too.
Test plan:
Copy your kohastructure to xx.sql.
Run sync_db_comments.pl -schema xx.sql. You will see usage.
Run sync_db_comments.pl -schema xyz.sql. You will see a warn and
the usage statement.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Adds a schema parameter to the cmdline script now too.
Test plan:
Run sync_db_comments.pl with -schema file where file does not exist.
(On dev install) rename kohastructure.sql, try with[out] referring
to it using -schema. You could also use the standard path
intranet/cgi-bin/installer/data/mysql.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
The module returns messages. The script can print them in verbose
mode. Test script adjusted accordingly.
Test plan:
Run t/db_dependent/Koha/Database/Commenter.t
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Note: This is only done (and 'needed') for the command line, not
for the module subroutines.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Test plan:
Run sync_db_comments.pl --clear --renumber
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Test plan:
[1] Backup your database, if not done already.
[2] Check output of dry_run when clearing a table:
misc/maintenance/sync_db_comments.pl -clear -table items -dry
[3] Save output of misc/devel/update_dbix_class_files before changing
comments in order to compare later. (Commit your changes.)
You may not have changes after running (at least on a fresh
database). That's fine.
[4] Clear all comments:
misc/maintenance/sync_db_comments.pl -clear
[5] Renumber all comments:
misc/maintenance/sync_db_comments.pl -renum
[6] Reset all comments to schema. Make sure that script finds your
structure in installer/data/mysql folder.
misc/maintenance/sync_db_comments.pl -reset
[7] Run update_dbix_class_files again and inspect changes as compared
to previous run.
Can you explain them? You should only see changes related to
column comments. If you did not have changes in step 3, you
should not have them here too.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Some old-style code is making our tests fail when run in Debian Testing.
This patch addresses this.
To test:
1. Launch bookworm KTD:
$ KOHA_IMAGE=master-bookworm ktd up -d
2. Run:
$ ktd --shell
k$ prove t/00-testcritic.t
=> FAIL: It fails!
3. Apply the patch
4. Repeat 2
=> SUCCESS: Tests now pass!
5. Sign off :-D
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
In search_for_data_inconsistencies.pl, the test for authorized values is a list in one line :
* The Framework *VR* is using the authorised value's category *LOC*, but the following items.location do not have a value defined ({itemnumber => value }):
{94 => AV} {95 => AV} {96 => AV} {97 => AV} {98 => AV} {99 => AV} {100 => AV} {101 => AV} {102 => AV} {103 => AV}
It would be more clear with new lines, especially for scripts (grep, awk ...) :
* The Framework *VR* is using the authorised value's category *LOC*, but the following items.location do not have a value defined ({itemnumber => value }):
{94 => AV}
{95 => AV}
{96 => AV}
{97 => AV}
{98 => AV}
{99 => AV}
{100 => AV}
{101 => AV}
{102 => AV}
{103 => AV}
Test plan :
1) In koha-testing-docker
2) Delete in authorized values LOC the value AV
3) Run misc/maintenance/search_for_data_inconsistencies.pl
=> You see the new line in result
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Some of our scripts have a space in the "shebang" (first) line:
#! /usr/bin/perl
This is not illegal, and it does work, but it is good to be
consistent, so this patch removes the space.
To test:
- Run: grep -rn --include=*.pl '#! /usr/' *
- See the list of files that have a space in the shebang
- Apply the patch
- Run the command again, there should be no output, meaning there
are no more files with space in the shebang
- Have a look at the patch and check that it only changes the
shebangs
- Sign off
Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Field <jonathan.field@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Field <jonathan.field@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
In search_for_data_inconsistencies.pl when there are several inconsistencies on same framework, the second output of items contains the first one.
Looks like it is since Bug 21466
Test plan using koha-testing-docker :
1) Create 2 inconsistencies on the same item via SQL :
update items set itemlost = 9 where itemnumber=900;
update items set notforloan = 8 where itemnumber=900;
2) Without patch
3) Run ./misc/maintenance/search_for_data_inconsistencies.pl
=> You see duplicate output :
== Wrong values linked to authorised values ==
* The Framework *BKS* is using the authorised value's category *NOT_LOAN*, but the following items.notforloan do not have a value defined ({itemnumber => value }):
{900 => 8}
* The Framework *BKS* is using the authorised value's category *LOST*, but the following items.itemlost do not have a value defined ({itemnumber => value }):
{900 => 8} {900 => 9}
4) Apply patch
5) Run ./misc/maintenance/search_for_data_inconsistencies.pl
=> Fixed :D
== Wrong values linked to authorised values ==
* The Framework *BKS* is using the authorised value's category *LOST*, but the following items.itemlost do not have a value defined ({itemnumber => value }):
{900 => 9}
* The Framework *BKS* is using the authorised value's category *NOT_LOAN*, but the following items.notforloan do not have a value defined ({itemnumber => value }):
{900 => 8}
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
If an AV is linked to a MARC field mapped with a biblio column, the
search_for_data_inconsistencies.pl script might explode with
The method Koha::Biblioitem->title is not covered by tests!
Trace begun at /kohadevbox/koha/Koha/Object.pm line 875
Koha::Object::AUTOLOAD('Koha::Biblioitem=HASH(0x556b67fa7168)') called at misc/maintenance/search_for_data_inconsistencies.pl line 246
Test plan:
For a given framework, pick a biblio using it
Link 245$a with an authorised value category
Run the script
=> Notice that with this script applied you will see the warning
=> Without this patch you got the error
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch adds a section about the --since option in the help section
of borrowers-force-messaging-defaults.pl
To test:
0) Apply patch
1) run the script with the --help option, e.g.
./misc/maintenance/borrowers-force-messaging-defaults.pl --help
--> There should be an explanation of the --since option with examples
for specific and relative dates
2) Optionally, run the script with different options
--> The behaviour shouldn't have changed
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
JD Amended patch:
-# FIXME Special case here
- print "Biblio not found\n,";
+ print "Biblio not found\n";
- my $biblio = Koha::Biblio->find($hostbiblionumber);
+ my $biblio = Koha::Biblios->find($hostbiblionumber);
Rebased-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Avoid warns 'Use of uninitialized value in sprintf' by using
'0' if lower age is undefined
'unlimited' if upper are is undefined
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Also added category limits in message, for example (12-18)
Check patron :
- with no date of birth
- with invalid age in category having age required
- with invalid age in category having upper age limit
- with invalid age in category having age required and upper age limit
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This is a suggestion for rephrasing the message slightly:
Before:
* Patron borrowernumber=15 in category 'J' has invalid age '56'
After:
* Patron borrowernumber=37 has an invalid age of 37 for their category 'K'
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Nick Clemens
Martin Renvoize
Kyle M Hall
Hammat Wele
Matthias Le Gac
Lucas Gass
Tomas Cohen Arazi
Julian Maurice
Marcel de Rooy
Fridolin Somers
David Cook
Caroline Cyr La Rose
Jonathan Druart
Magnus Enger
Katrin Fischer