Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Bug 31143 created a script in cronjob and then moved it to maintenance. However, it is still referred as in cronjob in help strings. This patch updates the hint.
No test needed I guess. Only changing a string
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch adds a new test to the data inconsistencies script to report where
MARC records contain item fields.
These may not cause issues in Koha, however, in discovery harvests they have eben found to create errors
To test:
1 - Follow test plan on bug 39412 to generate records with item tags
2 - perl misc/maintenance/search_for_data_inconsistencies.pl
3 - Note new section and hint on how to fix
4 - perl misc/maintenance/touch_all_biblios.pl
5 - repeat 2
6 - Errors cleared
Signed-off-by: Magnus Enger <magnus@libriotech.no>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch changes the script to use 'Koha::PseudonymizedTransaction->new_from_statistic' directly
rather than calling 'Koha::Statistic->pseudonymize()' which generates a background job.
Additionally, some of the checks from 'pseudonymnize' and the background job are moved into the search
so we don't fetch any to pseudonymize that should not be.
To test:
1 - Generate some entries in statistics (checkin, checkout, renew, etc)
2 - Enable pseudonymization (see bug 28911 if you need bcrypt settings)
3 - perl misc/maintenance/pseudonymize_statistics.pl -v -c
4 - Statistics pseudonymized
5 - Check the background jobs table, note jobs were generated
6 - Apply patch
7 - perl misc/maintenance/pseudonymize_statistics.pl -v -c
8 - You get same count as before, and are warned of existing pseudonymized transactions
9 - enter 'Y' to continue
10 - Check the pseudonymized_transacations table
11 - Confirm same rows were generated twice
12 - Check the background jobs
13 - Confirm there were no new jobs generated (for pseudonymization)
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds a new option switch `--fix` (or `-f`) to make the script
try to fix the problems it found.
Bonus: `--help` option switch added.
To test:
1. Apply this patch
2. Run:
$ ktd --shell
k$ perl -MKoha::SearchEngine -MKoha::SearchEngine::Indexer -e 'my $i = Koha::SearchEngine::Indexer->new({index => $Koha::SearchEngine::BIBLIOS_INDEX}); $i->delete_index([1]);'
=> SUCCESS: Bibliographic record #deleted from ES index
3. Run:
k$ koha-mysql kohadev
> DELETE FROM biblio WHERE biblionumber=2 OR biblionumber=3;
=> SUCCESS: Bibliographic records 2 and 3 deleted directly from the
database, no reindex triggered by Koha.
4. Run:
k$ perl misc/maintenance/compare_es_to_db.pl
=> SUCCESS: The results from (2) and (3) are explained. [1]
5. Run:
k$ perl misc/maintenance/compare_es_to_db.pl --fix
=> SUCCESS: It says it is gonna fix things
6. Repeat 4
=> SUCCESS: The problems we generated are no longer reported! They've
been fixed!
7. Sign off :-D
[1] Note KTD ships some broken records on purpose so devs need to deal
with a not-perfect DB everyday, and catch misses in the code.
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch makes the script provide links to the staff interface instead
of the OPAC. This way, the links can just be picked to access the
problematic records for fixing them.
To test:
1. Run:
$ ktd --shell
k$ perl misc/maintenance/compare_es_to_db.pl
=> FAIL: Generated links point to the OPAC
2. Apply this patch
3. Repeat 1
=> SUCCESS: Generated links are useful as they point to the staff
interface
4. Sign off :-D
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This commit is generated using:
% perl misc/devel/tidy.pl
*within* ktd, to get the same version of perltidy than what will be used
by our CI (currently v20230309).
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
A few notes:
- I grabbed this list by looking for things in misc/ that had cronlogaction
- I didn't touch scripts that aren't in the cronjobs directory, except
fix_invalid_dates where I follow import_patrons and only log if confirmed
since they aren't crons, I figure this seems reasonable
- purge_suggestions I moved the logggin up and changed the addition of
effective days to a verbose message - this seems more consistent
To test:
1 - Enable CronjobLog
2 - On command line:
perl misc/cronjobs/cleanup_database.pl --old_reservers 550
3 - Error on command line
4 - In staff client, go to Tools->Log viewer, check 'Cronjobs' and submit
5 - No entries
6 - Apply patch
7 - Repeat and see run is logged with wrong parameter
8 - Fix parameter and confirm
9 - Spot check other files until you are satisfied
10 - Sign it off!
Signed-off-by: William Lavoie <william.lavoie@inLibro.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Improve messages clarity.
Show success message if no schema diff was found from audit.
Code refactor to ensure the title+message shown in UI is the same as the one shown in CLI and are all translatable.
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch updates the module as suggested by David to handle just the
diff code and leaves the display logic up to the templates and script.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: JesseM <jesse@bywatersolutions.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Test plan #1 ensure same script functionality (same as bug 34064):
0. Apply patch
1. vi ./installer/data/mysql/kohastructure.sql
2. Comment out some columns, change NULL status, or whatever you like
3. perl misc/maintenance/audit_database.pl \
--filename /kohadevbox/koha/installer/data/mysql/kohastructure.sql
4. Note that the output includes SQL commands to change the database
to match the new kohastructure.sql
5a. Try using koha-foreach and note that the database name appears above
the database comparison
5b. koha-foreach "perl misc/maintenance/audit_database.pl \
--filename /kohadevbox/koha/installer/data/mysql/kohastructure.sql"
Test plan #2:
1. After doing test plan #1, access the new 'database audit' tab, visit:
<staff_url>/cgi-bin/koha/about.pl?tab=database
2. Ensure the output is the same as test plan #1
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: JesseM <jesse@bywatersolutions.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch skips record indexing and real time holds queue updates when updating the localuse field
from statistics. A note is added to the script that the user should reindex if the localuse field is mapped.
Additionally a verbose option is added to the script, and doubled use of GetOptions is removed.
Lastly, a check is added to confirm the items value is being changed before the value is stored.
To test:
1 - Enable the real time holds queue
2 - Enable Elasticsearch
3 - perl misc/maintenance/update_localuse_from_statistics.pl --confirm
4 - Note all items are touched and reported
5 - Check the background jobs table - there are many jobs generated
6 - Apply patch
7 - perl misc/maintenance/update_localuse_from_statistics.pl --confirm
8 - Note 0 items are reported updated, no new background jobs
9 - perl misc/maintenance/update_localuse_from_statistics.pl --confirm --verbose
10 - No items reported
11 - Update some items in the DB
UPDATE items SET localuse = 99 WHERE itemnumber LIKE '%9';
12 - perl misc/maintenance/update_localuse_from_statistics.pl --confirm
13 - Only the number of items changed above reported
14 - UPDATE items SET localuse = 99 WHERE itemnumber LIKE '%9';
15 - perl misc/maintenance/update_localuse_from_statistics.pl --confirm --verbose
16 - Each item changed reported, and the amounts, and the total items updated.
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch identifies date fields in the database (timestamp, datetime, date) that are 0000-00-00 and sets them to NULL
Test plan -
Examples of setting up some 0000-00-00 date fields:
e.g. Borrowers table
update borrowers set dateexpiry="0000-00-00" where borrowernumber=49;
e.g. Items table
update items set datelastseen="0000-00-00" where itemnumber=12;
To identify the cases:
1) Set up some 0000-00-00 date fields
2) Run the misc/maintenance/search_for_data_inconsistencies.pl script
3) Run fix_invalid_dates.pl -v
To fix the cases:
1) Run fix_invalid_dates.pl -c -v
2) Notice the value has changed from 0000-00-00 to NULL
e.g. Borrowers table
select dateexpiry from borrowers where borrowernumber=49;
e.g. Items table
select datelastseen from items where itemnumber=12;
Signed-off-by: Shi Yao Wang <shi-yao.wang@inlibro.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Olivier V <olivier.vezina@inLibro.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
To test:
1 - Perform some transactions
2 - Enable Pseudonymization
3 - perl misc/maintenance/pseudonymize_statistics.pl -v
4 - Confirm test run an nothing changed
5 - perl misc/maintenance/pseudonymize_statistics.pl -v -c
6 - Confirm statistics are pseudonymized
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Amend: tidied the changes [tcohen]
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds a hint to the end of the script to notify the end user
that they may need to run the build_holds_queue cronjob if they are
using RealTimeHoldsQueue.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
If RealTimeHoldsQueue is on, touch_all_biblios triggers a update_holds_queue_for_biblios background job for each affected record. This will result in a as many background jobs being queued up as records! It makes far more sense for this script to not do that which gives the administrator the option for running the holds queue builder if the changes would affect holdability, or to not run it at all.
Test Plan:
1) Run touch_all_biblios.pl
3) Note a update_holds_queue_for_biblios background job is queued for each record touched
4) Apply this patch
5) Merge touch_all_biblios.pl again
6) Note that no update_holds_queue_for_biblios jobs were queued
Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
Signed-off-by: Emily Lamancusa <emily.lamancusa@montgomerycountymd.gov>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
Signed-off-by: Philip Orr <philip.orr@lmscloud.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
Signed-off-by: Philip Orr <philip.orr@lmscloud.de>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
To test:
1. APPLY PATCH and restart all
2. Run the update_localuse_from_statistics.pl script without a confirm flag, nothing happens.
3. Run the script and add the --confirm flag, stuff happens.
Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Emily Lamancusa <emily.lamancusa@montgomerycountymd.gov>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Andrew Fuerste-Henry <andrewfh@dubcolib.org>
Signed-off-by: Andrew Fuerste Henry <andrewfh@dubcolib.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This script allow to deduplicate authorities automatically.
Script is in misc/maintenance/
It works this way:
1) authorities are fetched from the database. You can limit fetched
results by authtypecode, or directly by specifying WHERE clause
2) for each authority:
2.1) build a Zebra query using the 'search_form' for the heading
2.2) run the query, retrieve the results
2.3) among duplicates, choose the one we want to keep (use
--choose-method option).
2.5) use C4::Authorities::merge to merge authorities
3) delete the merged authorities
Use --help for more informations on options
To be done:
1 - Move to module and cover with tests
2 - Add option to only merge unused authorities
3 - Expand 'ppn' option to be 'control-number' option and allow specifying field
4 - More?
1 & 2 I will attempt - 3 & 4 may be future enhancements
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Introducing $orders->filter_by_obsolete and $orders->cancel.
Test plan:
Run t/db_dependent/Koha/Acquisition/Orders.t
Create basket with few orders. Remove some biblio records.
Run acq_cancel_obsolete_orders.pl -c
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: AFHDubCoLib <andrewfh@dubcolib.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This script takes a date parameter in SQL format and pseudonymizes all statistics
found before this date.
Only values that can be found will be added, i.e. no deleted patron or item info
will be present.
Additionally - the values stored will be the current values from patrons and items, so
some info will be approximate, much as it is when joining from the statistics table for reporting.
To test:
1 - Perform some issues/returns/renewals/on-site checkouts
2 - Make sure Pseudonymization is disabled
3 - perl misc/maintenance/pseudonymize_statistics.pl
4 - Script ends and reports that preference is not active
5 - Enable the pref, and choose some borrower and item fields
NOTE: See bug 28911 if you need a bcrypt key for your koha-conf.xml
6 - perl misc/maintenance/pseudonymize_statistics.pl
7 - sudo koha-mysql kohadev
8 - SELECT * FROM pseudonymized_transactions
9 - Confirm data is correctly stored
10 - DELETE FROM pseudonymized_transactions;
11 - UPDATE statistics SET datetime = '2023-01-01 00:00:00';
12 - perl misc/maintenance/pseudonymize_statistics.pl -b "2022-12-31 23:59:59";
13 - SELECT * FROM pseudonymized_transactions;
14 - Confirm no entries were made
15 - Select different options in Pseudonmyization prefs, including borrower attributes
This wil require defining an attribute that can be kept for pseudonymization
16 - Confirm options are correctly pseudonymized
Signed-off-by: AFHDubCoLib <andrewfh@dubcolib.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Test plan:
1) Select a patron A
2) Change all the message prefs away from defaults.
3) Run misc/maintenance/borrowers-force-messaging-defaults.pl --doit --message-name Item_due
Verify that patron A prefs changed only for 'Item due'
Signed-off-by: Sam Lau <samalau@gmail.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Like filter on category Bug 19454, it would be helpful to filter on a specified library for misc/maintenance/borrowers-force-messaging-defaults.pl
Test plan:
1) Select two patrons A and B in say libraries L1 and L2.
2) Change the msg prefs for A and B away from defaults.
3) Run misc/maintenance/borrowers-force-messaging-defaults.pl -doit -library L1
Verify that patron A changed and patron B did not.
4) Run with both category and library filters
Signed-off-by: Sam Lau <samalau@gmail.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This change removes the no_batch_alters option.
When working with real world databases, you really want to use batch
alters to group together changes. Otherwise, the output becomes
too overwhelming/unwieldy.
On real world databases, I'm noticing big discrepancies, although
I think most of the time this is a difference in the collation
at the column level.
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This script can take a connected database handle and compare that database
schema against kohastructure.sql to see what changes the database would
need in order to match kohastructure.sql
NOTE: It uses SQL::Translation::Diff, which is installed with DBIx::Class.
WARNING: The diff doesn't seem to compare comments, so that difference
won't appear in the output. If we wanted, we could easily enhance the
audit_database.pl script to also compare comments.
WARNING: The output is a proposed series of SQL commands. While they
are useful to review, they won't always duplicate the changes done
by updatedatabase.pl, so it's important to carefully analyze the output.
The key purpose of this audit script is to just highlight the differences
between the two.
Test plan:
0. Apply patch
1. vi ./installer/data/mysql/kohastructure.sql
2. Comment out some columns, change NULL status, or whatever you like
3. perl misc/maintenance/audit_database.pl \
--filename /kohadevbox/koha/installer/data/mysql/kohastructure.sql
4. Note that the output includes SQL commands to change the database
to match the new kohastructure.sql
5a. Try using koha-foreach and note that the database name appears above
the database comparison
5b. koha-foreach "perl misc/maintenance/audit_database.pl \
--filename /kohadevbox/koha/installer/data/mysql/kohastructure.sql"
Signed-off-by: Emmi Takkinen <emmi.takkinen@koha-suomi.fi>
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Bug 26311 added patron invalid age in search_for_data_inconsistencies.pl
But this is not relevant for expired patrons, especially in child categories.
Check should skip expired patrons
1) Set a non-expired patron with invalid age
2) Run misc/maintenance/search_for_data_inconsistencies.pl
=> Check you see the patron
3) Edit patron to be expired
4) Run misc/maintenance/search_for_data_inconsistencies.pl
=> Check you do not see the patron
Signed-off-by: Caroline Cyr La Rose <caroline.cyr-la-rose@inlibro.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
[EDIT] If we do not want to test valid age for expired patrons, you should
obviously test expired first. Reversing the order in the test.
Putting it in the search criteria would be nicer but leads to more
complicated code.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch corrects a typo in the output of
search_for_data_inconsistencies.pl when a bibliographic record has no
title.
The patch also replaces biblio to bibliographic record in the same
sentence for terminology consistency.
To test:
- Have a bibliographic record without a title
- Run misc/maintenance/search_for_data_inconsistencies.pl
- Read output, make sure the sentence is correct
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch reformats the params passed to scroll_helper as defined here:
https://metacpan.org/pod/Search::Elasticsearch::Client::7_0::Scroll
To test:
1 - perl misc/maintenance/compare_es_to_db.pl
2 - It dies:
[Param] ** Unknown param (scroll_in_qs) in (search) request. , called from sub Search::Elasticsearch::Client::7_0::Direct::scroll_helper at misc/maintenance/compare_es_to_db.pl line 55.
3 - Apply patch
4 - Repeat
5 - It succeeds!
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Since bug 29486, misc/maintenance/search_for_data_inconsistencies.pl search for biblio.biblionumber in MARC record with $record->subfield().
This fails when field is a control field (< 10).
Idem for biblioitems.biblioitemnumber
Test plan :
1.0) On a UNIMARC database (biblio.biblionumber is on 001)
1.1) Run misc/maintenance/search_for_data_inconsistencies.pl
=> Without patch you get error : Control fields (generally, just tags below 010) do not have subfields, use data()
=> With patch no error
2.0) On a MARC21 database (biblio.biblionumber is on 999c)
2.1) Run misc/maintenance/search_for_data_inconsistencies.pl
=> Check you see no error
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Tomás Cohen Arazi
Baptiste Wojtkowski
Nick Clemens
Jonathan Druart
Koha Development Team
Pedro Amorim
Martin Renvoize
Wainui Witika-Park
Kyle M Hall
Hammat Wele
Matthias Le Gac
Lucas Gass
Julian Maurice
Marcel de Rooy
Fridolin Somers
David Cook
Caroline Cyr La Rose