Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
When a new PseudonymizedTransaction is created, consider the presence of illrequest_id and act on it accordingly.
Signed-off-by: Jeremy Evans <Jeremy.Evans@ukhsa.gov.uk>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This now conforms to the new pseudonymized_metadata_values instead of pseudonymized_borrower_attributes
Signed-off-by: Jeremy Evans <Jeremy.Evans@ukhsa.gov.uk>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch updates the query logic of create_from_statistic for borrower
attributes.
We replace the count of attributes types with a query to fetch all
attributes type codes where keep_for_pseudonymization is set.
We then check for content in that array before using it to perform a
filtered search on the patrons extended_attributes to only return
attributes set for pseudonymization.
This should result in the same skip of extended_attributes fetch should
no attribute types be set to pseudonymization, but it should also reduce
the 1 call to attribute types per attribute the patron has to just a
single filtered call for patron attributes attributes followed by the
store call for each pseudonymizated attribute.
Test plan:
As before, except there should be even fewer sql calls to the database
than before for the case where there are attributes.
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch simply checks the count of pseudonymized borrower attributes before
looping through them
To test:
1 - Define several borrower attributes, none kept for pseudonymization
2 - Add some values for those attributes to some patrons
3 - Generate some statistics for those patrons (checkin, checkout, renew, etc)
4 - Enable pseudonymization
5 - On command line: export DBIC_TRACE=1
6 - Run the pseudonymization script with a date in the future to ensure all transactions are captured:
perl misc/maintenance/pseudonymize_statistics.pl -v -c --before 2026-01-01
7 - Note the borrower attributes are fetched
8 - Apply patch
9 - repeat 6
10 - Note the attributes are only counted, not fetched
11 - Make one of the attributes kept
12 - repeat 6
13 - Note the attributes are counted and fetched
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This new method name better reflects the fact that it now creates - and stores - the PseudonymizedTransaction object
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Test plan:
Run the tests before and after applying this patch:
perl t/db_dependent/Koha/Pseudonymization.t
They should fail before this patch and pass after applying patch.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This commit is generated using:
% perl misc/devel/tidy.pl
*within* ktd, to get the same version of perltidy than what will be used
by our CI (currently v20230309).
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds a new routine to pseudonymize a statistic and adjusts
C4::Stats to use this new routine.
Additionally Koha::PseudonymizedTransaction->new_from_statistic is updated to check for
the existence of objects before using them (in the case of older stats where things may have been deleted)
Tests are added and can be run using:
1. Run:
$ ktd --shell
k$ prove t/db_dependent/Koha/PseudonymizedTransaction.t \
t/db_dependent/Koha/Pseudonymization.t
=> SUCCESS: New tests pass, old tests keep passing
Signed-off-by: AFHDubCoLib <andrewfh@dubcolib.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
On bug 17591 we discovered that there was something weird going on with
the way we export and use subroutines/modules.
This patch tries to standardize our EXPORT to use EXPORT_OK only.
That way we will need to explicitely define the subroutine we want to
use from a module.
This patch is a squashed version of:
Bug 17600: After export.pl
Bug 17600: After perlimport
Bug 17600: Manual changes
Bug 17600: Other manual changes after second perlimports run
Bug 17600: Fix tests
And a lot of other manual changes.
export.pl is a dirty script that can be found on bug 17600.
"perlimport" is:
git clone https://github.com/oalders/App-perlimports.git
cd App-perlimports/
cpanm --installdeps .
export PERL5LIB="$PERL5LIB:/kohadevbox/koha/App-perlimports/lib"
find . \( -name "*.pl" -o -name "*.pm" \) -exec perl App-perlimports/script/perlimports --inplace-edit --no-preserve-unused --filename {} \;
The ideas of this patch are to:
* use EXPORT_OK instead of EXPORT
* perltidy the EXPORT_OK list
* remove '&' before the subroutine names
* remove some uneeded use statements
* explicitely import the subroutines we need within the controllers or
modules
Note that the private subroutines (starting with _) should not be
exported (and not used from outside of the module except from tests).
EXPORT vs EXPORT_OK (from
https://www.thegeekstuff.com/2010/06/perl-exporter-examples/)
"""
Export allows to export the functions and variables of modules to user’s namespace using the standard import method. This way, we don’t need to create the objects for the modules to access it’s members.
@EXPORT and @EXPORT_OK are the two main variables used during export operation.
@EXPORT contains list of symbols (subroutines and variables) of the module to be exported into the caller namespace.
@EXPORT_OK does export of symbols on demand basis.
"""
If this patch caused a conflict with a patch you wrote prior to its
push:
* Make sure you are not reintroducing a "use" statement that has been
removed
* "$subroutine" is not exported by the C4::$MODULE module
means that you need to add the subroutine to the @EXPORT_OK list
* Bareword "$subroutine" not allowed while "strict subs"
means that you didn't imported the subroutine from the module:
- use $MODULE qw( $subroutine list );
You can also use the fully qualified namespace: C4::$MODULE::$subroutine
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
POD is missing for get_hash
POD is missing for new_from_statistic
POD is missing for 'object_class'
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Still listing the address..
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This is the commit where you will find useful information about this development.
The goal of this new feature is to add a way to pseudonymize patron's
data, in a way they could not be personally identifiable.
https://en.wikipedia.org/wiki/Pseudonymization
There are different existing way to anonymize patron's information in
Koha, but we loose the ability to make useful report.
This development proposes to have 2 different tables:
* 1 for transactions and patrons data (pseudonymized_transactions)
* 1 for patrons' attributes (pseudonymized_borrower_attributes)
Entries to pseudonymized_transactions are added when a new transaction
(checkout, checkin, renew, on-site checkout) is done.
Also, anonymized_borrower_attributes is populated if patron's attributes are
marked as "keep for pseudonymization".
To make those informations not identifiable to a patron, we are having a
hashed_borrowernumber column in pseudonymized_transactions. This hash will be
generated (Blowfish-based crypt) using a key stored in the Koha
configuration.
To make things configurable, we are adding 3 sysprefs and 1 new DB
column:
* syspref Pseudonymization to turn on/off the whole feature
* syspref PseudonymizationPatronFields to list the informations of the
patrons to sync
* syspref PseudonymizationTransactionFields to list the informations
of the transactions to copy
* DB column borrower_attribute_types.keep_for_pseudonymization that is a
boolean to enable/disable the copy of a given patron's attribute type.
Test plan:
1/ Turn on Pseudonymization
2/ Define in PseudonymizationPatronFields and
PseudonymizationTransactionFields the different fields you want to copy
3/ Go to the about page
=> You will see a warning about a missing config entry
4/ You need to generate a key and put it in the koha-conf.xml file. The
following command will generate one:
% htpasswd -bnBC 10 "" password | tr -d ':\n' | sed 's/$2y/$2a/'
Then edit $KOHA_CONF and add it before of the end of the config section (</config)
it should be something like:
<key>$2a$10$PfdrEBdRcL2MZlEtKueyLegxI6zg735jD07GRnc1bt.N/ZYMvBAB2</key>
5/ Restart memcached then plack (alias restart_all)
=> Everything is setup!
6/ Create a new transaction (checkin for instance)
=> Confirm that a new entry has been added to pseudonymized_transaction with the data
you expect to be copied
7/ Edit some patron attribute types and tick "Keep for pseudonymization"
8/ Create a new transaction
=> Confirm that new entries have been added to pseudonymized_borrower_attributes
11/ Delete the patrons
=> Confirm that the entries still exist in the pseudonymized_* tables
12/ Purge the patrons (ie. use cleanup_database.pl to remove them from
the deleted_borrowers table)
=> Confirm that the entries still exist in the pseudonymized_* tables
See bug 24152 to remove data from the anonymized_* tables
Sponsored-by: Association KohaLa - https://koha-fr.org/
Signed-off-by: Signed-off-by: Sonia Bouis <sonia.bouis@univ-lyon3.fr>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Tomas Cohen Arazi
Pedro Amorim
Martin Renvoize
Nick Clemens
Pedro Amorim
Koha Development Team
Jonathan Druart
Marcel de Rooy