We have a partner that exports UTF-8 CSV files, and is experiencing the same thing as the author of this article: https://www.freecodecamp.org/news/a-quick-tale-about-feff-the-invisible-character-cd25cd4630e7/
In short, Excel is inserting an invisible UTF-8 character at the start of the file, so that the column name "cardnumber" is actually named "\x{feff}cardnumber", causing "cardnumber" to be blank.
A simple solution is provided here: https://stackoverflow.com/questions/24390034/remove-bom-from-string-with-perl
Test Plan:
1) Download the example.csv file
2) Attempt to import it using the patron import tool
3) Note the invalid column name error
4) Apply this patch, restart all the things!
5) Attempt the import again
6) Assuming you have a branchcode MPL and a cataegory code S, the patron should import!
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Maggie Wong <maggie.wong@yccece.edu.hk>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Joy Nelson <joy@bywatersolutions.com>
(cherry picked from commit 91980f27a3)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
(cherry picked from commit 40ae65e326)
Signed-off-by: Hayley Mapley <hayleymapley@catalyst.net.nz>
To test:
1 - Apply unit test patch only
2 - prove the tests, they fail
3 - Attempt to import the attached sample file
4 - First patron may be imported, second will give error (or both will error)
5 - If only one error, import agian, now both have error
6 - Apply this patch
7 - Prove tests, they pass!
8 - Import the sample file
9 - It succeeds!
Signed-off-by: Claire Gravely <claire.gravely@bsz-bw.de>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
(cherry picked from commit cc60770c4e)
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
(cherry picked from commit 80cac150b8)
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
For all other methods we define a patron object and check the userid,
we just need to define the $patron here
To test:
0 - Apply unit test patch alone
1 - prove t/db_dependent/Koha/Patrons/Import.t
2 - It dies
3 - Apply this patch
4 - repeat tests
5 - It passes!
Signed-off-by: Devlyn Courtier <dcourtier@hccc.edu>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
(cherry picked from commit 8a9c82c3ec)
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
To test:
1 - Export your patrons
a - Create a report 'SELECT * FROM borrowers'
b - Run and save the report as csv (check your delimiter)
c - Delete the borrowernumebr column
2 - Use the Patron Import tool to import the csv from above
3 - Set matching to 'cardnumber'
4 - Set 'If matching record is already in the borrowers table:' to
Overwrite
5 - Import
6 - None are import because of matchign userid (their own)
7 - Apply patch
8 - Repeat
9 - Patrons are successfully overwritten
10 - prove -v t/db_dependent/Koha/Patrons/Import.t
11 - prove -v t/db_dependent/Koha/Patrons.t
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
(cherry picked from commit 9a2bd027e5)
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Incorrect date value: '' for column 'date_renewed'
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
We previously prove that the method and the subroutine were equivalent,
we know update the controller calls.
Test plan:
- Add and update a patron with different variations of userid
(automatically generated or not)
- Import patrons with and without userid, as well as with existing
userid
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
When a test was introduced to not add duplicate userids
it overlooked that if you were matching on userid you
do not want the update to be rejected.
Add a test to overcome this logical flaw if updating
by matching userid
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch will have to be squashed with previous ones, but for
readability I prefer to let it alone for now.
It reintroduces 2 missing changes:
commit fd12390365
Bug 15010: Import patron tool creates 'duplicate' restrictions (
debarments )
commit 011d8e9855
Bug 12543 - Add userid as matchpoint for "Import patrons" tool
Signed-off-by: Colin Campbell <colin.campbell@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Tests will have to be written for this change
Signed-off-by: Colin Campbell <colin.campbell@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Test Plan:
1) Apply this patch
2) Test importing patrons from command line,
options are availble with --help.
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Tested with minimal csv
(cardnumber,surname,firstname,categorycode,branchcode,password,userid)
Overwrite does not change category or branch.
Patrons are loaded, userid & password works
Updated license to GPLv3
No other koha-qa errors.
Signed-off-by: Benjamin Rokseth <benjamin.rokseth@kul.oslo.kommune.no>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Bug 12598 - Tidy import_borrowers.pl
Signed-off-by: Benjamin Rokseth <benjamin.rokseth@kul.oslo.kommune.no>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Bug 12598 - Move importing code to a subroutine
Signed-off-by: Benjamin Rokseth <benjamin.rokseth@kul.oslo.kommune.no>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Bug 12598 - Update command line script to use patron import subroutine
Signed-off-by: Benjamin Rokseth <benjamin.rokseth@kul.oslo.kommune.no>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Bug 12598 [QA Followup]
* Fix copyright on import_borrowers.pl
* Changes -c --csv to -f --file
* Adds -c --confirm option
* Renames misc/import_borrowers.pl to misc/import_patrons.pl
* Restore userid matchpoint option
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Bug 12598 - Fix merge to master. Backport 3 updates from latest import_borrowers.pl
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Bug 12598 - Started regression tests. Fix missing C4::Members::Attributes package
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Bug 12598 - More refactoring and regression tests in Koha::Patrons::Import
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Bug 12598 - Creating objects in misc/import_patrons.pl and tools/import_borrowers.pl
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Bug 12598 - Refactoring Koha::Patrons::Import includes bug fixed for critical date types and header column parsing
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Bug 12598 - Rebase + backport of 16426 plus fixing 16426
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Bug 12598: catch warnings raised by import_patrons in tests
Signed-off-by: Colin Campbell <colin.campbell@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Kyle M Hall
Nick Clemens
Jonathan Druart
Colin Campbell