It turns out the modules had the option expected built in, we just didn't call
it. This patch set expands the options for passing to the export.
To test:
1 - Go to Tools->Export data
2 - Perform an export and check 'Remove non-local items'
3 - Note that file contains only 'local' records, but includes all items
on those records
4 - Apply patch
5 - Note Tools->Export data has a new option to remove records not owned
by logged in branch
6 - Export as before checking records option, file should be as before
7 - Now check 'Remove items not owned by logged in branch'
8 - File should now only have local items (may have empty records)
9 - Check both boxes and recieve only 'local' records and items
Signed-off-by: Scott Kehoe <scott@masslibsystem.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Change parameters to a hashref.
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Looks good to me.
Two calls in migration_tools/22_to_30 still in old style.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
The way the export options are displayed at the bottom of the checkouts table
was not consistent.
Prior to this patch set, they are display if ExportRemoveFields or
ExportWithCsvProfile is set.
It does not make any sense, the user could want to export the checkouts in
iso2709 format without having to define a csv profile and fill the pref.
Moreover the behavior of this pref did not match its description: it's used as
a default CSV profile when exporting records from the export tools or the
command line.
This patch set adds a new pref ExportCircHistory and remove
ExportWithCsvProfile. The new pref is set if ExportWithCsvProfile or
ExportRemoveFields were set.
A new dropdown list with the CSV profile list will be displayed in the
export area, at the bottom of the checkouts table.
Note that now --csv_profile_id is mandatory for the export command line
(misc/export_records.pl) if the export format is csv.
Test plan:
0/ Do not execute the DB entry
1/ Clear both ExportWithCsvProfile and ExportRemoveFields prefs
2/ Execute the DB entry
3/ ExportCircHistory should not be set and the export options should not
be displayed at the bottom of the checkouts table.
4/ Remove the pref
DELETE FROM systempreferences WHERE variable='ExportCircHistory';
and reinsert the previous one, with a value:
INSERT INTO systempreferences (variable, value) VALUES
('ExportWithCsvProfile', 'something');
Execute the DB entry again
=> The now pref should be now set
5/ Export some checkouts using the CSV entry
6/ Note that the export tool and commandline script still work using the
csv format. You have to provide a --csv_profile_id option to make it
work.
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Fixes a TODO for logging unsupported record_type in _get_record_for_export.
Also logs a warning when the record_type parameter is not supplied at all in sub export.
Replaces a warn by a log message about an invalid record for format iso2709.
Also adds a log message about an invalid record for format xml.
Adds a trivial unit test for passing no record_type to export.
Test plan:
See also first patch.
Run t/db_dependent/Exporter/Record.t.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Also tested the log messages for iso2709 and xml by manipulating
the record_type parameter with:
$params->{record_type}='xx';
before calling _get_record_for_export in Record.pm.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
If export_records.pl encounters a record that fails to load correctly as
a MARC::Record the export process will fail and only the records
encountered up to that point will be exported.
Test Plan:
1) Pick a record in your database, and replace the biblioitems.marcxml field with some random characters
2) Export your records using the command:
misc/export_records.pl --format xml --record-type bibs --filename /tmp/biblios.xml
3) Note you get an error and the script ends prematurely
4) Apply this patch
5) Repeat step 2
6) Note the full record set is exported ( except for your bad record )
7) Note an error is still generated for your bad record ( useful for
tracking down bad records ).
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
The previous calls were wrong, but there is something bad with the DB
structure: export_format.profile should be a unique key.
This patch fixes the previous calls and add a FIXME not to forget to fix
the DB structure.
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Previous test where done with all patches applied,
including this one, and all worked.
No errors
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
This subroutine returned the export_format_id for a given profile name.
This can be done easily with the Koha::CsvProfiles->search method.
Test plan:
Export records using the misc/export_records.pl script and the
export tool.
If you are exporting using the MARC format, the profile filled in the pref
ExportWithCsvProfile will be used (or the one passed in parameter of
misc/export_records.pl).
If you are exporting using the CSV format, you can choose a profile in
the dropdown list.
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Exported using tool & cmd, marc & csv. Pref is used.
No errors
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
The existing Koha::Authority does not use Koha::Objects and does not
exclusively deal with DB objects.
It makes much sense to move it to let the space free for a
Koha::Authority and Koha::Authorities modules based on Koha::Object.
Signed-off-by: Marc Véron <veron@veron.ch>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Brendan Gallagher brendan@bywatersolutions.com
Why a refactoring was need for this script?
The export tool (tools/export.pl) can be called from the command line
and some parts of code were unnecessarity complicated (just look at the
code, you will understand).
Worse still, the script does not provide the same options for both
interface. For instance you cannot export records given a range of
biblionumbers, authids, callnumbers, etc. from the commandline.
What does this patch?
1/ Important: The script tools/export.pl does not work anymore if called from
the command line (should be in the release notes).
2/ The code used to generated a file (csv, iso2709 or xml) has been moved to a new
module (Koha::Exporter::Record) and tests have been provided.
3/ No change is done on the web interface
4/ Some new options have been added to the commandline script
(misc/export.pl):
- starting_authid
- ending_authid
- authtype
- starting_biblionumber
- ending_biblionumber
- itemtype
- starting_callnumber
- ending_callnumber
- start_accession
- end_accession
5/ There is a change in the behavior if an error occurs:
Can't call method "as_usmarc" on an undefined value at Koha/Exporter/Record.pm line 114.
record (number 5530) is invalid and therefore not exported because its reopening generates warnings above at Koha/Exporter/Record.pm line 117.
Before this patch, they were not displayed (using the command line).
What does not do this patch?
It does not provide the 'clean', 'timestamp' and 'deleted_barcodes' options to
the web interface (same as before).
What about the perfs?
With a DB with ~800 biblios (MARC21)
Before: perl tools/export.pl 14.79s user 0.83s system 71% cpu 21.905 total
After: perl misc/export.pl 17.19s user 0.84s system 75% cpu 24.018 total
With a DB with ~6400 biblios (UNIMARC)
Before: perl tools/export.pl 26.55s user 0.76s system 76% cpu 35.498 total
After: perl misc/export.pl 26.78s user 0.84s system 80% cpu 34.494 total
How to test this patch?
Test plan:
A. Web interface:
1/ On the current master, export some records, biblios and authorities (with
the 3 differents exports) playing with the different filters (item type,
libraries, callnumber, accession date, don't export items, remove
non-local items, don't export fields, etc.).
2/ Apply this patch, export again the same records, and compare the
generated files. They must be identical!
3/ Confirm that the export features on the checkout list
(circ/circulation.pl) works as before this patch.
B. The command line
1/ On the current master, export some records, biblios and authorities (with
the 2 differents exports) playing with the different options (date,
deleted_barcodes, clean).
2/ Apply this patch, export again the same records, and compare the
generated files. They must be identical!
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Nick Clemens
Mark Tompsett
Jonathan Druart
Marcel de Rooy
Kyle M Hall