This patch correct the call to filter the record
To test:
1 - Enable OAI
2 - Set OAI-PMH:ConfFile to '/var/lib/koha/kohadev/OAI.yaml'
3 - Edit/create the file above to have content:
format:
marcxml:
metadataPrefix: marcxml
metadataNamespace: http://www.loc.gov/MARC21/slimhttp://www.loc.gov/standards/marcxml/schema/MARC21slim
schema: http://www.loc.gov/MARC21/slimhttp://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd
include_items: 1
expanded_avs: 1
4 - save and restart_all
5 - View some records:
http://localhost:8080/cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=marcxml
6 - Note the items show coded values
7 - Apply the regressions tests patch
8 - Run:
$ ktd --shell
k$ prove t/db_dependent/Koha/OAI/Server/Repository.t
=> FAIL: Tests fail!
9 - Apply this patch
10 - Restart all
11 - Refresh and confirm items now have descriptions
12 - Repeat 8
=> SUCCESS: Tests pass!
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Edit (tcohen): Amended the test plan to include the added regression
tests run
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Added head2 right before sub with missing POD
Signed-off-by: Baptiste Wojtkowski <baptiste.wojtkowski@biblibre.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This commit is generated using:
% perl misc/devel/tidy.pl
*within* ktd, to get the same version of perltidy than what will be used
by our CI (currently v20230309).
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
We can now call metadata_record directly on the Koha::Biblio object.
This aptch updates all modules and controllers to use
Koha::Biblio->metadata_record directly where appropriate. The
exceptions are where we don't require any filtering or the filtering
makes sense to do after the initial fetch.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
If a bib record is hidden by OpacHiddenItems,
mark it as deleted in the OAI-PMH, so that it's metadata
is not consumed in a new OAI-PMH harvest, and it's marked
for removal if the metadata was consumed in previous harvested.
Test plan:
0. Setup
vi /etc/koha/sites/kohadev/oai.conf
format:
marcxml:
metadataPrefix: marcxml
metadataNamespace: http://www.loc.gov/MARC21/slimhttp://www.loc.gov/standards/marcxml/schema/MARC21slim
schema: http://www.loc.gov/MARC21/slimhttp://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd
include_items: 1
Set sysprefs:
OAI-PMH: Enable
OAI-PMH:ConfFile: /etc/koha/sites/kohadev/oai.conf
OAI-PMH:AutoUpdateSets: Enable
OAI-PMH:AutoUpdateSetsEmbedItemData: Enabled
OpacHiddenItems:
barcode: [3999900000001]
OpacHiddenItemsHidesRecord: Hide
Adminsitration->OAI set configuration->New set
Choose anything for setSpec and setName
On new set: Actrion->define mappings
Field: 952
Subfield: p
Operator: is equal to
Value: 3999900000001
perl misc/migration_tools/build_oai_sets.pl -r -v -i
1. koha-plack --restart kohadev
2. Go to http://localhost:8080/cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=marcxml
3. Confirm KOHA-OAI-TEST:1 is marked as deleted and not metadata visible
4. Go to http://localhost:8080/cgi-bin/koha/oai.pl?verb=ListIdentifiers&metadataPrefix=marcxml
5. Confirm KOHA-OAI-TEST:1 is marked as deleted
6. Go to http://localhost:8080/cgi-bin/koha/oai.pl?verb=GetRecord&metadataPrefix=marcxml&identifier=KOHA-OAI-TEST:1
7. Confirm record is marked as deleted and no metadata visible
8. View Sets and confirm the record appears in the set but as deleted record
9. Change OpacHiddenItemsHidesRecord syspref to "Don't hide" and repeat above steps
10. Confirm that the record metadata is now visible and the record is not marked as deleted
Signed-off-by: Jake Deery <jake.deery@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
For some Unimarc biblio records, the OAI Server returns a bad XML
document, ie a document with extended characters, such as accents, not
encoded in UTF8. It seems to come from somewhere in MARC::Record library
trying to deduce biblio record endoding from some fields. It may work
for MARC21 records. It doesn't for Unimarc records. So it's necessary to
send to the method transforming record into XML the marcflavour. This
way there is no encoding guessing from Unimarc biblio records content.
https://bugs.koha-community.org/show_bug.cgi?id=34467
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch uses the new record_strip_nonxml routine to attempt to display
the record when it contains invalid characters
Rather than silently strippg these, we warn in the logs, then add an 'about'
container to the response. It is displayed nicely in the web view or sent as "INVALID_METADATA" in
the xml response
The 'error' codes for OAI seem to be at the request level, and the offered codes don't have a match
for a bad record. Adding the about when we can recover seems the most generous response
To test:
Test plan, assumes using KTD default data - otherwise you need to find and import a record with encoding issues:
1 - Enable OAI-PMH system preference
2 - Browse to:
http://localhost:8080/cgi-bin/koha/oai.pl?verb=ListRecords&resumptionToken=marcxml/350////0/0/352
3 - 500 error:
Invalid data, cannot decode metadata object (biblio_metadata.id=368, biblionumber=369, format=marcxml, schema=MARC21, decoding_error=':8: parser error : PCDATA invalid Char value 31...
4 - Apply patch, restart all
5 - Reload the page
6 - It loads!
7 - Click 'Metadata' for record 369 - it succeeds!
8 - Check the logs - confirm you see a warning of the record problem
9 - Confirm 369 has an about section
10 - Check the individul 'GetRecord' response as well
http://localhost:8080/cgi-bin/koha/oai.pl?verb=GetRecord&metadataPrefix=oai_dc&identifier=KOHA-OAI-TEST:369
Signed-off-by: Sam Lau <samalau@gmail.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
The statement is already inside an if block with the same condition.
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch removes the last remaining uses of transformMARCXML4XSLT and
replaced them with the equivilent RecordProcessor usage.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Includes:
Bug 29697: (follow-up) Use flag embed_items
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
On bug 17591 we discovered that there was something weird going on with
the way we export and use subroutines/modules.
This patch tries to standardize our EXPORT to use EXPORT_OK only.
That way we will need to explicitely define the subroutine we want to
use from a module.
This patch is a squashed version of:
Bug 17600: After export.pl
Bug 17600: After perlimport
Bug 17600: Manual changes
Bug 17600: Other manual changes after second perlimports run
Bug 17600: Fix tests
And a lot of other manual changes.
export.pl is a dirty script that can be found on bug 17600.
"perlimport" is:
git clone https://github.com/oalders/App-perlimports.git
cd App-perlimports/
cpanm --installdeps .
export PERL5LIB="$PERL5LIB:/kohadevbox/koha/App-perlimports/lib"
find . \( -name "*.pl" -o -name "*.pm" \) -exec perl App-perlimports/script/perlimports --inplace-edit --no-preserve-unused --filename {} \;
The ideas of this patch are to:
* use EXPORT_OK instead of EXPORT
* perltidy the EXPORT_OK list
* remove '&' before the subroutine names
* remove some uneeded use statements
* explicitely import the subroutines we need within the controllers or
modules
Note that the private subroutines (starting with _) should not be
exported (and not used from outside of the module except from tests).
EXPORT vs EXPORT_OK (from
https://www.thegeekstuff.com/2010/06/perl-exporter-examples/)
"""
Export allows to export the functions and variables of modules to user’s namespace using the standard import method. This way, we don’t need to create the objects for the modules to access it’s members.
@EXPORT and @EXPORT_OK are the two main variables used during export operation.
@EXPORT contains list of symbols (subroutines and variables) of the module to be exported into the caller namespace.
@EXPORT_OK does export of symbols on demand basis.
"""
If this patch caused a conflict with a patch you wrote prior to its
push:
* Make sure you are not reintroducing a "use" statement that has been
removed
* "$subroutine" is not exported by the C4::$MODULE module
means that you need to add the subroutine to the @EXPORT_OK list
* Bareword "$subroutine" not allowed while "strict subs"
means that you didn't imported the subroutine from the module:
- use $MODULE qw( $subroutine list );
You can also use the fully qualified namespace: C4::$MODULE::$subroutine
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Should be XSLT::Base now.
Removes old XSLT_Handler stub too (from bug 23290).
Result of a git grep | sed statement.
Test plan:
Run qa tools (so modules compile).
Run t/db_dependent/Breeding.t
Run t/db_dependent/Koha/XSLT/Base.t (This test fails when only this patch
has been applied; see subsequent patch.)
Enable XSLT use on results and details display. Check search results and
detail view on OPAC and staff.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
TEST PLAN
---------
apply unit test patch
prove t/db_dependent/OAI/Server.t
-- should fail
apply this patch
prove t/db_dependent/OAI/Server.t
-- should pass
run koha qa test tools
Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Replacing some code in the OAI modules by a call to an existing module.
Note that the xmldoc format is used in the transform call to get a xml
document object.
The stylesheet method of Repository now only returns the name of the xsl
file to be used instead of a cached xslt object. Similar functionality
inside XSLT_Handler is used when calling transform.
Note: We still lack unit tests in this area. I did not see the need for
adding something for stylesheet since it only returns a simple string.
The other change is made in Record::new; there are no tests for this
module yet and the heart of the change here is actually tested already
in XSLT_Handler.t.
Note: I benchmarked calls to Repository in the old and the new situation
and did not see significant changes.
Test plan:
[1] Run t/db_dependent/OAI/Server.t
[2] Run oai.pl with ListRecords and marcxml.
[3] Run oai.pl with ListRecords and oai_dc.
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Change parameters to a hashref.
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Looks good to me.
Two calls in migration_tools/22_to_30 still in old style.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
- Fixed date handling to use UTC as specs require.
- Added support for second precision in time stamps.
- Added support for marc21 metadata prefix as recommended in the
guidelines (synonym for marcxml).
- Improved performance of database queries especially for large
collections.
- Unified functionality of ListRecords and ListIdentifiers to a common
base class.
- If items are included in the records, their timestamps are taken into
account everywhere so that whichever is the most recent (timestamp of
biblioitem or any of its items) is considered the record's timestamp.
- Fixed OAI.xslt to show correct record range.
- Incorporated extended tests from Bug 17493 and their tweaks from Bug 15108.
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Olli-Antti Kivilahti <olli-antti.kivilahti@jns.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
This patch introduces an XSLT file grabbed from DSpace and a couple
css files.
To test:
- Point to your dev Koha's oai endpoint. For example:
http://localhost:8080/cgi-bin/koha/oai.pl?verb=ListMetadataFormats
=> FAIL: Look how ugly it is for humans.
- Apply this patch
- Reload
=> SUCCES: It is styled and provides really useful links to interact with
the OAI server verbs.
- Sign off :-D
Sponsored-by: Universidad Empresarial Siglo 21
Signed-off-by: Frédéric Demians <f.demians@tamil.fr>
Works as described. Nice looking.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
Koha OAI server has been done in one unique .pl file because there
wasn't any object model or rules in the Koha project when it has been
coded. This patch modularized existing classes, putting each class in a
separate file in Koha::OAI::Server namespace. UT begining.
Add new dependency: Capture::Tiny
Signed-off-by: Hector Castro <hector.hecaxmmx@gmail.com>
OAI server moduralized succefully. Works for Debian Jessie and
Wheezy. Test pass successfully
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Nick Clemens
Jonathan Druart
Koha Development Team
Tomas Cohen Arazi
Martin Renvoize
David Cook
Frédéric Demians
Joonas Kylmälä
Julian Maurice
Marcel de Rooy
Ere Maijala
Mark Tompsett