In the case when a biblio is deleted, we should not suppress its
orphaned records. Therefore, invert the conditions in the WHERE clause
to not suppress the record if no biblio could be found.
Test plan:
1. Enable the system preferences "OAI-PMH" and "OpacSuppression".
Ensure "OpacSuppressionByIPRange" is blank.
2. Navigate to
/cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=oai_dc
3. Select a displayed record, and delete its corresponding biblio.
4. Refresh the OAI page, verify the record has vanished.
5. Apply patch.
6. Refresh the OAI page, verify the record now appears and is marked as
deleted. It may have moved to the top of the list.
Sponsored-by: Open Fifth
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Pedro Amorim <pedro.amorim@openfifth.co.uk>
As we're aliasing the primary tables as `main`, we need to use that name
to refer to their columns.
Test plan:
1. Apply patch.
2. Prove t/db_dependent/OAI/Server.t and verify no tests fail.
Sponsored-by: Open Fifth
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Pedro Amorim <pedro.amorim@openfifth.co.uk>
Rather than fetching each biblio before choosing to suppress it, this
patch performs this filtering in the initial SQL query that fetches the
biblionumbers.
Note: for queries including items, this additional filtering is
performed in each of the three subqueries, necessitating three
quasi-joins against biblio. If we were to hoist this filtering to the
outer layer, we would also need to remove the LIMIT from each subquery
so as to allow each of them to be fairly weighted. As the preceding
comment implies keeping the subqueries separate has been beneficial for
performance, I've elected not to hoist the WHERE.
Test plan (specific URLs given for KTD, with thanks to David Cook):
1. Apply patch.
2. Navigate to http://localhost:8081/cgi-bin/koha/cataloguing/addbiblio.pl?biblionumber=2
3. Set 942$n to "Yes" and save the record.
4. Disable the system preference "OpacSuppression", and ensure
"OpacSuppressionByIPRange" is blank.
5a. Verify you can successfully access
http://localhost:8080/cgi-bin/koha/oai.pl?verb=GetRecord&metadataPrefix=oai_dc&identifier=KOHA-OAI-TEST:2
5b. Verify KOHA-OAI-TEST-2 appears on
http://localhost:8080/cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=oai_dc
6. Enable the system preference "OpacSuppression", and ensure
"OpacSuppressionByIPRange" is blank.
7. Repeat steps 5a-5b, verifying that:
- the OAI GetRecord page returns "There is no biblio record with
this identifier".
- the OAI ListRecords page does not show KOHA-OAI-TEST-2.
Sponsored-by: Open Fifth
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Pedro Amorim <pedro.amorim@openfifth.co.uk>
CGI.pm 4.68 (shipped with Debian 13/Trixie) changed the behavior of
self_url() to include a trailing slash when there's no path component.
This causes OAI-PMH tests to fail as they expect URLs without trailing
slashes.
This patch strips the trailing slash from both the requestURL field in
OAI responses and the baseURL field in Identify responses to maintain
compatibility across CGI.pm versions.
Test plan:
1. Start a 'trixie' KTD:
$ KOHA_IMAGE=main-trixie ktd --proxy --name trixie up -d
$ ktd --name trixie --wait-ready 120
2. Run the tests:
$ ktd --name trixie --shell
k$ prove t/db_dependent/OAI/Server.t
=> FAIL: Tests fail!
3. Apply this patch
5. Repeat 2
6. Repeat 1-2 with a 'bookworm' image
=> SUCCESS: Tests pass too!
7. Sign off :-D
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch correct the call to filter the record
To test:
1 - Enable OAI
2 - Set OAI-PMH:ConfFile to '/var/lib/koha/kohadev/OAI.yaml'
3 - Edit/create the file above to have content:
format:
marcxml:
metadataPrefix: marcxml
metadataNamespace: http://www.loc.gov/MARC21/slimhttp://www.loc.gov/standards/marcxml/schema/MARC21slim
schema: http://www.loc.gov/MARC21/slimhttp://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd
include_items: 1
expanded_avs: 1
4 - save and restart_all
5 - View some records:
http://localhost:8080/cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=marcxml
6 - Note the items show coded values
7 - Apply the regressions tests patch
8 - Run:
$ ktd --shell
k$ prove t/db_dependent/Koha/OAI/Server/Repository.t
=> FAIL: Tests fail!
9 - Apply this patch
10 - Restart all
11 - Refresh and confirm items now have descriptions
12 - Repeat 8
=> SUCCESS: Tests pass!
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Edit (tcohen): Amended the test plan to include the added regression
tests run
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Added head2 right before sub with missing POD
Signed-off-by: Baptiste Wojtkowski <baptiste.wojtkowski@biblibre.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This commit is generated using:
% perl misc/devel/tidy.pl
*within* ktd, to get the same version of perltidy than what will be used
by our CI (currently v20230309).
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
We can now call metadata_record directly on the Koha::Biblio object.
This aptch updates all modules and controllers to use
Koha::Biblio->metadata_record directly where appropriate. The
exceptions are where we don't require any filtering or the filtering
makes sense to do after the initial fetch.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
If a bib record is hidden by OpacHiddenItems,
mark it as deleted in the OAI-PMH, so that it's metadata
is not consumed in a new OAI-PMH harvest, and it's marked
for removal if the metadata was consumed in previous harvested.
Test plan:
0. Setup
vi /etc/koha/sites/kohadev/oai.conf
format:
marcxml:
metadataPrefix: marcxml
metadataNamespace: http://www.loc.gov/MARC21/slimhttp://www.loc.gov/standards/marcxml/schema/MARC21slim
schema: http://www.loc.gov/MARC21/slimhttp://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd
include_items: 1
Set sysprefs:
OAI-PMH: Enable
OAI-PMH:ConfFile: /etc/koha/sites/kohadev/oai.conf
OAI-PMH:AutoUpdateSets: Enable
OAI-PMH:AutoUpdateSetsEmbedItemData: Enabled
OpacHiddenItems:
barcode: [3999900000001]
OpacHiddenItemsHidesRecord: Hide
Adminsitration->OAI set configuration->New set
Choose anything for setSpec and setName
On new set: Actrion->define mappings
Field: 952
Subfield: p
Operator: is equal to
Value: 3999900000001
perl misc/migration_tools/build_oai_sets.pl -r -v -i
1. koha-plack --restart kohadev
2. Go to http://localhost:8080/cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=marcxml
3. Confirm KOHA-OAI-TEST:1 is marked as deleted and not metadata visible
4. Go to http://localhost:8080/cgi-bin/koha/oai.pl?verb=ListIdentifiers&metadataPrefix=marcxml
5. Confirm KOHA-OAI-TEST:1 is marked as deleted
6. Go to http://localhost:8080/cgi-bin/koha/oai.pl?verb=GetRecord&metadataPrefix=marcxml&identifier=KOHA-OAI-TEST:1
7. Confirm record is marked as deleted and no metadata visible
8. View Sets and confirm the record appears in the set but as deleted record
9. Change OpacHiddenItemsHidesRecord syspref to "Don't hide" and repeat above steps
10. Confirm that the record metadata is now visible and the record is not marked as deleted
Signed-off-by: Jake Deery <jake.deery@ptfs-europe.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
For some Unimarc biblio records, the OAI Server returns a bad XML
document, ie a document with extended characters, such as accents, not
encoded in UTF8. It seems to come from somewhere in MARC::Record library
trying to deduce biblio record endoding from some fields. It may work
for MARC21 records. It doesn't for Unimarc records. So it's necessary to
send to the method transforming record into XML the marcflavour. This
way there is no encoding guessing from Unimarc biblio records content.
https://bugs.koha-community.org/show_bug.cgi?id=34467
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch uses the new record_strip_nonxml routine to attempt to display
the record when it contains invalid characters
Rather than silently strippg these, we warn in the logs, then add an 'about'
container to the response. It is displayed nicely in the web view or sent as "INVALID_METADATA" in
the xml response
The 'error' codes for OAI seem to be at the request level, and the offered codes don't have a match
for a bad record. Adding the about when we can recover seems the most generous response
To test:
Test plan, assumes using KTD default data - otherwise you need to find and import a record with encoding issues:
1 - Enable OAI-PMH system preference
2 - Browse to:
http://localhost:8080/cgi-bin/koha/oai.pl?verb=ListRecords&resumptionToken=marcxml/350////0/0/352
3 - 500 error:
Invalid data, cannot decode metadata object (biblio_metadata.id=368, biblionumber=369, format=marcxml, schema=MARC21, decoding_error=':8: parser error : PCDATA invalid Char value 31...
4 - Apply patch, restart all
5 - Reload the page
6 - It loads!
7 - Click 'Metadata' for record 369 - it succeeds!
8 - Check the logs - confirm you see a warning of the record problem
9 - Confirm 369 has an about section
10 - Check the individul 'GetRecord' response as well
http://localhost:8080/cgi-bin/koha/oai.pl?verb=GetRecord&metadataPrefix=oai_dc&identifier=KOHA-OAI-TEST:369
Signed-off-by: Sam Lau <samalau@gmail.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
The statement is already inside an if block with the same condition.
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch removes the last remaining uses of transformMARCXML4XSLT and
replaced them with the equivilent RecordProcessor usage.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Includes:
Bug 29697: (follow-up) Use flag embed_items
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
The query was missing the "ORDER BY" and "LIMIT" clauses, which could make retrieve a wrong set of records.
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
This simple patch makes our OAI-PMH server return the appropriate error
code when no sets are defined.
To test:
1. Apply the regression tests
2. Run:
$ kshell
k$ t/db_dependent/OAI/Server.t
=> FAIL: Tests fail. Boo!
3. Apply this patch
4. Repeat 2
=> SUCCESS: Tests pass! YAY!
5. Sign off :-D
Bonus: you can check with the tool mentioned on the bug report, that
the error is gone. No sets need to be defined.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Nind <david@davidnind.com>
JD amended patch
- ) unless scalar @{$sets} > 0;
+ ) unless @$sets;
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
The tests are using 'http://localhost' and the regex is setting $baseURL
to empty in this case. It works on the oai.pl front-end.
This patch changes the regex.
To test:
1. Run:
$ kshell
k$ t/db_dependent/OAI/Server.t
=> FAIL: Boo! Warning!
2. Apply this patch
3. Repeat 2
=> SUCCESS: Oh, cool. No warning :-D
4. Sign off :-D
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
For OAI-PMH, all date and time values must be in the format
"YYYY-MM-DDThh:mm:ssZ" and in UTC. Currently,
Identify.earliestDatestamp uses the SQL format "YYYY-MM-DD hh:mm:ss".
This patch fixes that.
To test:
1) Get a Koha instance with OAI-PMH enabled.
2) Visit /cgi-bin/koha/oai.pl?verb=Identify, view source for the page
and observe that earliestDate is in the wrong format
"YYYY-MM-DD hh:mm:ss"
3) Apply patch
4) Visit /cgi-bin/koha/oai.pl?verb=Identify, view the source for the page
and observe that earliestDate is now in the correct format
"YYYY-MM-DDThh:mm:ssZ"
5) Sign off
Sponsored-by: Reformational Study Centre
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
When the OAI-PMH provider was configured to include item information
in the returned records, deleting an item would cause the record to
be reported as deleted. The provider also did some useless checks when
determining the timestamp of a deleted record. These checks only had
a performance impact.
To test:
1) Create /var/lib/koha/kohadev/OAI.yaml with:
format:
marcxml:
metadataPrefix: marcxml
metadataNamespace: http://www.loc.gov/MARC21/slimhttp://www.loc.gov/standards/marcxml/schema/MARC21slim
schema: http://www.loc.gov/MARC21/slimhttp://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd
include_items: 1
2) Set preference OAI-PMH to Enable
3) Set preference OAI-PMH:ConfFile to /var/lib/koha/kohadev/OAI.yaml
4) Delete an item from a record
5) View the page: http://localhost:8080/cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=marcxml
6) Note the record is now listed as deleted
7) Run prove -v t/db_dependent/OAI/
Sponsored-by: The National Library of Finland
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
TO test:
1 - Enable OAI-PMH server in Koha
2 - Set OAI-PMH:MaxCount to 5
3 - Define 6 sets in Admin->OAI sets configuration
Can setSpec and setName to 1,2,3....
4 - Browse to:
http://localhost:8080/cgi-bin/koha/oai.pl?verb=ListSets
5 - Click 'Show more'
6 - Keep seeing the same 5 - note the page says 'Results fetched -4-0'
7 - Apply patch
8 - Restart all the things
9 - Reload the ListSets URL above
10 - Click 'Show more'
11 - See the 6th set and note no 'Show more'
12 - Set OAI-PMH:MaxCount to 1
13 - Go to the base ListSets url again
14 - Confirm you can 'Show more' until you reach the end
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
On bug 17591 we discovered that there was something weird going on with
the way we export and use subroutines/modules.
This patch tries to standardize our EXPORT to use EXPORT_OK only.
That way we will need to explicitely define the subroutine we want to
use from a module.
This patch is a squashed version of:
Bug 17600: After export.pl
Bug 17600: After perlimport
Bug 17600: Manual changes
Bug 17600: Other manual changes after second perlimports run
Bug 17600: Fix tests
And a lot of other manual changes.
export.pl is a dirty script that can be found on bug 17600.
"perlimport" is:
git clone https://github.com/oalders/App-perlimports.git
cd App-perlimports/
cpanm --installdeps .
export PERL5LIB="$PERL5LIB:/kohadevbox/koha/App-perlimports/lib"
find . \( -name "*.pl" -o -name "*.pm" \) -exec perl App-perlimports/script/perlimports --inplace-edit --no-preserve-unused --filename {} \;
The ideas of this patch are to:
* use EXPORT_OK instead of EXPORT
* perltidy the EXPORT_OK list
* remove '&' before the subroutine names
* remove some uneeded use statements
* explicitely import the subroutines we need within the controllers or
modules
Note that the private subroutines (starting with _) should not be
exported (and not used from outside of the module except from tests).
EXPORT vs EXPORT_OK (from
https://www.thegeekstuff.com/2010/06/perl-exporter-examples/)
"""
Export allows to export the functions and variables of modules to user’s namespace using the standard import method. This way, we don’t need to create the objects for the modules to access it’s members.
@EXPORT and @EXPORT_OK are the two main variables used during export operation.
@EXPORT contains list of symbols (subroutines and variables) of the module to be exported into the caller namespace.
@EXPORT_OK does export of symbols on demand basis.
"""
If this patch caused a conflict with a patch you wrote prior to its
push:
* Make sure you are not reintroducing a "use" statement that has been
removed
* "$subroutine" is not exported by the C4::$MODULE module
means that you need to add the subroutine to the @EXPORT_OK list
* Bareword "$subroutine" not allowed while "strict subs"
means that you didn't imported the subroutine from the module:
- use $MODULE qw( $subroutine list );
You can also use the fully qualified namespace: C4::$MODULE::$subroutine
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Includes the following optimizations:
- Use next biblionumber instead of large offset in the queries.
- Use unions instead of subqueries
- Avoid fetching item timestamps when items are not included.
Test plan:
1. Without the patch, try harvesting a Koha database with (and without for good measure) `include_items: 1` in the OAI-PMH configuration file pointed to by preference OAI-PMH:ConfFile and take note of performance. For useful metrics the database must be large enough to not fit in InnoDB buffers or OS file cache.
2. Apply the patch.
3. Run tests: prove -v t/db_dependent/OAI
4. Try again the harvesting from step 1 and compare performance with step 1.
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Test plan:
0) apply the patchset
1) setup oai server with oaiconf file defining xsl_file for metadata
format marcxml or marc21
2) play with your oai and confirm that records are transformed as
expected
3) prove t/db_dependent/OAI/Server.t
Signed-off-by: Martin Šťastný <martin.stastny@teamlibrary.cz>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Should be XSLT::Base now.
Removes old XSLT_Handler stub too (from bug 23290).
Result of a git grep | sed statement.
Test plan:
Run qa tools (so modules compile).
Run t/db_dependent/Breeding.t
Run t/db_dependent/Koha/XSLT/Base.t (This test fails when only this patch
has been applied; see subsequent patch.)
Enable XSLT use on results and details display. Check search results and
detail view on OPAC and staff.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This seems not to be needed. Just at the safe side.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
To test:
1 - Edit/create a file in the koha home dir '/var/lib/koha/{instance}/OAI.yaml' with content like below:
format:
marcxml:
metadataPrefix: marcxml
metadataNamespace: http://www.loc.gov/MARC21/slimhttp://www.loc.gov/standards/marcxml/schema/MARC21slim
schema: http://www.loc.gov/MARC21/slimhttp://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd
include_items: 1
2 - Set the systempreferences below to indicated values:
OAI-PMH => Enable
OAI-PMH:ConfFile => /var/lib/koha/{instace}/OAI.yaml
3 - View a record with items and confirm the items show:
http://localhost:8080/cgi-bin/koha/oai.pl?verb=GetRecord&metadataPrefix=marcxml&identifier=KOHA-OAI-TEST:4
4 - Delete the items, then delete the record
5 - Reload the URL from step 3
6 - Internal Server Error:
Empty String at /kohadevbox/koha/Koha/OAI/Server/Record.pm line 58.
at /usr/share/perl/5.24/Carp.pm line 167
7 - Apply patch
8 - restart_all
9 - Repeat
10 - Reload URL from step 3
11 - Success
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
TEST PLAN
---------
apply unit test patch
prove t/db_dependent/OAI/Server.t
-- should fail
apply this patch
prove t/db_dependent/OAI/Server.t
-- should pass
run koha qa test tools
Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Since bug 17196, biblioitems.timestamp is not always updated after a
change in the MARC record.
Filtering should be based on biblio_metadata.timestamp instead.
Test plan:
1. prove t/db_dependent/OAI/Server.t
2. Verify that it SUCCEEDS
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Not sure what happened here, but this is an easy fix.
Test plan:
Try oai.pl with marcxml, marc21 and oai_dc again.
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Replacing some code in the OAI modules by a call to an existing module.
Note that the xmldoc format is used in the transform call to get a xml
document object.
The stylesheet method of Repository now only returns the name of the xsl
file to be used instead of a cached xslt object. Similar functionality
inside XSLT_Handler is used when calling transform.
Note: We still lack unit tests in this area. I did not see the need for
adding something for stylesheet since it only returns a simple string.
The other change is made in Record::new; there are no tests for this
module yet and the heart of the change here is actually tested already
in XSLT_Handler.t.
Note: I benchmarked calls to Repository in the old and the new situation
and did not see significant changes.
Test plan:
[1] Run t/db_dependent/OAI/Server.t
[2] Run oai.pl with ListRecords and marcxml.
[3] Run oai.pl with ListRecords and oai_dc.
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Change parameters to a hashref.
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Looks good to me.
Two calls in migration_tools/22_to_30 still in old style.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
- Fixed date handling to use UTC as specs require.
- Added support for second precision in time stamps.
- Added support for marc21 metadata prefix as recommended in the
guidelines (synonym for marcxml).
- Improved performance of database queries especially for large
collections.
- Unified functionality of ListRecords and ListIdentifiers to a common
base class.
- If items are included in the records, their timestamps are taken into
account everywhere so that whichever is the most recent (timestamp of
biblioitem or any of its items) is considered the record's timestamp.
- Fixed OAI.xslt to show correct record range.
- Incorporated extended tests from Bug 17493 and their tweaks from Bug 15108.
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Olli-Antti Kivilahti <olli-antti.kivilahti@jns.fi>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Two discussions on koha-devel lead to the same conclusion:
biblioitems.marcxml should be moved out this table
- biblio and biblioitems
http://lists.koha-community.org/pipermail/koha-devel/2013-April/039239.html
- biblioitems.marcxml & biblioitems.marc / HUGE performance issue !
http://lists.koha-community.org/pipermail/koha-devel/2016-July/042821.html
There are several goals to do it:
- Performance
As Paul Poulain wrote, a simple query like
SELECT publicationyear, count(publicationyear) FROM biblioitems GROUP BY publicationyear;
takes more than 10min on a DB with more than 1M bibliographic records
but only 3sec (!) on the same DB without the biblioitems.marcxml field
Note that priori to this patch set, the biblioitems.marcxml was not
retrieved systematically, but was, at least, in
C4::Acquisition::GetOrdersByBiblionumber and C4::Acquisition::GetOrders
- Flexibility
Storing the marcxml in a specific table would allow use to store several
kind of metadata (USMARC, MARCXML, MIJ, etc.) and different formats (marcflavour)
- Clean code
It would be a first step toward Koha::MetadataRecord for bibliographic
records (not done in this patch set).
Test plan:
- Update the DBIC Schema
- Add / Edit / Delete / Import / Export bibliographic records
- Add items
- Reindex records using ES
- Confirm that the following scripts still work:
* misc/cronjobs/delete_records_via_leader.pl
* misc/migration_tools/build_oai_sets.pl
- Look at the reading history at the OPAC (opac-readingrecord.pl)
- At the OPAC, click on a tag, you must see the result
Note: Changes in Koha/OAI/Server/ListRecords.pm is planned on bug 15108.
Signed-off-by: Mason James <mtj@kohaaloha.com>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Zeno Tajoli <z.tajoli@cineca.it>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
This was hardcoded to 0001-01-01.
To test:
- Make sure you have a couple of records, with different timestamps
- Enable OAI-PMH
- Check http://localhost:2201/cgi-bin/koha/oai.pl?verb=Identify
and verify that "Earliest Registered Date" is 0001-01-01
- Apply the patch
- Re-check the "Earliest Registered Date" and verify that it is now
the lowest timestamp in your biblio table
- Manipulate the timestamps and verify that "Earliest Registered Date"
changes accordingly
- Sign off!
Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
This patch introduces an XSLT file grabbed from DSpace and a couple
css files.
To test:
- Point to your dev Koha's oai endpoint. For example:
http://localhost:8080/cgi-bin/koha/oai.pl?verb=ListMetadataFormats
=> FAIL: Look how ugly it is for humans.
- Apply this patch
- Reload
=> SUCCES: It is styled and provides really useful links to interact with
the OAI server verbs.
- Sign off :-D
Sponsored-by: Universidad Empresarial Siglo 21
Signed-off-by: Frédéric Demians <f.demians@tamil.fr>
Works as described. Nice looking.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
This path fixes the value of xml schema variable and the link to Library
of Congress. For Dublin Core format those values are defined in
UNIMARCslim2OAIDC.xsl and MARC21slim2OAIDC.xsl. For MARCXML those
values are written into the code
To test it:
1) Setup OAI-PMH
2) Test the site inserting the BaseURL into http://validator.oaipmh.com
3) See the problem with ListMetadataFormats [link:
http://validator.oaipmh.com/#ListMetadataFormats]
4) Insert the patch
5) See the problem is over
Signed-off-by: Frédéric Demians <f.demians@tamil.fr>
Invalid before patch, valid after.
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Brendan Gallagher brendan@bywatersolutions.com
Olivia Reynolds
David Cook
Tomás Cohen Arazi
Nick Clemens
Jonathan Druart
Koha Development Team
Martin Renvoize
Frédéric Demians
Joonas Kylmälä
Ere Maijala
Rudolf Byker
Julian Maurice
Josef Moravec
Marcel de Rooy
Mark Tompsett
Magnus Enger
Zeno Tajoli