AutoCreateAuthorities builds a brand-new authority record whenever a bib
subject heading (6XX) has no match, but always lets the record fall back
to C4::AuthoritiesMarc::AddAuthority()'s default 008/040, which hardcodes
008 position 11 ("Subject heading system/thesaurus") to 'a' (LCSH) no
matter what thesaurus the source heading actually uses. This is the first
step towards fixing that.
Adds two things to Koha::Authority:
- $MARC21_THESAURUS_TO_CONTROL_FIELD_008_11: a single-source-of-truth
lookup mapping a thesaurus name (as already derived by C4::Heading from
MARC indicator 2 / subfield $2, e.g. 'lcsh', 'mesh', or a raw $2 code
such as 'fast') to its MARC21 authority 008/11 code, per the LOC ad008
code list. This table previously only existed once, for a different
purpose, in Koha::SearchEngine::Elasticsearch::QueryBuilder (used for
building thesaurus search queries) - it's promoted here so both that
and the authority-creation fix that follows can read from one place
instead of two copies that can silently drift apart.
- default_marc21_008($thesaurus): builds the default 34-character body of
a new MARC21 authority's 008 field, honouring the
MARCAuthorityControlField008 system preference. When $thesaurus is
given, position 11 is set from the table above instead of whatever the
default/syspref value carries, falling back to 'z' (Other) for a
thesaurus with no dedicated code.
AddAuthority() is refactored to build its own (thesaurus-less, LCSH
default) 008 via this new method too, replacing its previous inline copy
of the same defaulting logic - pure refactor, no behaviour change.
Nothing thesaurus-aware calls it yet.
Sponsored-by: Heythrop Library <https://heythroplibrary.co.uk/>
Signed-off-by: Clemens Gresser <cgresser@jesuit.org.uk>
Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Pedro Amorim <pedro.amorim@openfifth.co.uk>
MARC21 4xx/5xx $w is a control subfield with up to four positions;
only position 0 encodes the hierarchical relationship. The previous
regex required the whole subfield to equal g or h, so values such as
"g " or "gnna" would fall into the middle bucket and lose the
broader/narrower distinction.
This case is hypothetical and likely rare in practice (most catalogs
store a single character in $w), but the correctness improvement still
makes sense and aligns with how BuildSummary reads the same subfield
elsewhere via substr($w, 0, 1).
To test:
- prove -v t/db_dependent/AuthoritiesMarc.t
- Verify BuildSummary/_marc21_sort_hierarchy_alpha subtest passes,
including the new multi-position $w case.
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Pedro Amorim <pedro.amorim@openfifth.co.uk>
- koha-qa pod_coverage check flagged the new private helper
- sub as uncovered. Document it briefly to satisfy the check.
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Pedro Amorim <pedro.amorim@openfifth.co.uk>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Pedro Amorim <pedro.amorim@openfifth.co.uk>
Resolve critic warn:
"return" statement followed by "sort" at line 1238, column 5. Behavior is undefined if called in scalar context. (Severity: 5)
Tidied with lastest perltidy version.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Pedro Amorim <pedro.amorim@openfifth.co.uk>
Test plan (MARC21):
[1] Create an authority record, say a term. Include two 450s, the first
one starting with Z and second with A. Include five 550s, where two
are narrower ($w==h), two broader ($w==g) and one without $w. Make
sure that the terms will sort differently than the field order (go
back in alphabet when filling subsequent $a's or so).
[2] Perform an authority search that includes the new authority.
[3] Verify that the terms in the Summary column for this authority
are sorted first on hierarchy, second alphabetically. The one
without hierarchy should be in the middle.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Pedro Amorim <pedro.amorim@openfifth.co.uk>
Authority MODIFY action log diffs were emitting bogus entries for every
field with a default ' ' indicator on every save, producing output like:
_marc.fields.1.040.ind1 ' ' ' '
_marc.fields.1.040.ind2 ' ' ' '
_marc.fields.2.150.ind1 ' ' ' '
_marc.fields.3.667.ind1 ' ' ' '
...
even when only a single subfield was edited.
Root cause: Struct::Diff treats a utf8-flagged ' ' as different from a
non-utf8 ' ' SV, even though Perl's eq returns true. ModAuthority was
sourcing the two sides of the diff from different parse paths:
- $oldrecord = GetAuthority($authid)
-> MARC::Record->new_from_xml(...) -> indicators arrive utf8-flagged
- $record (the "after" side) was the in-memory MARC built by the
editor from CGI parameters -> indicators are bare byte strings
The XML round-trip through AddAuthority normalises the stored copy, but
the in-memory $record we passed to logaction had not been re-parsed, so
every default indicator looked like a change to Struct::Diff.
Fix: after AddAuthority commits the new state, re-read it via
$new_authority->record and pass that to _authority_log_data. Both sides
of the diff now traverse the same new_from_xml path, utf8 flags match,
and Struct::Diff falls silent on unchanged indicators.
This is the symmetric counterpart of how biblios already work in
C4::Biblio: ModBiblio sources both $original_marc and $updated_marc_log
through Koha::Biblio::Metadata::record_strip_nonxml (which also calls
new_from_xml), so biblios don't hit this.
The deeper asymmetry between in-memory editor MARC and stored MARC
(default indicators only get normalised on the XML round-trip) is left
for a separate bug; this patch keeps the action log diff clean without
touching the storage path.
Test plan:
1. prove t/db_dependent/AuthoritiesMarc.t
2. Enable AuthoritiesLog. Edit any authority via the staff UI making a
single subfield change. Open Tools -> Log viewer for that authority
and verify the diff column lists only the changed subfield (plus the
005 timestamp), with no rows for ind1/ind2 of untouched fields.
3. Re-run the same edit a second time changing nothing else; the diff
should again be limited to 005 + the touched subfield.
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Mirrors the bug 42188 main commit for biblios. Replace the custom
_marc_record_to_diffable helper with MARC::Record::MiJ's to_mij_structure
so AUTHORITIES action log diffs use the same MARC-in-JSON shape as
CATALOGUING diffs. Add Koha::Authority::_unblessed_for_log to expose
only the auth_header columns that exist independently of the MARC
record (authtypecode, origincode, linkid), avoiding duplication of
MARC-derived data (heading, authtrees), raw marcxml, and system
timestamps. The explicit whitelist also removes the need for the
post-hoc empty-field stripping added in a previous follow-up.
Test plan:
1. prove t/db_dependent/AuthoritiesMarc.t
2. Enable AuthoritiesLog. Create, edit, and delete an authority via
the staff UI, then inspect action_logs.diff in the DB:
- _marc contains MiJ shape { leader: "...", fields: [ { tag: {...} }, ... ] }
- Changes to heading fields produce diffs on
_marc.fields[N].TAG.subfields[0].a
- The diff no longer carries a redundant top-level heading entry
(it's MARC-derived and now only appears in _marc).
3. prove t/db_dependent/Log.t
4. prove t/db_dependent/Koha/Authority.t
5. prove t/db_dependent/Koha/Biblio.t
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
The AUTHORITIES action log entries for authority operations previously
stored minimal data: ADD and DELETE logged just the string "authority",
while MODIFY logged a raw MARC-formatted dump of the pre-change record
as a plain text string ("authority BEFORE=>..."). None of these populated
the diff column.
This patch updates AddAuthority, ModAuthority and DelAuthority in
C4/AuthoritiesMarc.pm to follow the same pattern introduced for biblios
in bug 40135: passing combined hashrefs to logaction so that the diff
column is automatically populated with Struct::Diff JSON.
The combined hashref contains the auth_header table columns (authtypecode,
heading, origincode, etc. — excluding the raw marcxml blob) plus a _marc
key built by the new _marc_record_to_diffable helper, which converts a
MARC::Record into a tag => array-of-formatted-strings structure readable
by the existing diff viewer.
A new _authority_log_data helper packages the authority unblessed data
and MARC record into the combined hashref, keeping the three call sites
concise.
For ADD, the logaction is now emitted after the authority record has been
fully stored (after _after_authority_action_hooks), so the diff captures
the complete saved state. For MODIFY, the before-state is captured from
Koha::Authorities before calling AddAuthority, and the after-state is
read back once AddAuthority has committed the new data. For DELETE, the
MARC is captured before the authority is removed.
The diff column for MODIFY will now show both authority table column
changes (e.g. heading) and MARC field-level changes (e.g. _marc.151.0)
in the existing diff viewer table in the log viewer.
Tests are added to verify that ADD, MODIFY, and DELETE all populate the
diff column and that _marc content is captured correctly.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Test plan:
Read the patch.
Search for both parameters to verify that they are not used.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
UNIMARC instances include the following two authority types
by default (among others): Corporate Body Name (code: 'CO')
and Collective Title (code: 'CO_UNI_TI').
The problem is that when searching for Corporate Body Name
authorities (in the OPAC or in the Staff interface), the
results will include Collective Title authorities as well.
It has to do with Corporate Body Name authorities using
an authtypecode (CO) that happens to be the prefix of the
authtypecode (CO_UNI_TI) used by Collective Titles.
NOTE: Elasticsearch is not affected by this.
The fix is to additionally pass the completeness attribute
to the RPN query sent to Zebra so that it will try to match
on what is literally found in the entire field's phrase
index (i.e. the value of UNIMARC 152$b), instead of performing
a partial match.
More information can be found at the official documentation:
https://software.indexdata.com/zebra/doc/querymodel-rpn.html#querymodel-bib1-completeness
Test plan (tailored for KTD):
0) Launch a UNIMARC Koha instance using Zebra:
ktd --marcflavour unimarc --search-engine zebra up
1) Import the Sample MARC record attached to this Bug,
containing a single Collective Title authority
(adding one manually would take a long time due to
the many mandatory subfields).
2) Perform a generic authority search (without entering
anything in the search box) for all 'Corporate Body
Name' authorities. Note the number of results returned
(83), and also notice that in page 4 of the results you
get a stray 'Collective Title' authority ("Selected
works", the one you imported in Step 1).
3) Apply this patch (and `restart_all`).
4) Repeat the same search as in Step 2. This time, the
number of results returned should be one less than
before (82), and you should only see 'Corporate Body
Name' authorities in the results.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
When merging (or updating) authorities, the biblios using the auths might also be updated. If this happens, Koha used to create one background job for each biblio. If an auth is used in a lot of biblios, this creates a lot of background jobs, each to index one single biblio.
This patch fixes that by collecting the affected biblios and putting all of them into one reindex background job.
Test plan:
* Apply the patch
* run the unit test t/db_dependent/Authority/Merge.t
* the test should now have 16 tests in total, with a new test #15 output like this:
1..3
ok 1 - 3 bibliographic record updated while merging two authorities
ok 2 - we have only one more background job
ok 3 - we have the correct biblio ids in the job to be indexed
ok 15 - merge with ModBiblio only creates one indexer background job for multiple biblios
ok 16 - no warnings
Manual test plan (rather cumbersome):
* Start KTD with ElasticSearch
* Create an PERSON authority (eg "100a Test")
* Create three biblios which use that authority in eg 100
* Access the DB to get the count of background jobs
* `ktd --dbshell`
* `select count(*) from background_jobs;`
* the count should be the same as the number of biblios you created (eg 3). But whatever it is, remeber it
* Edit the authority (change 100 to "Test new")
* check if the change is reflected in the linked biblios (100a = "Test new")
* Check the Background jobs in the DB again:
* `select count(*) from background_jobs;`
* should be three more than previously (eg 6)
* You can use this SQL to verify that the last three jobs each contain one biblionumber (which is what we want to fix):
* `select json_extract(data,'$.record_ids'),enqueued_on from background_jobs order by enqueued_on desc limit 3;`
+-----------------------------------+---------------------+
| json_extract(data,'$.record_ids') | enqueued_on |
+-----------------------------------+---------------------+
| [2299] | 2026-03-25 13:19:34 |
| [2300] | 2026-03-25 13:19:34 |
| [2301] | 2026-03-25 13:19:34 |
+-----------------------------------+---------------------+
* Now apply the patch and run restart_all
* Edit the authority again, setting 100a to "Test new patched".
* The biblios should change accordingly
* Check the Background jobs in the DB again:
* `select count(*) from background_jobs;`
* This should now be only 1 more then the last time (so eg 7)
* Check the details of the background jobs:
* `select json_extract(data,'$.record_ids'),enqueued_on from background_jobs order by enqueued_on desc limit 3;`
* The most recent entry should now contain three items in one row, and the following two should be the same as in the previous query (i.e. the old jobs from the edit before the patch)
+-----------------------------------+---------------------+
| json_extract(data,'$.record_ids') | enqueued_on |
+-----------------------------------+---------------------+
| [2299, 2300, 2301] | 2026-03-25 13:29:17 |
| [2300] | 2026-03-25 13:19:34 |
| [2299] | 2026-03-25 13:19:34 |
+-----------------------------------+---------------------+
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Baptiste Wojtkowski <baptiste.wojtkowski@biblibre.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
To test:
Before applying patch
1. Create an authority record, fill field 400$w with "n"
2. Search for the authority you just created
3. Observe a warning has been logged
"substr outside of string"
4. Apply patch
5. Refresh the search results from step 2
6. Observe no warnings have been logged
Signed-off-by: Harrison Hawkins <harrisonhawkins@catalyst.net.nz>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
So it is done before the other use statements.
Also uniformize the pattern, must now be:
```
use Modern::Perl;
use base 'Exporter';
BEGIN {
our @EXPORT_OK = qw(subroutine list);
}
```
Rebased-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Bug38936 exported redirect_if_opac_suppressed
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
The reference to this bug report may be helpful to understand
later why we did so.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Currently, when removing 040 $f from authority record with 008/11 =
'z' (MARC 21), merge function removes the content of $2 subfield in
linked 6XX fields, leaving the 6XX $2 subfield empty. This is not
correct. We should not have empty subfields in a MARC record.
Instead, the old content of the $2 subfield should be retained (if
auth 008/11 is still = 'z').
Test plan:
==========
1. Have an authority record with 008/11 = 'z' and 040 $f = 'whatever'.
2. Link the record to a bibliographic 6XX.
3. Edit the authority record, removing 040 $f.
4. Go to bibliographic record and in modal "MARC preview" from Normal
view notice that 6XX $2 exists but is empty. This is wrong.
5. Apply the patch ; restart all.
6. Repeat p. 1.-4. Note that the content of 6XX $2 has not been
removed.
Sponsored-by: Ignatianum University in Cracow
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
In order to remove false positives with the QA script we should replace some tabs with spaces.
There are more but I wanted to make sure no regression would be
introduced
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
- Rename LanguageToReportOnMerge system preference to LanguageToUseOnMerge
- Use "use" instead of "report" in system preference description
- Mark this feature as UNIMARC-only.
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Currently, when reporting authority field to the bibliographic records,
the merge function uses the first matching authority field.
However, authorities can provide multiple versions of the same field, with
different languages, identified by a code in the $7 field.
French reference for languages codes for example, but any code system could be
used:
https://documentation.abes.fr/sudoc/formats/unmb/DonneesCodees/CodesZone104.htm#$d
This patch introduces the LanguageToReportOnMerge system preference, which
allows to specify a language code.
When set, if there is an authority field with the given language code, it will
be used to report to the bibliographic record.
If the system preference is not set, or there is no matching field, the default
behavior will apply: the first field will be used.
The $7 in the authority field can be in a short or long form:
- the short form is 2 characters long. Example: ba
- the long form is 8 characters long, with the short form in position 5 and 6
(starting from 1). Example: ba0yba0y
Test plan:
----------
1) Edit the 'Personal Name' authority type framework to make the 100 field
repeatable
2) Add an authority with several languages, like this (the order is important):
100 _7ba0yda0y
_a伊東豊雄
100 _7ba0yba0y
_aItō
3) Add a bibliographic record linked to that authority.
(Click on the "tag editor" icon next to the 100$a field in the bibliographic record to
search for the authority, then Search main heading ($a only): contains Itō)
Note that you can choose which version you want when searching for the
authority, under the "Get it!" column
For this example, choose the latin version: Itō
You should now have this in your bibliographic record:
100 _aItō
_92646
4) Edit the authority, you don't have to change anything, just edit and save.
Note that your bibliographic record now looks like this:
100 _7ba0yda0y
_a伊東豊雄
_92646
The first 100 field of the authority has been reported back to the bibliographic
record, regardless of what you chose when selecting the authority.
5) Apply the patch, set the LanguageToReportOnMerge system preference to: 'ba'
6) Edit the authority again, and check that your bibliographic record now looks
like this, according to the system preference:
100 _7ba0yba0y
_aItō
_92646
7) Edit the authority again, but set the $7 of the latin field to its short
form:
100 _7ba0yda0y
_a伊東豊雄
100 _7ba
_aItō
8) Check that the bibliographic record still has the latin field reported:
100 _7ba
_aItō
_92646
9) prove koha/t/db_dependent/Authority/Merge.t
10) Sign-off :)
Sponsored-by: ENSA
Works smoothly, both with UNIMARC/MARC21
Signed-off-by: Frédéric Demians <f.demians@tamil.fr>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Added head2 right before sub with missing POD
Signed-off-by: Baptiste Wojtkowski <baptiste.wojtkowski@biblibre.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Currently, the function C4::AuthoritiesMarc::merge calls ModBiblio for
all bibliographic records linked to the modified authority record. It
could seem a just behavior but we know that this could lead to
performance problems. This is why AuthorityMergeLimit
systempreference with merge_authorities.pl was introduced (earlier:
dontmerge).
Although this partially solves the performance problems, for
moderately large and large catalogs (500+ K) and authority records
linked with a large number of bibliographic records, performance
problems still occur, even with overnight processing with
merge_authorities.pl. This problem is particularly apparent when
updating authority records from central databases, whose modifications
are beyond our control.
It turns out, however, that many such modifications to authority
records involve fields other than the heading field (1XX in MARC 21),
so these updates in fact do not modify bibliographic records.
However, with the current merge function, every touch on an
authority record forces a call to the ModBiblio function. The
number of relatively expensive ModBiblio calls could be
reduced by checking whether there has been an actual change in
the field associated with the authority record.
However, with IncludeSeeFromInSearches or IncludeSeeAlsoFromInSearches
preferences enabled, the record should be refreshed in the search
engine just in case (should the 4XX, 5XX fields have changed, which we
don't know at this point).
Test plan:
==========
1. Make sure to have CataloguingLog set to Log and
IncludeSeeFromInSearches set to Include.
2. Have a bibliographic record with an authority record linked. Open
the authority record in a separate tab, open it in the editor and
save, without modifying it. Repeat this 2-3 times. In the
bibliographic record tab open the Modification log and see a number of
entries - for each “empty” modification of the linked authority
record.
3. Apply the patch ; restart_all.
4. Repat p. 2. Note that there are no new entries in the log.
5. Edit the authority record again, this time putting in one of the
proper 4XX $a subfield an uncommon word (but do not touch the 1XX
field). Save the authority record. In the bibliographic record
tab control that there are no new entries in the modification log.
Make a search for the word put in 4XX field. You should get the
bibliographic record(s) linked to the edited authority record, even
though the bibliographic record itself has not been modified.
6. Edit the authority record again, this time changing the 1XX field.
Save the authority record. In the bibliographic record tab control
that there are is new entry in the modification log.
Sponsored-by: Ignatianum University in Cracow
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Test plan:
Delete an authority from the interface.
Check presence in the table deletedauth_header, verify that
the correct authid and authtype have been inserted.
Bonus: Check leader position 05 in deleted table.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This commit is generated using:
% perl misc/devel/tidy.pl
*within* ktd, to get the same version of perltidy than what will be used
by our CI (currently v20230309).
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
And copy fix across to Zebra search as well.
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Thomas Klausner <domm@plix.at>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Some file tidying to pass the QA tests
Signed-off-by: Aleisha Amohia <aleishaamohia@hotmail.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Before this patch if a record had a 751 and a 781 you could have fields repeated.
This patch reduces the scope of the fields to subdivision variable as it is only used in processing 7xx
fields and should not be shared between fields.
I also add unit tests
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Aleisha Amohia <aleishaamohia@hotmail.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Currently when searching for a geographic authority record the search will fail if the record has a heading in a 78X field. The system tries to do a regex match against an undefined variable causing an error. This patch makes that regex match conditional on the variable being defined to allow the search to succeed.
Test plan:
1) Navigate to Authorities
2) In the search bar at the top of the page, click on the dropdown options and in the 'Authority type' field, select 'Geographic Name'
3) Click search
4) You should have a list of authorities
5) Click on any authority record and then click edit and select to edit the record
6) Click on the "7" button
7) Click on the green text next to the '781' field to get the list of fields
8) In field 'v' enter any string you like
9) Click save
10) Repeat steps 1-3, this time it should display an error message for an Unmatched [ in regex
11) Apply patch
12) restart_all
13) Refresh the page, the results should show and the string you entered in the 'v' field should display on the record you edited
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Aleisha Amohia <aleishaamohia@hotmail.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This patch updates the test files to correctly create authorities and/or correctly
mock the needed data for authorities.
the tests in t/db_dependent/Authority/Merge.t cover the case of 'Default' authorities
which don't have valid headings - so I had to add handling for blanking the heading field
when Default type is used.
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch adds a new heading field containing the display form of the authority record
NOTE: If trying to save an auhority in the 'DEFAULT' framework, you will get an error, you should not be
using DEFAULT for authorities and we should remove from the list on another bug
To test:
1 - Apply patches
2 - Update database
3 - Restart all
4 - Create a new authority, save.
5 - Do this for various types
6 - View the db records:
SELECT * FROM auth_header\G
7 - Note new heading field is populated correctly
8 - Edit your new authorities
9 - Confirm the heading field is updated correctly
10 - Import some authorities and confirm heading generated correctly
11 - Import auth via Z39.50 and confirm heading generated correctly
Signed-off-by: Andrew Fuerste-Henry <andrewfh@dubcolib.org>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
If autolinking is enabled, there are scenarios where the authorities merge process
updates the record, then sends it to ModBiblio which then AutoLinks the record, undoing
the changes in the record (as the 'from' authority is not removed until the process is complete)
This patch simply disables the autolinking process when merging authorities
To test:
Set system preferences
AutoCreateAuthorities = don't generate
RequireChoosingExistingAuthority = don't require
AutoLinkBiblios = Do
CatalogModuleRelink = Do
LinkerKeepStale = Don't
LinkerModule = first match
LinkerRelink = Do
1. Create an authority record
1.1. Go to Authorities
1.2. Click on "New authority" and choose an authority type (I chose Corporate name)*
1.3. Fill in the mandatory fields (000, 003, 005, 008, 040)
1.4. In tab 1, click on the label of the main heading (110 for Corporate name)
1.5. Fill in subfield
1.6. Copy content of subfield
1.7. Click on "Save"
1.8. Note the authority record number
2. Duplicate the authority record
2.1. Click on "Edit" and choose "Edit as new (duplicate)"
2.2. Click on "Save"
2.3. Click on "No: Save as new authority"
2.4. Note the authority record number (should be the number from step 1.8 +1)
3. Link one of the authority records to a bibliographic record
3.1 In another tab, do a catalog search
3.2. Click on "Edit record" under one of the search results
3.3. Go to the relevant tab (for corporate name, I used field 710 in tab 7)
3.3.a. If there are no empty fields for that tag, click the "Repeat this tag" button (two rectangles)
3.4. Paste the text previously copied in subfield
3.5. Click on "Link authorities automatically" at the top of the page
3.6. Note the authority record number in subfield
--> The linked authority record should be the first authority record (step 1.8)
3.7. Click on "Save"
4. Merge the two authority records, keep the one that is not linked to the bibliographic record
4.1. Go back to the tab with the authorities
4.1. Go to Authorities
4.2. Search for the aurhorities previous created
4.2. Click on "Actions" and choose "Merge" for both records
4.3. Select the record number that is NOT linked to the bibliographic record
4.4. Click on "Next"
4.5. Click on "Merge"
5. Check the bibliographic record
5.1. Go back to the tab with the bibliographic record and refresh the page (Ctrl+F5)
5.2. Hover the mouse cursor over the name or term, without clicking
5.3. Note the record number that appears at the bottom of the screen
--> The record number was not changed
6. Apply the patch
6.1. Run prove t/AuthoritiesMarc_MARC21.t
6.2. Repeat Previous steps 1, 2, 3, 4, 5
--> The record number is change
Signed-off-by: Barbara Petritsch <barbara.petritsch@wienmuseum.at>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Test plan:
prove -v t/db_dependent/Authority/Merge.t
Signed-off-by: David Nind <david@davidnind.com>
Amended-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Replace $biblio->id with $biblio->biblionumber
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
We are calling ->field with an undef parameter.
MARC::Record does not like that and says:
Use of uninitialized value $tag in hash element at /usr/share/perl5/MARC/Record.pm line 202.
For now, a fix at our side.
Test plan:
Run SearchAuthorities without passing an authtypecode.
Without this patch, it results in a warning per hit.
With this patch, the noise should be gone.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This patch replaces the uses of the static method with the new filter.
It also moves the tests to the relevant place and removes the (now)
unused method.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
I have some trouble to get the tidyness test on
AuthoritiesMarc.pm passing. There are several outstanding
patches touching this file, so I didn't dare to perltidy
the whole file just yet.
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Instead of hardcoding the separator we take the value from systempreference.
Also adjust the unit test to expect the "equalterm" array.
Test plan same as before.
Signed-off-by: Christian Nelson <christian.nelson@uwasa.fi>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Currently when a MARC21 authority has 7XX field they do not display on
authority search result page. 7XX is equal term of 1XX based on LC rules.
Applying this patch will enable that feature, which is important for non latin
catalogues.
Test plan:
1) Add a new authority with 1XX and 7XX fields.
2) Try to search that authority and check if 7XX values are displayed
3) Apply this patch
4) Try to search that authority and you will see that 7XX values are displayed
like "1XX = 7XX"
Sponsored-by: Keratsini-Drapetsona Municipal Library, Greece
Mentored-by: Andreas Roussos
Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>
Works for me!
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This enhancement adds a system preference StripWhitespaceChars which,
when enabled, will strip leading and trailing whitespace characters from
all fields when cataloguing both bibliographic records and authority
records. Whitespace characters that will be stripped are:
- spaces
- newlines
- carriage returns
- tabs
To test:
1. Apply patch and install database updates
2. Go to Administration, system preferences, find the new
StripWhitespaceChars preference. It should be "Don't strip" by default.
Change it to "Strip".
3. Search for a biblio record and edit it. Put some leading or trailing
whitespace characters in input fields and textarea fields and save.
4. Confirm these characters are removed when you save the record.
5. Repeat steps 3 and 4 for authority records.
6. Confirm tests pass t/db_dependent/Biblio/ModBiblioMarc.t
Sponsored-by: Educational Services Australia SCIS
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Bug 30358: (follow-up) Also strip inner newlines
This patch amends the StripWhitespaceChars system preference to also
strip inner newlines (line breaks and carriage returns) when enabled.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Bug 30358: (follow-up) Inner newlines should be replaced with a space
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Bug 30358: (follow-up) Fixing tests and including for inner newlines
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Bug 30358: (follow-up) Clarify syspref wording about fields affected
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Bug 30358: (follow-up) Consider field has multiple subfields of same key
To test:
1) Click the clone subfield button to make multiple subfields with the
same key, i.e. 500$a$a$a
2) Save the record and confirm that the fields contain the correct data
after whitespaces are stripped.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Bug 30358: (follow-up) Put multiple subfields fix on auth side
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Bug 30358: (follow-up) stripWhitespaceChars subroutine and tests
To test:
Confirm test plan above still works as expected and tests pass in
t/Koha_MetadataRecord.t
Sponsored-by: Catalyst IT
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Bug 30358: (follow-up) Fixing ModBiblioMarc.t tests
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Bug 30358: (follow-up) Do not strip whitespace from control fields
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Bug 30358: (follow-up) Simplify regex
The regex does the following:
1. Replace newlines and carriage returns with a space
2. Replace leading and trailing whitespace with nothing (strip)
Signed-off-by: Hammat Wele <hammat.wele@inlibro.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
C4::AuthoritiesMarc method GetAuthorizedHeading is not exported thus it is called in other modules :
> git grep GetAuthorizedHeading
C4/AuthoritiesMarc.pm:=head2 GetAuthorizedHeading
C4/AuthoritiesMarc.pm: $heading = &GetAuthorizedHeading({ record => $record, authid => $authid })
C4/AuthoritiesMarc.pm:sub GetAuthorizedHeading {
C4/Breeding.pm: $heading = C4::AuthoritiesMarc::GetAuthorizedHeading({ record => $marcrecord });
C4/ImportBatch.pm: $row->{'authorized_heading'} = C4::AuthoritiesMarc::GetAuthorizedHeading( { authid => $row->{'candidate_match_id'} } );
C4/ImportBatch.pm: my $authorized_heading = C4::AuthoritiesMarc::GetAuthorizedHeading({ record => $marc_record });
This patch adds it to be exported.
For example for use in Koha plugins.
Test plan :
Check import of authorities from a file is OK
Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch joins authority queries together with OR like Elasticsearch.
To the best of the author's knowledge, this code only gets triggered
when using record matching rules, but there may be other situations
where it's used.
Authority search in cataloguing plugins use a different path, and
authority search in authority home doesn't allow multiple queries,
so they'll never trigger this OR joining of queries.
Test plan:
0. Apply patch and koha-plack --restart kohadev
1. Create a record matching rule with the following:
Matching rule code: NID
Description: NormdatenID
Match threshold: 100
Record type: Authority record
Search index: Any
Score: 101
Tag: 035
Subfields: a
Offset: 0
Length: 0
Normalization rule: None
Click "Remove this match check"
2. Stage gnd.xml (from BZ 20596 attachments)
- Choose "MARCXML" for "Format"
- Choose "Authority" for "Record type"
- Choose "NID" for "Record matching rule"
- Click "Stage for import"
3. Note the job gets to "100% Finished"
(Before this patch, it would say "0% Failed")
4. View the batch
5. Import the record
6. Repeat steps 2-4 and note that the record matching rule
worked
7. Go to http://localhost:8081/cgi-bin/koha/authorities/authorities-home.pl
8. Search for "e"
9. Note that the search works
10. Go to http://localhost:8081/cgi-bin/koha/cataloguing/addbiblio.pl?frameworkcode=
11. Click tab 6
12. Click on the tag editor next to "a Personal Name"
13. Type "e" into each box and click "Search"
14. Note that the search works
Signed-off-by: Jan Kissig <jan.kissig@th-wildau.de>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
The test
$MARCfrom->field('1..', '2..'))[0]->tag ne ($MARCto->field('1..', '2..'))[0]->tag
is not completely consistent with following code in sub merge.
I decided to get the authtype code from the old record that comes
from Koha and should include the type.
Remaining changes refer to indentation/comments.
Test plan:
Run t/db_dependent/Authority/Merge.t
Note: The test actually could be extended a bit for mocking MARC
flavor, but needs some additional framework support to work.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
In those rare cases when authorities are updated by an external agency (or
even internally, by reviewing and correcting an exported authority file)
when the heading tag will be changed (seems odd but happens:
111 Congress ==> 110 Corporate body.Congress ;
100 Person ==> 110 Corporate body (a company named with person's name ;
151 City--object ==> 150 Object (city) etc.)
and then the authority record in Koha database will be updated with
bulkmarcimport or by calling directly ModAuthority from a custom script,
the merge function "doesn't know" that the change to the authority type
has been made and, consequently, doesn't adequately change the tag in
related fields in biblio records (as it would if two different records
with different authtypecode were merged with Koha interface).
This is because at the moment when merge function is being called
by ModAuthority:
Koha::Authority::Types->find($autfrom->authtypecode)
Koha::Authority::Types->find($authto->authtypecode)
both have the same value (because $mergefrom == $mergeto).
Therefore in case when $mergefrom == $mergeto and the heading tag changes,
$authtypefrom and $authfrom calculated in merge in the ordinar way are
misleading and should not be taken unto consideration.
Test plan:
==========
1. run t/db_dependent/Authority/Merge.t
2. you should see problems in "Test update A with modified heading tag
(changing authtype)"
3. apply the patch
4. run t/db_dependent/Authority/Merge.t again
5. the test should pass
Alternatively:
1. have an authority record used in biblio;
export it to file;
change 1XX heading tag to a different (but reasonable) value
and possibly change also the content of the heading
(one can delete also 942 but it doesn't matter);
make bulkmarcimport.pl -a -update -file <modified_auth_file> and
see that the tag in biblio record has not been changed (whereas
the type of authority record did change);
2. make orders in database (so that the authority type and the tag of
the field in biblio record correspond); apply the patch;
3. repeat the test from 1.
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
This patch replaces the use of $dbh->do by corresponding use of
Koha::Authority object
Test plan:
1. Verify that creating/modifying authority still work
2. prove t/db_dependent/AuthoritiesMarc.t
Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Martin Renvoize
Paul Derscheid
Marcel de Rooy
Lisette Scheer
Andreas Roussos
Thomas Klausner
Lari Taskula
Jonathan Druart
Tomas Cohen Arazi
Janusz Kaczmarek
Matthias Meusburger
David Cook
Koha Development Team
Aleisha Amohia
Matt Blenkinsop
Nick Clemens
Kyle M Hall
Katrin Fischer
George Veranis
Fridolin Somers
Julian Maurice