This patch makes the usage POD for the EmbedItems filter actually be
usable. The filter is used in opac-ISBDdetail.pl and the POD is updated
to how it is used there.
Not much to test.
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Fridolin Somers <fridolin.somers@biblibre.com>
Test plan:
1. Create an authority with at least a 1XX$a and a 4XX$a, for instance:
100 $a Foo
400 $a Bar
2. Create a biblio and add a link to this authority using the
cataloguing plugin
3. Disable syspref IncludeSeeFromInSearches
4. Reindex the biblio. You should be able to find it when searching
'Foo' but not when searching 'Bar'
5. Enable syspref IncludeSeeFromInSearches
6. Reindex the biblio. You should be able to find it when searching
'Foo' and also when searching 'Bar'
without the patch, 'Bar' doesn't yeld results. With it, it does.
7. prove t/db_dependent/Koha/SearchEngine/Elasticsearch.t
Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Lucy Vaux-Harvey <lucy.vaux-harvey@ptfs-europe.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
On bug 17591 we discovered that there was something weird going on with
the way we export and use subroutines/modules.
This patch tries to standardize our EXPORT to use EXPORT_OK only.
That way we will need to explicitely define the subroutine we want to
use from a module.
This patch is a squashed version of:
Bug 17600: After export.pl
Bug 17600: After perlimport
Bug 17600: Manual changes
Bug 17600: Other manual changes after second perlimports run
Bug 17600: Fix tests
And a lot of other manual changes.
export.pl is a dirty script that can be found on bug 17600.
"perlimport" is:
git clone https://github.com/oalders/App-perlimports.git
cd App-perlimports/
cpanm --installdeps .
export PERL5LIB="$PERL5LIB:/kohadevbox/koha/App-perlimports/lib"
find . \( -name "*.pl" -o -name "*.pm" \) -exec perl App-perlimports/script/perlimports --inplace-edit --no-preserve-unused --filename {} \;
The ideas of this patch are to:
* use EXPORT_OK instead of EXPORT
* perltidy the EXPORT_OK list
* remove '&' before the subroutine names
* remove some uneeded use statements
* explicitely import the subroutines we need within the controllers or
modules
Note that the private subroutines (starting with _) should not be
exported (and not used from outside of the module except from tests).
EXPORT vs EXPORT_OK (from
https://www.thegeekstuff.com/2010/06/perl-exporter-examples/)
"""
Export allows to export the functions and variables of modules to user’s namespace using the standard import method. This way, we don’t need to create the objects for the modules to access it’s members.
@EXPORT and @EXPORT_OK are the two main variables used during export operation.
@EXPORT contains list of symbols (subroutines and variables) of the module to be exported into the caller namespace.
@EXPORT_OK does export of symbols on demand basis.
"""
If this patch caused a conflict with a patch you wrote prior to its
push:
* Make sure you are not reintroducing a "use" statement that has been
removed
* "$subroutine" is not exported by the C4::$MODULE module
means that you need to add the subroutine to the @EXPORT_OK list
* Bareword "$subroutine" not allowed while "strict subs"
means that you didn't imported the subroutine from the module:
- use $MODULE qw( $subroutine list );
You can also use the fully qualified namespace: C4::$MODULE::$subroutine
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch makes all places in Koha that rely on OpacHiddenItems
actually use C4::Context->yaml_preference instead of manually calling
the YAML libraries and handling it.
To test:
1. Apply this patch
2. Run:
$ kshell
k$ prove t/db_dependent/Items* \
t/db_dependent/Koha/Item* \
t/db_dependent/Koha/Filter/EmbedItems.t
=> SUCCESS: Tests pass!
3. Try hiding things with opac-ISBDdetail.pl and opac-search.pl
=> SUCCESS: Things work the same!
4. Sign off :-D
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch adds "1;" to the end of the following perl files:
Koha/Filter/MARC/EmbedItemsAvailability.pm
Koha/Filter/MARC/EmbedSeeFromHeadings.pm
Koha/Filter/MARC/Null.pm
Koha/Item/Search/Field.pm
Koha/SearchEngine.pm
misc/translator/VerboseWarnings.pm
t/lib/Koha/Plugin/Test.pm
This indicates the succesful execution of the initialization code.
Test plan:
Ensure that there are no other perl files that need "1;", but dont have
it.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
There is a difference between YAML::Load and YAML::XS::Load
From YAML::XS pod:
"YAML::XS only deals with streams of utf8 octets"
Test plan:
We are going to test 1 occurence and QA will confirm others don't
contain typos.
0. Don't apply the patches
1. Create a new itemtype with code=❤️
2. Create a new item using this itemtype (to biblionumber=1 will work)
3. Fill OpacHiddenItems with
itype: [❤️]
4. Search for "street shuffle" or any terms that will return the biblio
Notice that the item is there (there is an error in logs)
5. Apply the patches
6. Repeat 4 and confirm that the item is now hidden
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
From tht YAML pod:
"""
This module has been released to CPAN as YAML::Old, and soon YAML.pm will be changed to just be a frontend interface module for all the various Perl YAML implementation modules, including YAML::Old.
If you want robust and fast YAML processing using the normal Dump/Load API, please consider switching to YAML::XS. It is by far the best Perl module for YAML at this time. It requires that you have a C compiler, since it is written in C.
"""
See also
https://gitlab.com/koha-community/qa-test-tools/-/merge_requests/35
Test plan:
Try some place where YAML::XS is not used and confirm that it works
correctly
QA note: This patch removes some uses of YAML that were not useful
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This takes care of more occurences of staff client and changes it to
staff interface, including in code comments.
To test:
- I think in this case careful code review is what we look for.
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch makes the ViewPolicy filter use the 'params' accessor instead
of relying of ->{options} which has no accessor. This will allow
interacting with the filter object be similar through all the filters in
the chain.
To test, we just need to verify no behaviour change takes place:
1. Run:
$ kshell
k$ prove t/db_dependent/Filter_MARC_ViewPolicy.t
=> SUCCESS: Tests pass
2. Apply this patch
3. Repeat 1
=> SUCCESS: Tests still pass!
4. Sign off :-D
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
MARC::Record::append_fields takes a list of MARC::Field (not an arrayref)
Use $record->subfield() instead of $record->field()->subfield() to avoid errors
when field doesn't exist
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
This patch makes the EmbedItemsAvailability filter handle the biblio.biblionumber > control field mapping gracefully.
Right now, it was assuming a regular data field was used, and such is not the case for UNIMARC (001).
To test:
- Apply the tests patch
- Run:
$ sudo koha-shell kohadev
k$ cd kohaclone
k$ prove t/db_dependent/Koha/Filter/EmbedItemsAvailability.t
=> FAIL: Tests fail to run
- Apply this patch
- Run:
k$ prove t/db_dependent/Koha/Filter/EmbedItemsAvailability.t
=> SUCCESS: Tests pass!
- Sign off :-D
Sponsored-by: ByWater Solutions
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Removes the loop by moving the onloan test to the query.
Test plan:
Run the test again.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
This patch adds a new filter for MARC records to be used with
Koha::RecordProcessor. It's purpose is to inject the information about
items not-onloan in a fixed subfield, 999$x.
To test:
- Apply the patch
- Run:
$ prove t/db_dependent/Koha/Filter/EmbedItemsAvailability.t
=> SUCCESS: Tests pass!
- Sign off :-D
Sponsored-by: ByWater Solutions
Followed test plan, test passes OK
Signed-off-by: Marc Véron <veron@veron.ch>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
GetMarcStructure() currently uses Koha::Cache in the "safe" mode
(returning deep copy of the result data structure by default), which
causes numerous performance issues in many Koha scripts. Switching
it to the "unsafe" mode globally (2nd patch from Bug 16140) resolves
those issues, but ensuring that it is regression-free (and that it
will stay that way in the future) is far from easy. This patch
proposes a bit more manageable solution, it introduces
a possibility to use "unsafe" variant selectively (only in those
places in the code where GetMarcStructure() is called repetitively).
That way, amount of the code that needs to be audited for possible
problems gets vastly reduced, without any performance trade-offs.
Test plan:
1) Have a look at the code and audit the parts affected by this patch
for possible regressions
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Amended the POD of GetMarcStructure, removing a TODO.
NOTE: GetAuthorisedValueDesc, as called in C4::XSLT::transformMARCXML4XSLT
and by GetISBDView, GetMarcAuthors in C4::Biblio, may autovivify some hash
entries in tagslib.
Same for Koha/Filter/MARC/ViewPolicy.pm, sub filter.
No reason however to worry; our use of this structure in Koha does not
depend on the existence of intermediate hash keys. (We seem to be safe as
long as $tagslib->{$tag}->{$subfield}->{tab} and/or hidden are not filled.)
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Add should_hide_marc to ViewPolicy filter
Add should_hide_marc tests to t/db_dependent/Filter_MARC_ViewPolicy.t
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
This patch:
- makes the Koha::RecordProcessor code more clear
by removing the unnecessary newrecord variable.
- revises the filter to be more clear about the
expectation that operations are done directly on
the record parameter.
TEST PLAN
---------
prove t/RecordProcessor.t
prove t/db_dependent/Filter_MARC_ViewPolicy.t
run koha qa test tools
Signed-off-by: Hector Castro <hector.hecaxmmx@gmail.com>
Works as advertised. NO koha-qa errors
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
This patch:
- improves perlcritic messages in the filter and tests.
- changes should display logic to should hide logic
to simplify filter.
- perltidies the scripts
- debugs the issues outstanding on the comprehensive tests
provided in the second commit.
Signed-off-by: Hector Castro <hector.hecaxmmx@gmail.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
TEST PLAN
---------
0) Apply 15777 and 15871 as required.
1) Apply patch
2) prove -v t/db_dependpent/RecordProcessor_ViewPolicy.t
-- all should pass.
3) koha qa test tools.
BONUS
4) perlcritic -1 t/db_dependpent/RecordProcessor_ViewPolicy.t
-- using the koha qa test tools perlcriticrc, it should pass
though I am excluding checking print return values.
(see bug 15646)
Signed-off-by: Hector Castro <hector.hecaxmmx@gmail.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
Mainly a
perl -p -i -e 's/^.*3.07.00.049.*\n//' **/*.pm
Then some adjustements
Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>
Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
The existing Koha::Authority does not use Koha::Objects and does not
exclusively deal with DB objects.
It makes much sense to move it to let the space free for a
Koha::Authority and Koha::Authorities modules based on Koha::Object.
Signed-off-by: Marc Véron <veron@veron.ch>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Brendan Gallagher brendan@bywatersolutions.com
If the system preference IncludeSeeFromInSearches is enabled, records
exported for zebra indexing are being additionally processed by
EmbedSeeFromHeadings record filter (right now used only in rebuild_zebra.pl
script). This filter embeds 'see from' fields (extracted from authority
records linked with the given biblio via $9 subfields) into target MARC
record, which is then subsequently indexed in zebra.
Currently all fields containing $9 are getting the same exact treatment
by this filter. But on the export stage when the filter is applied, MARC
record being processed already does have holdings data fields added in
the previous stage (usually 952 / 995, depending on the MARC format).
Problem is that holdings data fields use to have $9 subfields in them
as well (mapped to item.itemnumber by default). As a consequence, some
(great many in the typical setup) records exported for zebra indexing
may have surplus "see from" fields added erroneously in semi-random
fashion, so biblio searches would often return some completely
unexpected additional results.
EmbedSeeFromHeadings record filter should not process holdings fields
when dealing with MARC records intended for zebra indexing.
To reproduce:
1) database with as many sample or real-world biblio, item and authority
records as possible is recommended for testing purposes
2) enable IncludeSeeFromInSearches
3) export a bunch of biblio records for zebra (e.g.:
misc/migration_tools/rebuild_zebra.pl -I -b -x -k -length=1000),
inspect the result xml records in /tmp/<whatever> file; observe that at
the end of many records, here and there some extra "see from" (= 1st
indicator: 'z') fields tend to appear, which shouldn't be there ;)
To test:
4) apply patch
5) redo 3)
6) compare results from 3) and 5) with diff
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
I introduced a regression test for this. You should run the tests
without/with the patch and verify that the patch actually fixes the problem.
Good job Jacek! I'm sure writing the regression test would take less time
than such a detailed commit message!
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Chris Nighswonger <cnighswonger@foundations.edu>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
http://bugs.koha-community.org/show_bug.cgi?id=9987
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
In Germany 020$9 is used to store the ISBN with hyphens. In rare cases this
can break indexing in combination with IncludeSeeFromInSearches.
To reproduce:
1) Change your frameworks:
* Make sure 084 $a is visible
* Create a 084 $9 that you link to PERSO_NAME authorities.
2) Create a record or edit an existing one:
* Use the plugin on 084 to link the record to an authority.
* Make sure 400 is filled in your authority record.
Preparations all done, now test indexing:
3) Make sure IncludeSeeFromInSearches is OFF
* Reindexing should work without any problems.
4) Activate IncludeSeeFromInSearches
* Reindexing should throw an error and stop:
201.Subfields are only for data fields (generally,
just tags >= 010 at /home/koha/Koha/Filter/MARC/EmbedSee
FromHeadings.pm line 94
5) Apply patch and reindex.
* No error should occur.
* Test 4xx fields from other tags were properly indexed.
What causes the problem is the fact, that Koha tries to store
the date from the 400 in the bibliographic record for indexing.
For this it calculates a tag in the bibliographic record from the
tag it found the authority link in and the 4xx tags used in the
authority record. If you have an authority linked to a 0xx field
and the matching authority has a 4xx where xx < 10, the calculated
tag will be a control field. The attempt to add subfields to this
control field, is causing the error and breaks the indexing process.
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Comment: Work as described, and is a long description :)
Test:
1) Enabled 084 (all subfields visible), added 084_9 (missing from
default framework) linked to PERSO_NAME
2) Edited authority record and added some data to 400
3) Edited bib record, linked 084 to previous auth record
4) With IncludeSeeFromInSearches OFF, re-index without problem
5) With IncludeSeeFromInSearches ON, re-index problems!!!
====================
exporting biblio
====================
1...Subfields are only for data fields (generally, just tags >= 010) at...
Abort
6) Applied patch
7) With IncludeSeeFromInSearches ON, re-index without problem
...
====================
5001....................................................................................................
Records exported: 5060
====================
...
====================
2001....................................................................................................
Records exported: 2024
====================
CLEANING
====================
Good!
*) That's "works as described", don't know if other tests are needed.
No errors from koha-qa
Signed-off-by: Elliott Davis <elliott@bywatersolions.com>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
This patch adds the Koha::Indexer::RecordNormalizer and
Koha::Indexer::MARC::RecordNormalizer::EmbedSeeFromHeadings packages
to enable the inclusion of alternate forms of headings in bibliographic
searches. When the new syspref IncludeSeeFromInSearches is turned on
(default is off) rebuild_zebra.pl will insert see from headings from
authority records into bibliographic records when indexing, so that a
search on an obsolete term will turn up relevant records.
To test:
1) Enable IncludeSeeFromInSearches
2) Add a heading that has an alternate form to a record (for example,
"Cooking" has the alternate form "Cookery," if you have authority
records from LC)
3) Index the zebraqueue (or reindex if you haven't indexed your system
yet)
4) Confirm that if you search for "Cookery" you get the record you
just modified
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Rebased on master 5 August 2012
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Rebased on master 11 September 2012
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Also checked:
- Verified database update works correctly
- Checked system preference and its description
- Checked staff/opac detail pages with feature on/off
- Checked staff/opac search facets
- Downloaded and tested records in various formats
- Tried different searches for 'see from' entries of authorities
- Ran all unit tests
No problems found.
Tomas Cohen Arazi
Nick Clemens
Julian Maurice
Jonathan Druart
James O'Keeffe
Katrin Fischer
Josef Moravec
Marcel de Rooy
Jacek Ablewicz
Mark Tompsett
Jonathan Druart
Katrin Fischer
Jared Camins-Esakov