041e4368f9
After Bug 38416 Elasticsearch indexing explodes with oversized
records, especially with UTF encoded data.
In Koha::SearchEngine::Elasticsearch::marc_records_to_documents a
following snippet has been introduced:
my $usmarc_record = $record->as_usmarc();
my $decoded_usmarc_record = MARC::Record->new_from_usmarc($usmarc_record);
But if $record is oversized (> 99999 bytes), it is OK for MARC::Record
object, but not for $record->as_usmarc. The produced ISO 2709 string
is not correct and hence cannot be properly converted back to
MARC::Record object by new_from_usmarc.
The result in this case can be like:
UTF-8 "\x85" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35.
Since it is done without any eval / try, the whole reindex procedure
(for instance rebuild_elasticsearch.pl) is being randomly interrupted
with no explanation.
Test plan:
==========
Hard to reproduce. But the explanation together with discussion in Bug
38416 (from 2024-12-15) explains and justifies the need of this added
eval.
1. Have a standard KTD installation with Elasticsearch.
2. Use the provided test record - add it to Koha with
./misc/migration_tools/bulkmarcimport.pl -b -file test.xml -m=MARCXML
(have patience).
During load process you should see a message like:
UTF-8 "\xC4" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35.
3. The record should get biblionumber 439. Check in librarian interface with
http://<your_addreess>:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=439
that the record has been imported.
However, you should not be able to make a search for this record.
4. Try to reindex with:
./misc/search_tools/rebuild_elasticsearch.pl -b -bn 439
You should get a message like:
UTF-8 "\xC4" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35.
Again, no search results.
5. Apply the patch ; restart_all.
6. Repeat reindex with:
./misc/search_tools/rebuild_elasticsearch.pl -b -bn 439
There should be no warning now and you should be able to find the record.
Signed-off-by: Magnus Enger <magnus@libriotech.no>
Followed the test plan. Works as advertised.
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
(cherry picked from commit
|
||
|---|---|---|
| C4 | ||
| Koha | ||
| acqui | ||
| admin | ||
| api | ||
| authorities | ||
| basket | ||
| bin | ||
| bookings | ||
| catalogue | ||
| cataloguing | ||
| circ | ||
| clubs | ||
| course_reserves | ||
| debian | ||
| docs | ||
| erm | ||
| errors | ||
| etc | ||
| ill | ||
| installer | ||
| koha-tmpl | ||
| labels | ||
| lib | ||
| members | ||
| misc | ||
| offline_circ | ||
| opac | ||
| patron_lists | ||
| patroncards | ||
| plugins | ||
| pos | ||
| preservation | ||
| recalls | ||
| reports | ||
| reserve | ||
| reviews | ||
| rotating_collections | ||
| serials | ||
| services | ||
| skel | ||
| suggestion | ||
| svc | ||
| t | ||
| tags | ||
| tools | ||
| virtualshelves | ||
| xt | ||
| .editorconfig | ||
| .eslintrc.json | ||
| .gitignore | ||
| .htaccess | ||
| .mailmap | ||
| .perlcriticrc | ||
| .perltidyrc | ||
| .prettierrc.js | ||
| .proverc.dist | ||
| .stylelintrc.json | ||
| INSTALL | ||
| Koha.pm | ||
| LICENSE | ||
| MANIFEST.SKIP | ||
| Makefile.PL | ||
| README | ||
| README.md | ||
| README.robots | ||
| about.pl | ||
| app.psgi | ||
| build-resources.PL | ||
| changelanguage.pl | ||
| cpanfile | ||
| cypress.config.ts | ||
| fix-perl-path.PL | ||
| gulpfile.js | ||
| help.pl | ||
| kohaversion.pl | ||
| mainpage.pl | ||
| package.json | ||
| rewrite-config.PL | ||
| rspack.config.js | ||
| tsconfig.json | ||
| webpack.config.js | ||
| yarn.lock | ||
README.md
Koha is a free software integrated library system (ILS).
Koha is distributed under the GNU GPL version 3 or later.
Note: Koha does not accept pull requests from git hosting sites.
Note: This project has its own bug tracker, to report a bug or submit a patch visit http://bugs.koha-community.org.
For guidelines on submitting patches for Koha please visit https://wiki.koha-community.org/wiki/SubmitingAPatch
The developers handbook can be found at https://wiki.koha-community.org/wiki/Developer_handbook
