gitea
/
Koha-clms
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Bug 38913: (bug 38416 follow-up) Elasticsearch indexing explodes with oversized records 

After Bug 38416 Elasticsearch indexing explodes with oversized
records, especially with UTF encoded data.

In Koha::SearchEngine::Elasticsearch::marc_records_to_documents a
following snippet has been introduced:

my $usmarc_record = $record->as_usmarc();
my $decoded_usmarc_record = MARC::Record->new_from_usmarc($usmarc_record);

But if $record is oversized (> 99999 bytes), it is OK for MARC::Record
object, but not for $record->as_usmarc. The produced ISO 2709 string
is not correct and hence cannot be properly converted back to
MARC::Record object by new_from_usmarc.

The result in this case can be like:

UTF-8 "\x85" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35.

Since it is done without any eval / try, the whole reindex procedure
(for instance rebuild_elasticsearch.pl) is being randomly interrupted
with no explanation.

Test plan:
==========
Hard to reproduce. But the explanation together with discussion in Bug
38416 (from 2024-12-15) explains and justifies the need of this added
eval.

1. Have a standard KTD installation with Elasticsearch.
2. Use the provided test record - add it to Koha with
   ./misc/migration_tools/bulkmarcimport.pl -b -file test.xml -m=MARCXML
   (have patience).
   During load process you should see a message like:
   UTF-8 "\xC4" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35.
3. The record should get biblionumber 439. Check in librarian interface with
   http://<your_addreess>:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=439
   that the record has been imported.
   However, you should not be able to make a search for this record.
4. Try to reindex with:
   ./misc/search_tools/rebuild_elasticsearch.pl -b -bn 439
   You should get a message like:
   UTF-8 "\xC4" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35.
   Again, no search results.
5. Apply the patch ; restart_all.
6. Repeat reindex with:
   ./misc/search_tools/rebuild_elasticsearch.pl -b -bn 439
   There should be no warning now and you should be able to find the record.

Signed-off-by: Magnus Enger <magnus@libriotech.no>
Followed the test plan. Works as advertised.
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
(cherry picked from commit c7127fabf4)
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Alex Buckley <alexbuckley@catalyst.net.nz>
This commit is contained in:
Janusz Kaczmarek 2025-01-16 21:04:28 +00:00 committed by Alex Buckley
parent 4214c68f17
commit 041e4368f9
1 changed files with 3 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
Koha/SearchEngine/Elasticsearch.pm
View File

@ -793,8 +793,9 @@ sub marc_records_to_documents {
my $usmarc_record = $record->as_usmarc();
#NOTE: Try to round-trip the record to prove it will work for retrieval after searching
my $decoded_usmarc_record = MARC::Record->new_from_usmarc($usmarc_record);
if ( $decoded_usmarc_record->warnings() ) {
my $decoded_usmarc_record;
eval { $decoded_usmarc_record = MARC::Record->new_from_usmarc($usmarc_record); };
if ( $@ || $decoded_usmarc_record->warnings() ) {
#NOTE: We override the warnings since they're many and misleading
@warnings = (