gitea
/
Koha-clms
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Bug 42485: Disable date and numeric detection on ES mappings 

When ElasticsearchMARCFormat is set to ARRAY, the marc_data_array
field uses dynamic:true mapping. Elasticsearch auto-detects field
types based on the first document indexed. If a MARC subfield value
looks like a date (e.g. 2024-01-15 in 952$e), ES locks that
subfield to type date. Subsequent records with non-date values in
the same subfield fail to index with a document_parsing_exception.

This patch sets date_detection and numeric_detection to false at
the mapping root level, so ES treats all dynamically mapped fields
as text instead of guessing types.

Test plan:
1. $ ktd --search-engine es8 --name test-es --proxy up -d
   $ ktd --name test-es --wait-ready 180
   $ ktd --name test-es --shell
2. k$ koha-mysql kohadev -e "UPDATE systempreferences
     SET value='ARRAY'
     WHERE variable='ElasticsearchMARCFormat';"
   k$ echo 'flush_all' | nc -q1 memcached 11211
   k$ koha-mysql kohadev -e "DELETE FROM biblio_metadata;
     DELETE FROM items; DELETE FROM biblioitems;
     DELETE FROM biblio;"
   k$ rebuild_elasticsearch.pl -d -r --biblios
3. Add a record with date-like 952$e and index it:
   k$ koha-shell kohadev -p -c 'perl -MC4::Biblio
     -MMARC::Record -MMARC::Field -MKoha::Item -e "
     my \$r = MARC::Record->new();
     \$r->append_fields(
       MARC::Field->new(q{245},q{1},q{0},a=>q{Record one}),
       MARC::Field->new(q{942},q{ },q{ },c=>q{BK}));
     my (\$bn) = C4::Biblio::AddBiblio(\$r, q{});
     Koha::Item->new({biblionumber=>\$bn,
       biblioitemnumber=>\$bn, homebranch=>q{CPL},
       holdingbranch=>q{CPL}, itype=>q{BK},
       booksellerid=>q{2024-01-15}})->store;
     print qq{Added biblio \$bn\n};"'
   k$ rebuild_elasticsearch.pl --biblios -bn <biblionumber>
4. Check the mapping for 952$e:
   k$ curl -s es:9200/koha_kohadev_biblios/_mapping/field/\
     marc_data_array.fields.952.subfields.e
=> FAIL: type is "date" (should be "text")
5. Add a record with non-date 952$e and index it:
   k$ koha-shell kohadev -p -c 'perl -MC4::Biblio
     -MMARC::Record -MMARC::Field -MKoha::Item -e "
     my \$r = MARC::Record->new();
     \$r->append_fields(
       MARC::Field->new(q{245},q{1},q{0},a=>q{Record two}),
       MARC::Field->new(q{942},q{ },q{ },c=>q{BK}));
     my (\$bn) = C4::Biblio::AddBiblio(\$r, q{});
     Koha::Item->new({biblionumber=>\$bn,
       biblioitemnumber=>\$bn, homebranch=>q{CPL},
       holdingbranch=>q{CPL}, itype=>q{BK},
       booksellerid=>q{My Vendor Name}})->store;
     print qq{Added biblio \$bn\n};"'
   k$ rebuild_elasticsearch.pl --biblios -bn <biblionumber> -v -v -v
=> FAIL: document_parsing_exception - failed to parse date field
   [My Vendor Name]
6. Apply this patch, repeat from step 2
7. Repeat step 4:
=> SUCCESS: type is "text"
8. Repeat step 5:
=> SUCCESS: No indexing errors
9. Sign off :-D

Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This commit is contained in:
Tomás Cohen Arazi 2026-04-28 09:05:47 -03:00 committed by Lucas Gass
parent 0a44bf4e8e
commit 78850b4a15
No known key found for this signature in database
GPG Key ID: 60B447398B0D36CA
1 changed files with 9 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

9
Koha/SearchEngine/Elasticsearch.pm
View File

@ -272,6 +272,15 @@ sub get_elasticsearch_mappings {
$mappings->{properties}{'match-heading'} = _get_elasticsearch_field_config( 'search', 'text' );
$mappings->{properties}{'subject-heading-thesaurus'} = _get_elasticsearch_field_config( 'search', 'text' );
}
# Disable date and numeric detection to prevent Elasticsearch from
# auto-detecting field types in marc_data_array. Without this, a
# subfield value that looks like a date (e.g. "2024-01-15") causes
# ES to lock that subfield to type "date", and subsequent non-date
# values fail to index. (Bug 42485)
$mappings->{date_detection} = JSON::PP::false;
$mappings->{numeric_detection} = JSON::PP::false;
$all_mappings{ $self->index } = $mappings;
}
$self->sort_fields( \%{ $sort_fields{ $self->index } } );