gitea
/
Koha-clms
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Bug 39694: Make `es_indexer_daemon.pl` use batch_size to limit DB poll size 

This patch makes the indexer poll the DB in configurable batches. Pretty
much like Rabbit does.

The only change is that it will now pass a LIMIT <batch_size> to the
query.

To test:
0. Have KTD started with ES (skip if you have an instance running):
   $ ktd --es7 up -d
1. Stop the indexer and rabbit:
   $ ktd --shell
  k$ sudo koha-es-indexer --stop kohadev
  k$ sudo service rabbitmq-server stop
2. On a separate terminal, watch the queue:
  k$ watch -n 0.5 'echo "SELECT COUNT(*) FROM background_jobs WHERE status=\"new\"" | koha-mysql kohadev'
=> SUCCESS: It should be empty
3. Add a bunch of things to index (this works with default KTD data):
  k$ for i in $(seq 1 438; seq 1 438) ; do echo "INSERT INTO background_jobs (status,queue,data) VALUES ('new','elastic_index','{\"record_server\":\"biblioserver\",\"record_ids\":[$i]}')" | koha-mysql kohadev ; done
4. Start the ES indexer:
  k$ sudo koha-es-indexer --start kohadev
5. Look at the queue size
=> FAIL: It shrinks too fast [1]
6. Apply this patch
7. Restart the indexer
  k$ perl misc/workers/es_indexer_daemon.pl &
8. Repeat 3
9. Look at the queue size:
=> SUCCESS: It shrinks 'slowly'
10. Sign off :-D

[1] This is a problem in certain scenarios, not in all of them. This is
the best I come up with in order to test the behavior change. Anyone can
set a new size of their taste.

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
This commit is contained in:
Tomas Cohen Arazi 2025-04-21 15:16:32 -03:00 committed by Katrin Fischer
parent 5937dd436c
commit 20b600d450
No known key found for this signature in database
GPG Key ID: 0EF6E2C03357A834
1 changed files with 6 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
misc/workers/es_indexer_daemon.pl
View File

@ -64,7 +64,8 @@ use Koha::BackgroundJobs;
use Koha::SearchEngine;
use Koha::SearchEngine::Indexer;
my ( $help, $batch_size );
my $help;
my $batch_size = 10;
my $not_found_retries = {};
my $max_retries = $ENV{MAX_RETRIES} || 10;
@ -76,8 +77,6 @@ GetOptions(
pod2usage(0) if $help;
$batch_size //= 10;
warn "Not using Elasticsearch" unless C4::Context->preference('SearchEngine') eq 'Elasticsearch';
my $logger = Koha::Logger->get( { interface => 'worker' } );
@ -172,7 +171,10 @@ while (1) {
}
} else {
@jobs = Koha::BackgroundJobs->search( { status => 'new', queue => 'elastic_index' } )->as_list;
@jobs = Koha::BackgroundJobs->search(
{ status => 'new', queue => 'elastic_index' },
{ rows => $batch_size }
)->as_list;
commit(@jobs);
@jobs = ();
sleep 10;