If Plack is running with several workers we must not use Cache::Memory
as L2 cache.
If a value is set from a worker, it will not be available from other
workers as the Cache::Memory instance is not shared (of course!)
Moreover we now have Koha::Cache::Memory::Lite that does the same job,
so we should not expect performance regressions by removing it.
See also the email sent to koha-devel for more info
http://lists.koha-community.org/pipermail/koha-devel/2018-December/045004.html
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
(cherry picked from commit 4bf033c6d6)
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
If memcached or the connection to it is misconfigured, show simple warnings to help
identify the problem.
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
This patch does the following:
[1] Move some POD lines from Cache to Caches.
[2] Correct C4::Plugins to Koha::Plugins in POD line of Koha::Plugins
[3] POD Koha/AuthorisedValue.pm: lib_opac moved to opac_description
[4] The POD in Koha/Patron.pm uses head2 and head3 inconsistently.
Ran s/^=head2/=head3/ on those lines (7 substitutions on 7 lines)
[5] Correct a copied POD line from reports/issues_stats.pl in
reports/reserve_stats.pl.
[6] Correct a test description in t/db_dependent/Koha/Authorities.t.
You should never delete the library :)
[7] Correct typo shouild in a comment of rebuild_zebra.pl
Test plan:
[1] Read the patch. Does it make sense?
[2] Run perldoc Koha/Cache.pm and Koha/Caches.pm
[3] Run t/db_dependent/Koha/Authorities.t
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
From the POD of Koha::Cache->set_in_cache:
# This is a bit of a hack to support the old API in case things still use it
Let's remove this hack and update old calls.
Test plan:
Look at the results of
git grep set_in_cache
and confirm that there are no more version of the old call (without
hashref as third param)
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Resolve typo inifinite too.
Adds a few lines in order to stress that the thawed key of the L1
cache SHOULD ONLY be used for unsafe calls, and not be mixed with
regular (safe) calls.
Test plan:
Nothing to test, but verify the quality of the added comments.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
This patch removes an occurence of an unused var in Context.pm and
also prevents a warning if memcached_servers is not set:
Use of uninitialized value in split at /home/vagrant/kohaclone/Koha/Cache.pm line 91.
t also tidies small things.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
TODO: Would be better to provide a better way to know if memcache is correctly set
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Move C4::Context->read_config_file to Koha::Config
That permits to reuse it from Koha::Cache without needing C4::Context
(and introduce a circular deps).
TODO: Add decent POD to Koha::Config
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
MEMCACHED_SERVERS and MEMCACHED_NAMESPACE should be moved back to
koha-conf instead of an ENV var to let command line scripts use
the cache mechanism.
TODO:
- Add the entries to koha-conf.xml
- Remove occurrences of MEMCACHED_SERVERS in package, apache and plack files
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Jacek Ablewicz <abl@biblos.pk.edu.pl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
https://bugs.koha-community.org/show_bug.cgi?id=11921
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
We need to define several namespaces for our cache system.
For instance sysprefs, koha conf (koha-conf.xml) and unit tests
should be defined in a separate namespace.
This will permit to
- launch the tests without interfering with other cache values
- and flush the sysprefs cache without flushing all other values
To do so, we need to store different Koha::Cache objects at a package
level. That's why this patch adds a new Koha::Caches module.
FIXME: There is an architecture problem here: the L1 cache should be
defined in Koha::Cache
Signed-off-by: Jacek Ablewicz <abl@biblos.pk.edu.pl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
https://bugs.koha-community.org/show_bug.cgi?id=11921
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Sereal is not packaged for jessie, so let's use Sereal::Encoder and
Sereal::Decoder instead.
Signed-off-by: Mirko Tietgen <mirko@abunchofthings.net>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
This manages to eke out a bit more performance on my machine.
Signed-off-by: Jacek Ablewicz <abl@biblos.pk.edu.pl>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Bug 16044 introduced two-level caching mechanism to Koha::Cache. By
default, get_from_cache() returns a deep copy of the data structures
stored in L1 cache (aka "safe mode"). For extremely big and/or complex
data structures (like MARC framework hash-of-hashes-of-hashes returned
by GetMarcStructure() ), deep-cloning is currently sub-optimal;
this patch addresses that particular issue. It also provides an
ability to intermix "safe" and "unsafe" cache feches, in such
way that they don't interfere with each other (i.e., eliminating
the risk involved with "unsafe" calls possibly compromising
further "safe" calls).
Test plan:
1) apply patch
2) flush L2 cache if necessary (restart memcached daemon)
3) profile GetMarcStructure() calls before / after patch, e.g. by
running some script which calls it often (like catalogue search
w/ XSLT processing turned on, etc.) - GetMarcStructure() should be
faster than before, in all possible circumstances (eg. 18 msec
per call -> 10 msec p/call)
4) after testing, before returning to the master branch, flush L2 cache
again (restart memcached daemon) - otherwise all system preferences
returned from L2 cache would be suffixed with '-CF0'
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Jesse Weaver <jweaver@bywatersolutions.com>
No need to cache undef in L2.
Prevent numerous warnings like:
Use of uninitialized value in subroutine entry at /usr/share/koha/masterclone/Koha/Cache.pm line 284
Test plan:
Run t/Cache.t
Run t/db_dependent/sysprefs.t
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Amended patch: remove trailing space char and replace 'you' with 'we' to
match existing wordings.
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jesse Weaver <jweaver@bywatersolutions.com>
This reverts commit 6b319a2d48.
Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Tested with memcached and without MEMCACHED_SERVERS.
Cache.t, Context.t and sysprefs.t pass now.
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Storable dclone() is over 3x faster then Clone::clone() when used for
creating a deep copies of the big, complex data structures like MARC
frameworks. See also Bug 16044 comments #18 & #20.
This is a trivial version of Bug 16166. Performance test
results (see Bug 16140 comment #23 for test setup description):
master
146.29 (14.64+14.53+14.51+14.57+14.72+14.56+14.53+15.06+14.56+14.61)
master + Bug 16221
96.5 (9.77+9.63+9.77+9.77+9.68+9.05+9.68+9.83+9.63+9.69)
master + Bug 16166
90.7 (9.15+9.10+9.16+8.41+9.19+9.19+9.11+9.19+9.02+9.18)
Bug 16166 is a bit faster and more comprehensive implementation, but
also more complex and harder to test/evaluate.
Test plan:
1) apply patch
2) profile GetMarcStructure() calls before / after patch, e.g. by
running some script which calls it often (like catalogue search
w/ XSLT processing turned on, etc.)
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Cache::Memory is loaded and Koha::Cache::_initialize_memory is called
even if a memcached cache has been correctly initialize, it does not
make sense.
It should only be initialize if needed.
Test plan:
Correctly configure a memcache server and confirm that the memory cache
is not initialize.
Do not configure correctly a memcache server and confirm that the cache
system default on Cache::Memory
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Since bug 11998, the following warnings are raised:
Name "Tie::Hash::FIELDS" used only once: possible typo at /usr/share/perl/5.22/fields.pm line 135.
Name "Cache::RemovalStrategy::LRU::FIELDS" used only once: possible typo at /usr/share/perl/5.22/fields.pm line 135.
Name "Cache::RemovalStrategy::FIELDS" used only once: possible typo at /usr/share/perl/5.22/fields.pm line 135.
It comes from the Koha::Cache
103 if ( can_load( modules => { 'Cache::Memory' => undef } ) ) {
Test plan:
perl -wc C4/AuthoritiesMarc.pm
should return green
QA note: this may have an impact on performance but I have not found any other workarounds.
NOTE: I applied 15870 to master, got the error (-v 2 -c 5), applied 16104,
rebased it in front, error disappeared (still only -c 5), rebased it at the end,
and signed off 16104.
Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Brendan Gallagher <bredan@bywatersolutions.com>
Could be useful later.
Signed-off-by: Jacek Ablewicz <abl@biblos.pk.edu.pl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>
Signed-off-by: Brendan Gallagher <bredan@bywatersolutions.com>
Koha::Cache->set_in_cache should deep copy (if needed) to avoid the
value which has been set in cache to be unintentionally modified later.
Signed-off-by: Jacek Ablewicz <abl@biblos.pk.edu.pl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>
Signed-off-by: Brendan Gallagher <bredan@bywatersolutions.com>
When a value exists in L2 cache but not in L1 cache, it should be deep
copied if needed (i.e. not a scalar). Otherwise the calling code is able
to modify the value in cache.
Note that is theoretical, it's possible that no code does that.
Signed-off-by: Jacek Ablewicz <abl@biblos.pk.edu.pl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@unc.edu.ar>
Signed-off-by: Brendan Gallagher <bredan@bywatersolutions.com>
The whole patch set is not very pertinent if the L1 cache is not
populated when L2 is fetched!
This patch fixes this inconsistency.
Signed-off-by: Jesse Weaver <jweaver@bywatersolutions.com>
Signed-off-by: Jacek Ablewicz <abl@biblos.pk.edu.pl>
Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
If the caller/developer knows what he is doing, he can decide not to
deep copy the structure. It will be faster but unsafe!
If the structure is modified, the cache will also be updated.
This option must be used with care and is not the default behavior.
Signed-off-by: Jesse Weaver <jweaver@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
To avoid the cache to be modified unfortunately, the default behavior of
get_from_cache will be to deep copy if we are getting a structure.
If the item is a scalar, it's simply returned.
Signed-off-by: Jesse Weaver <jweaver@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
Signed-off-by: Jesse Weaver <jweaver@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Brendan A Gallagher <brendan@bywatersolutions.com>
Signed-off-by: Chris Nighswonger <cnighswonger@foundations.edu>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
http://bugs.koha-community.org/show_bug.cgi?id=9987
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
1) Removed 'use C4::Context;' because it can lead to introduction
of circular reference in the near future
2) Put fastmmap initialization code into an eval {} block, to catch
various kinds of errors which can still occur during it's init in
some [less usual] Koha setups and/or more unusual circumstances
3) Do not include UID in the sharefile name (it will be constructed
using namespace + database name + database host instead).
Test plan addendum:
s/and UID//
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Koha::Cache package does not take into account that, when using
fastmmap caching variant, mmaped cache file created in /tmp
(typically: /tmp/sharefile-koha-koha), would only be further
accessible to the one given OS user - the one which created it.
In many Koha setups, in the circumstances when various system scripts
are executed by 2+ users with diffrent UIDs (like multi-tenant servers,
for example) this may cause many kinds of issues. Observable symptom
is usually the appearance of the below error when searching, or looking
at MARC Framework pages and a few other places:
Open of share file /tmp/sharefile-koha-koha failed: Permission denied
at /usr/lib/perl5/Cache/FastMmap.pm line 640.
This patch:
- disables initialisation of fastmmap caching subsystem unless it is
explicitly requested by the user (CACHING_SYSTEM=fastmmap)
- disables fastmmap cache usage for command line scripts
(i.e. when GATEWAY_INTERFACE environment variable is not defined)
- adds the database name, host name and an ID of the OS user to the
mmaped file name created in /tmp, to prevent various kinds of
unintentional conflicts and/or permission problems from happening
To test:
1) remove the /tmp/sharefile-koha-* file[s] (if any)
2) do something which would lead to its re-creation (e.g., performing
any search in OPAC should be sufficient to cause that)
3) observe that /tmp/sharefile-koha-koha got created
4) remove it
5) apply patch
6) redo step 2)
7) observe that aforementioned file is no longer created in /tmp
8) set CACHING_SYSTEM environment variable to 'fastmmap'
9) redo step 2), observe that /tmp/sharefile-koha-* file got created
and that it's name now contains hostname, database name and UID
10) ensure that everything still works like it should and that there
are no regressions of any kinds anywhere in the system ;)
Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
This fixes a minor POD error detected, and also adds the Koha/ directory
to the regular test case so that the in-repo tests can find these
things.
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
This makes Koha::Cache behave better by default. It will use memcached
if available to do shared caching, if that's not available it will fall
back to in-memory caching. It also allows for a singleton accessor to
allow a single cache to be shared within a process.
* Added tests to confirm UTF8-cleanness.
* Added minor fixups to stop warnings.
Test plan:
* The t/Cache.t file runs successfully with and without the
MEMCACHED_SERVERS envvar set (and memcached running in the
environment.)
Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
At the moment we cache numerous pieces of information in module-level
variables which then do not get updated in other threads/processes when
they are changed by the user. This is a serious usability issue.
Examples of this include the way we treat sysprefs (there is now a
method to disable the syspref cache, but by default it is enabled),
notices, frameworks, field mappings, and koha-conf.xml, at least.
This patch sets the stage for eliminating this problem by making it
possible to convert module-level cache variables into variables that
are actually backed by whatever caching system may be configured. This
is done through a special Koha::Cache::Object class which can be tied
to the variables that are being used for caching and provided with a
constructor method/closure to allow the cache to be reloaded when it
expires. For example:
my $cache = Koha::Cache->new();
my $data = 'whatever';
my $variable = Koha::Cache->create_scalar(
{
'key' => 'whatever',
'timeout' => 2,
'constructor' => sub { return $data; },
}
);
print "$$variable\n"; # Prints "whatever"
The one change this necessitates for accessing the data is that the
variable must be dereferenced an additional time before use (i.e.
$$variable instead of $variable). There is no difference when the
variable tied is a hash (created with Koha::Cache->create_hash). This
is a small price to pay for Koha working in a multi-threaded, persistent
environment. This change will also make caching easier in general.
CHI was incompatible with the variable tying, so this patch also removes
the dependency on CHI, using instead Cache::Memcached::Fast,
Cache::FastMmap, and Cache::Memory, when they are available.
To test:
1) Apply patch.
2) Run unit test t/Cache.t (after setting the MEMCACHED_SERVERS and
CACHING_SYSTEM environment variables). As no changes were made to
the tests already in that file, this passing demonstrates there
are no regressions.
3) With memcached caching enabled (you must set the MEMCACHED_SERVERS
and CACHING_SYSTEM environment variables) and DEBUG turned on (i.e.
the DEBUG environment variable set to 1), try running a report via
the web service ([intranet]/cgi-bin/koha/svc/report?id=1 and check
your web server logs to confirm that there are messages like
"get_from_cache for intranet:report:id:1" in them.
4) If the reports worked, sign off.
NOTE: Technically you could test this without needing memcached by
installing libcache-fastmmap-perl and setting CACHING_SYSTEM to
'fastmmap' instead of 'memcached'. You could also install libcache-perl
and set CACHING_SYSTEM to 'memory' but there would be little point as
the cached variables would go out of scope in between runs.
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
This reverts commit 215abc8024.
The 3 patches for bug 8089 have been reverted, because they break
jenkins & Koha.
A follow-up has been provided, but it does not solve the problem on my
test server, it just changes the error message.
After a discussion with jared, Dobrica should work on another patch, so
the best option is to revert.
1. Replace all instances of memoize_memcached with appropriate calls
into Koha::Cache:
* reports/guided_reports.pl
* C4::Biblio::GetMarcStructure
* C4::Languages::getFrameworkLanguages
* C4::Languages::getAllLanguages
* C4::SQLHelper::GetPrimaryKeys
* C4::SQLHelper::_get_columns
2. Replace all references to memcached with the appropriate calls into
Koha::Cache in C4::Context.
Test plan :
* have DEBUG env set to 1
* reach addbiblio page to test the patch in Biblio.pm, or setup more than 1
language
* you should see in the logs that you're reading and writing from cache
* run the test suite twice both with and without the following environment
variables set:
export MEMCACHED_SERVERS=127.0.0.1:11211
export MEMCACHED_NAMESPACE=KOHA
export CACHING_SYSTEM=memcached
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
I'm unsure about some of the caching times 10000 is a long long time,
but other than that, works fine.
Implements cache handlers for Memcached, mmap shared-file persistent,
and in-process memory caches. If CHI is unavailable, Koha::Cache::Memcached
will fall back to using Cache::Memcached::Fast, or caching will be skipped
without croaking.
To test: run t/Cache.t with the following options, before and after
installing CHI:
2. Tests 3-9 should be skipped with the following:
export CACHING_SYSTEM=
2. You should receive two failures with the following if CHI is not installed:
export CACHING_SYSTEM=memory
3. You should receive two failures with the following if CHI is not installed:
export CACHING_SYSTEM=fastmmap
4. You will need memcached activated for the following to work (but it
will work both with and without CHI):
export CACHING_SYSTEM=memcached
export MEMCACHED_SERVERS=127.0.0.1:11211
export MEMCACHED_NAMESPACE=KOHA
5. You should receive two failures with the following:
export CACHING_SYSTEM=thisdoesntexist
Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Works as advertised, now we need a follow up to add the new dependency.
http://bugs.koha-community.org/show_bug.cgi?id=8029
This patch introduces some new features for caching system:
* the type of caching is retrieved from ENV variable in httpd configuration if not forced when the ->new() is called.
* if $ENV{DEBUG} is ON, you'll have feedback when something is set or read from cache
* the Koha::Cache->is_cache_active is now available and will return 1 if there is a caching system available. It's a replacement for C4::Context->ismemcached
* fixes the mistake for debug/compress_threshold and expire_time parameters
The 2 report web service modules have been updated to use this new API
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Unit test for Koha/Cache.pm (which includes Koha/Cache/Memcached.pm)
Note that in order to test Koha::Cache you must export the environment variable
MEMCACHED_SERVERS. For example:
$ export MEMCACHED_SERVERS=127.0.0.1:11211
Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Jonathan Druart
Olli-Antti Kivilahti
Marcel de Rooy
Kyle M Hall
Tomas Cohen Arazi
Jesse Weaver
Jacek Ablewicz
Tomas Cohen Arazi
Jonathan Druart
Robin Sheat
Jared Camins-Esakov
Paul Poulain
Chris Hall