This should pass:
perl /kohadevbox/misc4dev/run_tests.pl --instance=kohadev --db-password=password --run-db-compare-only --compare-with=HEAD
This is a squash of the following:
1) Fix charset in kohastructure.sql
2) Fix NULL => DEFAULT NULL
3) Reorder tables in kohastructure.sql
4) typos
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This follow-up removes the security vulnerability of storing plaintext
passwords in the sip_accounts table. The SIP2 system now relies entirely
on Koha's secure password authentication.
Changes:
- Remove plaintext password comparison from login_core() and telnet_transport()
- Remove login_password column from sip_accounts table
- Update database schema files and atomic update
- Remove password field from API definitions and Vue form
- Update all tests to remove login_password references
Authentication now works in two stages:
1. Access control: Check if userid exists in sip_accounts (allowlist)
2. Authentication: Validate password against hash in borrowers table
This maintains SIP access control while eliminating plaintext password
storage. Staff users authenticate with their Koha password (securely hashed).
Test plan:
1. Run updated tests to verify they pass
2. Verify SIP2 authentication works with staff user credentials
3. Confirm no plaintext passwords are stored
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Sponsored-by: ByWater Solutions
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch updates the database to include a record_table field in the
additional_field_values table and adds an index on that field.
This should facilitate more performant joins.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch adds a new boolean field 'po_is_basketname' to the vendor_edi_accounts
table to allow configuration of how basket names are used in EDIFACT.
Changes:
- Added atomicupdate for existing installations
- Updated kohastructure.sql for new installations
Signed-off-by: Hannah Dunne-Howrie <hdunne-howrie@westminster.gov.uk>
Sponsored-by: Westminster City Council <https://www.westminster.gov.uk/>
Sponsored-by: Royal Borough of Kensington and Chelsea <https://www.rbkc.gov.uk/>
Sponsored-by: OpenFifth <https://openfifth.co.uk/>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Add UNIQUE KEY entries to table definition
The UNIQUE KEY are required as a hold group 'might have' a target hold, not 'has many' target holds.
This update and the other things this patch updates in atomicupdate + kohastructure should fix the error thrown by command:
perl /kohadevbox/misc4dev/run_tests.pl --instance=kohadev --db-password=password --run-db-compare-only --compare-with=HEAD
This command no longer produces any errors.
Also in this patch:
Renamed table from 'hold_group_target_holds' to 'hold_groups_target_holds' and move it alphabetically to be after 'hold_groups' in kohastructure.sql
This is cleaner, as it matches the 'hold_groups' related table, and allows for alphabetical sorting on kohastructure.sql that makes sense i.e. hold_groups is created first, then hold_group_target_holds is created second with an FK to the former.
All of this passes:
prove ./t/db_dependent/Koha/Hold.t
prove ./t/db_dependent/Reserves/HoldGroup.t
prove ./t/db_dependent/Circulation*
prove ./t/db_dependent/Hold*
prove ./t/db_dependent/Reserves*
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
If a hold that is a hold group target, i.e. it's 'waiting' or 'in transit', is reverted it needs to remove itself as its own hold group target
Tests added here as well:
prove t/db_dependent/Koha/Hold.t
Signed-off-by: Anneli Österman <anneli.osterman@koha-suomi.fi>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Hold group target hold:
The hold that is set as the hold group's target. If a hold group has a
hold target, subsequent check-ins that would 'trap' items to the
remaining holds in the group are instead skipped.
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Anneli <anneli.osterman@koha-suomi.fi>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
The default sort order for Koha's patron search includes many columns, some of which are not indexed:
[% CASE 'name-address' %] {
"data":"me.surname:me.preferred_name:me.firstname:me.middle_name:me.othernames:me.street_number:me.address:me.address2:me.city:me.state:me.postal_code:me.country",
Depending on the number of patrons in the database, and the specific configuration of the server, this can end up generating a query that creates a temp table in the hundreds of gigabytes of data or more. Not only does this risk crashing a database server with insufficient disk space, it's also incredibly slow.
We've found a compound index prevents the issue and also makes the search much faster. InndoDB indexes have a fixed maximum length which is too small ( 3072 bytes ) to encompass the full data for all the fields we need to sort by.
The following works well as a comprise to keep the index length within the maximum:
CREATE INDEX idx_borrowers_sort_order ON borrowers (
surname(100), -- 400
preferred_name(80), -- 320
firstname(80), -- 320
middle_name(50), -- 200
othernames(50), -- 200
streetnumber(20), -- 80
address(100), -- 400
address2(75), -- 300
city(75), -- 300
state(40), -- 160
zipcode(20), -- 80
country(40) -- 160
);
Test Plan:
1) Apply this patch
2) Run updatedatabase.pl
3) Perform a patron search
4) No change in behvior should be noted!
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch adds database migration to move EDI transport configurations
from vendor_edi_accounts to the new file_transports system introduced
in Bug 39190.
Changes:
- Add file_transport_id column to vendor_edi_accounts with foreign key
- Add 'local' transport type to file_transports enum
- Remove old transport fields from vendor_edi_accounts in kohastructure
- Migration script converts existing EDI configurations to file transports
- Handles FILE -> local transport type conversion
Sponsored-by: ByWater Solutions
Signed-off-by: Hannah Dunne-Howrie <hdunne-howrie@westminster.gov.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
As suggested by David, this one adds an index on the new column
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Update status field to store longtext so we can sore more details
information about connection statuses from test connection
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch updates the database schema to add booking_id foreign key
columns to both issues and old_issues tables, enabling the linkage
between checkouts and the bookings that generated them.
Schema changes:
- Add booking_id column to issues and old_issues tables
- Add foreign key constraints referencing bookings table
- Preserve referential integrity with appropriate CASCADE actions
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch explicitly defines indexes for foreign keys.
This isn't really needed for the atomic update, but we've included
it for consistency.
It is needed in kohastructure.sql, because MySQL/MariaDB silently/implicitly
adds these indexes for foreign keys, which means that the database
audit tool will show a discrepancy between kohastructure.sql and
the installed DB schema.
Also fix a discrepancy between atomic update and kohastructure.sql
Test plan:
0. Apply the patch
1. Use both methods to update the DB schema
2. Go to http://localhost:8081/cgi-bin/koha/about.pl?tab=database
3. Note that about.pl isn't telling you to drop the indexes
for reserves_ibfk_hg and old_reserves_ibfk_hg now
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Pedro Amorim <pedro.amorim@openfifth.co.uk>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
I have added the "Check for previous checkouts" option to item types. This option is visible only when the system preference "CheckPrevCheckout" is set to either :
- "Unless overridden by patron category or by item type, do"
or
- "Unless overridden by patron category or by item type, do not."
This option takes precedence over patron and patron category preferences.
TEST PLAN
1) Apply the patch.
2) Perform an atomic update (installer/data/mysql/updatedatabase.pl)
3) Set the "CheckPrevCheckout" system preference to "Unless overridden by patron category or by item type, do".
4) Go to "Koha administration > Item types > + New item type".
5) Fill in the necessary fields; the "Check for previous checkouts" field should be set to "Inherit from system preferences".
6) Click the "Save changes" button.
7) Create an item with the type you have created.
8) Check out this item for a patron, then check it in.
9) Check out the same item for the same patron again.
10) The message "Patron has previously checked out this title: <Title>. Check out anyway?" should appear.
11) Go to "Koha administration > Item types".
12) Edit the created item type.
13) Change the value of the "Check for previous checkouts" field to "No and try to override system preferences".
14) Click the "Save changes" button.
15) Check out the same item for the same patron.
16) The message should not appear.
17) Run the tests "Borrower_PrevCheckout.t" (prove t/db_dependent/Patron/Borrower_PrevCheckout.t) and "ItemTypes.t" (prove t/db_dependent/Koha/ItemTypes.t).
18) Verify that all tests pass successfully.
Signed-off-by: Laurence <laurence.rault@biblibre.com>
Signed-off-by: Emmi Takkinen <emmi.takkinen@koha-suomi.fi>
Signed-off-by: Lisette Scheer <lisette@bywatersolutions.com>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
00:40:25 koha-1 | < `checkprevcheckout` enum('yes', 'no', 'inherit') NOT NULL DEFAULT 'inherit' COMMENT 'produce a warning for this patron if this item has previously been checked out to this patron if ''yes'', not if ''no'', defer to category setting if ''inherit''.',
00:40:25 koha-1 | ---
00:40:25 koha-1 | > `checkprevcheckout` enum('yes','no','inherit') NOT NULL DEFAULT 'inherit' COMMENT 'produce a warning for this patron if this item has previously been checked out to this patron if ''yes'', not if ''no'', defer to category setting if ''inherit''.',
00:40:25 koha-1 | ERROR - DB structures are not identical
Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Some sysprefs have NULL as their default value, and systempreferences.value allows NULL values.
However C4::Context->set_preference does not unset a syspref if "undef" is passed.
Additionally some of the sysprefs that have NULL as default will generate warnings in the logs (and test suite output).
opac/opac-suggestions.pl: for my $f ( split( /\s*\,\s*/, C4::Context->preference("OPACSuggestionUnwantedFields") ) ) {
Here is the current list:
+-------------------------------+-------+
| variable | value |
+-------------------------------+-------+
| decreaseLoanHighHoldsDuration | NULL |
| decreaseLoanHighHoldsValue | NULL |
| ILLHiddenRequestStatuses | NULL |
| ILLOpacbackends | NULL |
| ILLSendStaffNotices | NULL |
| ListOwnerDesignated | NULL |
| MarcFieldDocURL | NULL |
| MaxFine | NULL |
| OPACSuggestionUnwantedFields | NULL |
| ReportsExportLimit | NULL |
| SerialsSearchResultsLimit | NULL |
+-------------------------------+-------+
Test plan:
Code review required here, please confirm the above and that the
different occurrences do not use "undef" as a value.
Note that some tests now do not longer generate warnings
(eg. t/db_dependent/selenium/opac_ill_requests.t)
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
On bug 40178 and bug 40179 we defined checkprevcheckout attributes as
ENUM at the REST API level, so the specs prevent us from inserting
values that are no yes, no, inherit in DB.
However the DB schema didn't have that constraint.
This has been caught by some tests that were failing:
eg. t/db_dependent/selenium/patrons_search.t with "Not in enum list: yes, no, inherit."
Test plan:
Apply this patch, run `updatedatabase`, `dbic` and `restart_all`
`prove t/db_dependent/selenium/patrons_search.t` should now pass.
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Pedro Amorim
Martin Renvoize
Pedro Amorim
Martin Renvoize
Tomas Cohen Arazi
Kyle M Hall
Lucas Gass
Jonathan Druart
Jake Deery
Paul Derscheid
David Cook
Aleisha Amohia
Shi Yao Wang
Hammat Wele
Katrin Fischer