_upload_file, _download_file, _list_files and _rename_file's 'no directory configured' checks still called add_message() directly, skipping the status persistence this bug added elsewhere in the file. Route them through _abort_operation() so all error paths in Local.pm stay consistent.
prove t/db_dependent/Koha/File/Transport/Local.t
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Pedro Amorim <pedro.amorim@openfifth.co.uk>
Two related inconsistencies remained across the three transport
backends' error-handling paths:
1) Only Koha::File::Transport::SFTP's _abort_operation() persisted an
"errors" status snapshot to the status column on every failed
operation. FTP never did, and Local had no _abort_operation() at
all - so an SFTP transport's displayed status could reflect a
single transient failure from real usage (e.g. an EDI cron job)
while the exact same failure on FTP/Local left the stored status
untouched (stale, possibly still showing the last explicit
test_connection() result). This is exactly the kind of unclear
status reporting bug 42656 is about.
2) FTP's _abort_operation() carried its extra detail under an
operation_params key, while SFTP (and Local's plain add_message
calls) used path. The change_directory() success payload had the
same split: FTP used directory, SFTP and Local used path. The
admin template's change_directory row display depended on FTP's
mismatched key names, which meant it silently failed to show the
tested directory for SFTP transports.
Adds a shared _record_error($operation, \%payload) to the
Koha::File::Transport base class - add_message() plus status
persistence - and routes all three backends' _abort_operation() (Local
gains one) through it. Renames FTP's operation_params/directory keys
to path throughout, matching SFTP and Local, and simplifies the
template to read operation.detail.path unconditionally instead of
ORing two differently-named keys.
_record_error() persists the full trace of operations recorded on the
transport object so far this session (every success leading up to the
failure), not just the failing operation in isolation - the same
"full trace" shape TestTransport::process() already builds for an
explicit test_connection() run. A long-lived transport used across a
multi-step operation (e.g. an EDI download loop touching several files
before one fails) will therefore show every successful step leading up
to the failure. It also logs the failure - including the built trace -
via Koha::Logger at warn level, so it's visible in the Koha logs
without needing to query the transport's status column.
Test plan:
1. prove -r t/db_dependent/Koha/File/ t/db_dependent/Koha/BackgroundJob/TestTransport.t t/db_dependent/Koha/Edifact/Transport.t t/db_dependent/api/v1/file_transports.t
=> All tests successful (48 tests total), including:
- new "_abort_operation persists status consistently" subtests
in FTP.t, SFTP.t and Local.t that reload the transport from
the DB after a simulated failure and confirm the status
column was updated with the correct operation code and path
detail;
- a new "_record_error() tests" subtest in Transport.t proving
the persisted trace includes a prior success message plus the
failure (not just the failure alone), and that a warn-level
log line mentioning the operation and error was emitted.
2. koha-qa.pl -v 2 --more-tests
=> [PASS], no critic/pod warnings.
Signed-off-by: Pedro Amorim <pedro.amorim@openfifth.co.uk>
This patch displays the directory tested in the background jobs results
It updates the language when test results are not stored for the transport and
links to the job when results are stored
To test:
1 - Setup an ftp file transport with good credentials
2 - Enter non-existent directories in incoming/outgoing
3 - Save
4 - Verify on save 'Never used' shows for status
5 - Refresh the page, status now:
Connection ok
Change directory ok
Change directory failed
Change directory ok
Change directory failed
6 - Apply patch, restart all
7 - Edit/save the transport again, change nothing
8 - Confirm the page says 'No test results stored'
9 - Reload the page
10 - Confirm results are now:
Connection ok
Change directory ok
Change directory failed
Change directory ok
Change directory failed
Test job details
11 - Click on the job details link
12 - Confirm actual directory and errors are shown
13 - Edit the transport, use existing directories
14 - Save, reload
15 - Confirm the success results and link t background job
Signed-off-by: John Vinke <jvinke@austinseminary.edu>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Signed-off-by: Pedro Amorim <pedro.amorim@openfifth.co.uk>
Koha::File::Transport's three backends disagree on what
change_directory(undef) means: FTP navigates to the server root,
SFTP falls back to the SFTP session's own home directory, and Local
fell back to the configured download_directory or, if that's also
unset, silently to '.' - the Koha process's own working directory,
which has no relation to any configured transport location.
test_connection() calls change_directory(undef) unconditionally for
both the download and upload directory checks on every config save
(via the auto-enqueued TestTransport background job). For a Local
transport configured with only one of download_directory/
upload_directory, the untested direction's list_files() call would
silently list the Koha process's own cwd (e.g. the Koha install root)
instead of failing cleanly.
Add a private _working_directory($configured_field) helper that
returns the tracked change_directory() override or the given
configured field, but never falls back to '.'. _upload_file,
_download_file, _list_files and _rename_file now use it and record a
clear "not configured" error instead of silently acting on an
unrelated directory when neither is set.
FTP's root and SFTP's home-directory fallbacks are left as-is: unlike
Local's '.', they navigate to a real, meaningful location for that
specific connection rather than leaking unrelated filesystem content,
so they aren't part of this fix.
Test plan:
1. prove t/db_dependent/Koha/File/Transport/Local.t
=> All tests successful, including the new subtest proving that
list_files()/rename_file()/download_file() fail cleanly (with a
"not configured" error) rather than falling back to '.' when
their respective directory isn't configured, while the
configured direction (upload, in the test) still works.
2. prove -r t/db_dependent/Koha/File/ t/db_dependent/Koha/Edifact/Transport.t t/db_dependent/api/v1/file_transports.t
=> All tests successful (43 tests total), no regressions.
3. koha-qa.pl -v 2 --more-tests
=> [PASS], no critic/pod warnings.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Pedro Amorim <pedro.amorim@openfifth.co.uk>
_change_directory(), _list_files() and _is_connected() each called
$self->{connection}->pwd directly for their log payloads/connectivity
check, duplicating exactly what _current_directory() now provides as a
proper, documented internal accessor. Route them through it instead -
no behaviour change, just removing the duplication now that the
wrapper exists.
Test plan:
1. prove -r t/db_dependent/Koha/File/
=> All tests successful (32 tests), including the existing
current_directory() and list_files()/change_directory() coverage.
2. koha-qa.pl -v 2 --more-tests
=> [PASS], no critic/pod warnings.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Pedro Amorim <pedro.amorim@openfifth.co.uk>
FTP, Local and SFTP list_files() were unified to return the same flat
shape, but consumers still had no portable way to ask a transport for
its current remote directory - the underlying libraries' own methods
of that name don't agree on what "cwd" means. Net::SFTP::Foreign's
cwd() is a pure read-only accessor, but Net::FTP's cwd() is a mutator:
called with no argument it defaults to "/", issues a real CWD command,
and returns a boolean rather than a path.
The koha-plugin-transport-browser plugin discovered this the hard way:
it read $transport->{connection}->cwd directly to capture the initial
working directory, which happened to work for SFTP but silently reset
the FTP connection to root and captured "1" (a boolean) as the "path"
on every request, breaking directory navigation for FTP transports.
Add a public current_directory() method to the Koha::File::Transport
base class, following the same connect-on-demand pattern as
list_files()/upload_file()/etc, with a _current_directory() hook each
subclass implements using its own real read-only accessor: pwd() for
FTP, cwd() for SFTP, and the tracked {current_directory} (falling back
to download_directory) for Local. Consumers should use this instead of
reaching into transport internals.
Test plan:
1. prove t/db_dependent/Koha/File/Transport.t
t/db_dependent/Koha/File/Transport/FTP.t
t/db_dependent/Koha/File/Transport/Local.t
t/db_dependent/Koha/File/Transport/SFTP.t
=> All tests successful, including the new current_directory()
tests subtest in each transport test file, which mock the
underlying libraries (Net::FTP / Net::SFTP::Foreign) to assert
each backend reads its own real read-only accessor.
2. koha-qa.pl -v 2 --more-tests
=> [PASS] for all four changed modules, no critic/pod warnings.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Pedro Amorim <pedro.amorim@openfifth.co.uk>
The FTP and Local transports both explicitly filter out '.' and '..'
pseudo-entries when listing a directory, but the SFTP transport never
did - it relied on the assumption that Net::SFTP::Foreign's ls() already
excludes them. It doesn't: ls() returns exactly whatever entries the
remote server includes in its SSH_FXP_READDIR response, with no default
filtering. Whether '.'/'..' show up therefore depends entirely on the
remote SFTP server implementation, producing inconsistent results between
otherwise identically configured SFTP accounts.
Test plan:
1. prove t/db_dependent/Koha/File/Transport/SFTP.t
=> fails before this patch (listing includes '.' and '..')
=> passes after this patch
2. prove -r t/db_dependent/Koha/File/Transport/ t/db_dependent/Koha/File/Transports.t
=> all still pass
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Pedro Amorim <pedro.amorim@openfifth.co.uk>
The MLSD path, SFTP, and Local always return all six documented
list_files() keys (filename, longname, size, perms, mtime, type) - undef
where the value is unknown, but the key always present. The NLST
fallback path (servers without MLSD support) only set filename, size,
mtime and type, omitting longname and perms entirely rather than
setting them to undef.
Functionally harmless for consumers reading $file->{perms} (Perl
returns undef for a missing hash key too), but it breaks the "same flat
shape" contract this bug established, and would trip up any consumer
using exists() instead of defined() to check for a key.
Test plan:
1. prove t/db_dependent/Koha/File/Transport/FTP.t
=> All tests successful, including the new assertion that longname
and perms keys are present (as undef) on the NLST fallback path.
2. koha-qa.pl -v 2 --more-tests
=> [PASS], no critic/pod warnings.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Pedro Amorim <pedro.amorim@openfifth.co.uk>
Net::SFTP::Foreign's ls() returned file metadata with size, permissions
and modification time nested inside a Net::SFTP::Foreign::Attributes
object, while FTP and Local returned flat hash keys and silently
dropped directories. Callers of list_files() needed backend-specific
parsing to get at file metadata or to know a directory even existed,
defeating the purpose of the shared transport API.
All three transports now return the same flat hashref shape per entry:
filename, longname, size, perms, mtime, type ('file'/'directory'/
'other'). Directories are included everywhere; '.' and '..' remain
excluded everywhere. FTP's MLSD path prefers the UNIX.mode fact for
exact Unix permissions, falling back to an approximation from the RFC
3659 perm capability fact when UNIX.mode isn't available, and parses
the modify fact into mtime; its NLST fallback (servers without MLSD)
probes each entry with SIZE/MDTM to recover size, mtime and a
best-effort type.
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Pedro Amorim <pedro.amorim@openfifth.co.uk>
Koha::File::Transport::FTP::_list_files parsed the LIST / "ls -l" output
with a non-greedy date capture that only consumed the first of the three
date tokens (the month), leaving the day and time glued to the front of
the filename. Every filename returned for an FTP transport was therefore
prefixed with the listing date, e.g.
-rw-r--r-- 1 user group 1234 Jan 01 12:00 QUOTES_413514.CEQ
=> filename "01 12:00 QUOTES_413514.CEQ"
Callers (Koha::Edifact::Transport, marc_ordering_process.pl) then asked
the server for a file that does not exist, so no files were downloaded,
renamed or deleted over FTP. SFTP and Local transports were unaffected.
This regressed when the EDI transport was moved onto the file_transports
system (bug 38489). The previous Koha::Edifact::Transport::ftp_download
listed files with Net::FTP->ls (NLST, bare names); the refactor switched
to Net::FTP->dir (LIST) so it could return the richer hashref shape used
by the SFTP and Local transports, but the parse was faulty.
The date field in "ls -l" output is a fixed run of three whitespace
separated tokens (Mon DD HH:MM or Mon DD YYYY). Consume it explicitly so
the filename is captured cleanly, preserving spaces and still skipping
directories. The LIST-based approach and the filename/longname/size/perms
return shape are kept so the structure stays consistent across transport
classes (relied on by consumers, including out-of-tree ones).
Test plan:
1) prove t/db_dependent/Koha/File/Transport/FTP.t
2) git checkout Koha/File/Transport/FTP.pm and re-run: the list_files()
subtest fails on the filename/size/perms assertions
3) prove t/db_dependent/Koha/Edifact/Transport.t
Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Andrew Fuerste Henry <andrew@bywatersolutions.com>
Signed-off-by: Pedro Amorim <pedro.amorim@openfifth.co.uk>
This patch fixes a critical bug where plain_text_password() was called
in list context during hash construction. When a transport has no
password, the method's bare 'return' statement returns an empty list ()
in list context rather than undef, causing all subsequent key-value
pairs to shift positions. This resulted in malformed connection
parameters being passed to Net::SFTP::Foreign and Net::FTP.
Example of the bug:
my %hash = (
user => 'john',
password => $self->plain_text_password, # Returns () when no password
timeout => 10
);
# Results in: (user => 'john', timeout => 10) instead of
# (user => 'john', password => undef, timeout => 10)
Changes to code:
- Wrap plain_text_password calls with scalar() to force scalar context
in both FTP.pm and SFTP.pm
- Make SFTP key and password authentication mutually exclusive using
ternary operator
- Prefer key-based auth over password when both are available (SFTP)
Test plan:
1. Run t/db_dependent/Koha/File/Transport/FTP.t - should pass
2. Run t/db_dependent/Koha/File/Transport/SFTP.t - should pass
3. Verify tests cover:
- Scalar context behavior with no password
- FTP authentication with proper parameter passing
- SFTP password-only authentication
- SFTP key-only authentication
- SFTP key preference when both key and password exist
Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Fix several API inconsistencies in the FTP transport class:
1. Message format: Use standardized `message => $operation` format
instead of hardcoded strings like "Upload succeeded"
2. Operation tracking: Add $operation variable to all methods for
consistent error reporting and logging
3. Error handling: Update _abort_operation() to accept operation
parameter and provide more detailed error payloads
4. Success payloads: Replace empty `detail => ''` payloads with
meaningful operation-specific information (host, port, file paths,
directory info, file counts)
All transport classes now have consistent APIs for:
- Standardized message formats
- Consistent operation tracking
- Uniform error handling
- Informative success payloads
Sponsored-by: ByWater Solutions
Signed-off-by: Hannah Dunne-Howrie <hdunne-howrie@westminster.gov.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Fix API inconsistency where different transport types returned different
formats from list_files():
- SFTP: returned array of hashrefs with detailed file information
- FTP: returned simple array of filenames
- Local: returned simple array of filenames
Now all transports return an array reference of hashrefs with consistent
structure containing filename, longname, and additional metadata.
This eliminates the need for workarounds like the ref($file) eq 'HASH'
check in EDI code and provides a more robust, consistent API.
Sponsored-by: ByWater Solutions
Signed-off-by: Hannah Dunne-Howrie <hdunne-howrie@westminster.gov.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch adds proper connection cleanup functionality to the file
transport system, addressing a gap identified during EDI migration.
Changes:
- Add disconnect() method to base Transport class interface
- Implement SFTP disconnect using connection->disconnect()
- Implement FTP disconnect using connection->quit()
- Add DESTROY methods to ensure connections are cleaned up on object destruction
- Local transport includes no-op disconnect for API consistency
This ensures that FTP/SFTP connections are properly closed, preventing
connection leaks and potential resource exhaustion issues.
Sponsored-by: ByWater Solutions
Signed-off-by: Hannah Dunne-Howrie <hdunne-howrie@westminster.gov.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch enhances the file transport system with file renaming
capabilities and adds a Local transport class for development/testing.
Changes:
- Add rename_file() method to base Transport class and SFTP/FTP subclasses
- Create Koha::File::Transport::Local for local directory operations
- Add 'local' transport type to polymorphic mapping
- Update transport instantiation to handle Local transport
- Add Local transport tests to verify polymorphic behavior
- Enhanced connect() method for Local transport validates directories
The rename_file functionality is needed for EDI file processing workflow
where files are renamed after download to mark them as processed.
Sponsored-by: ByWater Solutions
Signed-off-by: Hannah Dunne-Howrie <hdunne-howrie@westminster.gov.uk>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch:
* Removes unused logger object instantiation
* Adds missing Test::NoWarnings to the API tests
Signed-off-by: Tomás Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch renames the entire API structure from SFTP-specific to generic
file transport terminology to reflect the polymorphic nature of the system.
API Changes:
- Endpoints: /config/sftp_servers -> /config/file_transports
- Parameters: sftp_server_id -> file_transport_id
- Definitions: sftp_server -> file_transport
- Controller: Config::SFTP::Servers -> Config::File::Transports
File Changes:
- api/v1/swagger/definitions/sftp_server.yaml -> file_transport.yaml
- api/v1/swagger/paths/config_sftp_servers.yaml -> config_file_transports.yaml
- Koha/REST/V1/Config/SFTP/Servers.pm -> Koha/REST/V1/Config/File/Transports.pm
- t/db_dependent/api/v1/sftp_servers.t -> file_transports.t
Updated References:
- swagger.yaml: paths, definitions, parameters, tags
- Template JavaScript: API URLs and variable names
- Controller: package name, variable names, error messages
- Tests: endpoints, parameters, descriptions
- API mapping: to_api_mapping field name correction
To test:
1. Apply this patch
2. Run: yarn api:bundle (to regenerate swagger bundle)
3. Run: prove t/db_dependent/api/v1/file_transports.t
4. All API tests should pass
5. Verify admin interface uses new API endpoints
6. Test CRUD operations work correctly
The API now consistently uses 'file_transports' terminology matching
the polymorphic file transport system that supports FTP, SFTP, and
future transport types.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Resolved circular dependency chain:
Transport.pm → TestTransport.pm → Transports.pm → SFTP.pm → Transport.pm
Changed 'use Koha::BackgroundJob::TestTransport' to runtime 'require'
in the _post_store_trigger method to break the dependency cycle.
This fixes the C3 method resolution order failures in
Koha::File::Transport::FTP and Koha::File::Transport::SFTP.
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
We now use background jobs triggered by Transport->store to update the
connection status.
Status is now a json blob with more details, so we update the api
definition to reflect that.
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
We add triggers to store in the base class and then use them in the SFTP
transport class to write out the key file to the filesystem for later
use in connect.
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
This patch adds polymorphic transport classes to Koha for FTP and SFTP
file transport configurations.
We introduce a base class for interacting with the base configuration
storage and subclasses to impliment the protocol specific transport
handling.
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Lucas Gass <lucas@bywatersolutions.com>
Pedro Amorim
Martin Renvoize
Nick Clemens
Lucas Gass
Tomás Cohen Arazi
Martin Renvoize