Skip to content

[Enhancement]: Consider opt-in single-connection SFTP multiplexing for concurrent file transfers #188

Open

Description

@X1pheR

Problem

Current SFTPAdapter.openSession() uses a connection pool with limit: options?.concurrency ?? 1: concurrent transfers can create one SSH login per file in flight. The source even documents PERF-SFTP-MULTIPLEX as a future optimization: the ssh2 SFTP channel supports multiple concurrent request IDs, so a single SSH connection can potentially multiplex operations without N independent handshakes.

A maintained downstream currently uses one reusable SSH/SFTP client for concurrent source transfers, plus a bounded transient handshake retry. This reduces pressure on SFTP servers with strict concurrent-login limits, but may behave differently on certain SSH servers and is not automatically a drop-in replacement for the existing pool.

Proposed upstream design discussion

Consider an explicitly reviewable single-connection multiplex mode for SFTP source sessions, with the pool behavior preserved for compatibility until supported integration/performance evidence is available. The API/default choice should be made with maintainers, not silently changed by a fork.

Acceptance proposals:

  • A request for concurrency 8 with many files uses one SSH session in multiplex mode, while still having more than one file I/O operation in flight.
  • Connection lifecycle is fail-closed and race-safe: close exactly once, new requests after close or disconnect fail, mid-transfer disconnect is observable, and retry applies only to transient initial handshakes (not authentication).
  • Shared directory creation cache and concurrent file creation remain safe.
  • Existing pool mode retains current documented behavior and unit/integration expectations unless intentionally migrated.
  • Benchmarks compare multi-connection pooling and single-connection multiplexing on representative high-latency and restrictive SFTP servers, including large and many-small-file workloads.
  • No DB migrations, deployment-specific credentials or Homelab-only settings.

Source-tree directory scanning is separately covered by #186 / draft PR #187. This issue concerns file transfer sessions, not directory scans. Once an interface/default is accepted, an upstream-native PR can be prepared against the current dev baseline with test-first coverage.

Activity

  1. changed the title [-]Consider opt-in single-connection SFTP multiplexing for concurrent file transfers[/-] [+][Enhancement]: Consider opt-in single-connection SFTP multiplexing for concurrent file transfers[/+] on Oct 10, 2026
  2. self-assigned this
    on Oct 10, 2026
  3. Skyfay commented on Oct 10, 2026

    @Skyfay
    Owner

    Hey @X1pheR

    Thanks for this. It's already on my list, it's the PERF-SFTP-MULTIPLEX note in openSession(). The pool opens at most as many connections as Parallel transfers allows (4 by default, 8 at most) and reuses them for every file of the session, so this is about speed rather than something being broken. I'll treat it as an enhancement.

    It changes how transfers are pooled, so I'd like to do it myself as its own change and measure it against a few different servers. Please hold off on a PR for this one.

    A few things that will matter when I do:

    • The ssh2-sftp-client README advises against running requests in parallel over one connection, so it needs testing on more than OpenSSH.
    • Every request adds its own end, close and error listeners. With 12 requests in flight Node already prints a MaxListenersExceededWarning.
    • One connection also means one TCP connection and one SSH channel window for every file. Many small files should get faster, but a few large ones may get slower than over separate connections, so both need to be measured.

    The walk from #187 is a good first data point. With a single connection it was just as fast as with 8.

    I'll keep this issue open to track it.

    Thanks :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementAn improvement to an existing feature

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions