Skip to content

[Enhancement]: SFTP source-tree collection can create a connection storm under directory concurrency #186

Open

Description

@X1pheR

Problem

Current src/lib/adapters/storage/sftp.ts implements listTree() with breadth-first concurrent traversal backed by createConnectionPool({ limit: options?.concurrency ?? 1 }). With a wide tree and concurrency: 8, the pool may open eight SSH sessions merely to list the tree. On servers with strict handshake or connection limits, that can produce avoidable login storms and transient connection failures before any file transfer starts.

This is separate from SFTP file transfer session multiplexing (the openSession() path).

Proposal

Bound source-directory collection to one reusable SFTP/SSH connection while retaining concurrent directory requests on the SSH channel, a bounded retry only for transient handshake errors, and current abort/progress/path/exclusion/symlink behavior. Alternative: expose a configurable, low default connection ceiling while preserving a higher optional one for servers that need it. This requires maintainer input because upstream currently uses a connection pool by design.

Acceptance:

  • listTree(..., {concurrency: 8}) opens no more than one SSH session in the proposed default mode while fetching multiple child directories correctly.
  • No retry on authentication/unsupported-key errors, and at most one retry on timeout/reset during initial handshake.
  • Abort and failure close the sole client; no leaked connections.
  • Existing exclude, symlink and progress contracts remain unchanged.
  • Tests reproduce the behavior and fail before implementation, then pass with the new one-connection mode.
  • Source-only fix; no DB migration, deployment credentials or Homelab-specific configuration.

A downstream product currently carries this behavior and wants to converge upstream. A separate issue will cover file transfer session multiplexing rather than mixing this source-tree lifecycle with uploads/downloads.

Activity

  1. changed the title [-]SFTP source-tree collection can create a connection storm under directory concurrency[/-] [+][Enhancement]: SFTP source-tree collection can create a connection storm under directory concurrency[/+] on Oct 10, 2026
  2. self-assigned this
    on Oct 10, 2026
  3. Skyfay commented on Oct 10, 2026

    @Skyfay
    Owner

    Hey @X1pheR

    Thanks for this one. You're right that the tree walk opens one connection for every folder it reads in parallel, so with 8 parallel transfers it can log in 8 times. I wouldn't call it a storm though. The number is capped by Parallel transfers on the connection (4 by default, 8 at most), and the transfers afterwards use exactly as many connections anyway. So for a server with a strict limit, lowering that setting is already the fix today, and a cheaper walk alone wouldn't change that.

    That said, one connection for the walk is still worth it. I measured it against a real OpenSSH server with 249 folders and 20 ms of added latency:

    Parallel transfers dev #187
    4 4 logins, 17.2 s 1 login, 16.6 s
    8 8 logins, 9.3 s 1 login, 7.2 to 8.7 s

    So it's just as fast with a single login. I'll treat this as an enhancement rather than a bug, more in #187.

    One more thing: please open issues through the GitHub web interface with one of the official templates (Bug Report, Enhancement, Feature Request etc.). Your issues skipped them, so no labels were set and the fields I usually ask for are missing. I'm happy about the contributions, but please also give me a bit of time to go through the open ones before opening new ones, and test against a real server where you can. A few of the claims in #182 and #184 didn't hold up against a real Samba server.

    Thanks :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementAn improvement to an existing feature

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions