You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Enhancement]: SFTP source-tree collection can create a connection storm under directory concurrency #186
Current src/lib/adapters/storage/sftp.ts implements listTree() with breadth-first concurrent traversal backed by createConnectionPool({ limit: options?.concurrency ?? 1 }). With a wide tree and concurrency: 8, the pool may open eight SSH sessions merely to list the tree. On servers with strict handshake or connection limits, that can produce avoidable login storms and transient connection failures before any file transfer starts.
This is separate from SFTP file transfer session multiplexing (the openSession() path).
Proposal
Bound source-directory collection to one reusable SFTP/SSH connection while retaining concurrent directory requests on the SSH channel, a bounded retry only for transient handshake errors, and current abort/progress/path/exclusion/symlink behavior. Alternative: expose a configurable, low default connection ceiling while preserving a higher optional one for servers that need it. This requires maintainer input because upstream currently uses a connection pool by design.
Acceptance:
listTree(..., {concurrency: 8}) opens no more than one SSH session in the proposed default mode while fetching multiple child directories correctly.
No retry on authentication/unsupported-key errors, and at most one retry on timeout/reset during initial handshake.
Abort and failure close the sole client; no leaked connections.
Existing exclude, symlink and progress contracts remain unchanged.
Tests reproduce the behavior and fail before implementation, then pass with the new one-connection mode.
Source-only fix; no DB migration, deployment credentials or Homelab-specific configuration.
A downstream product currently carries this behavior and wants to converge upstream. A separate issue will cover file transfer session multiplexing rather than mixing this source-tree lifecycle with uploads/downloads.
changed the title [-]SFTP source-tree collection can create a connection storm under directory concurrency[/-][+][Enhancement]: SFTP source-tree collection can create a connection storm under directory concurrency[/+]on Oct 10, 2026
Thanks for this one. You're right that the tree walk opens one connection for every folder it reads in parallel, so with 8 parallel transfers it can log in 8 times. I wouldn't call it a storm though. The number is capped by Parallel transfers on the connection (4 by default, 8 at most), and the transfers afterwards use exactly as many connections anyway. So for a server with a strict limit, lowering that setting is already the fix today, and a cheaper walk alone wouldn't change that.
That said, one connection for the walk is still worth it. I measured it against a real OpenSSH server with 249 folders and 20 ms of added latency:
So it's just as fast with a single login. I'll treat this as an enhancement rather than a bug, more in #187.
One more thing: please open issues through the GitHub web interface with one of the official templates (Bug Report, Enhancement, Feature Request etc.). Your issues skipped them, so no labels were set and the fields I usually ask for are missing. I'm happy about the contributions, but please also give me a bit of time to go through the open ones before opening new ones, and test against a real server where you can. A few of the claims in #182 and #184 didn't hold up against a real Samba server.
Activity
[-]SFTP source-tree collection can create a connection storm under directory concurrency[/-][+][Enhancement]: SFTP source-tree collection can create a connection storm under directory concurrency[/+]Skyfay commentedon Oct 10, 2026
Hey @X1pheR
Thanks for this one. You're right that the tree walk opens one connection for every folder it reads in parallel, so with 8 parallel transfers it can log in 8 times. I wouldn't call it a storm though. The number is capped by Parallel transfers on the connection (4 by default, 8 at most), and the transfers afterwards use exactly as many connections anyway. So for a server with a strict limit, lowering that setting is already the fix today, and a cheaper walk alone wouldn't change that.
That said, one connection for the walk is still worth it. I measured it against a real OpenSSH server with 249 folders and 20 ms of added latency:
devSo it's just as fast with a single login. I'll treat this as an enhancement rather than a bug, more in #187.
One more thing: please open issues through the GitHub web interface with one of the official templates (Bug Report, Enhancement, Feature Request etc.). Your issues skipped them, so no labels were set and the fields I usually ask for are missing. I'm happy about the contributions, but please also give me a bit of time to go through the open ones before opening new ones, and test against a real server where you can. A few of the claims in #182 and #184 didn't hold up against a real Samba server.
Thanks :)