Skip to content

Commit 8e3b76f

Browse files
committed
ssh: keep server running while detached work remains
1 parent 54f4515 commit 8e3b76f

15 files changed

Lines changed: 268 additions & 68 deletions

File tree

‎experimental/ssh/README.md‎

Lines changed: 28 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -76,32 +76,34 @@ By default nothing outlives the session: when the last client disconnects, the s
7676
down after `--shutdown-delay` and the bootstrap notebook sweeps every process it parents,
7777
including work that was deliberately detached with `tmux`, `setsid` or `nohup`.
7878

79-
`databricks ssh connect --cluster=<id> --keep-detached-processes` changes that. On teardown
80-
the tunnel terminates only its own process group - the server and its `sshd` children - and
81-
then holds the job run open for as long as any detached process is still running. Things to
82-
know before using it:
83-
84-
- **It holds the cluster up.** A `RUNNING` job run suppresses autotermination, so the cluster
85-
keeps accruing DBUs until the last detached process exits. The bound is the run's own
86-
lifetime, `--server-timeout` (24h by default), which is therefore the knob to reach for when
87-
the cost is what matters; multi-day work still belongs in Jobs/DABs. Note also that
88-
reconnecting starts a new run rather than rejoining the one being held open, so each session
89-
with live detached work leaves its own run behind.
90-
- **Releasing the run is not the end of the work.** Once the hold is released the cluster
91-
starts its own autotermination countdown, which is the last thing that reaps survivors -
92-
detached work does not count as cluster activity, however busy it is.
93-
- **The notebook has to stay alive, not just the process.** Workspace filesystem access is
94-
authorized by walking the live process tree for a registered ancestor, and the bootstrap
95-
notebook is that ancestor. A detached process that outlives it keeps `/dbfs` and REST API
96-
access but loses `/Workspace` and `/Volumes` with `EPERM` - which is why the group-scoped
97-
teardown is tied to holding the run open and not enabled on its own. Work that finishes
98-
while the run is held never sees this; work still running when `--server-timeout` expires
99-
does, and from there every workspace path fails, including in a new window opened inside a
100-
surviving `tmux`, because the `tmux` server - not the shell - is what lost its registered
101-
ancestor. So size `--server-timeout` to the work you intend to leave behind.
102-
103-
Dedicated clusters only. On serverless the container is torn down with the run, so survivors
104-
die regardless and the flag is rejected.
79+
`--keep-detached-processes` prevents idle shutdown while detached work is still running.
80+
It works with dedicated clusters and serverless compute:
81+
82+
```sh
83+
databricks ssh connect --cluster=<id> --keep-detached-processes
84+
databricks ssh connect --name=my-session --keep-detached-processes
85+
```
86+
87+
When `--shutdown-delay` elapses with no SSH clients, the server checks for detached processes.
88+
If it finds any, it stays available for reconnection and checks again every 15 seconds.
89+
Once no detached work remains, it shuts down. A reconnect cancels the pending check; after
90+
the last client disconnects again, the full `--shutdown-delay` applies again. If the process
91+
tree cannot be read, the server postpones shutdown and retries rather than risking the work.
92+
93+
- **It keeps compute running.** On dedicated clusters the active job also
94+
suppresses autotermination. An idle `tmux` session counts as detached work even after the
95+
command in its pane finishes; close the session when you no longer need it.
96+
- **Reconnect to the same session.** Use the same cluster ID or serverless connection name.
97+
The existing server and notebook remain alive, so a reconnect can attach to the original
98+
`tmux` session rather than creating a replacement run.
99+
- **The maximum lifetime still applies.** `--server-timeout` (24h by default) bounds the job
100+
from its start, regardless of connected clients or detached work. The flag does not
101+
survive a job cancellation, notebook restart, or compute termination. Multi-day work
102+
belongs in Jobs/DABs.
103+
- **The notebook must remain alive too.** It anchors the detached processes' workspace
104+
filesystem access. If the SSH server exits for another reason, the bootstrap still
105+
preserves detached work and holds the run open, as before. This fallback preserves work,
106+
not SSH access: it does not restart the server inside that run.
105107

106108
`databricks ssh setup` takes the same flag and bakes it into the host's `ProxyCommand`, so
107109
`ssh <name>` sessions ask for it too. That is the only place a configured host can set it: the

‎experimental/ssh/cmd/connect.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -67,7 +67,7 @@ Connect to a dedicated cluster:
6767
cmd.Flags().IntVar(&maxClients, "max-clients", defaultMaxClients, "Maximum number of SSH clients")
6868
cmd.Flags().DurationVar(&serverTimeout, "server-timeout", defaultServerTimeout, "Maximum lifetime of the SSH server; it is terminated after this duration even if clients are connected")
6969
cmd.Flags().BoolVar(&autoStartCluster, "auto-start-cluster", true, "Automatically start the cluster if it is not running")
70-
cmd.Flags().BoolVar(&keepDetachedProcesses, "keep-detached-processes", false, "Keep processes detached from the SSH session (tmux, setsid, nohup) running after the tunnel shuts down. Holds the cluster up until they exit or --server-timeout elapses (dedicated clusters only)")
70+
cmd.Flags().BoolVar(&keepDetachedProcesses, "keep-detached-processes", false, "Keep the SSH server and detached processes (tmux, setsid, nohup) running while detached work remains, bounded by --server-timeout")
7171

7272
cmd.Flags().StringVar(&connectionName, "name", "", "Connection name to reuse across sessions (serverless only)")
7373
cmd.Flags().StringVar(&accelerator, "accelerator", "", "Serverless GPU accelerator type (GPU_1xA10 or GPU_8xH100)")

‎experimental/ssh/cmd/server.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ and proxies them to local SSH daemon processes.`,
4646
cmd.Flags().StringVar(&version, "version", "", "Client version of the Databricks CLI")
4747
cmd.Flags().BoolVar(&serverless, "serverless", false, "Enable serverless mode for Jupyter initialization")
4848
cmd.Flags().StringVar(&usagePolicyID, "usage-policy-id", "", "Usage policy ID the job was submitted with")
49-
cmd.Flags().BoolVar(&keepDetachedProcesses, "keep-detached-processes", false, "Whether the bootstrap notebook holds the job run open for detached processes after the server exits")
49+
cmd.Flags().BoolVar(&keepDetachedProcesses, "keep-detached-processes", false, "Keep the SSH server running while detached processes are alive")
5050

5151
cmd.PreRunE = func(cmd *cobra.Command, args []string) error {
5252
// The server can be executed under a directory with an invalid bundle configuration.

‎experimental/ssh/cmd/setup.go‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,7 @@ For serverless connections, use ` + "`databricks ssh connect`" + ` (no setup ste
3838
cmd.Flags().DurationVar(&shutdownDelay, "shutdown-delay", defaultShutdownDelay, "SSH server will terminate after this delay if there are no active connections")
3939
cmd.Flags().IntVar(&maxClients, "max-clients", defaultMaxClients, "Maximum number of SSH clients")
4040
cmd.Flags().DurationVar(&serverTimeout, "server-timeout", defaultServerTimeout, "Maximum lifetime of the SSH server; it is terminated after this duration even if clients are connected")
41-
cmd.Flags().BoolVar(&keepDetachedProcesses, "keep-detached-processes", false, "Keep processes detached from the SSH session (tmux, setsid, nohup) running after the tunnel shuts down. Holds the cluster up until they exit or --server-timeout elapses")
41+
cmd.Flags().BoolVar(&keepDetachedProcesses, "keep-detached-processes", false, "Keep the SSH server and detached processes (tmux, setsid, nohup) running while detached work remains, bounded by --server-timeout")
4242
cmd.Flags().BoolVar(&autoApprove, "auto-approve", false, "Skip confirmation prompts, recreating existing SSH host configs without asking")
4343

4444
cmd.PreRunE = func(cmd *cobra.Command, args []string) error {

‎experimental/ssh/internal/client/client.go‎

Lines changed: 4 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -130,11 +130,7 @@ type ClientOptions struct {
130130
AutoApprove bool
131131
// Id of the usage policy to use for the serverless SSH server job. Serverless only.
132132
UsagePolicyID string
133-
// Whether the bootstrap notebook holds the job run open after the SSH server shuts down,
134-
// for as long as processes detached from the session (tmux, setsid, nohup) keep running,
135-
// so they keep their workspace filesystem access intact. False, the default, keeps
136-
// today's behaviour: the run ends with the server and nothing outlives it. The run's own
137-
// timeout (--server-timeout) is what bounds the hold. Dedicated clusters only.
133+
// Whether detached processes prevent idle shutdown of the SSH server. Bounded by --server-timeout.
138134
KeepDetachedProcesses bool
139135
}
140136

@@ -148,11 +144,6 @@ func (o *ClientOptions) Validate() error {
148144
if o.UsagePolicyID != "" && o.ClusterID != "" {
149145
return errors.New("--usage-policy-id flag can only be used with serverless compute (--name flag)")
150146
}
151-
// On serverless the container goes away with the run, so nothing survives the server
152-
// however long the notebook holds the run open.
153-
if o.KeepDetachedProcesses && o.ClusterID == "" {
154-
return errors.New("--keep-detached-processes flag can only be used with a dedicated cluster (--cluster flag)")
155-
}
156147
if o.Accelerator != "" && o.Accelerator != "GPU_1xA10" && o.Accelerator != "GPU_8xH100" {
157148
return fmt.Errorf("invalid accelerator value: %q, expected %q or %q", o.Accelerator, "GPU_1xA10", "GPU_8xH100")
158149
}
@@ -261,9 +252,9 @@ func (o *ClientOptions) ToProxyCommand() (string, error) {
261252
} else {
262253
proxyCommand = fmt.Sprintf("%q ssh connect --proxy --cluster=%s --auto-start-cluster=%t --shutdown-delay=%s",
263254
executablePath, o.ClusterID, o.AutoStartCluster, o.ShutdownDelay.String())
264-
if o.KeepDetachedProcesses {
265-
proxyCommand += " --keep-detached-processes"
266-
}
255+
}
256+
if o.KeepDetachedProcesses {
257+
proxyCommand += " --keep-detached-processes"
267258
}
268259

269260
// Both of these are fixed when the server job is submitted, and for a host configured by

‎experimental/ssh/internal/client/client_test.go‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -122,9 +122,8 @@ func TestValidate(t *testing.T) {
122122
opts: client.ClientOptions{ConnectionName: "my-conn", UsagePolicyID: "pol-1"},
123123
},
124124
{
125-
name: "keep detached processes with serverless",
126-
opts: client.ClientOptions{ConnectionName: "my-conn", KeepDetachedProcesses: true},
127-
wantErr: "--keep-detached-processes flag can only be used with a dedicated cluster (--cluster flag)",
125+
name: "keep detached processes with serverless",
126+
opts: client.ClientOptions{ConnectionName: "my-conn", KeepDetachedProcesses: true},
128127
},
129128
{
130129
name: "keep detached processes with cluster ID",
@@ -350,6 +349,11 @@ func TestToProxyCommand(t *testing.T) {
350349
opts: client.ClientOptions{ClusterID: "abc-123", KeepDetachedProcesses: true, ShutdownDelay: 5 * time.Minute},
351350
want: quoted + " ssh connect --proxy --cluster=abc-123 --auto-start-cluster=false --shutdown-delay=5m0s --keep-detached-processes",
352351
},
352+
{
353+
name: "serverless keeping detached processes",
354+
opts: client.ClientOptions{ConnectionName: "my-conn", KeepDetachedProcesses: true, ShutdownDelay: 5 * time.Minute},
355+
want: quoted + " ssh connect --proxy --name=my-conn --shutdown-delay=5m0s --keep-detached-processes",
356+
},
353357
{
354358
name: "with metadata",
355359
opts: client.ClientOptions{ClusterID: "abc-123", ServerMetadata: "user,2222,abc-123"},

‎experimental/ssh/internal/client/ssh-server-bootstrap.py‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -275,8 +275,7 @@ def run_ssh_server():
275275
if usage_policy_id:
276276
server_args.append(f"--usage-policy-id={usage_policy_id}")
277277

278-
# The server does not hold the run open itself; it uses this to persist the mode for
279-
# reconnects and to warn about detached work it is about to leave behind when the mode is off.
278+
# The server uses this to defer idle shutdown while detached work is running.
280279
if keep_detached_processes:
281280
server_args.append("--keep-detached-processes")
282281

‎experimental/ssh/internal/proxy/connections.go‎

Lines changed: 12 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ func NewConnectionsManager(maxClients int, shutdownDelay time.Duration) *Connect
2424
connections: make(map[string]*proxyConnection),
2525
TimedOut: make(chan bool),
2626
}
27-
cm.startShutdownTimer()
27+
cm.startShutdownTimer(shutdownDelay)
2828
return cm
2929
}
3030

@@ -61,7 +61,7 @@ func (cm *ConnectionsManager) Remove(id string) {
6161
cm.removeConnection(id)
6262
count := cm.Count()
6363
if count <= 0 {
64-
cm.startShutdownTimer()
64+
cm.startShutdownTimer(cm.shutdownDelay)
6565
}
6666
}
6767

@@ -71,13 +71,21 @@ func (cm *ConnectionsManager) removeConnection(id string) {
7171
delete(cm.connections, id)
7272
}
7373

74-
func (cm *ConnectionsManager) startShutdownTimer() {
74+
func (cm *ConnectionsManager) ExtendIdleTimeout(delay time.Duration) {
75+
cm.connectionsMu.Lock()
76+
defer cm.connectionsMu.Unlock()
77+
if len(cm.connections) == 0 {
78+
cm.startShutdownTimer(delay)
79+
}
80+
}
81+
82+
func (cm *ConnectionsManager) startShutdownTimer(delay time.Duration) {
7583
cm.shutdownTimerMu.Lock()
7684
defer cm.shutdownTimerMu.Unlock()
7785
if cm.shutdownTimer != nil {
7886
cm.shutdownTimer.Stop()
7987
}
80-
cm.shutdownTimer = time.AfterFunc(cm.shutdownDelay, func() {
88+
cm.shutdownTimer = time.AfterFunc(delay, func() {
8189
cm.TimedOut <- true
8290
})
8391
}
Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
package proxy_test
2+
3+
import (
4+
"testing"
5+
"testing/synctest"
6+
"time"
7+
8+
"github.com/databricks/cli/experimental/ssh/internal/proxy"
9+
"github.com/stretchr/testify/assert"
10+
"github.com/stretchr/testify/require"
11+
)
12+
13+
func TestConnectionsManagerExtendIdleTimeout(t *testing.T) {
14+
synctest.Test(t, func(t *testing.T) {
15+
connections := proxy.NewConnectionsManager(1, time.Minute)
16+
connections.ExtendIdleTimeout(time.Second)
17+
time.Sleep(time.Second)
18+
assert.True(t, <-connections.TimedOut)
19+
require.True(t, connections.TryAdd("connected", nil))
20+
connections.ExtendIdleTimeout(time.Second)
21+
time.Sleep(time.Minute)
22+
synctest.Wait()
23+
select {
24+
case <-connections.TimedOut:
25+
t.Fatal("extended timeout fired while a client was connected")
26+
default:
27+
}
28+
})
29+
}
Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,13 @@
11
package server
22

33
var (
4-
WorkspaceToken = workspaceToken
5-
FuseUserInfo = fuseUserInfo
4+
WorkspaceToken = workspaceToken
5+
FuseUserInfo = fuseUserInfo
6+
WaitForIdleShutdown = waitForIdleShutdown
7+
ProcWithDetachedWork = procWithDetachedWork
8+
)
9+
10+
const (
11+
DetachedProcessCheckInterval = detachedProcessCheckInterval
12+
TestServerPid = testServerPid
613
)

0 commit comments

Comments
 (0)