Skip to content

Commit 3e43be0

Browse files
authored
Preserve SSH descendant filesystem access after bootstrap exit (#6645)
## Changes Register the SSH server process with the workspace-file and volume daemons. Check credentials and registration every minute, retry failures, and avoid updates when nothing changed. Bound daemon requests and keep SSH startup available when registration fails. ## Why SSH descendants need filesystem access when the bootstrap notebook exits while the server survives. Registration does not extend the server lifetime, preserve detached processes, or renew a fixed bootstrap credential. Document these limits and troubleshooting steps. ## Tests - SSH unit and acceptance suites, targeted race tests, full formatting, lint, repository checks, and builds for all supported platforms passed. - Dedicated compute passed SSH, PTY, a 1 MiB SCP round trip, concurrent sessions, and workspace/volume file operations. Registration recovery was observed, but a concurrent file stress test hit an unresolved missing-file error during recovery. After terminating the bootstrap process, file operations passed through 60 seconds before the driver proxy closed. _This PR was written with Codex._
1 parent fdb851c commit 3e43be0

14 files changed

Lines changed: 1111 additions & 0 deletions

File tree

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
* Preserve workspace-file and volume access for SSH server descendants when the bootstrap notebook exits and the server survives. ([#6645](https://github.com/databricks/cli/pull/6645))

‎experimental/ssh/FAILURE_MODES.md‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -156,3 +156,30 @@ The proxy-layer behaviors have unit tests that don't need a cluster:
156156
```shell
157157
go test ./experimental/ssh/...
158158
```
159+
160+
## Filesystem access after the bootstrap notebook exits
161+
162+
If SSH still connects but `/Workspace` or `/Volumes` operations return permission errors,
163+
check whether the tunnel server registered successfully. Its startup logs contain
164+
`Registered SSH server PID` for each filesystem daemon that accepted registration.
165+
`Failed to register SSH filesystem credentials` means at least one registration failed;
166+
the tunnel continues to run and retries in the background.
167+
168+
From an SSH session, read the process used for workspace-file authorization:
169+
170+
```sh
171+
cat /Workspace/.proc/self/metadata/pid
172+
ps -o pid,ppid,args -p <registered-pid>
173+
```
174+
175+
On dedicated compute, the reported PID should identify the tunnel server. If it identifies
176+
the bootstrap Python process, workspace-file access still depends on that process.
177+
The server checks this every 60 seconds and attempts to restore a lost registration.
178+
Volume registration is refreshed every 60 seconds, independently of this workspace-file check.
179+
An unavailable daemon produces a warning rather than preventing SSH startup.
180+
181+
This registration only covers the server and its descendants. A detached process that
182+
leaves that ancestry can still lose access. The compute can also terminate the server
183+
after the bootstrap notebook exits; registration does not extend either lifetime.
184+
An expired or revoked bootstrap credential cannot be repaired by re-registering it.
185+
In these cases, reconnect to start a new server after the existing server has stopped.

‎experimental/ssh/README.md‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,32 @@ file with `StrictHostKeyChecking yes`. Tunnel host keys therefore never land in
3636
`~/.ssh/known_hosts`, where a name - unique only within one workspace - would collide with an
3737
entry left by other compute.
3838

39+
### Filesystem access
40+
41+
On dedicated compute, the tunnel server registers its own process with the filesystem
42+
daemons. SSH sessions that remain descendants of that server can keep accessing
43+
`/Workspace` and `/Volumes` if the bootstrap notebook exits and the server survives.
44+
This does not keep the compute or server alive, preserve detached processes that leave
45+
the server's process tree, or restore them after a server restart.
46+
47+
The server checks its workspace-file registration and current credential every 60 seconds.
48+
It updates the workspace-file registration when the credential changes or the registered
49+
process is lost, and retries failed registrations. Unchanged workspace-file registrations
50+
do not generate updates. Volume registration is refreshed on every tick so it recovers
51+
after an independent UC-FUSE restart.
52+
The bootstrap supplies a fixed credential; checking it again cannot renew an expired or
53+
revoked credential. Registration entries may remain until compute shutdown. Matching
54+
the process start time prevents a reused PID from inheriting an old registration.
55+
56+
Registration failures are warnings. On compute where the daemon APIs are unavailable,
57+
including serverless, the tunnel starts and file access continues to depend on the
58+
bootstrap notebook as before. When the current-user lookup succeeds, registration also
59+
includes the workspace home directory so workspace files can prepopulate its directory cache.
60+
This does not change the session's working directory or filesystem permissions.
61+
See [filesystem troubleshooting](./FAILURE_MODES.md#filesystem-access-after-the-bootstrap-notebook-exits).
62+
3963
## Development
64+
4065
```shell
4166
./task build snapshot-release
4267
./cli ssh connect --cluster=<id> --releases-dir=./dist --debug # or modify ssh config accordingly
Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
// Package fuse registers the SSH server's credentials with the compute's filesystem daemons.
2+
package fuse
3+
4+
import (
5+
"bytes"
6+
"context"
7+
"encoding/json"
8+
"errors"
9+
"fmt"
10+
"net"
11+
"net/http"
12+
"strconv"
13+
"time"
14+
15+
"github.com/databricks/cli/libs/log"
16+
)
17+
18+
const (
19+
commandOrigin = "RemoteSshServer"
20+
requestTimeout = 2 * time.Second
21+
)
22+
23+
type request struct {
24+
APIToken string `json:"apiToken"`
25+
ProcStartTime uint64 `json:"procStartTime"`
26+
CommandOrigin string `json:"commandOrigin"`
27+
PIDNamespaceID uint32 `json:"namespaceId"`
28+
NotebookDir string `json:"notebookDir,omitempty"`
29+
AdditionalTags map[string]string `json:"additionalTags,omitempty"`
30+
}
31+
32+
type daemon struct {
33+
name string
34+
port int
35+
path string
36+
host string
37+
token string
38+
refreshUnchanged bool
39+
}
40+
41+
// Client tracks the last successful registration for each daemon. It must not be used concurrently.
42+
type Client struct {
43+
registration Registration
44+
hosts []string
45+
daemons []daemon
46+
http *http.Client
47+
readPID func() (int, error)
48+
}
49+
50+
// NewClient creates a client for the node-local filesystem daemons.
51+
func NewClient(r Registration) (*Client, error) {
52+
if r.PID <= 0 || r.PIDNamespaceID == 0 || r.StartTime == 0 {
53+
return nil, errors.New("FUSE registration requires a positive PID, PID namespace and process start time")
54+
}
55+
transport := http.DefaultTransport.(*http.Transport).Clone()
56+
// Credentials belong on this node, even when the server has HTTP_PROXY set.
57+
transport.Proxy = nil
58+
return &Client{
59+
registration: r,
60+
readPID: registeredPID,
61+
hosts: []string{"databricks.node.host.local", "node.host.local", "localhost"},
62+
daemons: []daemon{
63+
{name: "workspace files", port: 1021, path: fmt.Sprintf("/api/1/pid/%d", r.PID)},
64+
// UC-FUSE has no registration probe and its PUT does not invalidate WSFS's caches.
65+
{name: "volumes", port: 1015, path: fmt.Sprintf("/dbfs-fuse-api/1/pid/%d", r.PID), refreshUnchanged: true},
66+
},
67+
http: &http.Client{
68+
Timeout: requestTimeout,
69+
Transport: transport,
70+
CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse },
71+
},
72+
}, nil
73+
}
74+
75+
// Register refreshes volumes, updates changed credentials, and retries failed registrations.
76+
// Force also restores a lost workspace-files registration.
77+
func (c *Client) Register(ctx context.Context, token, userID, notebookDir string, force bool) error {
78+
if token == "" {
79+
return errors.New("cannot register an empty FUSE token")
80+
}
81+
body := request{
82+
APIToken: token,
83+
ProcStartTime: c.registration.StartTime,
84+
CommandOrigin: commandOrigin,
85+
PIDNamespaceID: c.registration.PIDNamespaceID,
86+
NotebookDir: notebookDir,
87+
}
88+
if userID != "" {
89+
body.AdditionalTags = map[string]string{"userId": userID}
90+
}
91+
encoded, err := json.Marshal(body)
92+
if err != nil {
93+
return fmt.Errorf("failed to encode FUSE registration: %w", err)
94+
}
95+
96+
var errs []error
97+
for i := range c.daemons {
98+
d := &c.daemons[i]
99+
if force {
100+
d.token = ""
101+
}
102+
if d.token == token && !d.refreshUnchanged {
103+
continue
104+
}
105+
if err := c.registerDaemon(ctx, d, encoded); err != nil {
106+
errs = append(errs, fmt.Errorf("%s: %w", d.name, err))
107+
continue
108+
}
109+
d.token = token
110+
log.Infof(ctx, "Registered SSH server PID %d with %s", c.registration.PID, d.name)
111+
}
112+
return errors.Join(errs...)
113+
}
114+
115+
func (c *Client) registerDaemon(ctx context.Context, d *daemon, body []byte) error {
116+
if d.host != "" {
117+
return c.put(ctx, d, d.host, body)
118+
}
119+
var errs []error
120+
for _, host := range c.hosts {
121+
if err := ctx.Err(); err != nil {
122+
return err
123+
}
124+
if err := c.put(ctx, d, host, body); err != nil {
125+
errs = append(errs, err)
126+
continue
127+
}
128+
// DNS resolution alone does not establish that this endpoint serves the daemon.
129+
d.host = host
130+
return nil
131+
}
132+
return errors.Join(errs...)
133+
}
134+
135+
func (c *Client) put(ctx context.Context, d *daemon, host string, body []byte) error {
136+
url := "http://" + net.JoinHostPort(host, strconv.Itoa(d.port)) + d.path
137+
req, err := http.NewRequestWithContext(ctx, http.MethodPut, url, bytes.NewReader(body))
138+
if err != nil {
139+
return fmt.Errorf("failed to build FUSE request: %w", err)
140+
}
141+
req.Header.Set("Content-Type", "application/json")
142+
resp, err := c.http.Do(req)
143+
if err != nil {
144+
return fmt.Errorf("failed to register with %s: %w", url, err)
145+
}
146+
defer resp.Body.Close()
147+
if resp.StatusCode != http.StatusOK {
148+
// Error bodies may echo credentials from the request.
149+
return fmt.Errorf("%s returned HTTP %d", url, resp.StatusCode)
150+
}
151+
return nil
152+
}

0 commit comments

Comments
 (0)