Skip to content

Commit d96f5bf

Browse files
authored
🔬 Spike: run Cloudflare Worker Shell and Worker JavaScript natively in Deno (#351) (#353)
* 🔬 Run Worker Shell and Worker JavaScript natively over Deno-local DOFS (#351) Both of Cloudflare Computer's imperative backends execute from one compiled Deno artifact against the #349 workspace, with no workerd, Wrangler, Docker, or second JavaScript runtime. The shell reuses Cloudflare's filesystem adapter byte-identical and drops the Dynamic Worker transport, since the workspace is in-process. The JavaScript backend is a port of WorkspaceRuntimeLoader: an in-memory module graph supplied as blob URLs with rewritten specifiers, run in a Deno Worker, with the workspace capability crossing as async postMessage RPC and nothing materialized to disk. Nine scenarios cover bidirectional coherence across restarts, workspace identity isolation, shell semantics, module graphs with dependencies, committed-state coherence, and explicit refusal of operations the host does not install. * 📝 Record both backend verdicts and the two-question recommendation (#351) Worker Shell is include-initially-scoped: it runs natively over the Deno-local workspace, reuses Cloudflare's adapter byte-identical, and its containment was measured — around fifty escape attempts with zero leaks and no native-execution path. Its one hazard is availability, not capability: a CPU-bound script starves the host event loop, so the boundary recommendation is a Deno Worker. Worker JavaScript is defer: the whole contract works, including an in-memory module graph with no materialization, but isolation holds only in a compiled artifact and CPU-bound user code cannot be preempted. The missing Deno primitive is named precisely. COMPARISON.md answers the two topology questions separately and states that bundled workerd is not necessary for this scope. --------- Co-authored-by: Taras Mankovski <[email protected]>
1 parent f3f7ad5 commit d96f5bf

23 files changed

Lines changed: 4163 additions & 4 deletions

‎deno.json‎

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,12 @@
11
{
2-
"workspace": ["packages/*", "site"],
3-
"exclude": ["scripts/tests/fixtures", "spikes"],
2+
"workspace": [
3+
"packages/*",
4+
"site"
5+
],
6+
"exclude": [
7+
"scripts/tests/fixtures",
8+
"spikes"
9+
],
410
"nodeModulesDir": "auto",
511
"lock": {
612
"frozen": true
@@ -60,6 +66,9 @@
6066
"analyze:dispatch": "deno run --allow-all packages/cli/src/deno.ts run .reviews/DispatchRepoAnalysis.md --component-dir .reviews/components --component-dir .reviews/policies --component-dir packages/core/components -j .reviews/journal.dispatch.jsonl",
6167
"spike:349": "deno task --cwd spikes/349-dofs all",
6268
"spike:349:build": "deno task --cwd spikes/349-dofs build",
63-
"spike:349:test": "deno task --cwd spikes/349-dofs test"
69+
"spike:349:test": "deno task --cwd spikes/349-dofs test",
70+
"spike:351": "deno task --cwd spikes/351-worker-backends all",
71+
"spike:351:build": "deno task --cwd spikes/351-worker-backends build",
72+
"spike:351:test": "deno task --cwd spikes/351-worker-backends test"
6473
}
65-
}
74+
}
Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
# Spike 351: Worker Shell and Worker JavaScript natively in Deno
2+
3+
Tests whether Cloudflare Computer's two imperative execution backends can run
4+
over the Deno-local DOFS Workspace from #349 — with no `workerd`, Wrangler,
5+
Docker, or other JavaScript runtime. Both are **optional imperative extensions**
6+
layered on the declarative durable core, not prerequisites for it; the spike
7+
supplies evidence for #346, not a product decision. Verdicts and measurements
8+
are in [evidence/EVIDENCE.md](evidence/EVIDENCE.md).
9+
10+
## Run it
11+
12+
From the repository root:
13+
14+
```bash
15+
deno task spike:351 # vendor-build + compile + test
16+
```
17+
18+
The compiled proof takes a workspace database path and one operation, so each
19+
invocation is a full host restart against the same durable state:
20+
21+
```bash
22+
dist/proof /tmp/ws.db fs-mkdir /workspace
23+
dist/proof /tmp/ws.db fs-write /workspace/a.txt alpha # host capability
24+
dist/proof /tmp/ws.db shell "cat a.txt" # Worker Shell
25+
dist/proof /tmp/ws.db js 'export default () => 42' # Worker JavaScript
26+
```
27+
28+
## Layout
29+
30+
- `vendor/worker-shell/adapter.ts` — Cloudflare's `WorkspaceFsAdapter`,
31+
byte-identical (MIT; see
32+
[PROVENANCE.md](vendor/worker-shell/PROVENANCE.md)). It is the whole reason
33+
the shell ports: zero `cloudflare:*` imports, and its filesystem contract is
34+
a structural interface rather than an RPC stub.
35+
- `host/shell.ts` — the shell backend with Cloudflare's transport removed:
36+
no `WorkerEntrypoint`, no Workers RPC, no Dynamic Worker loader, because the
37+
workspace is in-process.
38+
- `host/loader.ts` — a Deno port of `WorkspaceRuntimeLoader`: an in-memory
39+
module graph supplied as blob URLs with rewritten specifiers, run in a Deno
40+
Worker, with the filesystem capability crossing as async postMessage RPC.
41+
- `host/workspace.ts`, `host/workspace-fs.ts` — the DOFS workspace and the
42+
small shim supplying the three stub-only methods the adapter expects.
43+
- `tests/` — the scenario suite; `evidence/probes/` — the full probe ledgers
44+
with every command, escape attempt, and measurement.
45+
46+
The `@cloudflare/dofs` build is shared with the #349 spike rather than vendored
47+
twice: `deno task --cwd spikes/351-worker-backends vendor` builds it there and
48+
links it here. This directory stays outside the workspace via the root
49+
`deno.json` `exclude`, so the four verification gates keep their scope.
Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
import { main } from "effection";
2+
import { exec } from "@effectionx/process";
3+
4+
const here = new URL("./", import.meta.url).pathname;
5+
6+
main(function* () {
7+
yield* exec(Deno.execPath(), {
8+
arguments: [
9+
"compile",
10+
"--allow-all",
11+
"--unstable-worker-options",
12+
"--node-modules-dir=manual",
13+
"--include",
14+
"host/loader.ts",
15+
"--output",
16+
"dist/proof",
17+
"host/main.ts",
18+
],
19+
cwd: here,
20+
}).expect();
21+
console.log("built dist/proof");
22+
});
Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
{
2+
"exclude": [
3+
"vendor",
4+
"dist",
5+
"node_modules"
6+
],
7+
"nodeModulesDir": "manual",
8+
"tasks": {
9+
"all": "deno task vendor && deno task build && deno task test",
10+
"vendor": "deno run --allow-all vendor.ts",
11+
"build": "deno run --allow-all build.ts",
12+
"test": "deno test --allow-all tests/",
13+
"check": "deno check ."
14+
},
15+
"scopes": {
16+
"./vendor/": {
17+
"@cloudflare/dofs": "./host/dofs-types.ts"
18+
}
19+
}
20+
}
Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
# Comparison and recommendation for #346
2+
3+
#351 completes the evidence set: #347 measured bundled `workerd`, #349 measured
4+
the Deno-local DOFS Workspace, and this spike measures what imperative
5+
execution the Deno-local topology can carry on its own. The topology decision
6+
has two separate questions.
7+
8+
## Question 1 — is Deno-local DOFS sufficient for the declarative durable core?
9+
10+
**Yes, on the evidence available.** The core the workflow needs is a durable,
11+
coherent, identity-scoped filesystem, and #349 proved each property directly:
12+
Cloudflare's unmodified DOFS schema over a file-backed `node:sqlite` adapter,
13+
surviving full process restarts (including deletions and replacements),
14+
isolating workspaces by database path, refusing a newer on-disk schema rather
15+
than recreating it, and checkpointing to a single-file artifact on close — at
16+
110 MB and 0.08 s per operation, in one process with no supervision surface.
17+
18+
The declarative components map onto primitives that already exist in that
19+
layer: `<Dir>` onto `mkdir`, `<File>` onto `writeFile`/`readFile`, and
20+
`<Branch>`/`<Commit>` onto DOFS's own git support, which Cloudflare drives
21+
through the same filesystem (their `git` shell command is a thin shim over it).
22+
This spike does not implement those components, per its scope; it confirms the
23+
substrate exposes the operations they need, and that a host capability and an
24+
imperative backend observe the same committed state in both directions
25+
**[test]**.
26+
27+
The one caveat carried forward from #349: real FUSE — needed only for arbitrary
28+
native subprocesses, which `xmd workflow` does not require — is blocked
29+
in-process by a Deno uv-polyfill gap, and the userspace shim is development-only.
30+
Neither bears on the declarative core.
31+
32+
## Question 2 — which imperative capabilities can safely layer on top?
33+
34+
**Worker Shell: include initially, scoped.** It runs natively, reuses
35+
Cloudflare's adapter byte-identical, and its containment was measured rather
36+
than assumed — ~50 escape attempts, zero leaks, no native-execution path at
37+
all. Two conditions: run it inside a Deno Worker so a CPU-bound script cannot
38+
starve the host (the measured hazard: 6.4 s of wall time with the abort timer
39+
never firing), and describe it accurately as a workspace-scoped interpreter
40+
with 83 built-ins, never as native command execution.
41+
42+
**Worker JavaScript: defer.** The full contract works — in-memory module graph
43+
with no materialization, live-streamed output, capability access to DOFS,
44+
returned values, errors, timeouts, cancellation — and it is genuinely useful.
45+
But isolation holds only in a compiled artifact (a `deno run` host lets a
46+
"locked" worker import `jsr:`/`npm:`), and CPU-bound user code cannot be
47+
preempted at all. Since it is an optional extension over a declarative core,
48+
the cost of shipping it now exceeds its value; revisit when Deno offers a CPU
49+
or wall budget on `new Worker(...)`, or accept an out-of-process worker and its
50+
startup cost.
51+
52+
## Is bundled `workerd` still necessary?
53+
54+
**Not for this scope.** #347's decisive advantage was that it runs Cloudflare's
55+
real backends; this spike shows both of those execution models also run
56+
natively in Deno over the same workspace, at a fraction of the weight — 139.7 MB
57+
carrying *both* backends and DOFS, versus 191 MB for the workerd host, with
58+
operations in tens of milliseconds rather than tenths of a second, and no child
59+
process to supervise.
60+
61+
`workerd` remains the only option for the things this topology deliberately
62+
does not provide: Cloudflare Containers, the hosted Workers surface (KV, R2,
63+
Queues, Durable Object identity), and behavior guaranteed to match deployed
64+
Cloudflare. If those enter scope, #347's evidence stands ready; they are not
65+
required by the declarative workflow core.
66+
67+
## Recommendation
68+
69+
1. **Adopt** the Deno-local DOFS topology for the declarative durable workflow
70+
core.
71+
2. **Include** Worker Shell initially, scoped: inside a Deno Worker, described
72+
as a workspace-scoped interpreter, with unsupported operations refused
73+
explicitly.
74+
3. **Defer** Worker JavaScript until the preemption gap is resolved or an
75+
out-of-process cost is accepted.
76+
4. **Omit** bundled `workerd` from the local topology, keeping #347 as the
77+
documented path if hosted-Cloudflare fidelity or Containers become
78+
requirements.
79+
80+
The final selection is recorded on #346; these spikes supply the evidence, not
81+
the decision.

0 commit comments

Comments
 (0)