Skip to content

feat(codex): run remote workspaces without splitting app-server#101326

Draft
steipete wants to merge 1 commit into
mainfrom
codex/remote-execution-environment
Draft

feat(codex): run remote workspaces without splitting app-server#101326
steipete wants to merge 1 commit into
mainfrom
codex/remote-execution-environment

Conversation

@steipete

@steipete steipete commented Jul 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Closes #101282

Related: #101221, #94203

AI-assisted: yes. I understand the topology, compatibility boundaries, and fail-closed behavior in this change. Agent transcript: omitted.

What Problem This Solves

Remote Codex workspaces currently require operators to split Codex app-server from Gateway. That separates lifecycle, auth, hooks, thread state, local MCP integration, and filesystem ownership across hosts while depending on Codex's experimental app-server WebSocket transport.

Why This Change Was Made

This preview keeps Gateway and its managed Codex app-server together over stdio. Only Codex filesystem/process execution moves to one authenticated registry-backed exec-server environment. This matches Codex's supported control-plane ownership while using its existing Noise-authenticated remote executor path.

The implementation:

  • validates an HTTPS (or loopback) registry, SecretRef-compatible credentials, an absolute POSIX remoteWorkspaceRoot, and one explicit environment id;
  • clears inherited executor-routing variables, then injects the validated target only into the managed app-server child;
  • forces unified exec and disables remote hooks/notify, unsupported local stdio MCP servers, proxying, sandbox combinations, and Windows hosts;
  • checks managed config requirements and hooks before a remote thread mutation, and fails closed when local policy cannot be enforced remotely;
  • forks existing Codex rollouts when execution topology changes, preserves the source binding until replacement succeeds, and projects only history newer than the source coverage watermark;
  • keeps bounded media work local and prevents remote workspace paths from being persisted as Gateway-local paths; and
  • stores only durable routing fingerprints/bindings, never raw registry credentials.

This is intentionally one target, with no pool selection or failover in OpenClaw. The infrastructure remains responsible for registry availability and executor registration.

Security boundary: child-environment filtering reduces accidental credential leakage, but it is not isolation from code running as the same OS user as the exec-server. Such code can inspect the exec-server parent process. Executors therefore need a dedicated user/container/process boundary and narrowly scoped, rotatable credentials.

User Impact

Operators can keep one canonical Gateway/app-server control plane while running Codex shell and filesystem work on a different machine. The feature is opt-in under plugins.entries.codex.config.appServer.experimental.remoteExecution; existing local stdio and explicit remote app-server configurations are unchanged.

Thread resume, explicit rebind, command-driven resume/model changes, side questions, and conversation transfer now preserve the correct execution topology and history without replaying native-owned context.

Evidence

Final head: e0845291b548d93b2bf64eae3a68ebea673afaf8, rebased onto current origin/main and reduced from four follow-up commits to one commit.

  • Full Codex Docker suite on the rebased tree: 96 files / 2,025 tests passed in four one-worker shards (601 + 355 + 366 + 703).

    docker buildx build --target build --load -t openclaw:test-final .
docker run --rm openclaw:test-final node scripts/run-vitest.mjs run --config test/vitest/vitest.extension-codex.config.ts extensions/codex --maxWorkers 1 --shard 1/4
# repeated for shards 2/4, 3/4, and 4/4

  • Exact production Dockerfile build passed, including build:docker, CLI import guards, runtime post-build, plugin assets, package import closure, and Control UI. Runtime smoke returned OpenClaw 2026.6.11.

    docker buildx build --load -t openclaw:remote-final .
docker run --rm openclaw:remote-final node openclaw.mjs --version

  • Live Docker user-path proof used separate Gateway/app-server and Codex 0.142.5 exec-server containers with an authenticated registry/opaque Noise relay and a real gpt-5.5 turn. Resume worked; a later turn executed in /remote-workspace; direct child-secret probes were absent; the Gateway decoy workspace remained unchanged; registry register/connect/validate occurred; encrypted relay frames contained no plaintext probe. Result: 1 passed, 1 intentionally skipped in 202.12s.

  • A subsequent managed-account live run stopped before thread start when cloud PreToolUse hooks were present, proving the new fail-closed managed-hook guard. The positive relay run preceded that guard; the final Docker suite covers the guard and all resulting topology/history changes.

  • Crabbox pool run run_10e2c67bc313: 7 focused files / 426 tests passed. The broker became unavailable for later runs, so the final complete suite, production build, and live proof used Docker as the requested fallback.

  • oxfmt, direct oxlint, git diff --check, plugin manifest parsing, changelog attribution, docs MDX validation (699 files), and docs link audit (5,754 links, 0 broken) passed.

  • Two fresh autoreviews reported no accepted/actionable findings; the final review specifically confirmed that isolating the history-fork regression from unrelated dynamic-tool construction preserves the behavior under test.

Dependency contract checked directly at exact Codex tag rust-v0.142.5 (26de83050b20f7e0ee211b9739e52ae00ce8032a):

@openclaw-barnacle openclaw-barnacle Bot added docs Improvements or additions to documentation extensions: codex size: XL maintainer Maintainer-authored PR labels Jul 7, 2026
@clawsweeper

clawsweeper Bot commented Jul 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: found issues before merge. Reviewed July 7, 2026, 6:32 AM ET / 10:32 UTC.

Summary
The PR adds experimental Codex appServer.experimental.remoteExecution config and runtime handling so a Gateway-managed stdio app-server can route filesystem and process execution to one authenticated registry-backed exec-server environment, with docs, live Docker harness support, and tests.

PR surface: Source +1122, Tests +2051, Docs +191, Other +50. Total +3414 across 44 files.

Reproducibility: not applicable. This PR adds a new experimental Codex remote-execution mode, and current main does not expose the registry-backed remoteExecution surface. The PR body now supplies live proof for the new path rather than a current-main bug reproduction.

Review metrics: 2 noteworthy metrics.

  • Experimental config surface: 1 object added, 4 fields added, 2 SecretInput paths added. The new config and secret paths affect operator setup, credential handling, and compatibility before merge.
  • Persistent binding identity: 1 serialized field added. Persisting remoteExecutionFingerprint changes how existing Codex session bindings rotate and resume across topology changes.

Stored data model
Persistent data-model change detected: persistent cache schema: extensions/codex/src/app-server/shared-client.test.ts, persistent cache schema: extensions/codex/src/app-server/side-question.test.ts, persistent cache schema: extensions/codex/src/app-server/side-question.ts, persistent cache schema: extensions/codex/src/conversation-binding.ts, serialized state: extensions/codex/src/app-server/session-binding.test.ts, serialized state: extensions/codex/src/app-server/session-binding.ts. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #101282
Summary: This PR is the implementation candidate for the canonical remote-execution design issue; the merged 0.142 floor and remote WebSocket PRs are related prerequisites/overlaps rather than replacements.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • [P2] Remove the CHANGELOG.md entry before merge.
  • Get explicit maintainer security/product acceptance before undrafting or landing.

Risk before merge

  • [P1] The new experimental config deliberately fails closed for remote WebSocket app-server, sandbox exec-server, network proxy, active OpenClaw sandbox, Computer Use, local stdio MCP servers, and managed hook/config combinations, so operator upgrade expectations need maintainer acceptance.
  • [P1] Thread binding identity now includes remote execution topology and forks history when the topology changes; that is the right direction, but a mistake here can mis-associate session state or lose continuity across resume/rebind flows.
  • [P1] The PR routes process and filesystem execution through authenticated remote environments and injects registry credentials into the managed app-server child environment, so the executor isolation and credential-scoping model remains a maintainer-owned security boundary.
  • [P1] The PR is still a protected draft with the maintainer label; even with sufficient proof, this should not merge without explicit maintainer security/product review.

Maintainer options:

  1. Security review before undraft (recommended)
    Have a maintainer explicitly accept the executor isolation and registry-credential model, then remove the release-owned changelog entry before merge.
  2. Accept the preview risk as documented
    Maintainers may choose to merge the opt-in preview with its fail-closed operator limits if they are comfortable owning the experimental boundary.
  3. Pause this branch behind the design issue
    If the boundary is too broad for this release train, keep the canonical issue open and pause or close this draft until the scope narrows.

Next step before merge

  • [P2] This protected draft needs maintainer security/product review rather than an automation-owned repair; the only mechanical cleanup found is the non-blocking changelog entry.

Maintainer decision needed

  • Question: Should this PR proceed as the experimental Codex remote-execution preview with the documented fail-closed limits and executor isolation model, or should the canonical issue stay open until maintainers narrow or defer that boundary?
  • Rationale: The branch is technically plausible and now has live proof, but it adds registry credentials, remote process/filesystem execution, and session-binding rotation behavior that automation cannot accept as product/security direction.
  • Likely owner: steipete — steipete is assigned and has the strongest current history on the Codex app-server floor, binding, and lifecycle surfaces that define this security/product choice.
  • Options:
    • Sponsor the experimental preview after cleanup (recommended): Remove the release-owned changelog entry, keep the live proof in the PR context, and have maintainers explicitly own the documented security and compatibility limits before merge.
    • Narrow the preview boundary: Ask for a smaller PR that proves only the minimum remote-exec topology before carrying conversation binding, command resume, and side-question behavior together.
    • Defer behind the canonical issue: Keep the implementation draft open or pause it until upstream Codex registry contracts and OpenClaw operator policy are more settled.

Security
Needs attention: Needs maintainer attention because the diff intentionally adds a new remote process/filesystem execution boundary and registry credential routing, even though I found no discrete credential leak or supply-chain defect.

Review findings

  • [P3] Remove the release-owned changelog entry — CHANGELOG.md:9
Review details

Best possible solution:

Land the preview only after maintainers accept the remote-execution security boundary and fail-closed compatibility model, with the release-owned changelog entry removed and the live Docker proof retained in PR/release-note context.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this PR adds a new experimental Codex remote-execution mode, and current main does not expose the registry-backed remoteExecution surface. The PR body now supplies live proof for the new path rather than a current-main bug reproduction.

Is this the best way to solve the issue?

Unclear as merge-ready: the architecture matches the canonical design issue and the inspected Codex 0.142.5 stdio/Noise contract, but the best solution still requires maintainer security/product acceptance and removal of the release-owned changelog edit.

Full review comments:

  • [P3] Remove the release-owned changelog entry — CHANGELOG.md:9
    This prior finding is still present: CHANGELOG.md is release-owned for normal PRs, so keep this release-note context in the PR body or squash/commit message instead of editing the changelog in this branch.
    Confidence: 0.86

Overall correctness: patch is correct
Overall confidence: 0.78

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against cd5c3fc3b779.

Label changes

Label changes:

  • add proof: sufficient: Contributor real behavior proof is sufficient. The PR body now includes after-fix live Docker proof with separate Gateway/app-server and Codex exec-server containers, authenticated registry/Noise relay, a real model turn, resume, remote workspace execution, and decoy/secret checks.
  • add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • add status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (live_output): The PR body now includes after-fix live Docker proof with separate Gateway/app-server and Codex exec-server containers, authenticated registry/Noise relay, a real model turn, resume, remote workspace execution, and decoy/secret checks.
  • remove rating: 🧂 unranked krab: Current PR rating is rating: 🐚 platinum hermit, so this older rating label is no longer current.
  • remove status: 📣 needs proof: Current PR status label is status: ⏳ waiting on author.

Label justifications:

  • P2: This is a normal-priority experimental Codex operator feature with meaningful blast radius, not a current first-run blocker or emergency regression.
  • merge-risk: 🚨 compatibility: The preview introduces fail-closed behavior for several existing Codex app-server, sandbox, MCP, proxy, and hook configurations when remote execution is enabled.
  • merge-risk: 🚨 session-state: The PR changes Codex thread binding identity, rotation, fork, and resume behavior to account for remote execution topology.
  • merge-risk: 🚨 security-boundary: The diff routes process/filesystem execution through authenticated remote environments and handles registry credentials in the managed app-server environment.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (live_output): The PR body now includes after-fix live Docker proof with separate Gateway/app-server and Codex exec-server containers, authenticated registry/Noise relay, a real model turn, resume, remote workspace execution, and decoy/secret checks.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body now includes after-fix live Docker proof with separate Gateway/app-server and Codex exec-server containers, authenticated registry/Noise relay, a real model turn, resume, remote workspace execution, and decoy/secret checks.
Evidence reviewed

PR surface:

Source +1122, Tests +2051, Docs +191, Other +50. Total +3414 across 44 files.

View PR surface stats
Area Files Added Removed Net
Source 19 1360 238 +1122
Tests 21 2078 27 +2051
Docs 3 200 9 +191
Config 0 0 0 0
Generated 0 0 0 0
Other 1 50 0 +50
Total 44 3688 274 +3414

Security concerns:

  • [medium] Maintainer-owned remote execution boundary — extensions/codex/src/app-server/config.ts:1313
    The PR injects registry credentials into the managed app-server environment and routes filesystem/process execution through Noise remote environments; maintainers need to accept the documented executor isolation and credential-scope model before merge.
    Confidence: 0.84

What I checked:

  • Repository policy applied: Root AGENTS.md plus scoped extensions/, docs/, scripts/, and src/gateway/ guides were read; the Codex dependency-inspection, config-surface, security-boundary, protected-label, and changelog rules apply to this review. (AGENTS.md:16, cd5c3fc3b779)
  • Current main lacks the requested config: Current main exposes remoteWorkspaceRoot and sandboxExecServer but has no remoteExecution object or CODEX_EXEC_SERVER_NOISE_* config surface in Codex app-server runtime config. (extensions/codex/src/app-server/config.ts:278, cd5c3fc3b779)
  • PR adds remote-execution config and SecretInput paths: The manifest adds appServer.experimental.remoteExecution with registry URL, environment id, auth token, optional ChatGPT account id, and SecretInput contract paths for the credential fields. (extensions/codex/openclaw.plugin.json:261, e0845291b548)
  • PR injects authenticated Noise routing into the managed app-server: The runtime validates stdio, POSIX remote workspace root, HTTPS-or-loopback registry URL, non-cleared routing variables, and injects Noise registry env vars while setting raw CODEX_EXEC_SERVER_URL to none. (extensions/codex/src/app-server/config.ts:1243, e0845291b548)
  • PR adds live compatibility preflights: The new helper rejects remote execution when OpenClaw before-tool/trusted-tool policies, managed feature requirements, active legacy managed config, local stdio MCP servers, or managed command hooks would run on the wrong boundary. (extensions/codex/src/app-server/remote-execution.ts:21, e0845291b548)
  • PR changes session binding rotation behavior: Thread lifecycle now compares remoteExecutionFingerprint, forks when remote topology changes, and persists the remote fingerprint into the Codex binding. (extensions/codex/src/app-server/thread-lifecycle.ts:439, e0845291b548)

Likely related people:

  • steipete: Recent merged work raised the Codex app-server floor, moved thread bindings into SQLite plugin state, and added Codex supervisor infrastructure touching the same lifecycle and binding surfaces this PR changes. (role: recent Codex app-server lifecycle contributor; confidence: high; commits: e5798e547770, 0d981095d430, 9dd3bce549b4; files: extensions/codex/src/app-server/version.ts, extensions/codex/src/app-server/thread-lifecycle.ts, extensions/codex/src/conversation-binding.ts)
  • kevinslin: Authored the merged remote WebSocket app-server work that introduced the current remote app-server config, runtime identity, thread lifecycle, and docs surfaces this PR builds on and narrows away from for production remote execution. (role: recent remote app-server contributor; confidence: high; commits: bc5081c58730; files: extensions/codex/src/app-server/config.ts, extensions/codex/src/app-server/thread-lifecycle.ts, docs/plugins/codex-harness-reference.md)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
Review history (2 earlier review cycles)
  • reviewed 2026-07-07T04:18:11.088Z sha 9ffc6fd :: needs real behavior proof before merge. :: none
  • reviewed 2026-07-07T05:09:52.465Z sha e7f7694 :: needs real behavior proof before merge. :: [P3] Remove the release-owned changelog entry

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. merge-risk: 🚨 security-boundary 🚨 May affect sandboxing, authorization, credentials, or sensitive data. labels Jul 7, 2026
@steipete-oai
steipete-oai force-pushed the codex/remote-execution-environment branch from 9ffc6fd to e7f7694 Compare July 7, 2026 04:32
@steipete
steipete force-pushed the codex/remote-execution-environment branch from e7f7694 to e084529 Compare July 7, 2026 10:14
@openclaw-barnacle openclaw-barnacle Bot added gateway Gateway runtime scripts Repository scripts docker Docker and sandbox tooling labels Jul 7, 2026
@steipete

steipete commented Jul 7, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 7, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jul 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@steipete steipete

Labels

docker Docker and sandbox tooling docs Improvements or additions to documentation extensions: codex gateway Gateway runtime maintainer Maintainer-authored PR merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 security-boundary 🚨 May affect sandboxing, authorization, credentials, or sensitive data. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P2 Normal backlog priority with limited blast radius. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. scripts Repository scripts size: XL status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Feature]: Keep Codex app-server with Gateway for remote execution

1 participant

@steipete