Skip to content

feat(model): add the opencode_exec backend - #364

Open
Ivan Berlin (g0dm0d) wants to merge 1 commit into
microsoft:mainfrom
g0dm0d:feat/opencode-chat-backend
Open

Ivan Berlin (g0dm0d) wants to merge 1 commit into
microsoft:mainfrom
g0dm0d:feat/opencode-chat-backend

Conversation

@g0dm0d

Copy link
Copy Markdown

Summary

Adds OpenCode as a core SkillOpt backend, opencode_exec, for the optimizer and as an exec target, alongside the existing codex/claude/cursor/copilot harnesses. It uses the user's existing OpenCode login, so any provider configured in OpenCode can run a full train/eval loop with no separate provider API key. --backend opencode selects it for both roles; --optimizer_backend or --target_backend opencode_exec selects one.

OpenCode support so far lived only in SkillOpt-Sleep (#218). That backend targets OpenCode 1.x flags; this one targets OpenCode 2.x, which drops --pure, --dir and OPENCODE_CONFIG_CONTENT.

Changes

  • adds opencode_exec with opencode, opencode_chat and opencode_cli aliases, wired through common.py, backend_config.py, the model dispatcher, config.py, scripts/train.py, scripts/eval_only.py and the trainer
  • runs opencode run --standalone --format json with a private XDG_CONFIG_HOME; the credentials stay in the user's data directory
  • the optimizer is text-only: it runs in an empty temporary directory with every tool denied, and chat_optimizer_messages(..., tools=...) raises NotImplementedError
  • the target runs in the benchmark workspace with read tools; allow_file_edits adds the edit tool
  • outside the workspace, only validated data_dirs and image directories are reachable; tool steps are written to opencode_trace_steps.txt
  • exposes model.opencode_exec_path and model.opencode_exec_allow_shell (blank or null in the base config, so OPENCODE_EXEC_PATH / OPENCODE_EXEC_ALLOW_SHELL stay in control)
  • refuses OpenCode 1.x before any call
  • resolves the CLI path at import like the other exec backends (fix(model): resolve CLI exec paths for Windows .cmd shims #250)

Safety: every call starts from a global deny-all permission list and allows only what the run needs. Calls carry none of the user's MCP servers, global rules or skills, and the OPENCODE_DISABLE_* switches keep project AGENTS.md and Claude Code files out. Unlike codex_exec and cursor_exec, OpenCode has no OS sandbox, so file-edit rollouts get no shell unless opencode_exec_allow_shell is set explicitly.

Checked against OpenCode 2.2.5:

  • It reads its project directory from PWD, not the process cwd. Without setting PWD, a target ran in the caller's directory and wrote its file there.
  • A per-agent permissions list is dropped, so only the global list gates tools.
  • Context leaks in. A parent AGENTS.md up to the git root reached the agent, and with the skill tool enabled the user's global skills grew the system prompt from ~1.4k to ~13.6k tokens.

Both roles now guard against all three, with regression tests.

Two caveats worth knowing before use: each call starts a private OpenCode server, which adds roughly 1–3 s; and ChatGPT logins reject some older model ids, for example openai/gpt-5.5 ("not supported when using Codex with a ChatGPT account"), so set model.optimizer / model.target explicitly.

Validation

  • python -m pytest tests/test_opencode_exec_backend.py tests/test_opencode_exec_target.py -q: 33 passed, with a fake CLI replaying real OpenCode 2.2.5 JSONL
  • SKILLOPT_TEST_REAL_OPENCODE_EXEC=1 python -m pytest tests/test_opencode_exec_live.py -q: 4 passed on macOS with OpenCode 2.2.5 and openai/gpt-6.1-sol through a ChatGPT login. The tests check that:
    • the optimizer follows the system prompt;
    • the optimizer cannot use tools;
    • the target reads the skill and the task but cannot write;
    • file edits stay inside the workspace.
  • python -m pytest -q on Python 3.11 and 3.12: 2067 passed, 15 skipped and 2055 passed, 27 skipped
  • python -m pytest -q on Python 3.10: 2066 passed, 1 failed. The failure is test_officeqa_parallel_search.py::test_whole_mcp_request_has_a_deadline; it fails the same way on a clean main locally and passes on its own.
  • python -m mkdocs build --strict: passed
  • ruff check .: no new findings compared with main
  • git diff --check: passed

Not verified on Windows: external_directory patterns are built with os.path.join(dir, "*"), and I could not check how OpenCode matches backslash paths.

Scope and follow-ups

This PR covers the core backend for both roles and does not change the SkillOpt-Sleep OpenCode backend.

Follow-up work:

  • inject opencode_trace_steps.txt into reflection, like REFLACT_CODEX_TRACE_TO_OPTIMIZER does for Codex
  • Windows validation of the target's external-directory permissions

…n exec target

Run SkillOpt through the OpenCode CLI (2.x) with the user's existing
OpenCode login. `--backend opencode` fills both roles; `--optimizer_backend
opencode_exec` or `--target_backend opencode_exec` selects one. Aliases:
`opencode`, `opencode_chat`, `opencode_cli`. Wired through common.py,
backend_config.py, the dispatcher, config.py, train.py, eval_only.py and
the trainer, per docs/guide/new-backend.md.

Every call runs `opencode run --standalone --format json` with a private
XDG config home: a global deny-all permission list plus only what the run
needs, none of the user's MCP servers, and the OPENCODE_DISABLE_* switches
so project AGENTS.md files and Claude Code rules and skills stay out. The
data directory with the credentials is untouched. OpenCode 1.x is refused
before any call.

- Optimizer: text only, in an empty temporary directory, every tool denied.
  Tool calls through the message API raise NotImplementedError.
- Target: runs in the task workspace with read tools. allow_file_edits adds
  the edit tool; a shell only with model.opencode_exec_allow_shell, since
  OpenCode has no OS sandbox. Outside the workspace only data_dirs and image
  directories are reachable. Tool steps go to opencode_trace_steps.txt.

Checked against OpenCode 2.2.5:
- the project directory comes from PWD, not the process cwd, so PWD is set
  for both roles;
- a per-agent `permissions` list is dropped, so the global list is the gate;
- a parent AGENTS.md up to the git root and the user's global skills reach
  the agent unless the switches are set and the skill tool is denied.

Tests use a fake CLI replaying real 2.2.5 JSONL. Live smoke tests are opt-in
with SKILLOPT_TEST_REAL_OPENCODE_EXEC=1.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@g0dm0d

Copy link
Copy Markdown
Author

Ivan Berlin (Ivan Berlin (@g0dm0d)) please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@microsoft-github-policy-service agree [company="{your company}"]

Options:

  • (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.
@microsoft-github-policy-service agree
  • (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.
@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement

@microsoft-github-policy-service agree

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant