Skip to content

opencode RL literal bridge: rollout_details correlation + HARBOR_MODEL_ENDPOINT baseURL#7

Merged
penfever merged 5 commits into
mainfrom
feuer/opencode-literal-rollout-details
Jul 17, 2026
Merged

opencode RL literal bridge: rollout_details correlation + HARBOR_MODEL_ENDPOINT baseURL#7
penfever merged 5 commits into
mainfrom
feuer/opencode-literal-rollout-details

Conversation

@penfever

Copy link
Copy Markdown
Collaborator

The harbor half of the opencode RL literal bridge.

  • x-ot-trial-id per-trial correlation header + context.metadata['rollout_correlation_id'] (opencode.py) so the RL loop can attribute each shared-RecordProxy-log entry to its trial.
  • proxy.py records the trial id; new harbor.literal.rollout_build builds per-trial rollout_details (id-correlated, no bleed).
  • GAP C: the hosted_vllm provider id takes the openai-compatible baseURL/apiKey/header path (was vllm-only).
  • opencode reads its served-model baseURL from HARBOR_MODEL_ENDPOINT (harbor-specific), with OPENAI_BASE_URL as back-compat fallback — OPENAI_BASE_URL is reserved for genuine OpenAI traffic (LLM-judge verifiers) and must not be overloaded with a vLLM endpoint.

Local: 51 opencode tests + 9 rollout_build tests pass; ruff + IDE diagnostics clean.

Benjamin Feuer added 5 commits July 16, 2026 07:02
…ollout_details

OpenCode is a CLI agent that talks to vLLM over its own transport, bypassing
harbor Chat, so context.rollout_details is never populated. The RecordProxy
literal.jsonl was already parsed + merged into trajectory.json but not into
rollout_details, which the RL rollout loop (MarinSkyRL) consumes for TIS
logprob extraction.

Add a flag-gated (collect_rollout_details, default False) bridge that also
builds a RolloutDetail from the parsed literal per-turn pairs and sets
context.rollout_details in populate_context_post_run. Mirrors
Chat.rollout_details + the per-turn length-parity guard (8737426): a turn
whose logprobs length != completion_token_ids length (or absent) records an
empty logprob list so downstream TIS flags it rather than mis-pairing.

Flag off => byte-identical (rollout_details stays None); non-RL/datagen paths
unaffected. Adds S0 repro (rollout_details None despite non-empty literal log)
and S1 flag-on parity tests.
…iteral capture

RL runs many opencode trials concurrently against one served vLLM behind a
single shared RecordProxy, so their captured completions interleave in one log.
Content correlation (datagen's literal_correlator) cannot disambiguate GRPO
groups (n>1 rollouts of the IDENTICAL prompt -> every continuation ambiguous).

Add a per-trial correlation id transported as an HTTP header (confirmed to
forward through opencode-ai 1.18.2 @ai-sdk/openai-compatible options.headers):
- opencode.py mints a per-trial id when collect_rollout_details is on, stamps it
  into the vllm provider options.headers[x-ot-trial-id], and surfaces it on
  context.metadata[rollout_correlation_id] for the RL-loop glue.
- RecordProxy captures the x-ot-trial-id request header onto each JSONL entry
  (trial_id) on both the streaming and non-streaming paths.
- new harbor.literal.rollout_build: build_rollout_details_from_pairs (the shared
  Chat.rollout_details-mirroring parity builder, extracted from opencode.py) and
  build_rollout_details_for_trial (filter a shared log by trial_id, order by
  arrival -> one RolloutDetail; spans opencode auto-compaction boundaries).

Flag off => no id, no header, no metadata, trial_id=None (byte-identical;
datagen/content path untouched). Tests: n>1 identical-seed no-bleed, compaction
ordering, header round-trip, parity guard. 83 harbor unit tests green.
…teral-bridge header

The per-trial correlation-id header forwards through @ai-sdk/openai-compatible on
opencode-ai 1.18.2 (older 1.0.x dropped provider options, opencode #5674). RL runs
relying on the literal bridge must pin version 1.18.2 in the agent config. Comment
only; no behavior change.
…er (RL literal bridge GAP C)

The RL rollout generator forces model_name="hosted_vllm/<alias>"
(terminal_bench_generator), but the baseURL/apiKey/@ai-sdk/openai-compatible +
x-ot-trial-id correlation-header branch was gated on provider=="vllm" only, so it
never fired for opencode-in-RL: opencode got no baseURL and no per-trial header, the
shared RecordProxy could not attribute its log entries, and correlation no-op'd. Treat
"hosted_vllm" like "vllm" in the config-writer AND the two downstream env-selection
gates in run() (OPENAI_API_KEY/OPENAI_BASE_URL keys + the OPENCODE_DUMMY_KEY injection).

Adds real-path tests over the actual config-writer under model_name="hosted_vllm/..."
(header + npm + baseURL), byte-identical flag-off, and version/opencode_config reaching
the ctor. The prior header test used "vllm/..." so it passed while the real path no-op'd.
…(not OPENAI_BASE_URL)

_build_register_config_command sourced the self-hosted vLLM baseURL from
OPENAI_BASE_URL. That var is RESERVED for genuine OpenAI traffic — the LLM-judge
verifiers on the worker read it — so having the RL runner publish the vLLM
capability URL there silently misroutes every judge call to vLLM. Prefer the
harbor-specific HARBOR_MODEL_ENDPOINT (the var the RL runner already publishes for
the co-located served model); keep OPENAI_BASE_URL as a back-compat fallback so
nothing else regresses. Also forward HARBOR_MODEL_ENDPOINT into the sandbox env for
the vllm/openai providers, symmetric with the baseURL source (non-load-bearing: the
baseURL is baked into opencode.json). Adds 2 tests (prefers HARBOR_MODEL_ENDPOINT;
falls back to OPENAI_BASE_URL); 51 opencode tests pass.
@penfever
penfever merged commit c872216 into main Jul 17, 2026
4 checks passed
@penfever
penfever deleted the feuer/opencode-literal-rollout-details branch July 17, 2026 11:43
penfever pushed a commit that referenced this pull request Jul 18, 2026
…indow

Port of the runtime fix from harbor main 826c2e4 (PR #11) onto penfever/working
(the branch the TACC eval/datagen editable clones track; main's version sits atop
opencode PRs #7/#8 which working lacks). opencode defaults a custom (openai-compatible)
provider's max_tokens to ~32000 (opencode harbor-framework#1735); _build_register_config_command wrote
an EMPTY per-model dict, so against a 32768 served window every real prompt 400s
('maximum context length ... you requested 32000 output tokens') -> opencode exits 1
-> NonZeroAgentExitCodeError -> reward 0 on 100% of trials.

Writes a bounded per-model limit {context,output} into opencode.json: output =
model_info.max_output_tokens or 4096; window = model_info.max_input_tokens or 32768;
context = window - output, so prompt+output <= window always. model_info is the ctor
kwarg the eval path passes (eval_opencode_ctx32k.yaml: max_input 32768 / max_output
16384 -> limit {context:16384, output:16384}); safe class-default fallback otherwise.
Keeps working's env-based baseURL (OPENAI_BASE_URL), which the eval sbatch sets.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_0191M6UMWFw2eWr5y4GwC1Qg
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant