Skip to content

test: add omp (oh-my-pi) backend to skills-evals - #31

Merged
tsaqib merged 3 commits into
mainfrom
feat/omp-eval-backend
May 28, 2026
Merged

tsaqib merged 3 commits into
mainfrom
feat/omp-eval-backend

Conversation

@tsaqib

@tsaqib tsaqib commented May 28, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Wires omp (omp.sh, can1357/oh-my-pi) as a first-class eval backend alongside Claude Code, OpenCode, GitHub Copilot CLI, Pi, and Cline.
  • Appends a cline-style agentTarget row in constants.go:
    {"omp", "omp (oh-my-pi)", ".agents/skills", "", "", ""},
    .agents/skills is omp's documented agents skill-provider path (priority 70 in oh-my-pi/docs/skills.md, source at packages/coding-agent/src/discovery/agents.ts). omp's agents provider walks .agents/skills/ from cwd at project scope and $HOME/.agents/skills/ at user scope — same path Codex, Copilot, and Pi already share, so --agents codex,copilot,pi,omp collapses to a single install.
  • omp_driver.py runs omp -p --mode json single-shot per skill, with a --continue auto-yes loop. Parses message_end events and surfaces LLM errors (e.g. Insufficient Balance) explicitly instead of swallowing them.
  • Routing: DeepSeek via omp's built-in deepseek provider catalog (DEEPSEEK_API_KEY → api.deepseek.com). No BYOK shim needed.
  • Slash form: /skill:x-plan <task> and /skill:x-x per omp's getSkillSlashCommandName in packages/coding-agent/src/extensibility/skills.ts.

Coverage

  • init_test.go iterates agentTargets, so the new row is automatically exercised by TestRunInit_UserScope_EndToEnd (POSIX symlink + Windows copy fallback).
  • scripts/e2e_test.sh: two new cases — project-scope --agents=omp install lands at .agents/skills/, user-scope at ~/.agents/skills/. Local run: 550 cases passed.
  • scripts/e2e_test.ps1: same two cases mirrored for Windows parity.
  • skills-evals/tests/conftest.py: omp added to VALID_AGENT_KEYS, AGENT_INIT_VALUE_FOR_KEY["omp"] = "omp", AGENT_USER_SKILLS_REL_FOR_KEY["omp"] = .agents/skills.
  • skills-evals/src/skills_evals/omp_driver.py + 4 test files (smoke + todo + plan-extends + reminders-supersedes-todo).
  • constants_e2e_test.go: shell ↔ Go drift check now covers CLINE_SKILLS_REL (from main) and OMP_SKILLS_REL (this PR).
  • Workflow files (manual-omp-judge.yml, manual-omp-judge-user-scope.yml) already shipped on main as scaffolds — this PR fills in the driver and tests they invoke.

Rebase note

Rebased twice onto origin/main — first to absorb #29 (pi first-class registration), then again after #30 (cline first-class registration) merged. The omp registry row now sits at agentTargets[6], after cline at [5]. omp shares the cross-agent .agents/skills path with codex, copilot, and pi (idempotent install).

Test plan

  • manual-omp-judge.yml (project scope) on this branch — smoke + 3 scenarios green
  • manual-omp-judge-user-scope.yml (user scope) on this branch — smoke + 3 scenarios green

@tsaqib
tsaqib force-pushed the feat/omp-eval-backend branch from 6109b45 to f2b6577 Compare May 28, 2026 12:59
@tsaqib tsaqib mentioned this pull request May 28, 2026
2 of 3 tasks
tsaqib added 3 commits May 28, 2026 09:18
Wires omp (omp.sh, can1357/oh-my-pi) as a fourth agent backend
alongside Claude Code, OpenCode, and GitHub Copilot CLI. The driver is
single-shot per invocation (`omp -p --mode json`) because omp has no
documented stdin-based multi-turn protocol; the auto-yes loop is the
Copilot pattern (one `--continue` reply per gate).

Routing targets DeepSeek directly via the built-in `deepseek` provider
catalog (provider-models/descriptors.ts catalogDescriptor "deepseek"),
keyed on DEEPSEEK_API_KEY — same single-secret shape as OpenCode. The
auto-detected openai-completions compat layer in omp handles the
DeepSeek reasoning_content round-trip and max_tokens quirks.

Slash-command form is the easy-to-miss bit: omp registers skills as
`/skill:<name>` slash commands (see getSkillSlashCommandName in
packages/coding-agent/src/extensibility/skills.ts), not `/<name>`. The
suite invokes `/skill:x-plan <task>` and `/skill:x-x` for the planner
and executor; passing the bare `/x-plan` form would land as literal
text and the LLM would hallucinate a missing command.

Suite covers the smoke check plus three scenarios mirrored from the
Claude and Copilot paths (todo, plan-extends, reminders-supersedes-todo).
Workflow files (manual-omp-judge.yml, manual-omp-judge-user-scope.yml)
already shipped on main as scaffolds — this PR fills in the driver
and tests they invoke.

Known open question this PR exists to answer: whether omp's
`--mode json` event stream is stable enough to drive the auto-yes
detector without false negatives, and whether the slash-command form
resolves cleanly through omp's print-mode session.prompt() path. The
fixture passes `--agents claude` and lets omp's `.claude/skills/`
discovery (packages/coding-agent/src/discovery/claude.ts) pick the
SKILL.md files up — same transitional shape Copilot uses today.

Signed-off-by: Tanzim Saqib <[email protected]>
The first manual eval runs (workflows 26556757739 / 26556758520) had
the driver's `_extract_text` looking for either `type: "assistant"`
(Anthropic-shape) or `type: "text"` with a `part` envelope
(OpenCode-shape) — neither of which omp emits. omp's `--mode json`
wire emits the consolidated final assistant message as
`type: "message_end"` with `message.role: "assistant"` and the full
`message.content[]` array, while streaming `text_delta`/`thinking_delta`
events come wrapped in `message_update.assistantMessageEvent`.

Effect on the failing runs: the auto-yes detector got an empty string
every turn, so even when the agent emitted a 'Reply yes' gate the
driver short-circuited as `completed=True`. That alone wouldn't have
broken the passing tests (smoke + plan_extends both produced their
plans without needing auto-yes; the CI_DIRECTIVE keeps the agent
moving), but it masked a separate failure mode: when the model call
returned 402 / 429 / 500, omp's print-mode subprocess exited 0 and
emitted a final assistant message_end with `stopReason: "error"` and
empty `content[]`. Driver reported `completed=True, exit_code=0`, the
scenario test passed the early subprocess assertions, and the run
finally failed with the cryptic `expected 2 plan files, got 0` —
upstream cause invisible.

Fix:

  - `_extract_text` now reads `type: "message_end"` with role
    "assistant", iterates `message.content[]`, and pulls `text` items
    (filtering out `thinking` and `toolCall`).
  - `_summarize_event` learns the omp event types — message_end,
    message_update wrappers around assistantMessageEvent, turn/session
    boundaries — so CI logs are readable without re-paging the
    transcript.
  - New `_detect_stream_error` scans for the last assistant
    message_end and reports `stopReason: "error"` with the captured
    errorMessage. The driver loop runs it after every turn and flips
    `run.exit_code = 1` when fired, so the scenario test's
    `assert run.exit_code == 0` reports the LLM error directly.

No event-shape changes were needed for the smoke / plan_extends paths
that already passed — both happen to complete without needing the
auto-yes loop, so the prior empty-text bug never bit them.

Signed-off-by: Tanzim Saqib <[email protected]>
Adds omp (oh-my-pi, omp.sh / can1357/oh-my-pi) as a fifth row in the
agentTargets registry. `x-x init --agents omp` is now a real subcommand
flag value — the binary recognizes the key, the interactive picker
lists it, and the install path is bundled / unmerged through the same
loop as the other rows. No special-case branches anywhere in the
install/remove code.

Install path is `.agents/skills` at both scopes, the cross-agent open-
spec path Codex and Copilot already use. Rationale:

  - Officially documented as omp's `agents` skill provider in
    oh-my-pi/docs/skills.md ("priority 70 group ... agents") and
    backed by packages/coding-agent/src/discovery/agents.ts, which
    walks `.agent/` and `.agents/` at every cwd ancestor up to
    repoRoot plus `$HOME/.agents/` at user scope.
  - Symmetric across scopes — no userSkillsRel override needed.
    Codex and Copilot use the same path; install is idempotent so
    `--agents codex,copilot,omp` collapses to one shared
    `.agents/skills/` directory.
  - Cross-platform: node:path on omp's side joins to `.agents\skills`
    on Windows automatically; the Go side composes via filepath.Join.
    Verified through scripts/e2e_test.ps1 mirror.
  - Stays out of omp's native `.omp/...` tree (priority 100), which
    is omp's private session-storage area (`~/.omp/agent/`, owned
    end-to-end by omp's own runtime).

Skills-only — omp's settings live at `~/.omp/config.yml` (Settings →
Memory tab) and its model registry at `~/.omp/agent/models.yml`. Both
are user-owned end-to-end and outside the x-x install scope, same
reasoning as OpenCode and Copilot.

Mirrored through the full constants-drift chain per AGENTS.md:

  - constants.go agentTargets row 4
  - constants_e2e_test.go expected-map adds OMP_SKILLS_REL +
    COPILOT_SKILLS_REL (the latter was already declared in the shell
    file but never locked into the drift check)
  - scripts/e2e_test.sh OMP_SKILLS_REL readonly + two new cases
    (project + user scope) asserting on `.agents/skills/` presence,
    other agents' exclusive paths + per-agent config files absent
  - scripts/e2e_test.ps1 mirror of both readonly and cases, Windows
    backslash separators

skills-evals/tests/conftest.py: flip
AGENT_INIT_VALUE_FOR_KEY["omp"] from "claude" to "omp" (the binary now
recognizes the key) and AGENT_USER_SKILLS_REL_FOR_KEY["omp"] from
`.claude/skills` to `.agents/skills` so the user-scope post-install
log points at the actual install location.

Signed-off-by: Tanzim Saqib <[email protected]>
@tsaqib
tsaqib force-pushed the feat/omp-eval-backend branch from f2b6577 to b757fce Compare May 28, 2026 13:23
@tsaqib
tsaqib merged commit 8a13c52 into main May 28, 2026
2 checks passed
@tsaqib
tsaqib deleted the feat/omp-eval-backend branch May 28, 2026 13:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant