Repository navigation
Conversation
Append a third row to agentTargets in constants.go:
{"kilo", "Kilo Code", ".kilo/skills", "", ""}
Kilo discovers skills natively at `.kilo/skills/` (project) and
`~/.kilo/skills/` (user) per https://kilo.ai/docs/customize/skills. It
also auto-picks-up `.agents/skills/` (Open Agent Standard) and
`.claude/skills/` (compat mode), but `.kilo/skills/` is the path it
treats as the agent's own home, so that's where `x-x init` installs.
configSrc / configRel are intentionally empty. Kilo's `kilo.jsonc` is
provider-routing config, not a hook integration surface like Claude
`settings.json` or Codex `hooks.json`. Users opt into provider routing
through Kilo's `/connect` flow or by writing `kilo.jsonc` themselves —
shipping a default would impose a model+endpoint choice on every project
init, which is the wrong policy for a config file Kilo expects users to
own.
E2e parity:
- bash + PowerShell both gain a KILO_SKILLS_REL constant and an
`init --agents=kilo` single-agent install case asserting `.kilo/skills/`
appears and `.claude/`, `.agents/` do not.
- `reset_user_home` in the bash harness also wipes `$HOME/.kilo/` between
cases; without it, `.kilo/skills/` leaked from one user-scope init test
into the next and broke the "skills remove --user is a silent no-op"
case (kilo's leftover dir made skills-remove report a non-empty pass).
PS1 harness already nukes the whole sandbox HOME between cases, so no
parallel change there.
No new code-branch edits in init.go / skill.go — both already iterate
agentTargets and handle empty configSrc via the existing gate
(`if t.configSrc == ""`).
Local: go vet, go test ./... (1 pkg ok), scripts/e2e_test.sh (588 passed
0 failed).
Signed-off-by: Tanzim Saqib <[email protected]>
Second parallel agent backend after Claude Code. Follows the recipe in
docs/internal/adding-agent-eval-backend.md.
skills_evals/kilo_driver.py: drives `kilo run --auto --format json`
non-interactively. Kilo's CLI is one-shot per invocation in autonomous
mode (unlike Claude's stream-json which holds stdin open); the driver
keeps multi-turn working by re-entering with `--session <id>` (captured
from the first turn's JSON events) and the same `reply yes` confirmation
gate the Claude driver uses. One Popen per turn — simpler than Claude's
shared-pipe pump but with the same logging-everything policy. Wire
format is documented only as "machine-readable event stream"
(kilo.ai/docs/code-with-ai/platforms/cli); the parser is defensive,
session-ID capture is best-effort across several plausible event shapes,
and test_kilo_stream_smoke.py pins the actual schema once it runs.
tests/conftest.py: routing made agent-aware.
- Renames `_load_dotenv_and_route_claude` → `_load_dotenv_and_route_agents`
and sets both CLAUDE_* and KILO_* env defaults; each CLI ignores the
other's namespace, so always-set is safe.
- Mirrors DEEPSEEK_API_KEY into KILO_API_KEY (belt-and-suspenders for
paths that bypass the workspace config).
- New `X_X_AGENT_UNDER_TEST` env var (default "claude") drives the
`--agents` flag for `x-x init` and gates the per-workspace kilo.json
write. Workspace fixture also skips if the chosen agent's CLI isn't
on PATH (previously hard-coded to skip on missing claude).
- When the agent is kilo, the fixture writes `kilo.json` into the
workspace declaring an openai-compatible provider pointing at
api.deepseek.com/v1 with `{env:DEEPSEEK_API_KEY}` substitution
(documented Kilo pattern at kilo.ai/docs/code-with-ai/agents/custom-models).
Without this file kilo has no env-var-only route to a custom
OpenAI-compat endpoint.
tests/test_kilo_*.py: four scenarios ported from the Claude versions
(smoke, todo, plan_extends, reminders_supersedes_todo) with the driver
import swapped. The TASK strings, judge invocations, and plan-mechanic
assertions are agent-agnostic by design — they test the AGENT'S behavior
against the skill spec, not the wire format. Each Kilo test file's
docstring matches its Claude counterpart so they read as a pair.
.github/workflows/manual-kilo-judge.yml: copies the Claude workflow's
shape (60min timeout, setup-go + setup-node + setup-uv pinned versions,
artifact upload on failure). Replaces the Anthropic env block with
KILO_PROVIDER + KILOCODE_MODEL + X_X_AGENT_UNDER_TEST=kilo, installs
`@kilocode/cli@latest` (will pin after first successful smoke run), and
scopes pytest to `tests/test_kilo_*.py` explicitly.
manual-claude-judge.yml + manual-claude-judge-user-scope.yml: pytest
invocation scoped to claude + smoke tests. Without this both Claude
workflows would now collect the new test_kilo_*.py files and try to
spawn `kilo` (not on the Claude runner's PATH).
Local: `uv run pytest --collect-only` collects all 8 tests cleanly;
import-checked kilo_driver + the conftest module independently.
Signed-off-by: Tanzim Saqib <[email protected]>
3 tasks done
tsaqib
added a commit
that referenced
this pull request
May 26, 2026
Mirrors the project-scope manual-kilo-judge.yml with X_X_INSTALL_SCOPE=user added so `x-x init` lands skills under ~/.kilo/skills/ instead of <workspace>/.kilo/skills/. Matches the existing claude/codex/copilot patterns where every backend ships both scope variants. Same merge-safety story as the project-scope file: workflow_dispatch only, depends on skills-evals files from feat/kilo-eval-backend (#16) that aren't on main yet, no new secrets. Signed-off-by: Tanzim Saqib <[email protected]>
Mirrors the project-scope manual-kilo-judge.yml with X_X_INSTALL_SCOPE=user added so `x-x init` lands skills under ~/.kilo/skills/ instead of <workspace>/.kilo/skills/. Matches the existing claude/codex/copilot patterns where every backend ships both scope variants. Same file is being added to main via feat/kilo-workflows-only (#17) so the two PRs land identical content and don't conflict on merge. The user-scope conftest path was already wired in the previous commit on this branch — the workspace fixture reads X_X_INSTALL_SCOPE and logs the resolved skills dir under either .kilo/skills/ (project) or ~/.kilo/skills/ (user). No code change here, workflow file only. Signed-off-by: Tanzim Saqib <[email protected]>
tsaqib
added a commit
that referenced
this pull request
May 26, 2026
* ci: register kilo eval workflows on main GitHub Actions registers `workflow_dispatch` triggers from the default branch only. Until `manual-kilo-judge.yml` exists on `main`, it won't show up in the Actions tab and `gh workflow run` will refuse it — even when targeting the feature branch that holds the implementation. This is the small focused PR described in `docs/internal/adding-agent-eval-backend.md` §6, same shape as PR #12 (copilot) and PR #13 (codex). The file is copied verbatim from `feat/kilo-eval-backend` (PR #16). No code changes, no test changes, no doc changes. Safe to merge now: - `workflow_dispatch:` only — no push / pull_request / schedule triggers, so merging this PR cannot run anything on its own. - The job depends on skills-evals files (`kilo_driver.py`, `test_kilo_*.py`, the `X_X_AGENT_UNDER_TEST`-aware conftest) that don't exist on `main` yet. Dispatching against `main` would fail at the pytest step; the workflow is only meaningful when dispatched with `ref=feat/kilo-eval-backend`. - No new secrets. `DEEPSEEK_API_KEY` is already configured for the other manual-*-judge workflows. After merge: open Actions, pick "manual / kilo code <- llm judge", set **Use workflow from: feat/kilo-eval-backend**, and dispatch. Signed-off-by: Tanzim Saqib <[email protected]> * ci: add user-scope variant to the kilo registration Mirrors the project-scope manual-kilo-judge.yml with X_X_INSTALL_SCOPE=user added so `x-x init` lands skills under ~/.kilo/skills/ instead of <workspace>/.kilo/skills/. Matches the existing claude/codex/copilot patterns where every backend ships both scope variants. Same merge-safety story as the project-scope file: workflow_dispatch only, depends on skills-evals files from feat/kilo-eval-backend (#16) that aren't on main yet, no new secrets. Signed-off-by: Tanzim Saqib <[email protected]> --------- Signed-off-by: Tanzim Saqib <[email protected]>
Contributor
Author
|
Abandoning the kilo eval backend. First dispatch of |
tsaqib
added a commit
that referenced
this pull request
May 26, 2026
First dispatch of `manual / kilo code <- llm judge` against
feat/kilo-eval-backend showed the smoke green — kilo CLI installs,
`kilo run --auto --format json` accepts the openai-compatible DeepSeek
route declared in the per-workspace `kilo.json`, the driver's JSON event
parser handles the wire format, and the auto-yes loop terminates
cleanly. So the plumbing works.
All three scenarios (`test_kilo_todo`, `test_kilo_plan_extends`,
`test_kilo_reminders_supersedes_todo`) failed with
`len(plans)==0` — kilo "completed" each turn without ever invoking the
`x-plan` skill or writing a single plan file to `.x-plans/`. Two
plausible root causes, neither validated:
1. `/x-plan <task>` slash syntax is Claude-Code-specific. Kilo's
skill matcher (per kilo.ai/docs/customize/skills) keys off natural
language — "use the X skill" — so `/x-plan` likely lands as
literal text.
2. `kilo run --auto` may not load `.kilo/skills/` at all. The docs
say skills load "at session start"; whether one-shot `kilo run`
invocations walk the same loader as interactive sessions is
undocumented.
The auto-yes loop never triggers a plan write either way: with no skill
invocation, kilo just chats about the task and ends its turn without
the `reply yes` confirmation phrase the driver looks for. The driver
sees `completed=True`, exits clean, and the test asserts on an empty
`.x-plans/`.
Diagnosing between (1) and (2) requires the run's pytest-tmp artifact
or a local repro against `npm install -g @kilocode/cli`. Both are
multi-step rabbit holes for an integration whose value proposition
isn't worth that follow-up right now. Removing the workflow files so
they stop appearing in the Actions UI as broken.
Same shape as #20 (codex workflows removal). The driver, conftest
changes, agent-registry row in `constants.go`, and the four
`test_kilo_*.py` files on `feat/kilo-eval-backend` (closed in #16) are
preserved-but-parked; pick them up if the kilo invocation question
gets answered.
Signed-off-by: Tanzim Saqib <[email protected]>
4 of 6 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Second parallel agent backend in
skills-evals, alongside the existing Claude Code path. Followsdocs/internal/adding-agent-eval-backend.mdend-to-end.feat(init): appendkilotoagentTargets(.kilo/skills/, no per-agent config), bash + PowerShell e2e gain a--agents=kiloinstall case,reset_user_homein bash now wipes$HOME/.kilo/between cases.feat(skills-evals): newkilo_driver.py(onekilo run --auto --format jsonper turn, multi-turn via--session <id>captured from JSON events, samereply yesconfirmation gate as the Claude driver), conftest gainsKILO_ENV_DEFAULTS+ a per-workspacekilo.jsonwriting an openai-compatible DeepSeek provider, 4 ported scenario tests (test_kilo_{stream_smoke,todo,plan_extends,reminders_supersedes_todo}.py), and amanual-kilo-judge.ymlworkflow. Existing Claude workflows get their pytest invocation scoped so they don't pick up the new kilo tests.Why draft
Two pieces of Kilo's CLI surface are documented only at the surface level —
--format jsonis described as "machine-readable event stream" with no public schema, and the session-resume model (--continue/--session <id>) hasn't been validated end-to-end through the auto-yes loop yet. The smoke test (test_kilo_stream_smoke.py) is the canary that pins the actual shape on a real DeepSeek run. Until that's green in CI, the driver's session-ID capture and assistant-text extraction are best-effort with extensive defensive logging.Test plan
go vet ./...cleango test ./...clean (parameterized tests already iterateagentTargetsso no new Go unit tests required)bash scripts/e2e_test.sh— 588 passed, 0 failed (incl. new--agents=kilocase)pwsh scripts/e2e_test.ps1— local PS1 run pending (no pwsh in this env); CI'swindows-cli.ymlwill catch any drift on the PRuv run pytest --collect-onlycollects 8 tests (4 claude + 4 kilo)manual-kilo-judge.ymlfirst run on thefeat/kilo-eval-backendbranch via the Actions UI after merge tomain(per the doc's note that GitHub UI only registers workflows from the default branch — this PR adds the workflow but it can't be triggered from a feature branch)manual-claude-judge.ymlsmoke re-run to confirm the pytest-scope change didn't regress the Claude pathReferences
docs/internal/adding-agent-eval-backend.md— the recipe this PR follows--auto,--format json,--session/--continue.kilo/skills/,.agents/skills/,.claude/skills/)