Repository navigation
Commit 0633be9
committed
fix(harness): hermetic XDG isolation, KV cache flush, permission pre-grant, and regex assertions
Resolve four categories of test reliability issues identified through
iterative debugging of the black-box pytest harness against gpt-oss-20b
running under llama-server.
XDG directory isolation
- Move .xdg_config, .xdg_state, .xdg_cache from inside workspace/ to
siblings under tmp_path, preventing OpenCode's bun cache and
node_modules from polluting snapshot_text_files() and causing
spurious files_unchanged assertion failures
- Add XDG_DATA_HOME redirection to .xdg_data/ — without this, session
files written to ~/.local/share/opencode leaked between runs, causing
the second run to find a stale session referencing the previous
tmp_path and triggering a permission dialog against a non-existent
workspace (anomalyco/opencode#8538 root cause)
- Document the tmp_path layout in a block comment above xdg_base
KV cache flush fixture
- Add flush_llama_kv_cache autouse fixture to conftest.py, erasing all
llama-server slots via POST /slots/{id}?action=erase&id_slot={id}
before each test
- Without this, residual KV cache from prior tests caused the model to
reference stale workspace paths from earlier tmp_path directories
- Slot IDs are discovered dynamically from GET /slots so the fixture
adapts to any --parallel value
- Failure is a warning not an error — flush is best-effort
- Requires llama-server to be started with --slot-save-path (gates the
POST /slots endpoint regardless of whether save/restore is used)
- Add requests to requirements.txt
Permission pre-grant
- Add permission block to tools/opencode/opencode.json granting allow
for read, glob, grep, edit, write, and external_directory
- Eliminates the interactive permission dialog on every invocation:
OpenCode raises external_directory for all workspace paths because the
ephemeral workspace has no .git root, causing project ID to resolve
to "global" (anomalyco/opencode#8538)
- Permission state is in-memory only and not persisted across processes
(confirmed in permission/next.ts), so dialog would fire on every run
regardless of prior approvals
- Permission approval fallback (Tab+Enter for Allow always) retained in
run_opencode_tui with saw_post_submit_output reset after each approval
to prevent READY_RE match on post-approval redraw from prematurely
terminating the response-wait loop
Regex output assertions
- Add assert_matches_all and assert_not_matches_any functions accepting
regex patterns, wired to new YAML keys stdout_contains_re,
stdout_not_contains_re, stderr_contains_re, stderr_not_contains_re
- Required because TUI terminal wrapping splits long paths across lines,
breaking literal stdout_contains assertions for nested paths
- Update glob_discovery.yaml to use stdout_contains_re with pattern
src[\s/]*nested[\s/]*sample\.py, matching the path regardless of
how the TUI wraps it across line boundaries
Other
- Increase quiet_timeout from 2s to 5s to allow TUI rendering to
complete before the harness declares the response done
- Update write_file.yaml prompt to explicitly invoke the write tool,
preventing the model from responding with a plan instead of executing
- Add requests and pexpect to README install instructions1 parent 8385c90 commit 0633be9
4 files changed
Lines changed: 75 additions & 2 deletions
File tree
- opencode_pytest_harness
- tests
- cases
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
| 4 | + | |
4 | 5 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
7 | | - | |
| 7 | + | |
| 8 | + | |
8 | 9 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
| 4 | + | |
| 5 | + | |
4 | 6 | | |
5 | 7 | | |
6 | 8 | | |
| |||
11 | 13 | | |
12 | 14 | | |
13 | 15 | | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
14 | 19 | | |
15 | 20 | | |
16 | 21 | | |
| |||
50 | 55 | | |
51 | 56 | | |
52 | 57 | | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
84 | 84 | | |
85 | 85 | | |
86 | 86 | | |
87 | | - | |
| 87 | + | |
88 | 88 | | |
89 | 89 | | |
90 | 90 | | |
| |||
291 | 291 | | |
292 | 292 | | |
293 | 293 | | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
294 | 317 | | |
295 | 318 | | |
296 | 319 | | |
| |||
349 | 372 | | |
350 | 373 | | |
351 | 374 | | |
| 375 | + | |
| 376 | + | |
352 | 377 | | |
353 | 378 | | |
| 379 | + | |
| 380 | + | |
354 | 381 | | |
355 | 382 | | |
356 | 383 | | |
| |||
0 commit comments