QA Use-Case Suite

End-to-end QA scenarios for the thatch project. These are not unit tests — the regular unit suite lives in tests/*.test.ts and is run by mise run check. This suite verifies end-to-end behavior through the same interfaces a user would use (CLI, setup commands, module APIs), either with or without a live LLM session.

Directory structure

tests/qa/
  runner.ts          Shared library: UseCase interface, fixture setup, opencode runner
  auto/              Automatable use cases — no LLM, no model tokens, ~90 seconds
    index.test.ts    Barrel file that imports all auto use cases (for parallel execution)
    uc-NNN-name.ts   Individual use case definitions
  live/              Live-session use cases — spawn agent runs, cost tokens, up to 10 min each
    opencode/          The default leg: real opencode sessions (the docker sandbox or host binary)
    claude/            The CLAUDE leg: real `claude -p` sessions with the thatch plugin loaded - OPT-IN per run (never a default gate; model tokens + the host's logged-in claude CLI). Asserts only plugin-written surfaces (db/queue/logs), never the model's prose - see .claude/skills/thatch-plugin-qa/SKILL.md
    index.test.ts    Barrel file that imports all live use cases (for parallel execution)
    uc-NNN-name.ts   Individual use case definitions

auto/ vs live/

Put a use case in auto/ if it can be verified without a live LLM session. This means the scenario can be checked by:

• Running the CLI (bun run bin/thatch <subcommand>) and asserting on exit codes, stdout, or file artifacts written to disk
• Importing thatch modules directly (ThatchDB, ExtractionPipeline, extract-queue, setup, etc.) and calling functions with a temp DB
• Checking file presence and content after thatch setup --claude/--cursor

Put a use case in live/ if it requires a live agent session — the scenario needs the LLM to read a prompt, make tool calls, and respond. These use the default runViaOpencode helper (no custom run function).

Use manualOnly: true for use cases that cannot be automated at all (visual TUI verification, compaction triggers, real Claude Code/Cursor sessions).

Running

mise run qa          # auto first (&&), then live
mise run qa-auto     # only automatable (fast, no tokens)
mise run qa-live     # only live sessions
mise run qa-dry-run  # list all without spawning

Opencode-driven use cases resolve the opencode binary through the runner's discovery: a discovered v2 install wins over whatever PATH resolves (the plugin cannot load under a v1 binary).

Override the model with QA_MODEL=venice/<model-id>.

Sandbox isolation

Every spawned opencode process (runner, live sessions) runs inside a per-fixture sandbox. The runner sets all four XDG surfaces in the child env (runner.ts fixture env):

Surface	Override	What it isolates
XDG_STATE_HOME	dir/home/.local/state	Daemon discovery. The background daemon registers itself at $XDG_STATE_HOME/opencode/service.json (url, pid, password). This is the easy one to miss: without it a spawned opencode run JOINS THE REAL DAEMON - sessions land in the real session store and the plugin executes in the real daemon's env (real memories, real chat roster).
XDG_DATA_HOME	dir/home/.local/share	The session database. --standalone private servers still share the XDG-scoped session db, so data isolation needs its own override.
XDG_CONFIG_HOME	dir/config	Thatch's db/config + the plugin shim + installed skills.
THATCH_DB_PATH	dir/thatch.db	Thatch's own db directly (the plugin resolves this before the XDG default).

Anything spawning opencode OUTSIDE this runner (a manual reproduction of a live scenario, a sandboxed TUI smoke) must set the same four. Even then the launching shell's env leaks in (direnv's OPENCODE_CONFIG layers real MCP server configs into the sandbox, whose credentials live in the overridden data dir - spurious re-auth prompts). The docker sandbox (qa/opencode-sandbox/) ends that class: the container has exactly the env it is given.

Known coupling: the fixture PRE-COPIES skills from the real ~/.config/opencode/skills, and uc-014's opencode leg asserts COPY FIDELITY - the fixture's skill set must equal the live mirror's at run time (no floor, no count literal). A stale mirror therefore passes silently while exercising nothing new; refresh it after committing skill changes. Clear a stale master cache with rm -rf "$TMPDIR"/thatch-qa-master-* (the runner's roots are run-scoped; see the isolation section).

Docker mode (the default qa-auto/qa-live targets)

mise run qa-auto / mise run qa-live run through the qa/opencode-sandbox container: a pinned opencode v2 (2.0.23) with a throwaway XDG tree built per container. mise run qa-host-auto / qa-host-live run against the HOST binary instead (the CI-shaped mode). Details:

• The image builds on first use (and on demand: docker build -t thatch-qa-opencode qa/opencode-sandbox).
• The fixture dir, the master copy, the checkout, and the real node_modules mount at their HOST paths, and every fixture env var passes into the container explicitly (PATH excepted - the image's own PATH finds its binary).
• The docker leg runs a single unlabeled leg: the image pins opencode v2, so host-binary discovery does not apply.
• startServe use cases (UC-100) still spawn the serve process host-side.
• Serve mode (a long-lived headless server for live-session experiments across plugin reloads): run the container with serve --hostname 0.0.0.0 --port 4096 + -p 127.0.0.1:4096:4096 + the pinned OPENCODE_SERVER_PASSWORD + THATCH_DEBUG=1 + THATCH_WATCH_POLL_SECONDS=5 (the default 60s cadence makes mid-turn timing non-deterministic). The SDK/scripts talk to the mapped port with basic auth + the x-opencode-directory header. The reload-loss repro (qa/opencode-sandbox/reload-loss-repro.ts) is the worked example: it verifies pending-event durability across a plugin reload end to end.
• The default model is venice/z-ai-glm-5-3-flash (cheap and fast); override with QA_MODEL. The sandbox entrypoint's fallback config must be kept in sync with the runner's MODEL default.

Adding a use case

1. Create tests/qa/auto/uc-NNN-name.ts or tests/qa/live/<leg>/uc-NNN-name.ts (no .test.ts extension — only the barrel file uses that).
2. Import registerUseCase and UseCase (and QaContext if automatable) from ../runner. If the use case spawns opencode itself (custom run that shells out, or a serve), set spawnsOpencode: true so the runner resolves the opencode binary for it - and build run invocations with opencodeRunArgs (it carries the --standalone flag a fixture sandbox needs).
3. Define the scenario with name, preconditions, steps, and expected as string arrays joined by \n.
4. For automatable use cases, add a run(ctx: QaContext) function that verifies the scenario and returns "PASS", "FAIL", or "PARTIAL".
5. Call registerUseCase(useCase).
6. Add an import "./uc-NNN-name"; line to the corresponding index.test.ts barrel file.

The runner handles fixture setup (isolated repo copy, temp DB, env vars), dry-run skipping, manual-only skipping, timeouts, and result assertions.

Why barrel files?

Bun's --concurrent flag parallelizes tests within a single file, not across files. Each use case file calls registerUseCase() which calls test.concurrent(), but if each lives in its own .test.ts file, bun runs them one at a time. The index.test.ts barrel imports all use case modules so their test.concurrent() calls register in one file, and --concurrent --max-concurrency 5 runs 5 at once.

Isolation

Each qa-auto process gets its own run-scoped roots: the master copy at $TMPDIR/thatch-qa-master-<run-id> and the fixture tree under /tmp/thatch-qa/<run-id>/. Two concurrent qa-auto runs (two worktrees gating at once) are therefore independent - one run's setup can never delete the other's live fixtures (tests/qa-runner-isolation.test.ts pins this with real concurrent subprocesses).

Each use case runs inside a copy of the repo at /tmp/thatch-qa/<run-id>/<name>/. The copy is created via git archive HEAD (tracked files only) with node_modules symlinked from the real repo. Opencode's own npm deps and pre-installed skills are copied from ~/.config/opencode/. Each copy has its own temp database, config dirs, and home directory. The real repo, real config, and real database are never touched.

Requires VENICE_API_KEY in the environment for live-session use cases. Automatable use cases don't need it.