Skip to content

feat(guardrails): patterns L-O — test verification gaps beyond A-K (env-dependent skips, worktree env, runner API limits, env-var instability) #290

Closed

Description

@terisuke

Summary

A–K cover removal safety, permissions, planning, hygiene, falsification, self-check, CI tiering, and SSOT sync. Field work on Grift (2026-08-03, PRs anomalyco#1761/anomalyco#1762) surfaced four more failure classes that the current set cannot see. Each was observed live, not hypothesized.


Pattern L — Silent test skips are accepted as green

Observed (Grift anomalyco#1761/anomalyco#1762): DB-backed proofs (conversation turn seq, migration, dead-letter policy, share-link reentry) call t.Skip(...) when *_TEST_DATABASE_URL is unset. With PostgreSQL down (docker stopped), mise run test reported green while 7+ store proofs silently never ran. The implementer pushed with zero DB verification; CI caught the gap later.

Why not A–K: A–K assume tests run. Nothing counts skipped tests at the boundary.

Proposal: test-result-honesty guard —

  • /plan-light / implement verification steps must report skip counts (N passed, M skipped) and treat skipped == critical proof tests as NOT verified.
  • Plugin hook: after mise run test / go test / pytest, parse output for SKIP / t.Skip; if a known-critical suite skipped, emit a warning that verification is incomplete.
  • CI wiring test: assert that CI's own env always sets the DB URLs (so CI never skips what local could not run).

Pattern M — Worktree environments are not provisioned

Observed (Grift anomalyco#1762): A fresh git worktree add has no node_modules and no shared package-manager cache. Running vitest in the worktree fails with ERR_MODULE_NOT_FOUND until the agent manually symlinks ../../node_modules. Every new worktree repeats this ritual; agents waste turns discovering it.

Why not A–K: A is about deletion safety; nothing provisions a working environment.

Proposal: worktree-bootstrap guard —

  • Plugin hook on worktree creation (or a /repo-hygiene step): detect node_modules absence in the new worktree and emit the exact symlink commands (or refuse to start implementation until provisioned).
  • Document the monorepo convention (node_modules lives at root; worktrees share via symlink).

Pattern N — Test-runner API limits create vacuous tests

Observed (Grift anomalyco#1666): fireEvent.keyDown(textarea, { isComposing: true }) cannot set isComposing on the native event (read-only KeyboardEvent prop), so a guard reading event.nativeEvent.isComposing always saw false. Every IME test passed vacuously. Root cause: the test author did not know fireEvent drops these props; the fix required dispatching a real new KeyboardEvent('keydown', { isComposing: true }).

Why not F (falsification): F assumes a test CAN be written that detects the bug; here the harness API silently made detection impossible. F does not teach which event properties fireEvent cannot carry.

Proposal: harness-api-gotcha skill/guide —

  • Catalog the API limits that cause vacuous tests: fireEvent & read-only native event props (isComposing, keyCode), negative waitFor passing at 0ms, mock streams resolving immediately flipping isRunning false.
  • CI wiring test: the falsifiable-change skill must include a checklist item "does the harness actually deliver the event properties under test?"

Pattern O — Env-var-dependent test instability

Observed (Grift anomalyco#1762): apps/web streaming tests red on machines with VITE_CLERK_PUBLISHABLE_KEY set (Clerk session branch taken, buildHeaders throws tenant-mapping error before fetch). Pre-existing 3 failures only reproduced on the implementer's machine; CI stayed green because CI lacks the var. Fix: pin runtime-config in tests via vi.mock.

Why not A–K: D (env-check) checks dev environment readiness, not per-test env-var hermeticity.

Proposal: env-hermetic-tests guard —

  • Test files that exercise code paths gated by isClerkConfigured / isFirebaseAuthConfigured must pin those flags via mock (no reliance on ambient env).
  • CI wiring test: grep test files calling such gates; warn if no mock is present.

Acceptance (pattern I compliance)

Each guard must include: negative tests (guarded-pass), positive tests (guarded-fail), and falsification proof (disable → behavior changes).

Scope

Fork-side profile plugins/commands/skills + CI wiring tests. No upstream changes.

Refs

Grift PRs anomalyco#1761 (L), anomalyco#1762 (M/N/O); existing patterns #284 (A–J), #288 (K), epic #1

Activity

  1. terisuke commented on Aug 3, 2026

    @terisuke
    Author

    実装マージ済み: #291

    パターン 実装
    L test-honesty.ts + /test-honesty — skip 数と critical DB skip を助言
    M worktree-bootstrap.ts — worktree の node_modules 欠如を symlink 手順付きで助言
    N skill harness-api-gotcha + falsifiable-change チェックリスト
    O env-hermetic.ts + skill env-hermetic-tests — gate 無 mock を結線検出

    受入: 陰性/陽性/反証(OPENCODE_*_GUARD=off)付き。ハードブロックなし(I)。
    テスト 17 pass、local:check green、CI 全 green。

    ローカル entrypoint は .worktrees/cursor/test-reliability-guards に pin 済みです。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions