Skip to content

Gateway behavioral test suite is gated out of CI — recently-merged fixes have no behavioral coverage #2720

Description

@alexey-pelykh

Problem

The gateway behavioral test suite (src/gateway/**/*.test.ts, run via vitest.gateway.config.ts) does not run in CI on pull requests:

  • The unit test job uses vitest.unit.config.ts, which excludes src/gateway/** (vitest.unit.config.ts:18).
  • scripts/test-parallel.mjs only includes the gateway suite when REMOTECLAW_TEST_INCLUDE_GATEWAY === "1" (scripts/test-parallel.mjs:105) — which CI does not set.

The only gateway test that runs in CI is src/gateway/server.preauth-hardening.test.ts, in the dedicated test-gateway-preauth lane (.github/workflows/ci.yml:170). The CI comment states the reason directly:

The full gateway suite is gated behind REMOTECLAW_TEST_INCLUDE_GATEWAY=1 because of orthogonal pre-existing failures unrelated to this slice

So the entire gateway behavioral suite is dark in CI because of ~2 pre-existing failing tests — an all-or-nothing exclusion rather than skipping just the failing tests.

Impact

Two recently-merged gateway fixes have their only behavioral tests in this dark suite, so neither is protected by CI:

Fix Source Test file (not run in CI)
Surface error-only webchat agent runs (don't silently complete as success) src/gateway/server-methods/chat.ts (#2719) src/gateway/server-methods/chat.error-propagation.test.ts
Reject delivery to unconfigured plugin channels (fall back to the internal channel) src/gateway/server-methods/agent.ts (#2718) src/gateway/server.agent.gateway-server-agent-b.test.ts

Both fixes passed CI only via type-check (tsgo) and build — not behavior. A future edit to either file (including a large merge from upstream) could break the behavior and pass CI silently. The chat.ts error-surfacing fix in particular was previously lost in a large merge and went undetected for months, precisely because its test was not running in CI and passed vacuously against the reverted code.

Proposed fix (quarantine-and-re-enable)

  1. Identify the ~2 pre-existing failing gateway tests (the "orthogonal pre-existing failures" the CI comment refers to).
  2. Mark only those with explicit it.skip / describe.skip and a comment linking back to this issue, so the failures are visible and tracked rather than silently swept under the env-var gate.
  3. Run the rest of the gateway suite in CI on every PR — either by including src/gateway/** in the test job's path, or by adding a dedicated gateway lane (mirroring test-gateway-preauth) and wiring it into the aggregate CI job's required-results check (.github/workflows/ci.yml:280).

This converts a binary all-or-nothing gate into per-test quarantine and restores behavioral CI coverage for the rest of the suite, including the two fixes above.

Acceptance criteria

  • The gateway behavioral suite runs in CI on every PR (not gated behind REMOTECLAW_TEST_INCLUDE_GATEWAY=1).
  • The ~2 known-failing tests are explicitly skipped with a tracking reference — not silently excluded via the env-var gate.
  • src/gateway/server-methods/chat.error-propagation.test.ts and src/gateway/server.agent.gateway-server-agent-b.test.ts execute in CI and pass.
  • The aggregate CI job fails if the gateway suite fails (wired into the required-results check).

🤖 Generated with Claude Code

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions