Skip to content

gsd-ui-auditor: the sole screenshot path misreads a redirecting dev server as absent, prints "Screenshots captured" when every capture failed, and never tries the port fallback its own next line documents #4176

Closed as duplicate

Description

@0xdhx

Pre-submission checklist

  • I have searched existing issues and this bug has not already been reported

Closest prior art is #2526 (closed/completed by #2594), which removed the uncallable <playwright_mcp_approach> block. That PR is what made the block below the auditor's sole capture path; it touched none of the three lines reported here. This is a follow-on, not a duplicate.

GSD Version

1.12.0

Verified against pristine upstream, not the installed tree: agents/gsd-ui-auditor.md is byte-identical at tag v1.12.0 and at next (457 lines, decoded sha1 fd487dc2608726c0b4505cb7be7ec5139b7b3801), fetched via the contents API at each ref. Every line number below is exact at both refs.

Disclosure: I maintain an internal patched copy of this file, and it already fixes two of the three issues below. Everything reported here was re-derived and independently re-verified against canonical upstream content only — the local copy was deliberately excluded from the analysis. I am reporting these upstream rather than keeping the patch to myself.

Runtime

Claude Code

Operating System

Linux (WSL) — Ubuntu 24.04.4 LTS on kernel 6.6.114.1-microsoft-standard-WSL2

Node.js Version

v24.14.1

Installation Method

npx (user-scope install under the Claude Code config dir)

What happened?

agents/gsd-ui-auditor.md carries one screenshot path — <screenshot_approach>, lines 107-144 — and :367-369 makes it mandatory: "## Step 3: Detect Dev Server and Capture Screenshots / Run the screenshot approach from <screenshot_approach>." Since #2594 removed the MCP branch, and :4 grants tools: Read, Write, Bash, Grep, Glob, Skill (no MCP), this block is the agent's only way to produce visual evidence.

Its success path is unverified end to end. Three defects in 28 contiguous lines:

https://github.com/open-gsd/gsd-core/blob/v1.12.0/agents/gsd-ui-auditor.md#L107-L144

1. A redirecting dev server is misread as no dev server

113  DEV_STATUS=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:3000 2>/dev/null || echo "000")
115  if [ "$DEV_STATUS" = "200" ]; then
...
136    echo "No dev server at localhost:3000 — code-only audit"

curl is invoked without -L, so a dev server that redirects / returns 307/302 and fails the equality test. The audit silently degrades to code-only against a server that is serving normally. Real cases: Next.js i18n or middleware redirects, auth redirects, trailing-slash normalization. (Not Vite — its default / does not redirect.)

The = "200" equality is a second, separable problem: it also rejects 401/403 (auth-gated dev servers) and any 2xx that is not exactly 200. And :113 has no --max-time, so a port that accepts but never responds hangs the probe indefinitely.

Upstream already has a correct probe on another surface — gsd-core/references/checkpoints.md#L495 uses node -e "fetch(...).then(r=>{process.exit(r.ok?0:1)})", whose own comment reads "uses fetch() for cross-platform compatibility" — fetch follows redirects and r.ok accepts 200-299.

2. "Screenshots captured" prints even when every capture failed

120    npx playwright screenshot http://localhost:3000 \
121      "$SCREENSHOT_DIR/desktop.png" \
122      --viewport-size=1440,900 2>/dev/null
...
134    echo "Screenshots captured to $SCREENSHOT_DIR"

All three invocations discard stderr and no exit status is ever checked — there is no set -e, no trap, no ||/&&, no $? test, and no [ -f ... ] existence check anywhere in the file. Line 134 prints unconditionally. npx's non-interactive consent prompt makes an unprovisioned Playwright the common failure, not a rare one.

The false claim then reaches durable artifacts:

  • :302 — **Screenshots:** {captured / not captured (no dev server)}, the scored UI-REVIEW.md header
  • :409 — the same field on the orchestrator return

So a report can state screenshots were captured while the visual pillars were scored with no evidence, and an empty timestamped .planning/ui-reviews/ directory is left behind.

In fairness: :369 says "Record whether screenshots were captured". But nothing instructs a check, and the only signal the block emits is the unconditional echo — so an agent following the block faithfully has no way to record anything else.

3. The port fallback the next line documents is never attempted

142  Try port 3000 first, then 5173 (Vite default), then 8080.

Port 3000 is hard-coded at :113, :120, :125, :130, and :136's terminal verdict names it. The block emits "No dev server at localhost:3000 — code-only audit" before 5173 or 8080 are ever tried — directly contradicting the line immediately beneath it.

Two things I checked so this is not misread:

I have framed this as a contradiction rather than "the fallback does not exist" deliberately: the agent is an LLM with a Bash grant and could satisfy :142 by hand. The defect is that the block's literal control flow terminates first.

What did you expect?

  1. A dev server that answers on localhost:3000 should be detected whether or not / redirects, and the probe should not hang on an unresponsive port.
  2. Screenshots captured — and the Screenshots: field in UI-REVIEW.md — should reflect whether files were actually produced.
  3. The block should try 5173 and 8080 before declaring no dev server, or :142 should be removed.

Steps to reproduce

Repro status: defects 1 and 2 are executed against the verbatim upstream lines; defect 3 is source-derived and confirmed by reading :113-142.

These reproduce the block's own lines directly rather than a full /gsd-ui-review run, because the block is what is defective and running it verbatim is the sharper test.

Defect 1 — start any server on port 3000 whose / returns a 307 to a path that serves 200, then run line 113 verbatim:

$ DEV_STATUS=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:3000 2>/dev/null || echo "000")
$ echo "DEV_STATUS=$DEV_STATUS"
DEV_STATUS=307

$ if [ "$DEV_STATUS" = "200" ]; then echo CAPTURE; else echo "No dev server at localhost:3000 — code-only audit"; fi
No dev server at localhost:3000 — code-only audit

$ curl -s -o /dev/null -w "final=%{http_code} after %{num_redirects} redirect(s)\n" -L http://localhost:3000
final=200 after 1 redirect(s)

The server is serving. The auditor reports it absent.

Defect 2 — run the verbatim block with npx unreachable (or Playwright unprovisioned, which is the realistic case):

Screenshots captured to /tmp/uiaudit-repro
block rc=0

$ ls -A /tmp/uiaudit-repro
(empty)

Success line, exit 0, zero files.

Defect 3 — read :113 against :142. With nothing on 3000 and a dev server on 5173, the block prints the code-only verdict without probing 5173.

How often does this happen?

Every time (100% reproducible) for defects 1 and 3. Defect 2 fires whenever a capture fails for any reason.

Impact

Moderate. The audit still produces a UI-REVIEW.md — but one whose Screenshots: line can be false and whose visual pillars were scored with evidence that silently went missing. A wrong report is worse than an absent one, which is the reason I am filing this rather than living with it.

Suggested fix

All three are in the same 28-line region and any fix rewrites overlapping lines, so one change:

  • Replace the probe with a redirect-following, range-accepting, time-bounded check — follow redirects, accept 2xx (and decide deliberately about 401/403), and bound it with --max-time. The fetch-based probe linked above is an in-repo precedent.
  • Loop the probe over 3000, 5173, 8080 so :142 becomes true, and carry the resolved port into the capture commands instead of hard-coding 3000.
  • Check each capture's exit status, drop the blanket 2>/dev/null (or capture stderr for the report), and make :134 plus the Screenshots: fields at :302/:409 reflect what was actually produced — including a partial state, since one of three viewports can fail alone.

Happy to open a PR with a source-text regression test once this is labeled confirmed-bug — tests/playwright-ui-verify.test.cjs already asserts against this file's text and looks like the right home. Per CONTRIBUTING.md I am not writing code before that label.

Privacy Checklist

  • I have reviewed all pasted output for PII (usernames, paths, API keys) and redacted where necessary

Activity

  1. trek-e commented on Sep 2, 2026

    @trek-e
    Collaborator

    This was generated by AI during triage.

    Diagnosis

    Reproduced: yes — defect 1: ran line 113's probe verbatim (port substituted with a free one) against a Node server whose / returns 307 → /home (200); DEV_STATUS=307, the block's branch prints "No dev server … code-only audit" while curl -L on the same URL returns final=200 after 1 redirect(s). Defect 2: ran the verbatim capture block with npx stubbed to fail on stderr; output was "Screenshots captured to …", rc=0, zero files, empty timestamped directory left behind. Defect 3 is source-confirmed: no loop exists, 3000 is hard-coded at lines 113/120/125/130 and named in the verdict at 136, and a repo-wide grep for 5173 hits only this file's guidance line plus the two references/checkpoints.md copies (neither loaded by the auditor).
    Root cause: three defects in agents/gsd-ui-auditor.md's sole screenshot path: (1) :113 curls without -L and :115 requires exact "200", with no --max-time — a redirecting/auth-gating/unresponsive port is misread as "no dev server"; (2) :120-132 discard stderr with no exit-status or file checks and :134 echoes success unconditionally, propagating to the Screenshots: fields at :302 and :409; (3) :142 documents a 3000→5173→8080 fallback the block's control flow never attempts.
    Introduced by: long-standing lines; elevated to sole-capture-path by #2594 (commit 3274db275, 2026-07-28), whose hunk @@ -104,46 +104,6 @@ removed only <playwright_mcp_approach> and ended before <screenshot_approach> — verified via git show. Matches the issue's history claim exactly.
    Blast radius: moderate — every /gsd-ui-review run routes through this block (:367-369); worst case is a UI-REVIEW.md claiming screenshot evidence that does not exist plus orphaned empty .planning/ui-reviews/<ts>/ dirs; Vite/8080 deployments are always code-only. No code in src/ parses this file.
    Coupled (must not miss): the :302/:409 template fields must change with :134 (including partial-capture state); a port fix must carry the resolved port into the three capture commands, not just the probe; in-repo precedent at gsd-core/references/checkpoints.md:494-495 (fetch() in a run-with-timeout 30 loop — follows redirects, accepts 2xx, bounded) is the pattern to match.
    Regression test: source-text assertions in tests/playwright-ui-verify.test.cjs (it already asserts against this file's text at lines 42-48): (a) probe text contains redirect-following (-L or fetch() and a time bound; (b) an exit-status or file-existence check appears between the capture commands and any "captured" message; (c) 5173 and 8080 appear inside the probe/capture control flow, not only a guidance sentence.

    Agent Brief

    Category: bug
    Summary: The UI auditor's only screenshot path misclassifies live dev servers as absent, reports "Screenshots captured" after total capture failure, and never tries the port fallback its own next line documents.
    Current behavior: A dev server whose root path redirects (or returns any non-200, or accepts but never responds) is treated as absent and the audit silently degrades to code-only; when capture commands fail, the success message still prints and the review report's Screenshots field claims evidence that was never produced; only port 3000 is ever probed despite documented fallbacks.
    Desired behavior: Server detection follows redirects, accepts 2xx (with a deliberate decision on 401/403), and is time-bounded; the captured/not-captured signal reflects actual command exit statuses and files on disk, including partial capture; ports 3000, 5173, and 8080 are tried in order and the resolved port is used for every capture.
    Key interfaces: the screenshot_approach bash block, the dev-server probe, the npx playwright screenshot capture commands, the Screenshots field of the UI-REVIEW.md template and the completion-return template.
    Acceptance criteria: a redirecting dev server triggers capture rather than a code-only verdict; with capture commands failing, no "captured" claim reaches the report and the failure is recorded; with nothing on 3000 but a server on 5173, screenshots are taken from 5173; the report distinguishes full, partial, and no capture.
    Out of scope: restoring a Playwright-MCP capture branch (removed by design in #2594), changing the agent's tool grants, and any behavioral change to the code-only audit path itself.

  2. added
    bugSomething isn't working
    and removed
    needs-triageNew issue awaiting maintainer review
    on Sep 2, 2026
  3. github-actions commented on Sep 2, 2026

    @github-actions
    Contributor

    Branch fix/4176-gsd-ui-auditor-the-sole-screenshot-path- created.

    git fetch origin && git checkout fix/4176-gsd-ui-auditor-the-sole-screenshot-path-
  4. 91 remaining items

  5. added 5 commits that reference this issue on Sep 9, 2026
  6. trek-e commented on Sep 11, 2026

    @trek-e
    Collaborator

    Closing as a duplicate of #4631 — consolidating at the seam rather than fixing per-symptom.

    This is not a dismissal of the report. Your specifics are preserved verbatim in that epic's Absorbed issues table, and its Done when checklist requires a failing-first regression test for this exact defect before the epic can close.

    Why it is being consolidated: a scan of all 69 open confirmed-bug issues (2026-09-11) found this one shares a single root cause with seven other open issues — a gate hand-rolls a narrow derivation of what it should evaluate, the derivation comes back empty, and "empty" is reported as "pass". Fixing each occurrence where it was found leaves the remaining call sites carrying the same bug, which is the pattern that produced this cluster in the first place.

    The epic delivers one resolveEvaluationScope(), evidence read through the existing document seams instead of per-gate regex, and an Evidence type that distinguishes "none" from "could not read" so a gate can no longer pass vacuously, plus a machine-enforced anti-divergence ratchet so a new hand-written copy cannot be introduced — the approach that worked for #1372 (markdown sectionizer), #2143 (table + bounded mutation) and #2121 (phase-id), each of which closed its bug class structurally.

    Track the fix at #4631.

  7. added
    duplicateThis issue or pull request already exists
    on Sep 11, 2026
  8. added 7 commits that reference this issue on Oct 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingconfirmed-bugVerified reproducible bugduplicateThis issue or pull request already exists

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions