Repository navigation
Description
Pre-submission checklist
- I have searched existing issues and this bug has not already been reported
Closest prior art is #2526 (closed/completed by #2594), which removed the uncallable <playwright_mcp_approach> block. That PR is what made the block below the auditor's sole capture path; it touched none of the three lines reported here. This is a follow-on, not a duplicate.
GSD Version
1.12.0
Verified against pristine upstream, not the installed tree: agents/gsd-ui-auditor.md is byte-identical at tag v1.12.0 and at next (457 lines, decoded sha1 fd487dc2608726c0b4505cb7be7ec5139b7b3801), fetched via the contents API at each ref. Every line number below is exact at both refs.
Disclosure: I maintain an internal patched copy of this file, and it already fixes two of the three issues below. Everything reported here was re-derived and independently re-verified against canonical upstream content only — the local copy was deliberately excluded from the analysis. I am reporting these upstream rather than keeping the patch to myself.
Runtime
Claude Code
Operating System
Linux (WSL) — Ubuntu 24.04.4 LTS on kernel 6.6.114.1-microsoft-standard-WSL2
Node.js Version
v24.14.1
Installation Method
npx (user-scope install under the Claude Code config dir)
What happened?
agents/gsd-ui-auditor.md carries one screenshot path — <screenshot_approach>, lines 107-144 — and :367-369 makes it mandatory: "## Step 3: Detect Dev Server and Capture Screenshots / Run the screenshot approach from <screenshot_approach>." Since #2594 removed the MCP branch, and :4 grants tools: Read, Write, Bash, Grep, Glob, Skill (no MCP), this block is the agent's only way to produce visual evidence.
Its success path is unverified end to end. Three defects in 28 contiguous lines:
1. A redirecting dev server is misread as no dev server
113 DEV_STATUS=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:3000 2>/dev/null || echo "000")
115 if [ "$DEV_STATUS" = "200" ]; then
...
136 echo "No dev server at localhost:3000 — code-only audit"curl is invoked without -L, so a dev server that redirects / returns 307/302 and fails the equality test. The audit silently degrades to code-only against a server that is serving normally. Real cases: Next.js i18n or middleware redirects, auth redirects, trailing-slash normalization. (Not Vite — its default / does not redirect.)
The = "200" equality is a second, separable problem: it also rejects 401/403 (auth-gated dev servers) and any 2xx that is not exactly 200. And :113 has no --max-time, so a port that accepts but never responds hangs the probe indefinitely.
Upstream already has a correct probe on another surface — gsd-core/references/checkpoints.md#L495 uses node -e "fetch(...).then(r=>{process.exit(r.ok?0:1)})", whose own comment reads "uses fetch() for cross-platform compatibility" — fetch follows redirects and r.ok accepts 200-299.
2. "Screenshots captured" prints even when every capture failed
120 npx playwright screenshot http://localhost:3000 \
121 "$SCREENSHOT_DIR/desktop.png" \
122 --viewport-size=1440,900 2>/dev/null
...
134 echo "Screenshots captured to $SCREENSHOT_DIR"All three invocations discard stderr and no exit status is ever checked — there is no set -e, no trap, no ||/&&, no $? test, and no [ -f ... ] existence check anywhere in the file. Line 134 prints unconditionally. npx's non-interactive consent prompt makes an unprovisioned Playwright the common failure, not a rare one.
The false claim then reaches durable artifacts:
:302—**Screenshots:** {captured / not captured (no dev server)}, the scoredUI-REVIEW.mdheader:409— the same field on the orchestrator return
So a report can state screenshots were captured while the visual pillars were scored with no evidence, and an empty timestamped .planning/ui-reviews/ directory is left behind.
In fairness: :369 says "Record whether screenshots were captured". But nothing instructs a check, and the only signal the block emits is the unconditional echo — so an agent following the block faithfully has no way to record anything else.
3. The port fallback the next line documents is never attempted
142 Try port 3000 first, then 5173 (Vite default), then 8080.
Port 3000 is hard-coded at :113, :120, :125, :130, and :136's terminal verdict names it. The block emits "No dev server at localhost:3000 — code-only audit" before 5173 or 8080 are ever tried — directly contradicting the line immediately beneath it.
Two things I checked so this is not misread:
- No other file the agent loads implements the fallback. A repo-wide search for
5173returns three files: this one and the tworeferences/checkpoints.mdcopies, neither of which the auditor loads. - It is not orphaned text left by fix(#2526): remove gsd-ui-auditor's uncallable Playwright-MCP block #2594. The same line sits at
v1.8.0:182inside<screenshot_approach>, and fix(#2526): remove gsd-ui-auditor's uncallable Playwright-MCP block #2594's hunk (@@ -104,46 +104,6 @@) removes only<playwright_mcp_approach>, ending before<screenshot_approach>opens.
I have framed this as a contradiction rather than "the fallback does not exist" deliberately: the agent is an LLM with a Bash grant and could satisfy :142 by hand. The defect is that the block's literal control flow terminates first.
What did you expect?
- A dev server that answers on
localhost:3000should be detected whether or not/redirects, and the probe should not hang on an unresponsive port. Screenshots captured— and theScreenshots:field inUI-REVIEW.md— should reflect whether files were actually produced.- The block should try 5173 and 8080 before declaring no dev server, or
:142should be removed.
Steps to reproduce
Repro status: defects 1 and 2 are executed against the verbatim upstream lines; defect 3 is source-derived and confirmed by reading :113-142.
These reproduce the block's own lines directly rather than a full /gsd-ui-review run, because the block is what is defective and running it verbatim is the sharper test.
Defect 1 — start any server on port 3000 whose / returns a 307 to a path that serves 200, then run line 113 verbatim:
$ DEV_STATUS=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:3000 2>/dev/null || echo "000")
$ echo "DEV_STATUS=$DEV_STATUS"
DEV_STATUS=307
$ if [ "$DEV_STATUS" = "200" ]; then echo CAPTURE; else echo "No dev server at localhost:3000 — code-only audit"; fi
No dev server at localhost:3000 — code-only audit
$ curl -s -o /dev/null -w "final=%{http_code} after %{num_redirects} redirect(s)\n" -L http://localhost:3000
final=200 after 1 redirect(s)
The server is serving. The auditor reports it absent.
Defect 2 — run the verbatim block with npx unreachable (or Playwright unprovisioned, which is the realistic case):
Screenshots captured to /tmp/uiaudit-repro
block rc=0
$ ls -A /tmp/uiaudit-repro
(empty)
Success line, exit 0, zero files.
Defect 3 — read :113 against :142. With nothing on 3000 and a dev server on 5173, the block prints the code-only verdict without probing 5173.
How often does this happen?
Every time (100% reproducible) for defects 1 and 3. Defect 2 fires whenever a capture fails for any reason.
Impact
Moderate. The audit still produces a UI-REVIEW.md — but one whose Screenshots: line can be false and whose visual pillars were scored with evidence that silently went missing. A wrong report is worse than an absent one, which is the reason I am filing this rather than living with it.
Suggested fix
All three are in the same 28-line region and any fix rewrites overlapping lines, so one change:
- Replace the probe with a redirect-following, range-accepting, time-bounded check — follow redirects, accept 2xx (and decide deliberately about 401/403), and bound it with
--max-time. Thefetch-based probe linked above is an in-repo precedent. - Loop the probe over 3000, 5173, 8080 so
:142becomes true, and carry the resolved port into the capture commands instead of hard-coding 3000. - Check each capture's exit status, drop the blanket
2>/dev/null(or capture stderr for the report), and make:134plus theScreenshots:fields at:302/:409reflect what was actually produced — including a partial state, since one of three viewports can fail alone.
Happy to open a PR with a source-text regression test once this is labeled confirmed-bug — tests/playwright-ui-verify.test.cjs already asserts against this file's text and looks like the right home. Per CONTRIBUTING.md I am not writing code before that label.
Privacy Checklist
- I have reviewed all pasted output for PII (usernames, paths, API keys) and redacted where necessary
Activity
trek-e commentedon Sep 2, 2026
Diagnosis
Reproduced: yes — defect 1: ran line 113's probe verbatim (port substituted with a free one) against a Node server whose
/returns 307 →/home(200);DEV_STATUS=307, the block's branch prints "No dev server … code-only audit" whilecurl -Lon the same URL returnsfinal=200 after 1 redirect(s). Defect 2: ran the verbatim capture block withnpxstubbed to fail on stderr; output was "Screenshots captured to …", rc=0, zero files, empty timestamped directory left behind. Defect 3 is source-confirmed: no loop exists, 3000 is hard-coded at lines 113/120/125/130 and named in the verdict at 136, and a repo-wide grep for5173hits only this file's guidance line plus the tworeferences/checkpoints.mdcopies (neither loaded by the auditor).Root cause: three defects in
agents/gsd-ui-auditor.md's sole screenshot path: (1):113curls without-Land:115requires exact"200", with no--max-time— a redirecting/auth-gating/unresponsive port is misread as "no dev server"; (2):120-132discard stderr with no exit-status or file checks and:134echoes success unconditionally, propagating to theScreenshots:fields at:302and:409; (3):142documents a 3000→5173→8080 fallback the block's control flow never attempts.Introduced by: long-standing lines; elevated to sole-capture-path by #2594 (commit
3274db275, 2026-07-28), whose hunk@@ -104,46 +104,6 @@removed only<playwright_mcp_approach>and ended before<screenshot_approach>— verified via git show. Matches the issue's history claim exactly.Blast radius: moderate — every
/gsd-ui-reviewrun routes through this block (:367-369); worst case is aUI-REVIEW.mdclaiming screenshot evidence that does not exist plus orphaned empty.planning/ui-reviews/<ts>/dirs; Vite/8080 deployments are always code-only. No code insrc/parses this file.Coupled (must not miss): the
:302/:409template fields must change with:134(including partial-capture state); a port fix must carry the resolved port into the three capture commands, not just the probe; in-repo precedent atgsd-core/references/checkpoints.md:494-495(fetch()in arun-with-timeout 30loop — follows redirects, accepts 2xx, bounded) is the pattern to match.Regression test: source-text assertions in
tests/playwright-ui-verify.test.cjs(it already asserts against this file's text at lines 42-48): (a) probe text contains redirect-following (-Lorfetch() and a time bound; (b) an exit-status or file-existence check appears between the capture commands and any "captured" message; (c) 5173 and 8080 appear inside the probe/capture control flow, not only a guidance sentence.Agent Brief
Category: bug
Summary: The UI auditor's only screenshot path misclassifies live dev servers as absent, reports "Screenshots captured" after total capture failure, and never tries the port fallback its own next line documents.
Current behavior: A dev server whose root path redirects (or returns any non-200, or accepts but never responds) is treated as absent and the audit silently degrades to code-only; when capture commands fail, the success message still prints and the review report's Screenshots field claims evidence that was never produced; only port 3000 is ever probed despite documented fallbacks.
Desired behavior: Server detection follows redirects, accepts 2xx (with a deliberate decision on 401/403), and is time-bounded; the captured/not-captured signal reflects actual command exit statuses and files on disk, including partial capture; ports 3000, 5173, and 8080 are tried in order and the resolved port is used for every capture.
Key interfaces: the
screenshot_approachbash block, the dev-server probe, thenpx playwright screenshotcapture commands, the Screenshots field of the UI-REVIEW.md template and the completion-return template.Acceptance criteria: a redirecting dev server triggers capture rather than a code-only verdict; with capture commands failing, no "captured" claim reaches the report and the failure is recorded; with nothing on 3000 but a server on 5173, screenshots are taken from 5173; the report distinguishes full, partial, and no capture.
Out of scope: restoring a Playwright-MCP capture branch (removed by design in #2594), changing the agent's tool grants, and any behavioral change to the code-only audit path itself.
github-actions commentedon Sep 2, 2026
Branch
fix/4176-gsd-ui-auditor-the-sole-screenshot-path-created.git fetch origin && git checkout fix/4176-gsd-ui-auditor-the-sole-screenshot-path-91 remaining items
test(open-gsd#4176): an operator ends a word too — `#` after `;`, `|`…
test(open-gsd#4176): a `)` closing a process substitution or an extgl…
test(open-gsd#4176): an opener inside double quotes is literal text, …
test(open-gsd#4176): name every timeout in the new suite, using the s…
test(open-gsd#4176): hand bash a script FILE, not `-c` — Windows was …
trek-e commentedon Sep 11, 2026
Closing as a duplicate of #4631 — consolidating at the seam rather than fixing per-symptom.
This is not a dismissal of the report. Your specifics are preserved verbatim in that epic's Absorbed issues table, and its Done when checklist requires a failing-first regression test for this exact defect before the epic can close.
Why it is being consolidated: a scan of all 69 open
confirmed-bugissues (2026-09-11) found this one shares a single root cause with seven other open issues — a gate hand-rolls a narrow derivation of what it should evaluate, the derivation comes back empty, and "empty" is reported as "pass". Fixing each occurrence where it was found leaves the remaining call sites carrying the same bug, which is the pattern that produced this cluster in the first place.The epic delivers one
resolveEvaluationScope(), evidence read through the existing document seams instead of per-gate regex, and anEvidencetype that distinguishes "none" from "could not read" so a gate can no longer pass vacuously, plus a machine-enforced anti-divergence ratchet so a new hand-written copy cannot be introduced — the approach that worked for #1372 (markdown sectionizer), #2143 (table + bounded mutation) and #2121 (phase-id), each of which closed its bug class structurally.Track the fix at #4631.
fix(#5170): ui-auditor screenshot block follows redirects, probes the…
refactor(#5170): gsd-ui-auditor loads its static capture block from a…
fix(#5170): the ui-auditor's static capture fence stays inline in <sc…
fix(#5170): ui-auditor screenshot block follows redirects, probes the…
refactor(#5170): gsd-ui-auditor loads its static capture block from a…
fix(#5170): the ui-auditor's static capture fence stays inline in <sc…
refactor(#5170): typed gate evidence and a verdict-driven exit status…