Skip to content

verify-backends ignores run_acq's exit status when classifying USAi reachability, so a failed exec is recorded as pass #547

Closed

Description

@basiliskus

verify_backend runs the USAi reachability probe through run_acq, then classifies the
result purely by string-matching the captured output, and never looks at the command's exit
status:

run_acq "exec usai-reach" --backend "$backend" exec "$name" -- sh -c \
  "curl -sS -o /dev/null -w '%{http_code}' --max-time 15 $USAI_MODELS_URL"
local ustatus="$LAST_OUT"
case "$ustatus" in
  000|*"resolve"*|*"curl:"*|"")
    fail "${backend}: USAi host not reachable from guest (egress/net-rule?)"
    ...
  *)
    pass "${backend}: USAi host reachable from guest (HTTP $ustatus; auth rewritten on wire)"
    ;;
esac

scripts/verify-backends:612-623 on main (e0b1143b) — 610-623 including the run_acq
invocation above.

run_acq does return the command's status (return "$_rc"), so the information is
available; the caller discards it. Any output that is non-empty and happens not to contain
the substrings resolve or curl: lands in the *) branch and is recorded as pass —
including a backend error, a shell error, or a curl diagnostic whose wording changes
upstream. The pass message then interpolates that text as if it were an HTTP status.

Reproduced in isolation: a stand-in command printing noisy-error and exiting 7 was
classified pass.

This is the exact false-green class the live harness exists to prevent, in the one check
that proves egress actually works.

Suggested fix: check run_acq's return value first, and require ustatus to match a
three-digit HTTP status ([0-9][0-9][0-9]) before treating it as reachable — anything else
is a failure, not a pass.

One caveat on the status source, if this is fixed on top of
GSA-TTS/agentic-coding-quickstart#474 rather than on main: on main, run_acq's
verbose branch reads PIPESTATUS[0] after the enclosing LAST_OUT=$(... | tee /dev/stderr)
assignment, which reports the assignment's status (0) rather than the command's. So under
-v on main, the return value is not currently trustworthy either, and a fix that checks
it needs that corrected too. PR #474 already rewrites this path (_run_captured runs the
command directly and preserves the real status), so on that branch the rc is sound and only
the caller needs changing.

Provenance: surfaced during review of GSA-TTS/agentic-coding-quickstart#474, where it
was initially filed as a blocking finding against that PR. It is not that PR's defect — the
block is byte-identical to main and does not appear in its diff. Filing it here so it is
tracked against the right target.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions