Skip to content

ci(deploy-gate): single-poll race posts a permanently-stale 'gate: pending' after all checks pass (stuck PR #5476) #5479

Description

@koala73

Symptom

PR #5476 sat "CI pending" for ~30 minutes after every workflow had finished. All 25 checks were green, but the required gate commit status was stuck at pending — "Waiting for required PR gates: unit", blocking merge indefinitely.

Root cause (evidenced from run logs)

Deploy Gate (.github/workflows/deploy-gate.yml) triggers on workflow_run.completed for Test / Typecheck / Lint Code / Security Audit, polls commits/<sha>/check-runs exactly once, and posts the gate status. Two failure modes combine:

  1. Stale read, no retry. On PR feat(agent-readiness): SDKs in the agent view + homepage rel=alternate pointer (orank round 2) #5476 (head b33988c70), the unit check-run completed at 08:11:12Z. The final gate run evaluated at 08:12:19Z — 67s later — and the check-runs API still returned unit=pending (eventual consistency). Log from run 29990493776:
    changes=success docs-stats=success unit=pending ... security-audit=success
    pending=unit
    
    It posted gate=pending at 08:12:20Z. Since that was the LAST workflow-completion event for the SHA, nothing ever re-evaluated: the stale pending is permanent.
  2. Last-writer-wins across concurrent gate runs. Each of the four watched workflows fires its own gate run; a run triggered early but scheduled late can overwrite a later run's correct success with pending. Same terminal state, different trigger.

Recovery (works today)

gh run rerun <latest-gate-run-id> — the rerun re-polls (now consistent) and posts the true result. That's how #5476 was unstuck (gate flipped to success at 08:39:52Z).

Proposed fix

In the gate step, when pending is non-empty and failed is empty (i.e. nothing has actually failed — we're only waiting), retry the check-runs poll 3–4 times with a 30s sleep before posting pending. That bounds the stale-read window at ~2 minutes and turns the terminal-stale case into a self-healing one. Optionally guard last-writer-wins by skipping the status POST when the existing gate status for the SHA is already success with a newer updated_at evaluation basis.

Evidence

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions