Skip to content

infra: seccomp remediation redeploy fails (image_pull_failed on a stale state-file tag) — profile left unenforced #6512

Closed

Description

@deruelle

Summary

Apply deploy-pipeline-fix failed on main at 3e934cf (run 29450562340). Its #5875 item 4 step detected that the committed seccomp profile is not enforced by the running container, attempted a remediation redeploy, and the redeploy failed:

Committed seccomp profile sha256: 7654ef3466cf1d02f3888e334459e0a5eb974314fdd38e7d10819f030f5bad73
Baseline: host_present=false host_sha256='<none>' loaded_matches_host=false (committed='7654ef…', tag='latest').
STATE invariant not satisfied — a seccomp change is applied but the running container is not enforcing it. Redeploying to load it.
Redeploying running version as v0.214.7 (state-file .tag was 'latest').
Redeploy FAILED: web-platform v0.214.7 terminal reason='image_pull_failed' exit_code=1. The applied seccomp profile was NOT loaded.

Net effect: seccomp_profile_host_present=false — the profile is absent from the host and the mechanism that exists to load it cannot. The security control is silently not in force, and the only signal is one red workflow among a page of green.

This is not a production outage — which is why it will get ignored

Established while verifying the #6454 merge:

  • app.soleur.ai → HTTP 307, healthy throughout.
  • The release pipeline is fine. Web Platform Release, Version Bump and Release, and Apply web-platform infra all succeeded on the same commit, and web-v0.215.1 / web-v0.215.2 released and deployed normally afterward (21:13 / 21:23).
  • The image exists. v0.214.7 is present in GHCR alongside v0.214.6/.8/.9/.10, so the artifact is not the problem.

So prod health hides this completely. That combination — real control unenforced, everything green that anyone looks at — is the #6454 shape again: a gate reporting nothing wrong while checking nothing.

The suspicious part: .tag was 'latest'

The redeploy targeted v0.214.7, resolved from a state-file whose .tag was the literal string latest — while the deployed line was already on v0.215.x. Two questions worth separating:

  1. Why is the state-file tag latest rather than a pinned version? The fallback ("redeploy the running version") then resolved to a version that is several releases stale.
  2. Why did pulling that specific stale tag fail when the registry demonstrably serves the current tags? A retention/GC policy pruning older tags out of the pull path is one candidate (cf. the in-flight zot retention work), but this is unverified — do not treat it as the diagnosis.

A zot-credential hypothesis (#6484 rewrote the zot htpasswd surface ~2.5h earlier) was considered and rejected: zot is dark-launched (only when ZOT_* present) and subsequent releases pulled fine.

Not flaky-by-default

Apply deploy-pipeline-fix shows 3 failures in its last 8 runs — but it succeeded on the immediately preceding commit (9576e590b, 18:27) and failed at 21:03. Worth checking whether the intermittency shares this root cause rather than filing it as noise.

Suggested first steps

  • Determine why the deploy state-file carries .tag='latest' instead of a pinned version — that is what makes the remediation path aim at a stale image.
  • Confirm whether stale tags are still pullable by the host, or whether retention/GC removed them from the path the host uses.
  • Make the seccomp invariant's failure visible where someone reads it — host_present=false should surface as an alert, not solely as a red job on a merge commit.
  • Re-check the three recent intermittent failures against the above.

Found via

Post-merge verification of #6458 (wg-after-a-pr-merges-to-main-verify-all). Not caused by #6458 — its only server.tf change is 13 lines of comment (Apply web-platform infra succeeded on the same commit). Filed rather than fixed inline: the root cause is in the deploy state-file / registry-retention surface, unrelated to that PR, and diagnosis needs live infra state.

Activity

  1. added this to the Post-MVP / Later milestone on Jul 15, 2026
  2. soleur-ai commented on Jul 16, 2026

    @soleur-ai
    Contributor

    Automated Triage

    Priority: p1-high
    Type: bug
    Domain: engineering

    Reasoning: A security control (seccomp profile enforcement) is silently not in force in production and the remediation mechanism itself failed, though this is not currently a production outage. Significant risk with no clean workaround for the underlying deploy-state/registry issue, warranting high priority attention despite no active incident.

  3. deruelle commented on Jul 17, 2026

    @deruelle
    ContributorAuthor

    Reopening: this was auto-closed in error. PR #6521 (a learnings-only compound PR) merged at 08:42:38Z and GitHub's keyword parser matched the substring close #6512 inside the sentence "Does not close #6512" in that PR's body, closing this a second later. #6521 touched only one markdown learning file and explicitly disclaimed closing this issue. The seccomp remediation / stale-latest-state-file-tag bug is not fixed — reopening to actually work it.

  4. deruelle commented on Jul 17, 2026

    @deruelle
    ContributorAuthor

    Resolved by PR #6622 (merged, deployed, verified).

    Live verification (post-merge): /hooks/deploy-status reports host_present=true, loaded_matches_host=true, host_sha256=7654ef34… (== committed seccomp-bwrap.json) — the profile is enforced and correct on prod.

    Post-merge workflows all green: Apply deploy-pipeline-fix (item-4 re-enforced with the new on-host ci-deploy.sh), Apply sentry infra (the 2 new alerts created), Apply web-platform infra, and Web Platform Release (deploy + live-verify).

    What shipped: Fix 1 — the local-cache reload tier so a same-version seccomp redeploy survives a both-registries-fail by reusing the already-verified running image (with a same-version RepoTag guard, added after review, that prevents a stale-bits rollback on a new-version deploy). Fix 2a — scripts/seccomp-unenforced-alert.sh files a plain-language ci/seccomp-unenforced GitHub issue + a seccomp_remediation_failed Sentry alert on any item-4 terminal failure, so the next occurrence pages instead of hiding.

    Deferred (tracked): #6628 (Fix 2b standing watchdog + stale-but-enforcing residual), #6629 (delivery-leg RCA). PIR filed. ADR-079 amended (#6512, 2026-07-17).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    domain/engineeringPlugin code, CI/CD, infra, docs site (CTO)priority/p1-highDegraded functionality, no workaroundtype/bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions