Repository navigation
Description
Summary
Apply deploy-pipeline-fix failed on main at 3e934cf (run 29450562340). Its #5875 item 4 step detected that the committed seccomp profile is not enforced by the running container, attempted a remediation redeploy, and the redeploy failed:
Committed seccomp profile sha256: 7654ef3466cf1d02f3888e334459e0a5eb974314fdd38e7d10819f030f5bad73
Baseline: host_present=false host_sha256='<none>' loaded_matches_host=false (committed='7654ef…', tag='latest').
STATE invariant not satisfied — a seccomp change is applied but the running container is not enforcing it. Redeploying to load it.
Redeploying running version as v0.214.7 (state-file .tag was 'latest').
Redeploy FAILED: web-platform v0.214.7 terminal reason='image_pull_failed' exit_code=1. The applied seccomp profile was NOT loaded.
Net effect: seccomp_profile_host_present=false — the profile is absent from the host and the mechanism that exists to load it cannot. The security control is silently not in force, and the only signal is one red workflow among a page of green.
This is not a production outage — which is why it will get ignored
Established while verifying the #6454 merge:
app.soleur.ai→ HTTP 307, healthy throughout.- The release pipeline is fine.
Web Platform Release,Version Bump and Release, andApply web-platform infraall succeeded on the same commit, andweb-v0.215.1/web-v0.215.2released and deployed normally afterward (21:13 / 21:23). - The image exists.
v0.214.7is present in GHCR alongsidev0.214.6/.8/.9/.10, so the artifact is not the problem.
So prod health hides this completely. That combination — real control unenforced, everything green that anyone looks at — is the #6454 shape again: a gate reporting nothing wrong while checking nothing.
The suspicious part: .tag was 'latest'
The redeploy targeted v0.214.7, resolved from a state-file whose .tag was the literal string latest — while the deployed line was already on v0.215.x. Two questions worth separating:
- Why is the state-file tag
latestrather than a pinned version? The fallback ("redeploy the running version") then resolved to a version that is several releases stale. - Why did pulling that specific stale tag fail when the registry demonstrably serves the current tags? A retention/GC policy pruning older tags out of the pull path is one candidate (cf. the in-flight zot retention work), but this is unverified — do not treat it as the diagnosis.
A zot-credential hypothesis (#6484 rewrote the zot htpasswd surface ~2.5h earlier) was considered and rejected: zot is dark-launched (only when ZOT_* present) and subsequent releases pulled fine.
Not flaky-by-default
Apply deploy-pipeline-fix shows 3 failures in its last 8 runs — but it succeeded on the immediately preceding commit (9576e590b, 18:27) and failed at 21:03. Worth checking whether the intermittency shares this root cause rather than filing it as noise.
Suggested first steps
- Determine why the deploy state-file carries
.tag='latest'instead of a pinned version — that is what makes the remediation path aim at a stale image. - Confirm whether stale tags are still pullable by the host, or whether retention/GC removed them from the path the host uses.
- Make the seccomp invariant's failure visible where someone reads it —
host_present=falseshould surface as an alert, not solely as a red job on a merge commit. - Re-check the three recent intermittent failures against the above.
Found via
Post-merge verification of #6458 (wg-after-a-pr-merges-to-main-verify-all). Not caused by #6458 — its only server.tf change is 13 lines of comment (Apply web-platform infra succeeded on the same commit). Filed rather than fixed inline: the root cause is in the deploy state-file / registry-retention surface, unrelated to that PR, and diagnosis needs live infra state.
Activity
soleur-ai commentedon Jul 16, 2026
Automated Triage
Priority: p1-high
Type: bug
Domain: engineering
Reasoning: A security control (seccomp profile enforcement) is silently not in force in production and the remediation mechanism itself failed, though this is not currently a production outage. Significant risk with no clean workaround for the underlying deploy-state/registry issue, warranting high priority attention despite no active incident.
compound: absent-by-design config is not drift (merge_queue / #6512 s…
deruelle commentedon Jul 17, 2026
Reopening: this was auto-closed in error. PR #6521 (a learnings-only compound PR) merged at 08:42:38Z and GitHub's keyword parser matched the substring
close #6512inside the sentence "Does not close #6512" in that PR's body, closing this a second later. #6521 touched only one markdown learning file and explicitly disclaimed closing this issue. The seccomp remediation / stale-latest-state-file-tag bug is not fixed — reopening to actually work it.docs: create plan and tasks for feat-one-shot-6512-seccomp-remediatio…
docs: deepen plan for #6512 (gates, precedent-diff, live-citation ver…
docs: create plan and tasks for feat-one-shot-6512-seccomp-remediatio…
docs: deepen plan for #6512 (gates, precedent-diff, live-citation ver…
fix(infra): seccomp reload survives a both-registries-fail; unenforce…
fix(infra): local-cache tier must only rescue a SAME-VERSION reload (#…
review: address multi-agent findings — label bootstrap, vacuous test,…
compound: reuse-on-fetch-failure fallback must gate on version, not p…
compound: archive #6512 feature spec artifacts
incident: PIR for #6512 seccomp remediation unenforced-and-invisible
fix(infra): seccomp reload survives both-registries-fail + unenforced…
deruelle commentedon Jul 17, 2026
Resolved by PR #6622 (merged, deployed, verified).
Live verification (post-merge):
/hooks/deploy-statusreportshost_present=true, loaded_matches_host=true, host_sha256=7654ef34…(== committedseccomp-bwrap.json) — the profile is enforced and correct on prod.Post-merge workflows all green: Apply deploy-pipeline-fix (item-4 re-enforced with the new on-host
ci-deploy.sh), Apply sentry infra (the 2 new alerts created), Apply web-platform infra, and Web Platform Release (deploy + live-verify).What shipped: Fix 1 — the
local-cachereload tier so a same-version seccomp redeploy survives a both-registries-fail by reusing the already-verified running image (with a same-version RepoTag guard, added after review, that prevents a stale-bits rollback on a new-version deploy). Fix 2a —scripts/seccomp-unenforced-alert.shfiles a plain-languageci/seccomp-unenforcedGitHub issue + aseccomp_remediation_failedSentry alert on any item-4 terminal failure, so the next occurrence pages instead of hiding.Deferred (tracked): #6628 (Fix 2b standing watchdog + stale-but-enforcing residual), #6629 (delivery-leg RCA). PIR filed. ADR-079 amended
(#6512, 2026-07-17).