Skip to content

Commit 641349a

Browse files
luisleo526CORE-TOPclaudeci-cx
authored
ci: dispatch-only post-runner capability canary (fleet-post-capability.yml) (#520)
* ci: add dispatch-only post-runner capability canary (fleet-post-capability.yml) New post VMs (pf-gcp-post-4/5/6) must pass this canary before laneboss promotes them into the pf-gcp-post pool. workflow_dispatch only (no push, pull_request or schedule trigger), permissions: {}, runs only on the holding label pf-gcp-vm-<vm>; it builds nothing and runs no engine suite. File installed verbatim from laneboss evidence (sha256 2e0e3e6b6497bc9f...). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * ci: run the post canary on the pf-gcp-post label (holding runner group) The engine preflight pins every post-reachable job to [self-hosted, pf-gcp-post]. The new VMs keep that label but sit in the holding runner group pf-gcp-post-hold, whose restricted_to_workflows is exactly this file on main, so real post jobs cannot reach them before the canary passes (laneboss read-back + idle-under-queue proof). The file's hostname and runner-name guards still pin the run to the chosen VM. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * ci: spell the canary runs-on exactly as the preflight pin compares it Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * tests: teach test_ci_preflight.py about the dispatch-only post-fleet canary workflow (#520) fleet-post-capability.yml adds a 19th GCP-routed workflow job. The full preflight (not the PR-CI preflight stage) failed in test_gcp_sites_reject_cross_class_routes: len(sites) 19 != 18, and the exact (workflow, job) pair comparison was stale by the same job. The canary is recorded in its own DISPATCH_ONLY_CANARY_JOB_IDS, not in FLEET_JOB_IDS: that table also drives test_gcp_sites_require_every_routed_job (a deleted job must yield "is missing job X"), a finding scripts/ci_preflight.py only produces for workflows with a pinned-job entry, which the canary deliberately does not have. The count becomes 19 and the pair comparison stays exact (FLEET minus LIGHT plus the canary pair). New test_gcp_sites_pin_canary_to_the_dispatch_fleet (green on the base by design, a preservation control): the canary's sites are exactly [('capability', 'post')], the finding text is the post rule's, and routing the job to the PR fleet yields that finding. Known gap: no control fails if the canary's job is deleted (the checker has no required-job entry for it). Not run on the author's machine; the lab preflight run is the evidence. --------- Co-authored-by: CORE-TOP <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> Co-authored-by: ci-cx <ci-cx@invalid>
1 parent 2446e86 commit 641349a

2 files changed

Lines changed: 198 additions & 2 deletions

File tree

Lines changed: 178 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,178 @@
1+
# One-off workflow template: install verbatim on the engine main branch at
2+
# .github/workflows/fleet-post-capability.yml. Do not dispatch it from this repo.
3+
name: fleet-post-capability
4+
run-name: post-capability:${{ inputs.vm }}
5+
on:
6+
workflow_dispatch:
7+
inputs:
8+
vm:
9+
description: New VM, still in the holding group with its VM label only
10+
required: true
11+
type: choice
12+
options:
13+
- pf-gcp-post-4
14+
- pf-gcp-post-5
15+
- pf-gcp-post-6
16+
permissions: {}
17+
concurrency:
18+
group: post-capability-${{ inputs.vm }}
19+
cancel-in-progress: false
20+
defaults:
21+
run:
22+
shell: bash
23+
jobs:
24+
capability:
25+
name: post-capability-${{ matrix.slot }}
26+
strategy:
27+
fail-fast: false
28+
matrix:
29+
slot: [a, b]
30+
runs-on: ["self-hosted","pf-gcp-post"]
31+
env:
32+
TARGET_VM: ${{ inputs.vm }}
33+
SLOT: ${{ matrix.slot }}
34+
steps:
35+
- name: Verify both runner identities and VM capabilities
36+
run: |
37+
set +x
38+
set -euo pipefail
39+
umask 077
40+
fail() { printf '::error::%s\n' "$*" >&2; exit 1; }
41+
case "$TARGET_VM" in pf-gcp-post-4|pf-gcp-post-5|pf-gcp-post-6) ;; *) fail 'VM outside the plan' ;; esac
42+
[[ "$(hostname -s)" = "$TARGET_VM" ]] || fail 'Wrong VM'
43+
[[ "$EUID" != 0 && "$(id -un)" = runner ]] || fail 'Capabilities must run as the normal runner user'
44+
[[ "$GITHUB_EVENT_NAME" = workflow_dispatch && "$GITHUB_REF" = refs/heads/main ]] || fail 'Wrong event or branch'
45+
[[ "$RUNNER_NAME" = "$TARGET_VM-r1" || "$RUNNER_NAME" = "$TARGET_VM-r2" ]] || fail 'Wrong runner'
46+
case "$SLOT" in a) peer=b ;; b) peer=a ;; *) fail 'Wrong matrix slot' ;; esac
47+
shared=/var/tmp/pf-post-canary-$GITHUB_RUN_ID-$GITHUB_RUN_ATTEMPT
48+
install -d -m 0700 "$shared"
49+
# Neither job can finish before the other starts, so one runner cannot
50+
# execute both jobs sequentially and masquerade as coverage of two runners.
51+
printf '%s\n' "$RUNNER_NAME" > "$shared/$SLOT.runner"
52+
ready=0
53+
for ((poll=0; poll<120; poll++)); do
54+
if [[ -s "$shared/$peer.runner" ]]; then ready=1; break; fi
55+
sleep 5
56+
done
57+
[[ "$ready" = 1 ]] || fail 'Second local runner did not start in ten minutes'
58+
peer_runner=$(cat "$shared/$peer.runner")
59+
[[ "$peer_runner" != "$RUNNER_NAME" &&
60+
( "$peer_runner" = "$TARGET_VM-r1" || "$peer_runner" = "$TARGET_VM-r2" ) ]] || fail 'Both jobs used the same runner or a wrong runner'
61+
sudo -n true
62+
sudo apt-get update
63+
sudo DEBIAN_FRONTEND=noninteractive apt-get install -y libeigen3-dev tzdata-legacy ccache
64+
docker info --format '{{.OSType}}' | grep -Fx linux
65+
# Only VM prerequisites: this does not build or run the engine suite.
66+
python3 - <<'PY'
67+
import ctypes
68+
import fcntl
69+
import mmap
70+
import os
71+
import struct
72+
import subprocess
73+
import tempfile
74+
75+
def require(condition, message):
76+
if not condition:
77+
raise RuntimeError(message)
78+
79+
roots = ("/dev/shm", "/tmp", "/var/tmp")
80+
magic = {
81+
root: int(subprocess.check_output(
82+
["stat", "-f", "-c", "%t", root], text=True).strip(), 16)
83+
for root in roots
84+
}
85+
def require_tmpfs(root):
86+
require(magic[root] == 0x01021994, "required tmpfs is absent")
87+
def require_disk(root):
88+
require(magic[root] not in (0x01021994, 0x858458F6),
89+
"required disk fallback is memory-backed")
90+
91+
# RED controls use the real filesystems. Neither a no-op assertion nor
92+
# a constant success can reject both deliberately wrong classifications.
93+
rejected = 0
94+
try:
95+
require_tmpfs("/var/tmp")
96+
except RuntimeError:
97+
rejected += 1
98+
try:
99+
require_disk("/dev/shm")
100+
except RuntimeError:
101+
rejected += 1
102+
require(rejected == 2, "filesystem negative controls did not both reject")
103+
require_tmpfs("/dev/shm")
104+
require_disk("/var/tmp")
105+
require(os.path.isdir("/proc/self/fd"), "descriptor inspection is unavailable")
106+
before = len(os.listdir("/proc/self/fd"))
107+
108+
libc = ctypes.CDLL(None, use_errno=True)
109+
libc.inotify_init1.argtypes = [ctypes.c_int]
110+
libc.inotify_init1.restype = ctypes.c_int
111+
libc.inotify_add_watch.argtypes = [ctypes.c_int, ctypes.c_char_p, ctypes.c_uint32]
112+
libc.inotify_add_watch.restype = ctypes.c_int
113+
notify = libc.inotify_init1(os.O_NONBLOCK | os.O_CLOEXEC)
114+
require(notify >= 0, "inotify_init1 failed")
115+
observed = set()
116+
expected = {}
117+
try:
118+
for root in roots:
119+
wd = libc.inotify_add_watch(notify, os.fsencode(root), 0x100 | 0x80)
120+
require(wd >= 0, "cannot watch a scratch candidate")
121+
fd, path = tempfile.mkstemp(prefix="pf-post-canary-", dir=root)
122+
expected[(wd, os.path.basename(path))] = root
123+
try:
124+
payload = b"scratch-capability:" + bytes(range(256)) * 16
125+
require(os.write(fd, payload) == len(payload), "short scratch write")
126+
os.fsync(fd)
127+
os.lseek(fd, 0, os.SEEK_SET)
128+
require(os.read(fd, len(payload)) == payload, "scratch bytes changed")
129+
finally:
130+
os.close(fd)
131+
os.unlink(path)
132+
while True:
133+
try:
134+
events = os.read(notify, 65536)
135+
except BlockingIOError:
136+
break
137+
require(events, "unexpected empty inotify read")
138+
offset = 0
139+
while offset < len(events):
140+
wd, mask, cookie, length = struct.unpack_from("iIII", events, offset)
141+
require(not mask & (0x4000 | 0x8000), "inotify overflow or lost watch")
142+
name = os.fsdecode(events[offset + 16:offset + 16 + length].split(b"\0", 1)[0])
143+
if (wd, name) in expected and mask & (0x100 | 0x80):
144+
observed.add(expected[(wd, name)])
145+
offset += 16 + length
146+
require(observed == set(roots), "creation watches did not witness all candidates")
147+
finally:
148+
os.close(notify)
149+
150+
# Matches the jailed fixture's 32 MiB memfd and grow/shrink/seal seals.
151+
arena = os.memfd_create("pf-post-canary-arena", os.MFD_CLOEXEC | os.MFD_ALLOW_SEALING)
152+
try:
153+
os.ftruncate(arena, 32 * 1024 * 1024)
154+
seals = fcntl.F_SEAL_GROW | fcntl.F_SEAL_SHRINK | fcntl.F_SEAL_SEAL
155+
fcntl.fcntl(arena, fcntl.F_ADD_SEALS, seals)
156+
require(fcntl.fcntl(arena, fcntl.F_GET_SEALS) & seals == seals,
157+
"memfd seals are missing")
158+
with mmap.mmap(arena, 1024 * 1024) as window:
159+
window[:16] = b"memfd-capability"
160+
window.flush()
161+
require(os.pread(arena, 16, 0) == b"memfd-capability",
162+
"sealed arena did not preserve its mapped bytes")
163+
require(os.readlink("/proc/self/fd/" + str(arena)).startswith("/memfd:"),
164+
"jailed arena is not a memfd")
165+
finally:
166+
os.close(arena)
167+
require(len(os.listdir("/proc/self/fd")) == before, "capability probe leaked descriptors")
168+
print("PASS: negative controls; tmpfs /dev/shm; disk /var/tmp; writable candidates; inotify; /proc/self/fd; sealed memfd")
169+
PY
170+
printf 'passed\n' > "$shared/$SLOT.passed"
171+
complete=0
172+
for ((poll=0; poll<240; poll++)); do
173+
if [[ -s "$shared/$peer.passed" ]]; then complete=1; break; fi
174+
sleep 5
175+
done
176+
[[ "$complete" = 1 ]] || fail 'Peer capability check did not finish in twenty minutes'
177+
printf 'PASS: both distinct local runners completed their capability checks\n'
178+

‎scripts/test_ci_preflight.py‎

Lines changed: 20 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,12 @@
6060
'release-ruleset-probe.yml': ('authz', 'probe'),
6161
'promote-baseline.yml': ('promote',),
6262
}
63+
# Dispatch-only post-fleet canary: gcp_sites() discovers its single job and the
64+
# generic post rule routes it. ci_preflight has no pinned-job entry, so it stays
65+
# outside FLEET_JOB_IDS, which also drives the missing-job deletion control.
66+
DISPATCH_ONLY_CANARY_JOB_IDS = {
67+
'fleet-post-capability.yml': ('capability',),
68+
}
6369
LIGHT_JOB_IDS = {
6470
'ci.yml': ('authz', 'changes', 'build-gate'),
6571
'ci-post.yml': ('authz', 'changes', 'build-gate'),
@@ -3381,10 +3387,12 @@ def test_verifier_looking_step_name_cannot_hide_an_unsized_call_from_mutated_fil
33813387

33823388
def test_gcp_sites_reject_cross_class_routes(self):
33833389
sites = list(self.gcp_sites())
3384-
self.assertEqual(len(sites), 18)
3390+
self.assertEqual(len(sites), 19)
33853391
self.assertCountEqual([(name, job) for name, job, *_ in sites],
33863392
[(name, job) for name, jobs in FLEET_JOB_IDS.items()
3387-
for job in jobs if job not in LIGHT_JOB_IDS.get(name, ())])
3393+
for job in jobs if job not in LIGHT_JOB_IDS.get(name, ())]
3394+
+ [(name, job) for name, jobs in DISPATCH_ONLY_CANARY_JOB_IDS.items()
3395+
for job in jobs])
33883396
self.assertEqual(ci_preflight.runner_findings(self.workflow_sources()), [])
33893397
for name, job, fleet, before, finding in sites:
33903398
correct = {'pr': PR_RUNNER, 'post': POST_RUNNER, 'main': RELEASE_RUNNER}[fleet]
@@ -3401,6 +3409,16 @@ def test_gcp_sites_pin_probe_to_the_dispatch_fleet(self):
34013409
f'release-ruleset-probe.yml job probe must run on {POST_RUNNER} '
34023410
'(post fleet)')
34033411

3412+
def test_gcp_sites_pin_canary_to_the_dispatch_fleet(self):
3413+
sites = [site for site in self.gcp_sites() if site[0] == 'fleet-post-capability.yml']
3414+
self.assertEqual([(job, fleet) for _, job, fleet, *_ in sites],
3415+
[('capability', 'post')])
3416+
self.assertEqual(sites[0][-1],
3417+
f'fleet-post-capability.yml job capability must run on {POST_RUNNER}')
3418+
name, job, _, before, finding = sites[0]
3419+
findings = self.routed_file_findings(name, job, before, routed_line(before, PR_RUNNER))
3420+
self.assertIn(finding, findings)
3421+
34043422
def test_gcp_sites_reject_retired_and_void_labels(self):
34053423
for name, job, fleet, before, finding in self.gcp_sites():
34063424
runner = {'pr': PR_RUNNER, 'post': POST_RUNNER, 'main': RELEASE_RUNNER}[fleet]

0 commit comments

Comments
 (0)