Skip to content

Commit 440354b

Browse files
RomneyDaRomneyDa
committed
Merge remote-tracking branch 'origin/main' into pr-96143-conflicts
# Conflicts: # src/plugins/update.test.ts
2 parents d6c9fba + 941e8f1 commit 440354b

1,070 files changed

Lines changed: 41245 additions & 8015 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.agents/skills/openclaw-ci-limits/SKILL.md‎

Lines changed: 196 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,196 @@
1+
---
2+
name: openclaw-ci-limits
3+
description: Manage OpenClaw GitHub Actions and Blacksmith CI capacity, runner-registration budgets, fanout caps, main-push debounce, shard sizing, hosted-runner offload, queue health, and safe ramp-down/ramp-up changes. Use when tuning `.github/workflows/*`, `docs/ci.md`, CI runner labels, matrix `max-parallel`, ClawSweeper/Blacksmith burst protection, CodeQL runner placement, or investigating slow/queued OpenClaw CI.
4+
---
5+
6+
# OpenClaw CI Limits
7+
8+
Use this skill for CI capacity changes, not ordinary test failure triage. The
9+
goal is to keep OpenClaw fast while staying below GitHub's self-hosted runner
10+
registration edge limit.
11+
12+
## Core Facts
13+
14+
- The scarce resource is Blacksmith runner registrations, not Blacksmith vCPU
15+
capacity.
16+
- GitHub runner registrations are capped at 1,500 per 5 minutes per repository,
17+
organization, or enterprise. The `openclaw` organization shares one bucket.
18+
- Core REST quota does not draw down this bucket. Check
19+
`actions_runner_registration` separately; core quota can be healthy while
20+
runner registration is throttled.
21+
- Use 1,000 registrations per 5 minutes as the operating target. Leave the last
22+
third for other repos, retries, and burst overlap.
23+
- Jobs that route, notify, summarize, choose shards, or run short CodeQL quality
24+
scans should stay on GitHub-hosted runners unless measured evidence says
25+
Blacksmith is required.
26+
27+
## First Checks
28+
29+
Before changing CI, collect current pressure:
30+
31+
```bash
32+
ghx api rate_limit --jq '{core:.resources.core,graphql:.resources.graphql,search:.resources.search,actions_runner_registration:.resources.actions_runner_registration}'
33+
ghx run list -R openclaw/openclaw --limit 20 --json databaseId,status,conclusion,workflowName,event,headBranch,createdAt,updatedAt,url
34+
ghx run list -R openclaw/clawsweeper --limit 20 --json databaseId,status,conclusion,workflowName,event,headBranch,createdAt,updatedAt,url
35+
curl -fsS https://clawsweeper.openclaw.ai/api/status | jq '{generated_at,fleet,diagnostics:{errors:.diagnostics.errors}}'
36+
curl -fsS https://clawsweeper.openclaw.ai/api/exact-review-queue | jq '.'
37+
node scripts/ci-run-timings.mjs --latest-main
38+
node scripts/ci-run-timings.mjs --recent 10
39+
```
40+
41+
Read:
42+
43+
- `.github/workflows/ci.yml`
44+
- `.github/workflows/codeql-critical-quality.yml`
45+
- `docs/ci.md`
46+
- `test/scripts/ci-workflow-guards.test.ts`
47+
- touched planner files under `scripts/lib/*ci*`, `scripts/lib/*test-plan*`, or
48+
`scripts/ci-changed-scope.mjs`
49+
50+
## Diagnose The Bottleneck
51+
52+
Classify the issue before changing caps:
53+
54+
- **Runner-registration throttle:** many jobs queued before runner assignment,
55+
Blacksmith/GitHub reports 403/429 or spam-style 422 responses from
56+
`generate-jitconfig`, and API core quota is still healthy. Treat 422 as this
57+
signal only when the request payload is otherwise valid. Fix burstiness and
58+
Blacksmith job count.
59+
- **Blacksmith capacity:** Blacksmith dashboard shows actual concurrency caps or
60+
unavailable capacity. Do not solve this with GitHub workflow fanout alone.
61+
- **OpenClaw test runtime:** jobs start quickly but one lane dominates wall time.
62+
Use `$openclaw-test-performance` instead of runner tuning.
63+
- **Real failing CI:** one job fails after starting. Use `$github:gh-fix-ci` or
64+
`$openclaw-testing`, not this skill.
65+
- **ClawSweeper backlog:** exact-review queue grows while CI is healthy. Tune
66+
ClawSweeper workers in `openclaw/clawsweeper`, not OpenClaw CI.
67+
68+
## Registration Budget Math
69+
70+
Estimate worst-case registrations for a change before editing:
71+
72+
```text
73+
new Blacksmith registrations ~= number of Blacksmith jobs that can become queued
74+
inside one 5 minute window
75+
```
76+
77+
For matrix jobs, count every row that can start in the 5-minute window.
78+
`strategy.max-parallel` only caps simultaneous rows; short rows can turn over
79+
and register more runners before the window resets. Use job duration, retries,
80+
and queue turnover to justify any lower estimate. Add non-matrix Blacksmith jobs
81+
such as `preflight`, `security-fast`, `build-artifacts`, and platform lanes.
82+
83+
For repeated pushes, multiply by the number of runs expected to reach
84+
Blacksmith admission in the same 5-minute window, including runs canceled after
85+
admission. The debounce only suppresses pushes that arrive while
86+
`runner-admission` is still sleeping; once Blacksmith jobs register, those
87+
registrations are spent even if a later push cancels the run. If timing is
88+
uncertain, count every sequential push in the window.
89+
90+
Reject a change unless the org-level worst case stays below 1,000 registrations
91+
per 5 minutes with headroom for ClawSweeper, ClawHub, Clownfish, OpenClaw RTT,
92+
and Clawbench.
93+
94+
## Safe Levers
95+
96+
Prefer these in order:
97+
98+
1. Add or preserve concurrency groups that cancel superseded PR and canonical
99+
`main` runs before Blacksmith work starts.
100+
2. Keep the `runner-admission` hosted debounce for canonical `main` pushes.
101+
Change `OPENCLAW_MAIN_CI_DEBOUNCE_SECONDS` only with evidence.
102+
3. Move high-frequency, short, non-build jobs to `ubuntu-24.04`.
103+
4. Reduce matrix rows by bundling related tests inside one runner job when the
104+
combined job stays under timeout and keeps useful failure names.
105+
5. Lower `strategy.max-parallel` for bursty Blacksmith matrices.
106+
6. Right-size runners from timing evidence. Use fewer/larger jobs only when
107+
elapsed time improves enough to justify registration count.
108+
7. Split truly slow tests with `$openclaw-test-performance`; do not hide a slow
109+
test problem by registering more runners.
110+
111+
Do not:
112+
113+
- add another Blacksmith installation expecting a higher registration bucket;
114+
- move CodeQL Critical Quality back to Blacksmith;
115+
- raise all `max-parallel` values at once;
116+
- make manual `workflow_dispatch` runs cancel normal push/PR validation;
117+
- delete coverage just to reduce runner count;
118+
- treat cancelled superseded runs as failures without checking the newest run
119+
for the same ref.
120+
121+
## Current OpenClaw Knobs
122+
123+
These are intentionally guarded by `test/scripts/ci-workflow-guards.test.ts`:
124+
125+
- `CI` concurrency key version and `cancel-in-progress` for PRs and canonical
126+
`main` pushes.
127+
- `runner-admission` on `ubuntu-24.04` with
128+
`OPENCLAW_MAIN_CI_DEBOUNCE_SECONDS=90`.
129+
- `preflight` and `security-fast` needing `runner-admission`.
130+
- CI matrix caps: fast/check lanes at 8, compact Node PR plan at current caps,
131+
Windows and Android at 2.
132+
- `build-artifacts` on `blacksmith-16vcpu-ubuntu-2404`.
133+
- lower-weight Node/check shards on `blacksmith-4vcpu-ubuntu-2404`.
134+
- heavy retained Linux/Android shards on `blacksmith-8vcpu-ubuntu-2404`.
135+
- CodeQL Critical Quality on `ubuntu-24.04` with no `blacksmith-` labels.
136+
137+
When changing one knob, update `docs/ci.md` and the guard test in the same PR.
138+
139+
## Validation
140+
141+
For workflow-only or docs/skill-only changes in a Codex worktree:
142+
143+
```bash
144+
node scripts/run-vitest.mjs test/scripts/ci-workflow-guards.test.ts
145+
node scripts/check-workflows.mjs
146+
node scripts/docs-list.js
147+
./node_modules/.bin/oxfmt --check .github/workflows/ci.yml .github/workflows/codeql-critical-quality.yml docs/ci.md test/scripts/ci-workflow-guards.test.ts .agents/skills/openclaw-ci-limits/SKILL.md .agents/skills/openclaw-ci-limits/agents/openai.yaml
148+
git diff --check
149+
```
150+
151+
If `pnpm docs:list` tries to reconcile dependencies in a linked Codex worktree,
152+
stop and use `node scripts/docs-list.js`.
153+
154+
For a PR before requesting maintainer approval:
155+
156+
```bash
157+
.agents/skills/autoreview/scripts/autoreview --mode branch --base origin/main
158+
ghx pr checks <pr> -R openclaw/openclaw --watch --interval 15
159+
```
160+
161+
Use hosted exact-head gates for CI workflow tuning. Do not burn local
162+
`pnpm test` on unrelated full-suite proof.
163+
164+
Only after the maintainer explicitly asks you to prepare or land the PR, run the
165+
repo-native mutating wrapper:
166+
167+
```bash
168+
scripts/pr review-init <pr>
169+
scripts/pr review-artifacts-init <pr>
170+
scripts/pr review-validate-artifacts <pr>
171+
OPENCLAW_TESTBOX=1 scripts/pr prepare-run <pr>
172+
```
173+
174+
`prepare-run` can push a prepared commit to the PR branch. Only run
175+
`scripts/pr merge-run <pr>` after the maintainer has explicitly asked you to
176+
land the PR. Both commands mutate GitHub state.
177+
178+
## Post-Land Monitoring
179+
180+
After merge, watch at least one fresh main cycle and the adjacent repos:
181+
182+
```bash
183+
ghx run list -R openclaw/openclaw --limit 20 --json databaseId,status,conclusion,workflowName,event,headBranch,createdAt,updatedAt,url
184+
for repo in openclaw/clawsweeper openclaw/clawhub openclaw/clownfish openclaw/openclaw-rtt openclaw/clawbench; do
185+
ghx run list -R "$repo" --limit 12 --json databaseId,status,conclusion,workflowName,event,headBranch,createdAt,updatedAt,url
186+
done
187+
curl -fsS https://clawsweeper.openclaw.ai/api/exact-review-queue | jq '.'
188+
```
189+
190+
Report:
191+
192+
- exact PR/commit landed;
193+
- expected registration reduction or added headroom;
194+
- CI run status and slowest/queued jobs;
195+
- ClawSweeper queue pending, dispatching, leased, oldest pending age;
196+
- any real failures that remain outside runner registration.

‎.agents/skills/openclaw-ci-limits/agents/openai.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
interface:
2+
display_name: "OpenClaw CI Limits"
3+
short_description: "Tune OpenClaw CI fanout and runner budgets"
4+
default_prompt: "Use $openclaw-ci-limits to inspect OpenClaw CI pressure, tune runner-registration fanout safely, and document the exact validation before landing."

‎.github/labeler.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -118,6 +118,7 @@
118118
- any-glob-to-any-file:
119119
- "extensions/qa-lab/**"
120120
- "qa/scenarios/**"
121+
- "docs/maturity/**"
121122
- "docs/concepts/qa-e2e-automation.md"
122123
- "docs/concepts/personal-agent-benchmark-pack.md"
123124
- "docs/channels/qa-channel.md"

‎.github/workflows/ci-build-artifacts-testbox.yml‎

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -198,10 +198,19 @@ jobs:
198198
"+refs/heads/main:refs/remotes/origin/main"
199199
200200
node_bin="$(dirname "$(node -p 'process.execPath')")"
201-
sudo ln -sf "$node_bin/node" /usr/local/bin/node
202-
sudo ln -sf "$node_bin/npm" /usr/local/bin/npm
203-
sudo ln -sf "$node_bin/npx" /usr/local/bin/npx
204-
sudo ln -sf "$node_bin/corepack" /usr/local/bin/corepack
201+
link_node_tool() {
202+
local tool="$1"
203+
local source="$node_bin/$tool"
204+
local target="/usr/local/bin/$tool"
205+
if [ -e "$target" ] && [ "$(readlink -f "$source")" = "$(readlink -f "$target")" ]; then
206+
return
207+
fi
208+
sudo ln -sf "$source" "$target"
209+
}
210+
link_node_tool node
211+
link_node_tool npm
212+
link_node_tool npx
213+
link_node_tool corepack
205214
sudo tee /usr/local/bin/pnpm >/dev/null <<'PNPM'
206215
#!/usr/bin/env bash
207216
exec /usr/local/bin/corepack pnpm "$@"

‎.github/workflows/ci-check-arm-testbox.yml‎

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -116,10 +116,19 @@ jobs:
116116
"+refs/heads/main:refs/remotes/origin/main"
117117
118118
node_bin="$(dirname "$(node -p 'process.execPath')")"
119-
sudo ln -sf "$node_bin/node" /usr/local/bin/node
120-
sudo ln -sf "$node_bin/npm" /usr/local/bin/npm
121-
sudo ln -sf "$node_bin/npx" /usr/local/bin/npx
122-
sudo ln -sf "$node_bin/corepack" /usr/local/bin/corepack
119+
link_node_tool() {
120+
local tool="$1"
121+
local source="$node_bin/$tool"
122+
local target="/usr/local/bin/$tool"
123+
if [ -e "$target" ] && [ "$(readlink -f "$source")" = "$(readlink -f "$target")" ]; then
124+
return
125+
fi
126+
sudo ln -sf "$source" "$target"
127+
}
128+
link_node_tool node
129+
link_node_tool npm
130+
link_node_tool npx
131+
link_node_tool corepack
123132
sudo tee /usr/local/bin/pnpm >/dev/null <<'PNPM'
124133
#!/usr/bin/env bash
125134
exec /usr/local/bin/corepack pnpm "$@"

‎.github/workflows/ci-check-testbox.yml‎

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -105,10 +105,19 @@ jobs:
105105
"+refs/heads/main:refs/remotes/origin/main"
106106
107107
node_bin="$(dirname "$(node -p 'process.execPath')")"
108-
sudo ln -sf "$node_bin/node" /usr/local/bin/node
109-
sudo ln -sf "$node_bin/npm" /usr/local/bin/npm
110-
sudo ln -sf "$node_bin/npx" /usr/local/bin/npx
111-
sudo ln -sf "$node_bin/corepack" /usr/local/bin/corepack
108+
link_node_tool() {
109+
local tool="$1"
110+
local source="$node_bin/$tool"
111+
local target="/usr/local/bin/$tool"
112+
if [ -e "$target" ] && [ "$(readlink -f "$source")" = "$(readlink -f "$target")" ]; then
113+
return
114+
fi
115+
sudo ln -sf "$source" "$target"
116+
}
117+
link_node_tool node
118+
link_node_tool npm
119+
link_node_tool npx
120+
link_node_tool corepack
112121
sudo tee /usr/local/bin/pnpm >/dev/null <<'PNPM'
113122
#!/usr/bin/env bash
114123
exec /usr/local/bin/corepack pnpm "$@"

0 commit comments

Comments
 (0)