Skip to content

Commit 560aba5

Browse files
mmapsmmaps
authored
Merge branch 'main' into maintenance/update-20260608-210846-a13e19
2 parents e70d417 + 374076b commit 560aba5

710 files changed

Lines changed: 24212 additions & 4553 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.agents/skills/openclaw-ci-limits/SKILL.md‎

Lines changed: 196 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,196 @@
1+
---
2+
name: openclaw-ci-limits
3+
description: Manage OpenClaw GitHub Actions and Blacksmith CI capacity, runner-registration budgets, fanout caps, main-push debounce, shard sizing, hosted-runner offload, queue health, and safe ramp-down/ramp-up changes. Use when tuning `.github/workflows/*`, `docs/ci.md`, CI runner labels, matrix `max-parallel`, ClawSweeper/Blacksmith burst protection, CodeQL runner placement, or investigating slow/queued OpenClaw CI.
4+
---
5+
6+
# OpenClaw CI Limits
7+
8+
Use this skill for CI capacity changes, not ordinary test failure triage. The
9+
goal is to keep OpenClaw fast while staying below GitHub's self-hosted runner
10+
registration edge limit.
11+
12+
## Core Facts
13+
14+
- The scarce resource is Blacksmith runner registrations, not Blacksmith vCPU
15+
capacity.
16+
- GitHub runner registrations are capped at 1,500 per 5 minutes per repository,
17+
organization, or enterprise. The `openclaw` organization shares one bucket.
18+
- Core REST quota does not draw down this bucket. Check
19+
`actions_runner_registration` separately; core quota can be healthy while
20+
runner registration is throttled.
21+
- Use 1,000 registrations per 5 minutes as the operating target. Leave the last
22+
third for other repos, retries, and burst overlap.
23+
- Jobs that route, notify, summarize, choose shards, or run short CodeQL quality
24+
scans should stay on GitHub-hosted runners unless measured evidence says
25+
Blacksmith is required.
26+
27+
## First Checks
28+
29+
Before changing CI, collect current pressure:
30+
31+
```bash
32+
ghx api rate_limit --jq '{core:.resources.core,graphql:.resources.graphql,search:.resources.search,actions_runner_registration:.resources.actions_runner_registration}'
33+
ghx run list -R openclaw/openclaw --limit 20 --json databaseId,status,conclusion,workflowName,event,headBranch,createdAt,updatedAt,url
34+
ghx run list -R openclaw/clawsweeper --limit 20 --json databaseId,status,conclusion,workflowName,event,headBranch,createdAt,updatedAt,url
35+
curl -fsS https://clawsweeper.openclaw.ai/api/status | jq '{generated_at,fleet,diagnostics:{errors:.diagnostics.errors}}'
36+
curl -fsS https://clawsweeper.openclaw.ai/api/exact-review-queue | jq '.'
37+
node scripts/ci-run-timings.mjs --latest-main
38+
node scripts/ci-run-timings.mjs --recent 10
39+
```
40+
41+
Read:
42+
43+
- `.github/workflows/ci.yml`
44+
- `.github/workflows/codeql-critical-quality.yml`
45+
- `docs/ci.md`
46+
- `test/scripts/ci-workflow-guards.test.ts`
47+
- touched planner files under `scripts/lib/*ci*`, `scripts/lib/*test-plan*`, or
48+
`scripts/ci-changed-scope.mjs`
49+
50+
## Diagnose The Bottleneck
51+
52+
Classify the issue before changing caps:
53+
54+
- **Runner-registration throttle:** many jobs queued before runner assignment,
55+
Blacksmith/GitHub reports 403/429 or spam-style 422 responses from
56+
`generate-jitconfig`, and API core quota is still healthy. Treat 422 as this
57+
signal only when the request payload is otherwise valid. Fix burstiness and
58+
Blacksmith job count.
59+
- **Blacksmith capacity:** Blacksmith dashboard shows actual concurrency caps or
60+
unavailable capacity. Do not solve this with GitHub workflow fanout alone.
61+
- **OpenClaw test runtime:** jobs start quickly but one lane dominates wall time.
62+
Use `$openclaw-test-performance` instead of runner tuning.
63+
- **Real failing CI:** one job fails after starting. Use `$github:gh-fix-ci` or
64+
`$openclaw-testing`, not this skill.
65+
- **ClawSweeper backlog:** exact-review queue grows while CI is healthy. Tune
66+
ClawSweeper workers in `openclaw/clawsweeper`, not OpenClaw CI.
67+
68+
## Registration Budget Math
69+
70+
Estimate worst-case registrations for a change before editing:
71+
72+
```text
73+
new Blacksmith registrations ~= number of Blacksmith jobs that can become queued
74+
inside one 5 minute window
75+
```
76+
77+
For matrix jobs, count every row that can start in the 5-minute window.
78+
`strategy.max-parallel` only caps simultaneous rows; short rows can turn over
79+
and register more runners before the window resets. Use job duration, retries,
80+
and queue turnover to justify any lower estimate. Add non-matrix Blacksmith jobs
81+
such as `preflight`, `security-fast`, `build-artifacts`, and platform lanes.
82+
83+
For repeated pushes, multiply by the number of runs expected to reach
84+
Blacksmith admission in the same 5-minute window, including runs canceled after
85+
admission. The debounce only suppresses pushes that arrive while
86+
`runner-admission` is still sleeping; once Blacksmith jobs register, those
87+
registrations are spent even if a later push cancels the run. If timing is
88+
uncertain, count every sequential push in the window.
89+
90+
Reject a change unless the org-level worst case stays below 1,000 registrations
91+
per 5 minutes with headroom for ClawSweeper, ClawHub, Clownfish, OpenClaw RTT,
92+
and Clawbench.
93+
94+
## Safe Levers
95+
96+
Prefer these in order:
97+
98+
1. Add or preserve concurrency groups that cancel superseded PR and canonical
99+
`main` runs before Blacksmith work starts.
100+
2. Keep the `runner-admission` hosted debounce for canonical `main` pushes.
101+
Change `OPENCLAW_MAIN_CI_DEBOUNCE_SECONDS` only with evidence.
102+
3. Move high-frequency, short, non-build jobs to `ubuntu-24.04`.
103+
4. Reduce matrix rows by bundling related tests inside one runner job when the
104+
combined job stays under timeout and keeps useful failure names.
105+
5. Lower `strategy.max-parallel` for bursty Blacksmith matrices.
106+
6. Right-size runners from timing evidence. Use fewer/larger jobs only when
107+
elapsed time improves enough to justify registration count.
108+
7. Split truly slow tests with `$openclaw-test-performance`; do not hide a slow
109+
test problem by registering more runners.
110+
111+
Do not:
112+
113+
- add another Blacksmith installation expecting a higher registration bucket;
114+
- move CodeQL Critical Quality back to Blacksmith;
115+
- raise all `max-parallel` values at once;
116+
- make manual `workflow_dispatch` runs cancel normal push/PR validation;
117+
- delete coverage just to reduce runner count;
118+
- treat cancelled superseded runs as failures without checking the newest run
119+
for the same ref.
120+
121+
## Current OpenClaw Knobs
122+
123+
These are intentionally guarded by `test/scripts/ci-workflow-guards.test.ts`:
124+
125+
- `CI` concurrency key version and `cancel-in-progress` for PRs and canonical
126+
`main` pushes.
127+
- `runner-admission` on `ubuntu-24.04` with
128+
`OPENCLAW_MAIN_CI_DEBOUNCE_SECONDS=90`.
129+
- `preflight` and `security-fast` needing `runner-admission`.
130+
- CI matrix caps: fast/check lanes at 8, compact Node PR plan at current caps,
131+
Windows and Android at 2.
132+
- `build-artifacts` on `blacksmith-16vcpu-ubuntu-2404`.
133+
- lower-weight Node/check shards on `blacksmith-4vcpu-ubuntu-2404`.
134+
- heavy retained Linux/Android shards on `blacksmith-8vcpu-ubuntu-2404`.
135+
- CodeQL Critical Quality on `ubuntu-24.04` with no `blacksmith-` labels.
136+
137+
When changing one knob, update `docs/ci.md` and the guard test in the same PR.
138+
139+
## Validation
140+
141+
For workflow-only or docs/skill-only changes in a Codex worktree:
142+
143+
```bash
144+
node scripts/run-vitest.mjs test/scripts/ci-workflow-guards.test.ts
145+
node scripts/check-workflows.mjs
146+
node scripts/docs-list.js
147+
./node_modules/.bin/oxfmt --check .github/workflows/ci.yml .github/workflows/codeql-critical-quality.yml docs/ci.md test/scripts/ci-workflow-guards.test.ts .agents/skills/openclaw-ci-limits/SKILL.md .agents/skills/openclaw-ci-limits/agents/openai.yaml
148+
git diff --check
149+
```
150+
151+
If `pnpm docs:list` tries to reconcile dependencies in a linked Codex worktree,
152+
stop and use `node scripts/docs-list.js`.
153+
154+
For a PR before requesting maintainer approval:
155+
156+
```bash
157+
.agents/skills/autoreview/scripts/autoreview --mode branch --base origin/main
158+
ghx pr checks <pr> -R openclaw/openclaw --watch --interval 15
159+
```
160+
161+
Use hosted exact-head gates for CI workflow tuning. Do not burn local
162+
`pnpm test` on unrelated full-suite proof.
163+
164+
Only after the maintainer explicitly asks you to prepare or land the PR, run the
165+
repo-native mutating wrapper:
166+
167+
```bash
168+
scripts/pr review-init <pr>
169+
scripts/pr review-artifacts-init <pr>
170+
scripts/pr review-validate-artifacts <pr>
171+
OPENCLAW_TESTBOX=1 scripts/pr prepare-run <pr>
172+
```
173+
174+
`prepare-run` can push a prepared commit to the PR branch. Only run
175+
`scripts/pr merge-run <pr>` after the maintainer has explicitly asked you to
176+
land the PR. Both commands mutate GitHub state.
177+
178+
## Post-Land Monitoring
179+
180+
After merge, watch at least one fresh main cycle and the adjacent repos:
181+
182+
```bash
183+
ghx run list -R openclaw/openclaw --limit 20 --json databaseId,status,conclusion,workflowName,event,headBranch,createdAt,updatedAt,url
184+
for repo in openclaw/clawsweeper openclaw/clawhub openclaw/clownfish openclaw/openclaw-rtt openclaw/clawbench; do
185+
ghx run list -R "$repo" --limit 12 --json databaseId,status,conclusion,workflowName,event,headBranch,createdAt,updatedAt,url
186+
done
187+
curl -fsS https://clawsweeper.openclaw.ai/api/exact-review-queue | jq '.'
188+
```
189+
190+
Report:
191+
192+
- exact PR/commit landed;
193+
- expected registration reduction or added headroom;
194+
- CI run status and slowest/queued jobs;
195+
- ClawSweeper queue pending, dispatching, leased, oldest pending age;
196+
- any real failures that remain outside runner registration.

‎.agents/skills/openclaw-ci-limits/agents/openai.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
interface:
2+
display_name: "OpenClaw CI Limits"
3+
short_description: "Tune OpenClaw CI fanout and runner budgets"
4+
default_prompt: "Use $openclaw-ci-limits to inspect OpenClaw CI pressure, tune runner-registration fanout safely, and document the exact validation before landing."

‎.github/labeler.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -118,6 +118,7 @@
118118
- any-glob-to-any-file:
119119
- "extensions/qa-lab/**"
120120
- "qa/scenarios/**"
121+
- "docs/maturity/**"
121122
- "docs/concepts/qa-e2e-automation.md"
122123
- "docs/concepts/personal-agent-benchmark-pack.md"
123124
- "docs/channels/qa-channel.md"

‎.github/workflows/ci-build-artifacts-testbox.yml‎

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -198,10 +198,19 @@ jobs:
198198
"+refs/heads/main:refs/remotes/origin/main"
199199
200200
node_bin="$(dirname "$(node -p 'process.execPath')")"
201-
sudo ln -sf "$node_bin/node" /usr/local/bin/node
202-
sudo ln -sf "$node_bin/npm" /usr/local/bin/npm
203-
sudo ln -sf "$node_bin/npx" /usr/local/bin/npx
204-
sudo ln -sf "$node_bin/corepack" /usr/local/bin/corepack
201+
link_node_tool() {
202+
local tool="$1"
203+
local source="$node_bin/$tool"
204+
local target="/usr/local/bin/$tool"
205+
if [ -e "$target" ] && [ "$(readlink -f "$source")" = "$(readlink -f "$target")" ]; then
206+
return
207+
fi
208+
sudo ln -sf "$source" "$target"
209+
}
210+
link_node_tool node
211+
link_node_tool npm
212+
link_node_tool npx
213+
link_node_tool corepack
205214
sudo tee /usr/local/bin/pnpm >/dev/null <<'PNPM'
206215
#!/usr/bin/env bash
207216
exec /usr/local/bin/corepack pnpm "$@"

‎.github/workflows/ci-check-arm-testbox.yml‎

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -116,10 +116,19 @@ jobs:
116116
"+refs/heads/main:refs/remotes/origin/main"
117117
118118
node_bin="$(dirname "$(node -p 'process.execPath')")"
119-
sudo ln -sf "$node_bin/node" /usr/local/bin/node
120-
sudo ln -sf "$node_bin/npm" /usr/local/bin/npm
121-
sudo ln -sf "$node_bin/npx" /usr/local/bin/npx
122-
sudo ln -sf "$node_bin/corepack" /usr/local/bin/corepack
119+
link_node_tool() {
120+
local tool="$1"
121+
local source="$node_bin/$tool"
122+
local target="/usr/local/bin/$tool"
123+
if [ -e "$target" ] && [ "$(readlink -f "$source")" = "$(readlink -f "$target")" ]; then
124+
return
125+
fi
126+
sudo ln -sf "$source" "$target"
127+
}
128+
link_node_tool node
129+
link_node_tool npm
130+
link_node_tool npx
131+
link_node_tool corepack
123132
sudo tee /usr/local/bin/pnpm >/dev/null <<'PNPM'
124133
#!/usr/bin/env bash
125134
exec /usr/local/bin/corepack pnpm "$@"

‎.github/workflows/ci-check-testbox.yml‎

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -105,10 +105,19 @@ jobs:
105105
"+refs/heads/main:refs/remotes/origin/main"
106106
107107
node_bin="$(dirname "$(node -p 'process.execPath')")"
108-
sudo ln -sf "$node_bin/node" /usr/local/bin/node
109-
sudo ln -sf "$node_bin/npm" /usr/local/bin/npm
110-
sudo ln -sf "$node_bin/npx" /usr/local/bin/npx
111-
sudo ln -sf "$node_bin/corepack" /usr/local/bin/corepack
108+
link_node_tool() {
109+
local tool="$1"
110+
local source="$node_bin/$tool"
111+
local target="/usr/local/bin/$tool"
112+
if [ -e "$target" ] && [ "$(readlink -f "$source")" = "$(readlink -f "$target")" ]; then
113+
return
114+
fi
115+
sudo ln -sf "$source" "$target"
116+
}
117+
link_node_tool node
118+
link_node_tool npm
119+
link_node_tool npx
120+
link_node_tool corepack
112121
sudo tee /usr/local/bin/pnpm >/dev/null <<'PNPM'
113122
#!/usr/bin/env bash
114123
exec /usr/local/bin/corepack pnpm "$@"

‎.github/workflows/ci.yml‎

Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -100,6 +100,7 @@ jobs:
100100
run_macos_node: ${{ steps.manifest.outputs.run_macos_node }}
101101
macos_node_matrix: ${{ steps.manifest.outputs.macos_node_matrix }}
102102
run_macos_swift: ${{ steps.manifest.outputs.run_macos_swift }}
103+
run_ios_build: ${{ steps.manifest.outputs.run_ios_build }}
103104
run_android_job: ${{ steps.manifest.outputs.run_android_job }}
104105
android_matrix: ${{ steps.manifest.outputs.android_matrix }}
105106
steps:
@@ -204,6 +205,7 @@ jobs:
204205
OPENCLAW_CI_DOCS_CHANGED: ${{ github.event_name == 'workflow_dispatch' && 'true' || steps.docs_scope.outputs.docs_changed }}
205206
OPENCLAW_CI_RUN_NODE: ${{ github.event_name == 'workflow_dispatch' && 'true' || steps.changed_scope.outputs.run_node || 'false' }}
206207
OPENCLAW_CI_RUN_MACOS: ${{ github.event_name == 'workflow_dispatch' && 'true' || steps.changed_scope.outputs.run_macos || 'false' }}
208+
OPENCLAW_CI_RUN_IOS_BUILD: ${{ github.event_name == 'workflow_dispatch' && 'true' || steps.changed_scope.outputs.run_ios_build || 'false' }}
207209
OPENCLAW_CI_RUN_ANDROID: ${{ github.event_name == 'workflow_dispatch' && (inputs.release_gate || inputs.include_android) && 'true' || steps.changed_scope.outputs.run_android || 'false' }}
208210
OPENCLAW_CI_RUN_WINDOWS: ${{ github.event_name == 'workflow_dispatch' && 'true' || steps.changed_scope.outputs.run_windows || 'false' }}
209211
OPENCLAW_CI_RUN_NODE_FAST_ONLY: ${{ github.event_name == 'workflow_dispatch' && 'false' || steps.changed_scope.outputs.run_node_fast_only || 'false' }}
@@ -267,6 +269,8 @@ jobs:
267269
const runPluginContractShards = runNodeFull || runNodeFastPluginContracts;
268270
const runMacos =
269271
parseBoolean(process.env.OPENCLAW_CI_RUN_MACOS) && !docsOnly && isCanonicalRepository;
272+
const runIosBuild =
273+
parseBoolean(process.env.OPENCLAW_CI_RUN_IOS_BUILD) && !docsOnly && isCanonicalRepository;
270274
const runAndroid =
271275
parseBoolean(process.env.OPENCLAW_CI_RUN_ANDROID) && !docsOnly && isCanonicalRepository;
272276
const runWindows =
@@ -361,6 +365,7 @@ jobs:
361365
runMacos ? [{ check_name: "macos-node", runtime: "node", task: "test" }] : [],
362366
),
363367
run_macos_swift: runMacos,
368+
run_ios_build: runIosBuild,
364369
run_android_job: runAndroid,
365370
android_matrix: createMatrix(
366371
runAndroid
@@ -2162,6 +2167,76 @@ jobs:
21622167
done
21632168
exit 1
21642169
2170+
ios-build:
2171+
permissions:
2172+
contents: read
2173+
name: "ios-build"
2174+
needs: [preflight]
2175+
if: needs.preflight.outputs.run_ios_build == 'true'
2176+
runs-on: ${{ github.event_name == 'workflow_dispatch' && 'macos-26' || (github.repository == 'openclaw/openclaw' && 'blacksmith-12vcpu-macos-26' || 'macos-26') }}
2177+
timeout-minutes: 45
2178+
steps:
2179+
- name: Checkout
2180+
env:
2181+
CHECKOUT_REPO: ${{ github.repository }}
2182+
CHECKOUT_SHA: ${{ needs.preflight.outputs.checkout_revision }}
2183+
run: |
2184+
set -euo pipefail
2185+
git init "$GITHUB_WORKSPACE"
2186+
git -C "$GITHUB_WORKSPACE" config gc.auto 0
2187+
git -C "$GITHUB_WORKSPACE" remote add origin "https://github.com/${CHECKOUT_REPO}.git"
2188+
fetch_timeout_seconds=90
2189+
fetch_checkout_ref() {
2190+
git -C "$GITHUB_WORKSPACE" \
2191+
-c protocol.version=2 \
2192+
fetch --no-tags --prune --no-recurse-submodules --depth=1 origin \
2193+
"+${CHECKOUT_SHA}:refs/remotes/origin/checkout" &
2194+
local fetch_pid="$!"
2195+
local elapsed=0
2196+
while kill -0 "$fetch_pid" 2>/dev/null; do
2197+
if [ "$elapsed" -ge "$fetch_timeout_seconds" ]; then
2198+
kill -TERM "$fetch_pid" 2>/dev/null || true
2199+
sleep 10
2200+
kill -KILL "$fetch_pid" 2>/dev/null || true
2201+
wait "$fetch_pid" || true
2202+
return 124
2203+
fi
2204+
sleep 1
2205+
elapsed=$((elapsed + 1))
2206+
done
2207+
wait "$fetch_pid"
2208+
}
2209+
fetch_checkout_ref
2210+
git -C "$GITHUB_WORKSPACE" checkout --detach refs/remotes/origin/checkout
2211+
2212+
- name: Select Xcode 26
2213+
run: |
2214+
set -euo pipefail
2215+
for xcode_app in /Applications/Xcode_26.5.app /Applications/Xcode-26.5.0.app; do
2216+
if [ -d "$xcode_app/Contents/Developer" ]; then
2217+
sudo xcode-select -s "$xcode_app/Contents/Developer"
2218+
break
2219+
fi
2220+
done
2221+
xcodebuild -version
2222+
xcode_version="$(xcodebuild -version | awk 'NR == 1 { print $2 }')"
2223+
if [[ "$xcode_version" != 26.* ]]; then
2224+
echo "error: expected Xcode 26.x, got $xcode_version" >&2
2225+
exit 1
2226+
fi
2227+
swift --version
2228+
2229+
- name: Setup Node environment
2230+
uses: ./.github/actions/setup-node-env
2231+
with:
2232+
install-bun: "false"
2233+
2234+
- name: Install iOS Swift tooling
2235+
run: brew install xcodegen swiftlint swiftformat
2236+
2237+
- name: Build iOS app
2238+
run: pnpm ios:build
2239+
21652240
android:
21662241
permissions:
21672242
contents: read
@@ -2313,6 +2388,7 @@ jobs:
23132388
- checks-windows
23142389
- macos-node
23152390
- macos-swift
2391+
- ios-build
23162392
- android
23172393
if: ${{ !cancelled() && always() && github.event_name != 'push' && (github.event_name != 'pull_request' || !github.event.pull_request.draft) }}
23182394
runs-on: ubuntu-24.04

0 commit comments

Comments
 (0)