fix(scripts): retry Docker E2E after rejected resource caps by ooiuuii · Pull Request #98579 · openclaw/openclaw

ooiuuii · 2026-07-01T10:49:39Z

What Problem This Solves

Docker E2E and package-backed Docker lanes can fail before the test container starts when the local Docker daemon accepts the command but rejects OpenClaw's default --memory, --cpus, or --pids-limit cgroup controls during container create/start.

This can happen on nested, rootless, or otherwise constrained Docker hosts. Before this change, operators had to know to set OPENCLAW_DOCKER_E2E_DISABLE_RESOURCE_LIMITS=1 manually; failed named-container and --cidfile attempts could also leave artifacts that blocked the retry.

Why This Change Was Made

The Docker run helpers now retry once without OpenClaw-injected default resource caps only when Docker itself reports a daemon/runtime resource-limit create failure. The retry path:

preserves stderr for the original failure;
keeps ordinary Docker and container-command failures unchanged;
caches the runtime resource-cap disable for later runs in the same process;
cleans failed --name / --cidfile container artifacts before retrying;
stops cleanup parsing at Docker's image boundary, so --name or --cidfile inside the container command cannot trigger cleanup of an unrelated container.

The package fallback helper receives the same behavior because it can run without the shared container helper in package-backed lanes. Explicit user-provided Docker run arguments remain on retry; only OpenClaw's injected default caps are removed.

User Impact

Developers and CI operators can run Docker E2E lanes on more host configurations without manually disabling resource limits. Normal hosts keep the existing capped Docker runs, and non-resource failures such as missing images or container-command exits still fail immediately without retrying as a different command.

Evidence

Duplicate search before opening:

gh pr list -R openclaw/openclaw --state all --search "Docker cgroup resource limits pids limit fallback" --limit 20 returned only older unrelated sandbox/low-memory PRs (feat(sandbox): pluggable ISandboxProvider — 3-tier isolation with browser automation (Docker/gVisor/Firecracker) #41437, fix: improve npm install resilience on low-memory systems #39463).
gh issue list -R openclaw/openclaw --state all --search "Docker cgroup resource limits pids limit fallback" --limit 20 returned no issues.
Searching the exact new warning text returned no existing issue or PR for this behavior.

Local checks:

bash -n scripts/lib/docker-e2e-container.sh
bash -n scripts/lib/docker-e2e-package.sh
node scripts/check-docker-e2e-boundaries.mjs
node_modules/.bin/oxlint.cmd scripts/lib/docker-e2e-container.sh scripts/lib/docker-e2e-package.sh test/scripts/docker-build-helper.test.ts
node_modules/.bin/oxfmt.cmd --check test/scripts/docker-build-helper.test.ts
git diff --check
python .agents/skills/autoreview/scripts/autoreview --mode local

Focused local Vitest on Windows with Git Bash and /tmp temp paths:

node_modules/.bin/vitest.cmd run --config test/vitest/vitest.tooling-docker.config.ts test/scripts/docker-build-helper.test.ts -t "retries Docker runs without default resource limits when cgroups reject them|retries Docker runs when rootless daemons reject NanoCPUs|ignores Docker-like command arguments after the image|does not retry container command failures that only print resource-limit text|retries package-backed Docker runs without default resource limits when cgroups reject them|retries package-backed Docker runs when rootless daemons reject NanoCPUs|ignores package-backed Docker-like command arguments after the image|does not retry package-backed container command failures that only print resource-limit text"

Test Files  1 passed (1)
Tests  8 passed | 160 skipped (168)

Real constrained-daemon proof on 122.51.21.218:

Host: VM-0-5-ubuntu
Kernel: Linux 5.15.0-170-generic x86_64
System Docker after runtime recovery: server=29.6.1 cgroup=2 driver=overlayfs cgroupDriver=systemd
Method: started a temporary rootless Docker daemon as a systemd user service, built a local FROM scratch BusyBox image without pulling from a registry, and ran the helper against that real daemon.
Control: the system Docker daemon accepted both plain and resource-limited runs, proving the host was otherwise healthy.
Repro daemon: the temporary rootless Docker daemon accepted the plain run but rejected the resource-limited run with Docker's real NanoCPUs cgroup error.

SYSTEMD_ROOTLESS_READY=ok
SYSTEMD_ROOTLESS_INFO=server=29.6.1 rootless=[name=seccomp,profile=builtin name=rootless name=cgroupns] cgroup=2 driver=fuse-overlayfs cgroupDriver=systemd
SYSTEMD_ROOTLESS_PLAIN_STATUS=0
SYSTEMD_ROOTLESS_LIMITED_STATUS=125
SYSTEMD_ROOTLESS_LIMITED_ERR=docker: Error response from daemon: NanoCPUs can not be set, as your kernel does not support CPU CFS scheduler or the cgroup is not mounted

Helper proof against the same rootless daemon after this patch:

REMOTE_ENV=host-server=29.6.1 cgroup=2 driver=overlayfs cgroupDriver=systemd
ROOTLESS_ENV=server=29.6.1 rootless=[name=seccomp,profile=builtin name=rootless name=cgroupns] cgroup=2 driver=fuse-overlayfs cgroupDriver=systemd
HELPER_STATUS=0
HELPER_ERR=docker: Error response from daemon: NanoCPUs can not be set, as your kernel does not support CPU CFS scheduler or the cgroup is not mounted
HELPER_ERR=Run 'docker run --help' for more information
HELPER_ERR=Docker run resource limits were rejected by this daemon; retrying without default --memory/--cpus/--pids-limit flags.

Cleanup after the remote proof:

DOCKER_STATUS=server=29.6.1 containers=0 running=0 images=0
LEFTOVER_TEST_PROCS=
DISK_ROOT=40G total 16G avail 60% used

clawsweeper · 2026-07-01T11:47:12Z

Codex review: needs maintainer review before merge. Reviewed July 2, 2026, 11:23 PM ET / 03:23 UTC.

Summary
The PR adds a retry-without-default-resource-caps path to Docker E2E container and package helpers, plus focused shell/Vitest coverage for cgroup, NanoCPUs, cleanup, and false-positive cases.

PR surface: Tests +559, Other +413. Total +972 across 3 files.

Reproducibility: yes. The linked issue and PR body provide a high-confidence constrained-daemon path where plain Docker succeeds but resource-limited Docker fails, and current source shows main still injects those caps without retry.

Review metrics: 1 noteworthy metric.

Retry helper copies: 2 changed helper copies. The container helper and package fallback can each own Docker run resource injection, so their classifier, cleanup, and retry behavior must stay aligned before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #98601
Summary: This PR is the candidate fix for the canonical Docker E2E resource-cap rejection bug report; the other search hits are adjacent Docker/resource topics, not the same root cause.

Members:

canonical: [Bug]: Docker E2E fails when Docker rejects default resource caps #98601 - The PR body closes the issue, and the issue describes the same Docker E2E resource-cap rejection behavior and expected retry.
adjacent_distinct: fix: improve npm install resilience on low-memory systems #39463 - This closed unmerged PR concerned npm install resilience on low-memory systems, not Docker run cgroup/resource-limit rejection in E2E helpers.
adjacent_distinct: feat(sandbox): pluggable ISandboxProvider — 3-tier isolation with browser automation (Docker/gVisor/Firecracker) #41437 - This closed unmerged PR concerned broad sandbox provider architecture, not the focused Docker E2E helper retry path.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

none.

Risk before merge

[P1] This changes Docker E2E shell automation on constrained hosts; ordinary CI cannot exhaust every rootless/nested Docker daemon ordering, although the current PR covers the reported path with focused tests and real rootless-Docker proof.

Maintainer options:

Land with current constrained-Docker proof (recommended)
Accept the remaining automation-surface risk because the PR now covers status-125 resource-limit matching, image-boundary cleanup, false-positive non-retry behavior, package fallback, and real rootless-Docker output.
Request broader host proof
A maintainer can ask for one more nested or rootless host run if they want coverage beyond the reported NanoCPUs/cgroup daemon path before merge.
Pause the shell fallback
If maintainers decide automatic retry and cleanup is too much shell behavior for these helpers, pause or close this branch and keep the manual disable-resource-limits workaround.

Next step before merge

[P2] No repair job: there are no blocking findings, and the remaining action is maintainer review of an already-open fixing PR.

Security
Cleared: No dependency, workflow, permission, secret, or supply-chain change was found; the Docker cleanup behavior is bounded to the affected helper path and covered as functional automation risk.

Review details

Best possible solution:

Merge the focused helper retry fix after maintainer review, preserving normal Docker failures and keeping container and package helper behavior aligned.

Do we have a high-confidence way to reproduce the issue?

Yes. The linked issue and PR body provide a high-confidence constrained-daemon path where plain Docker succeeds but resource-limited Docker fails, and current source shows main still injects those caps without retry.

Is this the best way to solve the issue?

Yes. Retrying once only after Docker status 125 plus resource-limit stderr, while removing only OpenClaw-injected caps and preserving ordinary failures, is the narrowest maintainable fix for the reported helper behavior.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 9eec24df6710.

Label changes

Label changes:

add proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes after-fix terminal proof from a real rootless Docker daemon showing resource-cap rejection before retry and helper success after the patch.
add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The PR body includes after-fix terminal proof from a real rootless Docker daemon showing resource-cap rejection before retry and helper success after the patch.
remove rating: 🧂 unranked krab: Current PR rating is rating: 🐚 platinum hermit, so this older rating label is no longer current.
remove status: 📣 needs proof: Current PR status label is status: 👀 ready for maintainer look.

Label justifications:

P2: This fixes a medium-severity Docker E2E reliability problem with limited blast radius to developer and CI helper lanes.
merge-risk: 🚨 automation: The diff changes Docker E2E shell retry and cleanup behavior, which can affect CI/proof automation on host configurations not fully represented by ordinary checks.
rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The PR body includes after-fix terminal proof from a real rootless Docker daemon showing resource-cap rejection before retry and helper success after the patch.
proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes after-fix terminal proof from a real rootless Docker daemon showing resource-cap rejection before retry and helper success after the patch.

Evidence reviewed

PR surface:

Tests +559, Other +413. Total +972 across 3 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	0	0	0	0
Tests	1	559	0	+559
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	2	456	43	+413
Total	3	1015	43	+972

What I checked:

Repository policy read: Read the full root AGENTS.md plus scoped scripts/test guides; the review applies the OpenClaw PR proof, script-wrapper, best-fix, and automation-risk rules. (AGENTS.md:1, 9eec24df6710)
Current main still fails without retry: Current main injects default resource caps and directly returns the Docker run status; there is no retry when Docker rejects those caps. (scripts/lib/docker-e2e-container.sh:140, 9eec24df6710)
Package fallback still has the same main behavior: Current main package fallback also injects --memory, --cpus, and --pids-limit and calls Docker directly without the proposed retry classifier. (scripts/lib/docker-e2e-package.sh:88, 9eec24df6710)
Latest release is not fixed: Tag v2026.6.11 contains the same direct capped Docker run path, so this is not already shipped as fixed. (scripts/lib/docker-e2e-container.sh:140, e085fa1a3ffd)
PR head adds bounded fallback: The PR routes docker run through docker_e2e_docker_run_with_resource_fallback, requires Docker status 125 plus resource-limit stderr, cleans failed named/cidfile artifacts, and retries without only injected resource caps. (scripts/lib/docker-e2e-container.sh:363, 6a48258d1d68)
Package fallback aligned: The package helper has the same classifier, image-boundary cleanup parser, runtime-disable cache, and retry behavior for package-backed lanes that do not use the shared container helper. (scripts/lib/docker-e2e-package.sh:218, 6a48258d1d68)

Likely related people:

vincentkoc: Live path history shows repeated recent work on Docker E2E resource caps, pids-limit validation, timeouts, and package-helper behavior across the affected files. (role: recent area contributor; confidence: high; commits: 6af1b97b1de7, 5496044f6d7f, a372429a96bf; files: scripts/lib/docker-e2e-container.sh, scripts/lib/docker-e2e-package.sh, test/scripts/docker-build-helper.test.ts)
steipete: Live path history shows earlier Docker E2E helper extraction and simplification commits that shaped the current shared helper structure. (role: earlier helper refactor owner; confidence: medium; commits: 2c0449571c3b, 1c4262ef87e9, 8ac2dd4cd2a4; files: scripts/lib/docker-e2e-container.sh, scripts/lib/docker-e2e-package.sh)
RomneyDa: Live path history shows adjacent package-backed Docker proof and QA Lab migration work on the package helper path. (role: adjacent package-proof contributor; confidence: medium; commits: 0a6736af09f6; files: scripts/lib/docker-e2e-package.sh)
zyzo: Recent path history shows adjacent Docker helper test work in the same test file shortly before this PR. (role: recent adjacent test contributor; confidence: medium; commits: 98254634f18b; files: test/scripts/docker-build-helper.test.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

ooiuuii · 2026-07-01T13:59:15Z

@clawsweeper re-review

clawsweeper · 2026-07-01T13:59:18Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

ooiuuii · 2026-07-01T14:42:52Z

@clawsweeper re-review

clawsweeper · 2026-07-01T14:42:56Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

ooiuuii · 2026-07-03T03:11:57Z

@clawsweeper re-review

Current PR body now includes the requested real constrained-Docker proof from host 122.51.21.218: a temporary rootless Docker daemon accepts a plain run, rejects the injected NanoCPUs resource cap with Docker's real cgroup error, and this PR's helper retries without default resource caps and exits 0. The body also documents cleanup state after the proof and includes the post-image --name / --cidfile parser regression coverage requested in the prior review.

steipete · 2026-07-17T03:56:54Z

AI-assisted (approved by Peter)

Thank you for the patch and for your continued work across the repository.

We are closing this approach because silently retrying without resource caps weakens an explicit safety constraint and violates the repository's explicit-fallback doctrine. Operators who intentionally accept that tradeoff already have the supported opt-out: OPENCLAW_DOCKER_E2E_DISABLE_RESOURCE_LIMITS=1.

We landed #109587 to make the incompatible-runtime failure actionable: it preserves the original Docker error and status, keeps the capped run fail-closed, and points directly to that explicit opt-out.

fix(scripts): retry docker e2e without rejected resource caps

2c9745c

openclaw-barnacle Bot added scripts Repository scripts docker Docker and sandbox tooling size: L labels Jul 1, 2026

ooiuuii mentioned this pull request Jul 1, 2026

[Bug]: Docker E2E fails when Docker rejects default resource caps #98601

Open

fix: constrain docker e2e cleanup parsing

d24510f

fix: recognize rootless docker resource rejection

6a48258

openclaw-barnacle Bot added size: XL and removed size: L labels Jul 1, 2026

clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jul 3, 2026

clawsweeper Bot mentioned this pull request Jul 4, 2026

improve(qa): execute runtime scenarios through Docker #99705

Merged

steipete self-assigned this Jul 17, 2026

steipete closed this Jul 17, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(scripts): retry Docker E2E after rejected resource caps#98579

fix(scripts): retry Docker E2E after rejected resource caps#98579
ooiuuii wants to merge 3 commits into
openclaw:mainfrom
ooiuuii:fix/docker-cgroup-resource-limit-probe

ooiuuii commented Jul 1, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Jul 1, 2026 •

edited

Loading

Uh oh!

ooiuuii commented Jul 1, 2026

Uh oh!

clawsweeper Bot commented Jul 1, 2026 •

edited

Loading

Uh oh!

ooiuuii commented Jul 1, 2026

Uh oh!

clawsweeper Bot commented Jul 1, 2026 •

edited

Loading

Uh oh!

ooiuuii commented Jul 3, 2026

Uh oh!

steipete commented Jul 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

ooiuuii commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Problem This Solves

Why This Change Was Made

User Impact

Evidence

Uh oh!

clawsweeper Bot commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ooiuuii commented Jul 1, 2026

Uh oh!

clawsweeper Bot commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ooiuuii commented Jul 1, 2026

Uh oh!

clawsweeper Bot commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ooiuuii commented Jul 3, 2026

Uh oh!

steipete commented Jul 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ooiuuii commented Jul 1, 2026 •

edited

Loading

clawsweeper Bot commented Jul 1, 2026 •

edited

Loading

clawsweeper Bot commented Jul 1, 2026 •

edited

Loading

clawsweeper Bot commented Jul 1, 2026 •

edited

Loading