Skip to content

Build Docker images on native arm64 runners to fix release hang#296

Merged
hackerwins merged 1 commit into
mainfrom
fix/docker-publish-arm64-native
May 25, 2026
Merged

Build Docker images on native arm64 runners to fix release hang#296
hackerwins merged 1 commit into
mainfrom
fix/docker-publish-arm64-native

Conversation

@hackerwins

@hackerwins hackerwins commented May 25, 2026

Copy link
Copy Markdown
Collaborator

Summary

The v0.4.2 release Docker build hangs and fails (run 26367212237). Attempt 1 ran for exactly 6h (16:54:50 → 22:55:08) and was cancelled by GitHub's hard 6-hour job limit; attempt 2 was grinding toward the same fate. Previous releases (v0.4.1 / v0.4.0 / v0.3.x) all finished in ~2–3 min.

Root cause

docker-publish.yml built linux/amd64,linux/arm64 in a single job on an amd64 runner, so arm64 was produced under QEMU emulation. The Dockerfile pins the builder stage to --platform=$BUILDPLATFORM (native, fast), but the runtime stage is not pinned, so for the arm64 target these ran emulated:

  • pnpm install --frozen-lockfile --prod
  • npx [email protected] generate ← notoriously slow / hangs under QEMU arm64

These were normally inline-cache hits (cache-from: type=registry,ref=:latest), so the emulated RUNs never executed → ~3 min. The release was immediately preceded by #294 ("Bump deps", rewrote pnpm-lock.yaml 687 lines + 6 package.json) and #295 (version bump, touched lockfile + all package.json), which invalidated those cached layers. The release then executed the arm64 runtime stage emulated for the first time → past the 6h limit. type=inline cache also can't recover (no multi-arch / intermediate-stage caching).

Fix

Build each platform on its own native runner — no QEMU anywhere:

  • build matrix: linux/amd64ubuntu-24.04, linux/arm64ubuntu-24.04-arm (free on this public repo), each pushed by digest.
  • merge job downloads the digests and assembles the multi-arch manifest via docker buildx imagetools create, applying tags from docker/metadata-action.
  • Replace the broken type=inline cache with type=gha,mode=max scoped per arch.
  • Add timeout-minutes (30 build / 15 merge) so a hang fails fast instead of burning 6h.

Tag semantics are unchanged: push to main:latest; release → :<tag_name> + :latest. Pattern follows the official Docker "distribute build across multiple runners" reference.

Tradeoff: type=gha keeps Docker Hub clean (vs type=registry cache tags). GHA cache is ref-scoped, so release (tag-ref) builds get a cold cache and rebuild natively (~minutes, acceptable); main pushes reuse their scope and stay fast.

Test plan

  • pnpm verify:fast green (lint + 792 unit tests).
  • Workflow YAML parses; jobs build + merge.
  • CI on this PR runs Docker Publish (push to main path) — confirm both arch builds finish in minutes and the manifest merges.
  • After merge, re-cut/re-run the v0.4.2 release publish and confirm a multi-arch image (docker buildx imagetools inspect yorkieteam/wafflebase:v0.4.2 shows amd64 + arm64).

Note: the stuck run 26367212237 should be cancelled (gh run cancel 26367212237) so it stops burning CI.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes

    • Fixed Docker image publishing for ARM64 architecture builds that previously hung during execution.
  • Chores

    • Optimized Docker publish workflow with per-architecture build jobs and manifest assembly.
    • Added task documentation tracking Docker publish improvements and release v0.4.2.

Review Change Stack

The v0.4.2 release Docker build exceeded GitHub's 6-hour job limit and
was cancelled, while normal main-push builds finished in ~3 minutes. The
workflow built linux/amd64,linux/arm64 in one job on an amd64 runner, so
arm64 was produced under QEMU emulation. The Dockerfile builder stage is
pinned to $BUILDPLATFORM (native), but the runtime stage is not, so the
arm64 'pnpm install --prod' and 'npx prisma generate' ran emulated. These
were normally inline-cache hits; the preceding dependency bump (#294/#295)
rewrote pnpm-lock.yaml and package.json, invalidating the cache and forcing
the emulated steps to execute for the first time -> 6h timeout.

Build each platform on its own native runner (ubuntu-24.04 /
ubuntu-24.04-arm, free on this public repo), push by digest, and merge a
multi-arch manifest. Replace the broken type=inline cache with
type=gha,mode=max scoped per arch, and add timeout-minutes so a hang fails
fast instead of burning 6h.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
@coderabbitai

coderabbitai Bot commented May 25, 2026

Copy link
Copy Markdown
📝 Walkthrough

Walkthrough

Docker publish workflow reworked from single multi-arch image build to two-stage architecture-native builds with per-digest artifacts, followed by manifest-list merge. Task documentation explains the arm64 hang root cause and provides implementation checklist.

Changes

Docker Workflow Multi-Arch Refactor

Layer / File(s) Summary
Task documentation: arm64 hang issue and solution approach
docs/tasks/active/20260525-docker-publish-arm64-native-todo.md
Complete task writeup describing why Docker Publish v0.4.2 hangs on linux/arm64 (QEMU emulation in runtime stage after cache invalidation) and providing a detailed checklist to restructure the workflow for native per-arch builds with digest-based assembly, GHA scoped caching, and proper timeout handling.
Per-architecture build job matrix with digest extraction
.github/workflows/docker-publish.yml
Build job converted to matrix-driven architecture selection, choosing platform-specific runners (ubuntu-24.04 for amd64, arm64-capable runner for arm64), and building each architecture by digest using docker/build-push-action@v6, extracting the digest output and uploading per-arch digest artifacts.
Merge job: manifest assembly and tag computation
.github/workflows/docker-publish.yml
New merge job downloads digest artifacts from all architectures, logs into registry, computes Docker tags (release + latest tags) via docker/metadata-action@v5, assembles multi-arch manifest using docker buildx imagetools create with jq-derived tag arguments, and inspects the final :latest manifest.
Task index and latest task updates
docs/tasks/README.md
Task index updated to include two new active tasks (docker publish arm64 native and release v0.4.2, both 2026-05-25), and "Latest active task" reference changed to the docker publish arm64 native task.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~22 minutes

Possibly related PRs

  • wafflebase/wafflebase#208: Updates task index in docs/tasks/README.md to archive/rebalance shipped tasks, similar index management pattern as this PR's task tracking updates.

Poem

🐰 A rabbit builds in many shells,
Per-arch, side-by-side we dwell,
No QEMU hangs on native ground,
Digest by digest, arm64 found!
Manifest merged, the workflow sings. 🎵

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title directly and accurately reflects the main change: refactoring the Docker publish workflow to use native arm64 runners instead of QEMU emulation to fix release hangs.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/docker-publish-arm64-native

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/docker-publish.yml:
- Line 50: Replace tag-pinned GitHub Action refs with immutable commit SHAs for
all uses entries mentioned (actions/checkout, docker/login-action,
docker/setup-buildx-action, docker/build-push-action, actions/upload-artifact,
actions/download-artifact, docker/metadata-action). For each occurrence (e.g.,
the existing uses: entries like actions/checkout@v4 and
docker/build-push-action@v6), look up the corresponding repository's recommended
commit SHA for the desired tag/release and update the uses value to the full SHA
ref (e.g., actions/checkout@<sha>) to ensure immutable pinning; do this
consistently for the instances at the workflow locations that reference those
action names so the workflow uses specific commit SHAs instead of version tags.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 6be85f52-8eee-4962-a784-44cc53ebe31f

📥 Commits

Reviewing files that changed from the base of the PR and between fe1e3a4 and c18058e.

📒 Files selected for processing (3)
  • .github/workflows/docker-publish.yml
  • docs/tasks/README.md
  • docs/tasks/active/20260525-docker-publish-arm64-native-todo.md

# and push by digest only; the merge job assembles the tagged manifest.
- name: Build and push image by digest
id: build
uses: docker/build-push-action@v6

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Find non-SHA-pinned `uses:` references in this workflow.
rg -nP '^\s*uses:\s*[^@]+@(?!(?:[0-9a-fA-F]{40})$).+$' .github/workflows/docker-publish.yml

Repository: wafflebase/wafflebase

Length of output: 454


Pin GitHub Actions to immutable commit SHAs (including checkout).

In .github/workflows/docker-publish.yml, these uses: refs are tag-pinned (@v*) rather than SHA-pinned, weakening supply-chain integrity: actions/checkout@v4 (35), docker/login-action@v3 (38, 88), docker/setup-buildx-action@v3 (44, 94), docker/build-push-action@v6 (50), actions/upload-artifact@v4 (65), actions/download-artifact@v4 (81), and docker/metadata-action@v5 (99).

[NO DIFF NEEDED—SUGGESTED PATCH SHAPE ALREADY PROVIDED IN ORIGINAL COMMENT]

🧰 Tools
🪛 zizmor (1.25.2)

[error] 50-50: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)

(unpinned-uses)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/docker-publish.yml at line 50, Replace tag-pinned GitHub
Action refs with immutable commit SHAs for all uses entries mentioned
(actions/checkout, docker/login-action, docker/setup-buildx-action,
docker/build-push-action, actions/upload-artifact, actions/download-artifact,
docker/metadata-action). For each occurrence (e.g., the existing uses: entries
like actions/checkout@v4 and docker/build-push-action@v6), look up the
corresponding repository's recommended commit SHA for the desired tag/release
and update the uses value to the full SHA ref (e.g., actions/checkout@<sha>) to
ensure immutable pinning; do this consistently for the instances at the workflow
locations that reference those action names so the workflow uses specific commit
SHAs instead of version tags.

@github-actions

github-actions Bot commented May 25, 2026

Copy link
Copy Markdown
Contributor

Verification: verify:self

Result: ✅ PASS in 219.8s

Lane Status Duration
tokens:build ✅ pass 2.0s
sheets:build ✅ pass 13.1s
docs:build ✅ pass 11.7s
slides:build ✅ pass 13.2s
verify:fast ✅ pass 139.2s
frontend:build ✅ pass 16.9s
verify:frontend:chunks ✅ pass 0.3s
backend:build ✅ pass 4.7s
cli:build ✅ pass 2.0s
verify:entropy ✅ pass 16.7s

Verification: verify:integration

Result: ✅ PASS

@codecov

codecov Bot commented May 25, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@hackerwins
hackerwins merged commit 1d7a801 into main May 25, 2026
4 checks passed
@hackerwins
hackerwins deleted the fix/docker-publish-arm64-native branch May 25, 2026 00:24
hackerwins added a commit that referenced this pull request May 25, 2026
These tasks landed on main per recent commit log: docker-publish-arm64
(#296), import-progress-toast (#299), pptx-blipfill-fillrect-crop (#297),
release-v0.4.2 (#295), slides-textbox-autogrow (cf83a49), and
slides-textbox-edit-theme-color (#300). Move their docs to
archive/2026/05 and regenerate the task indexes.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
hackerwins added a commit that referenced this pull request May 25, 2026
These tasks landed on main per recent commit log: docker-publish-arm64
(#296), import-progress-toast (#299), pptx-blipfill-fillrect-crop (#297),
release-v0.4.2 (#295), slides-textbox-autogrow (cf83a49), and
slides-textbox-edit-theme-color (#300). Move their docs to
archive/2026/05 and regenerate the task indexes.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant