Skip to content

ci(nix-build): raise job timeout to 120 minutes for cold builds#6389

Merged
houko merged 2 commits into
mainfrom
ci/nix-build-cold-timeout
Jul 5, 2026
Merged

ci(nix-build): raise job timeout to 120 minutes for cold builds#6389
houko merged 2 commits into
mainfrom
ci/nix-build-cold-timeout

Conversation

@houko

@houko houko commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Problem

The Nix Build workflow has not gone green on main since June 11 (last success: run 27324913045).
Every push-to-main run since then either failed with Nix daemon disconnected unexpectedly (late June) or was cancelled at exactly 1h00m by the 60-minute job timeout (run 28433400325, run 28709165648 — both matrix legs).

Diagnosis

  • The /nix/store cache saved by cache-nix-action almost never survives to the next run: the Rust CI lanes create ~8 caches of 0.9–1.9 GiB each per day (≈9.6 GiB on July 4 alone), which churns through the repo's 10 GB Actions cache quota and LRU-evicts the nix-* entries. Every Nix Build run is therefore a cold build of the full crane deps + workspace closure.
  • The eviction was already biting on June 11: the librefang-desktop leg finished in 82 s (full cache hit) while the librefang-cli leg took 56 min (its cache had been evicted) — right at the edge of the 60-minute limit.
  • In the July 4 cancelled run the cli leg was still compiling workspace crates (librefang-acp, with librefang-api and the final link still ahead) at the 60-minute mark, putting a cold leg at roughly 80–95 minutes.
  • The Nix daemon disconnected failures from late June were the disk-exhaustion mode tracked in ci(nix-build): librefang-cli job repeatedly fails with 'Nix daemon disconnected unexpectedly' (runner disk exhaustion) #6081/ci(nix-build): free runner disk space before nix build #6082. That mode is gone: runners now report a 145 GB disk with ~89 GB free before the free-space step (July 4 run logs). The free-space step is kept as cheap defense.

Change

  • Raise timeout-minutes from 60 to 120 in .github/workflows/nix-build.yml, with a comment recording why (cold builds are the norm while the cache quota is contended; 120 covers an observed ~80–95 min cold leg with margin while still catching a hung build).
  • CHANGELOG entry under [Unreleased].

Verification

Out of scope, surfaced for a maintainer decision

The durable fix for the cache eviction is an external binary cache (Cachix / FlakeHub cache), which needs an account + repo secret and is deliberately not part of this PR.
Without it, Nix Build stays a cold ~80–95 min job on every Cargo.lock-touching push to main.

Nix Build has not gone green on main since June 11: the Rust CI lanes churn through the repo's 10 GB Actions cache quota daily, so the /nix/store cache is evicted before the next run and every leg builds the full crane deps + workspace closure from scratch.
A cold librefang-cli leg takes ~80-95 minutes and the 60-minute timeout cancelled it mid-workspace-compile on every recent push-to-main run.
Disk is no longer a factor (runners now have 145 GB with ~89 GB free before the cleanup step), so the #6081 daemon-crash mode is gone; the free-space step stays as cheap defense.
@github-actions github-actions Bot added no-rust-required This task does not require Rust knowledge size/XS < 10 lines changed area/ci CI/CD and build tooling labels Jul 5, 2026
@github-actions github-actions Bot added the area/docs Documentation and guides label Jul 5, 2026
@houko
houko enabled auto-merge (squash) July 5, 2026 05:43
@houko
houko merged commit f562ade into main Jul 5, 2026
33 of 35 checks passed
@houko
houko deleted the ci/nix-build-cold-timeout branch July 5, 2026 05:45
@houko houko mentioned this pull request Jul 10, 2026
houko pushed a commit that referenced this pull request Jul 10, 2026
Rewrap the new cli_pypi guard comments in release.yml and
release-cli.yml to one sentence per line, per CLAUDE.md's
prose-wrapping rule (new prose must break only at sentence
boundaries, not at a fixed column width) — the same fix #6418
applied to a similar release-workflow comment.

Add the missing [Unreleased] CHANGELOG entry for this fix, matching
the established convention of every other CI-only fix PR in this
repo (#6410, #6389, #6416, #6418) recording a changelog line.
houko added a commit that referenced this pull request Jul 10, 2026
…op deleting CLI assets (#6433)

* fix(ci): publish CLI wheels to PyPI only for stable/LTS releases

The `librefang` PyPI project has a 10 GB total-size quota.
Each release uploads ~8 per-platform CLI binary wheels (~250 MB), and because every beta/rc tag published too, 46 accumulated pre-releases consumed 10.35 GB and pushed the project over quota — the v2026.7.10 stable `CLI / PyPI` job then failed with `400 Project size too large`.

Gate the `cli_pypi` job (in both release.yml and the manual release-cli.yml re-run path) to stable and LTS tags only.
A tag is treated as a pre-release when it carries any `-suffix` (e.g. `-beta.10`, `-rc.1`); `-lts` is the sole non-pre-release suffix, matching create_release's existing `-lts` special-casing.
Pre-release CLI binaries are still attached to the GitHub Release, and `librefang-sdk` (a separate PyPI project) is unaffected.

* docs(ci): rewrap cli_pypi gate comments and add CHANGELOG entry

Rewrap the new cli_pypi guard comments in release.yml and
release-cli.yml to one sentence per line, per CLAUDE.md's
prose-wrapping rule (new prose must break only at sentence
boundaries, not at a fixed column width) — the same fix #6418
applied to a similar release-workflow comment.

Add the missing [Unreleased] CHANGELOG entry for this fix, matching
the established convention of every other CI-only fix PR in this
repo (#6410, #6389, #6416, #6418) recording a changelog line.

* fix(ci): stop desktop re-release from deleting CLI binary assets

The desktop job's "Delete existing assets for this target" step matched `*<rust_target>*`, and the cli_* jobs name their uploads `librefang-<rust_target>.tar.gz|.zip` (+ `.sha256`) with the SAME target triple.
So when a desktop job re-ran after its CLI counterpart had already uploaded, it deleted the CLI binary tarball as collateral.
This is why the v2026.7.10 macOS CLI tarballs (`librefang-x86_64-apple-darwin.tar.gz`, `librefang-aarch64-apple-darwin.tar.gz`) disappeared after the desktop macOS jobs re-ran for notarization, which then made the `CLI / PyPI` job fail with a 404 when it tried to download them to build the wheels.

Skip CLI-owned assets (`librefang-*`, `SHA256SUMS*`) before the target-glob delete.
Desktop bundles use Tauri's own naming (`LibreFang_<ver>_<x64|aarch64|amd64|arm64>…`) which never contains the triple, so the guard costs the desktop nothing while protecting the CLI artifacts on every platform.

* docs(changelog): note the desktop asset-cleanup fix (#6433)

---------

Co-authored-by: Evan <[email protected]>
Co-authored-by: Claude <[email protected]>
@houko houko mentioned this pull request Jul 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/ci CI/CD and build tooling area/docs Documentation and guides no-rust-required This task does not require Rust knowledge size/XS < 10 lines changed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant