Repository navigation
fix(update): bound the Cachix flush and recover a wedged daemon - #286
Merged
Merged
Conversation
Update run 40048292663 failed because validate-darwin-warm-rust (4) sat 3h in `Flush Cachix daemon` until the 360-minute job timeout. cachix (1.11.1 on the runners, unchanged through 1.13.0) builds its HTTP manager with responseTimeoutNone. On that runner the daemon stopped completing uploads at 06:05:45: 81 consecutive cargo-package outputs built after that point are on neither gkze nor cache.nixos.org, while shard 2's same 129 outputs are all on gkze. Daemon shutdown waits for the wedged narinfo processor before it replies, and each client handler keeps answering keep-alive pings, so `cachix daemon stop` never returns. - Every Cachix/Nix network call in flush now has a timeout inside a 40-minute flush budget; pushes run in retried, bounded chunks. - A stop that fails or times out kills the daemon, pushes every path the runner built (`ultimate` in `nix path-info --all --sigs`), and fails closed unless each one has a narinfo on gkze or cache.nixos.org. The old daemon.log `Pushing` leftovers missed queued paths that no worker had started. - Flush steps get timeout-minutes: 45 as a backstop. Co-authored-by: George Kontridze <[email protected]>
The flush proof now builds a third path into a frozen daemon, so the hook's push only reaches the socket backlog and `daemon stop` gets no reply. Flush must time stop out, kill the daemon, and republish that path; the flush step asserts it took that path, and assert-partial requires the wedged path on gkze. Co-authored-by: George Kontridze <[email protected]>
Co-authored-by: George Kontridze <[email protected]>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
gkze
added a commit
that referenced
this pull request
Oct 24, 2026
#287) * refactor(update): share exact-Bun node_modules updater mechanics Mux owned the only implementation of the exact-Bun contract: pin the manifest's packageManager Bun, hash its release archives, keep them through the pins-only stage, and refuse duplicate runtime entries. Move that into ExactBunNodeModulesHashUpdater so another Bun-locked source can reuse it. Candidate handoff restored only direct MappingMetadata subclasses, which would have dropped MuxSourceMetadata once it specialized the shared base. Walk the whole subclass tree. Co-authored-by: George Kontridze <[email protected]> * fix(update): evaluate companion hash probes against the parent's run result Flake-input hash probes overrode only the package's own sources entry, so a companion read its parent's pre-update workspace file. opencode-desktop's node_modules is opencode's derivation: once opencode pins an exact Bun, a packageManager bump would make desktop evaluate the stale pin and fail. Pass the companion parent's in-run entry into the probe overrides, as the source-backed updaters already do, and declare opencode-desktop a companion of opencode. Co-authored-by: George Kontridze <[email protected]> * fix(packages): install opencode node_modules with its exact pinned Bun Update run 40086686747 failed opencode and opencode-desktop on x86_64-linux: their node_modules FODs ran nixpkgs bun 1.4.2 against fork rev d5ea4ab, whose bun.lock the fork now regenerates with its packageManager Bun ([email protected], bumped upstream in anomalyco/opencode#58087). With the fork's root `vite: catalog:` devDependency, 1.4.2 re-resolves the vite 7/8 split and rejects --frozen-lockfile ("lockfile had changes"); 1.4.4 accepts the same tree for both filter sets on every target. nixpkgs ships 1.4.2 on every channel. Build opencode's node_modules and CLI with lib/exact-bun.nix at the input's packageManager version, maintained by the updater like mux and superset. Desktop inherits it through opencode's node_modules and buildBun. Drop the packageManager rewrite: exact-bun.nix now asserts that equality instead. sources.json is regenerated by the updaters for the still-locked a6fde022 ([email protected]): the same release archives nixpkgs fetches, so the carried node_modules hashes still build (verified on x86_64-linux). Co-authored-by: George Kontridze <[email protected]> * chore(ci): kick one Update run after the opencode exact-Bun fix Co-authored-by: George Kontridze <[email protected]> * fix(update): keep the newest metadata class when restoring candidates The subclass-tree walk visited classes last-defined-first, so a reloaded updater module's stale metadata class shadowed the live one and restored metadata failed isinstance checks. Visit in definition order, as the direct-subclass comprehension did. Co-authored-by: George Kontridze <[email protected]> * style: satisfy yamllint and nix fmt on inherited lines cachix-flush-proof.yml (#286) carried a 114-column comment line that fails lint-yaml-yamllint, and nix fmt marks opencode-desktop's unused node_modules override argument. Both already fail on main. * fix(ci): make bootstrap's flake eval work on shallow checkouts Update run 40089810663 failed prepare-pins at bootstrap with exit 1 and no diagnostic. 92e3ee5 added a `nix eval` of `(builtins.getFlake "git+file://<checkout>").inputs.nixpkgs.outPath`. Actions checks out with fetch-depth 1, and Determinate Nix (3.22.5 on the runners) rejects an explicit git+file URL to a clean shallow clone: "'<checkout>' is a shallow Git repository, so 'revCount' is not available". The `.` flake reference used by the preceding `nix build` and `nix develop` adds `shallow=1` implicitly, so only the eval failed. Bootstrap now takes its flake URL from the shared `local_flake_url`, which opts into `shallow=1`. That also covers the updaters that build the same URL and only avoided the failure because the tree was dirty or the devshell's Nix 2.34.9 tolerates it. The job launcher's entrypoint now reports a failed child's command and the tail of its captured stderr before exiting with its status, so a captured failure can no longer exit silently. * fix(packages): give superset's bun-lock refresh the shallow-safe URL update_bun_lock.py is a standalone template that cannot import lib.update.paths, so it carried its own copy of the git+file URL without the shallow=1 opt-in. Its test now asserts that the installable equals local_flake_url, and the neutils resolution-event test derives its expected URL from the same helper. --------- Co-authored-by: Cursor Agent <[email protected]> Co-authored-by: George Kontridze <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Update run 40048292663 failed only because
validate-darwin-warm-rust (4)finished validate at 06:12:32 and then sat inFlush Cachix daemonuntil the 360-minute job timeout cancelled it at 09:17.Root cause
The Cachix daemon on that runner wedged, and the flush's
cachix daemon stophad no bound, so it waited forever.rust_*outputs from shard 4 (taken from its artifactstderr.log) are on gkze. Of its 129 locally builtcargo-package-*FODs, 45 are on gkze and 84 are on neither gkze nor cache.nixos.org. In build order, everything after the 47th (built around 06:05:45) is missing: 81 in a row, plus 3 sporadic misses earlier. Shard 2 built the same 129 FODs, and all of them are on gkze.managerResponseTimeout = responseTimeoutNone(Cachix/Client/Env.hs). A stalled narinfo or upload request therefore never times out. Every queued path needs a narinfo answer, so the queue freezes.ClientStop,shutdownGracefullydrains for 60 s. It then callsNarinfoQuery.stop, which runsAsync.waiton the wedged processor thread before sendingDaemonExit. Meanwhile each client handler thread keeps answeringClientPingdirectly (Daemon/Listen.hs), so the stop client's 180 s stall guard never fires.flush_cachixrancachix daemon stop(and every latercachix push,pin, andpath-info) with no timeout. Nothing uploaded the flush log after the step, so the hang left no evidence.A second, latent gap: on the existing drain-timeout path, flush recovered only paths with a
Pushingline and no matchingPushedline indaemon.log. The daemon never logs queued paths that no worker has started, so those were silently lost.Fix
daemon stop300 s, pushes 20 min per attempt with 3 attempts in chunks of 1000, queries 120 s. Each flush step inupdate-native.ymlandupdate.ymlalso getstimeout-minutes: 45as a backstop.daemon stopfails or times out, flush logs the tail ofdaemon.logand terminates the daemon (SIGTERM plus SIGCONT, then SIGKILL; guarded by apscheck that the pid is still cachix). It then pushes every path this runner built itself: theultimatepaths fromnix path-info --all --sigs, excluding.drvs. The daemon starts before any build, so this covers everything the post-build hook ever queued.cachix pushskips). Only an HTTP 200 counts; anything else is retried, then treated as missing. Flush fails the job if any path is not served. No paths are dropped and nothing is waived.Flush proof extended
cachix-flush-proof.ymlnowSIGSTOPs the daemon and builds a third unique path, so the hook's push only reaches the socket backlog andstopgets no reply. The proof flush lowers the stop timeout to 30 s. In the same step (flush must stay the job's last step), it asserts the log showsstop returncode=124, the daemon termination, and the verified recovery.assert-partialalso requires the wedged path on gkze.Validation
ruff check,ruff format --checkandty checkare clean.jobs.pyhas 100% line and branch coverage.test_update_jobs_cachix_bounds.pyandtest_update_ci.pyinclude a real wedgeddaemon stop: a subprocess namedcachixgets SIGTERM, and the flush finishes in about 1 s.mainin this environment, which lacks Nix, Node and similar tooling..github/update-kickgets one line, so merging starts the next Update run.