fix(goal-runner): persist run state outside the state lock so GET /run never spuriously reports running:false#6083
Merged
Merged
Conversation
houko
enabled auto-merge (squash)
June 11, 2026 12:09
houko
force-pushed
the
fix/goal-run-state-trylock-flake
branch
from
June 11, 2026 12:12
e4a6472 to
3bb32ad
Compare
GoalRunner::state() snapshots a run with a non-blocking try_lock so an async HTTP handler never parks on a tick, and GET /api/goals/{id}/run renders a None snapshot as {"running": false} — indistinguishable from "no run exists".
The run loop, however, held that same state mutex across persist_run(), a synchronous SQLite write plus a potentially-blocking connection-pool checkout.
Under load a GET /run landing inside that write window lost the try_lock and surfaced a live run as not running; the goal_run_start_then_stop_with_agent integration test flaked on exactly this.
Update the in-memory state under the lock, release it, then persist the cloned snapshot outside the lock.
state is written only by the single loop task, so the snapshot stays consistent after the lock is released.
This shrinks the lock hold to a few synchronous field writes, so the try_lock no longer realistically contends.
CLAUDE.md prohibits multi-line comment blocks; each of the three new blocks added in the preceding commit is trimmed to one line that keeps the non-obvious lock-discipline invariant.
houko
force-pushed
the
fix/goal-run-state-trylock-flake
branch
from
June 11, 2026 12:25
3bb32ad to
0034a87
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The
Test / Ubuntu (shard 1/4)lane flaked onlibrefang-api::goals_routes_integration::goal_run_start_then_stop_with_agent(surfaced on the unrelated Dependabot PR #6080, which touches only dashboard npm deps):Root cause
GoalRunner::state()(goal_runner.rs) snapshots a run with a non-blockingtry_lock()so an async HTTP handler never parks on a tick.GET /api/goals/{id}/runrenders aNonesnapshot as{"running": false}— which is indistinguishable from "no run exists".The run loop, however, held that same
statemutex acrosspersist_run()— a synchronous SQLite write (save_run), including a potentially-blocking connection-pool checkout:Under load (CI runs ~1800 tests in parallel) a
GET /runlanding inside that write window loses thetry_lockand reports a live run as not running.This is also a real product issue: a dashboard polling
/runwould intermittently flicker "stopped" for an active goal run.Confirmed by code, not guesswork:
save_runis synchronousrusqlite(crates/librefang-memory/src/goal_run_store.rs), andstate()usestry_lock(crates/librefang-kernel/src/goal_runner.rs).The race is one short persist window per run, so it only surfaces under heavy parallelism — 150 isolated local iterations did not reproduce it.
Fix
Update the in-memory state under the lock, release it, then persist the cloned snapshot outside the lock — in both the success and the error tick paths.
stateis written only by the single loop task, so the cloned snapshot stays consistent after the lock is released.This shrinks the lock hold from "across a blocking SQLite write" to "a few synchronous field writes" (sub-microsecond), so the
try_lockno longer realistically contends.Also clarified the
state()doc comment to make the lock-discipline invariant explicit, so a future edit does not reintroduce a lock-across-I/O hold.Verification (Linux dev container)
cargo test -p librefang-kernel --lib goal_runner— 12 passed, 0 failedcargo test -p librefang-api --test goals_routes_integration— 26 passed, 0 failedcargo clippy -p librefang-kernel --lib -- -D warnings— cleancargo fmt -p librefang-kernel --check— cleanScope
The existing
goal_run_start_then_stop_with_agentintegration test is the regression guard — it exercisesstart → GET /run → assert running:trueand was the test that flaked.A bespoke unit test for this sub-millisecond race would itself be timing-dependent (probabilistic), which the repo's no-flaky-tests policy rules out; the fix is verified at the code level and by the now-stable integration test.
This is a kernel-internal fix, intentionally not bundled into Dependabot PR #6080 (npm-only) per the one-PR-one-concern rule.