Skip to content

fix(goal-runner): persist run state outside the state lock so GET /run never spuriously reports running:false#6083

Merged
houko merged 2 commits into
mainfrom
fix/goal-run-state-trylock-flake
Jun 11, 2026
Merged

fix(goal-runner): persist run state outside the state lock so GET /run never spuriously reports running:false#6083
houko merged 2 commits into
mainfrom
fix/goal-run-state-trylock-flake

Conversation

@houko

@houko houko commented Jun 11, 2026

Copy link
Copy Markdown
Contributor

Problem

The Test / Ubuntu (shard 1/4) lane flaked on librefang-api::goals_routes_integration::goal_run_start_then_stop_with_agent (surfaced on the unrelated Dependabot PR #6080, which touches only dashboard npm deps):

assertion `left == right` failed   (goals_routes_integration.rs:570)
  left: Some(false)    // actual run["running"]
 right: Some(true)

Root cause

GoalRunner::state() (goal_runner.rs) snapshots a run with a non-blocking try_lock() so an async HTTP handler never parks on a tick.
GET /api/goals/{id}/run renders a None snapshot as {"running": false} — which is indistinguishable from "no run exists".

The run loop, however, held that same state mutex across persist_run() — a synchronous SQLite write (save_run), including a potentially-blocking connection-pool checkout:

{
    let mut s = state.lock().await;
    s.iteration = iteration + 1;
    ...
    persist_run(&store, &s);   // blocking I/O while holding the lock
}

Under load (CI runs ~1800 tests in parallel) a GET /run landing inside that write window loses the try_lock and reports a live run as not running.
This is also a real product issue: a dashboard polling /run would intermittently flicker "stopped" for an active goal run.

Confirmed by code, not guesswork: save_run is synchronous rusqlite (crates/librefang-memory/src/goal_run_store.rs), and state() uses try_lock (crates/librefang-kernel/src/goal_runner.rs).
The race is one short persist window per run, so it only surfaces under heavy parallelism — 150 isolated local iterations did not reproduce it.

Fix

Update the in-memory state under the lock, release it, then persist the cloned snapshot outside the lock — in both the success and the error tick paths.
state is written only by the single loop task, so the cloned snapshot stays consistent after the lock is released.
This shrinks the lock hold from "across a blocking SQLite write" to "a few synchronous field writes" (sub-microsecond), so the try_lock no longer realistically contends.

Also clarified the state() doc comment to make the lock-discipline invariant explicit, so a future edit does not reintroduce a lock-across-I/O hold.

Verification (Linux dev container)

  • cargo test -p librefang-kernel --lib goal_runner — 12 passed, 0 failed
  • cargo test -p librefang-api --test goals_routes_integration — 26 passed, 0 failed
  • cargo clippy -p librefang-kernel --lib -- -D warnings — clean
  • cargo fmt -p librefang-kernel --check — clean

Scope

The existing goal_run_start_then_stop_with_agent integration test is the regression guard — it exercises start → GET /run → assert running:true and was the test that flaked.
A bespoke unit test for this sub-millisecond race would itself be timing-dependent (probabilistic), which the repo's no-flaky-tests policy rules out; the fix is verified at the code level and by the now-stable integration test.

This is a kernel-internal fix, intentionally not bundled into Dependabot PR #6080 (npm-only) per the one-PR-one-concern rule.

@github-actions github-actions Bot added size/S 10-49 lines changed area/docs Documentation and guides area/kernel Core kernel (scheduling, RBAC, workflows) labels Jun 11, 2026
@github-actions github-actions Bot added the has-conflicts PR has merge conflicts that need resolution label Jun 11, 2026
@houko
houko enabled auto-merge (squash) June 11, 2026 12:09
@houko
houko force-pushed the fix/goal-run-state-trylock-flake branch from e4a6472 to 3bb32ad Compare June 11, 2026 12:12
@github-actions github-actions Bot added ready-for-review PR is ready for maintainer review has-conflicts PR has merge conflicts that need resolution and removed has-conflicts PR has merge conflicts that need resolution ready-for-review PR is ready for maintainer review labels Jun 11, 2026
Evan and others added 2 commits June 11, 2026 21:25
GoalRunner::state() snapshots a run with a non-blocking try_lock so an async HTTP handler never parks on a tick, and GET /api/goals/{id}/run renders a None snapshot as {"running": false} — indistinguishable from "no run exists".
The run loop, however, held that same state mutex across persist_run(), a synchronous SQLite write plus a potentially-blocking connection-pool checkout.
Under load a GET /run landing inside that write window lost the try_lock and surfaced a live run as not running; the goal_run_start_then_stop_with_agent integration test flaked on exactly this.
Update the in-memory state under the lock, release it, then persist the cloned snapshot outside the lock.
state is written only by the single loop task, so the snapshot stays consistent after the lock is released.
This shrinks the lock hold to a few synchronous field writes, so the try_lock no longer realistically contends.
CLAUDE.md prohibits multi-line comment blocks; each of the three new
blocks added in the preceding commit is trimmed to one line that keeps
the non-obvious lock-discipline invariant.
@houko
houko force-pushed the fix/goal-run-state-trylock-flake branch from 3bb32ad to 0034a87 Compare June 11, 2026 12:25
@github-actions github-actions Bot added ready-for-review PR is ready for maintainer review and removed has-conflicts PR has merge conflicts that need resolution labels Jun 11, 2026
@houko
houko merged commit effff56 into main Jun 11, 2026
32 checks passed
@houko
houko deleted the fix/goal-run-state-trylock-flake branch June 11, 2026 12:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/docs Documentation and guides area/kernel Core kernel (scheduling, RBAC, workflows) ready-for-review PR is ready for maintainer review size/S 10-49 lines changed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants