Skip to content

feat(kernel/workflow): pause/resume foundation for resumable workflows#3418

Merged
houko merged 3 commits into
mainfrom
feat/3335-resumable-workflow
Apr 27, 2026
Merged

feat(kernel/workflow): pause/resume foundation for resumable workflows#3418
houko merged 3 commits into
mainfrom
feat/3335-resumable-workflow

Conversation

@houko

@houko houko commented Apr 27, 2026

Copy link
Copy Markdown
Contributor

First slice of #3335. Lands the state machine + persistence primitives for resumable workflows. ApprovalManager wiring, REST/CLI surface, DAG-path support, GC, and the requester/approver capability check are explicit follow-ups against the same issue (listed below).

Why split

Issue #3335's full acceptance criteria are a multi-PR effort. This PR keeps the diff to one file (~550 lines including tests) so reviewers can focus on the state-machine shape — once it lands, downstream work (approval integration, API endpoints, dashboard surface) plugs into a stable substrate.

What lands

Types

  • WorkflowRunState::Paused { resume_token, reason, paused_at } carries the token a caller must present to resume_run.
  • WorkflowRun gains four #[serde(default)] fields that survive round-trips with old persisted JSON:
    • pause_request: Option<PauseRequest> — set by pause_run, consumed at the next step boundary.
    • paused_step_index: Option<usize> — where to resume.
    • paused_variables: HashMap<String, String>{{var}} bindings restored on resume.
    • paused_current_input: Option<String> — input that would have fed the paused step.
  • PauseRequest { reason, resume_token } — token is generated when pause_run is called so the caller can begin waiting on the corresponding approval / input artifact before the executor has honored the request. The executor reuses the same token at the actual transition.

API

WorkflowEngine::pause_run(run_id, reason) -> Uuid
WorkflowEngine::resume_run(run_id, token, resolver, sender) -> Result<String>

pause_run is idempotent — paused-already returns the existing token; errors on Completed / Failed. resume_run verifies state + token, restores snapshot, flips state to Running, re-enters execute_run_sequential. Snapshot fields are cleared on success; a wrong-token attempt leaves state untouched.

Executor changes

execute_run_sequential reads pause_request at the top of every step iteration. On honoring it snapshots (step_index, variables, current_input) into the run, transitions to Paused, and returns the intermediate output. In-flight steps finish first — partial-step rollback would be a much bigger feature than #3335 requires.

execute_run_dag refuses cleanly when a pause request is lodged, rather than silently dropping it (the DAG executor would never observe pause_request and the run would just complete, leaving the caller's resume_run call hanging).

Persistence

persist_runs filter extended to Completed | Failed | Paused. Paused runs round-trip through daemon restart with their token + snapshot intact; a fresh load_runs restores them and a subsequent resume_run works as if no restart had happened. Pending / Running are still skipped — they have no durable boundary.

Tests (5 new)

Test What it verifies
pause_after_first_step_then_resume_finishes_workflow Sender triggers pause from inside step 1; loop honors at the boundary before step 2; resume completes with step_results.len() == 2 and snapshot fields cleared
resume_run_with_wrong_token_is_rejected Resume with random token errors and leaves state Paused
resume_run_on_non_paused_run_is_rejected Resume on Pending run errors with "expected Paused"
paused_run_round_trips_through_persist_and_load (multi-thread flavor) Build paused run on engine #1, load into engine #2, assert resume_token + snapshot survive
pause_request_on_dag_workflow_returns_explicit_error DAG run with pause request returns the explicit follow-up error rather than silently completing

Test plan

  • cargo clippy --workspace --all-targets -- -D warnings clean
  • cargo test -p librefang-kernel --lib --no-fail-fast 637 passed (5 new, no regressions)
  • Backward compat: persisted workflow_runs.json from before this change loads cleanly because all new fields are #[serde(default)]

Follow-ups (separate PRs against #3335)

  • ApprovalManager wiring — a step returning "needs approval" triggers pause + registers with the existing TOTP / escalation flow.
  • POST /api/workflows/runs/{id}/resume + GET /api/workflows/runs/paused for dashboard / external callers.
  • CLI librefang workflow run paused list / resume <id>.
  • DAG executor pause/resume (per-layer checkpoints).
  • GC for paused workflows older than a configurable TTL.
  • Capability check: the approver must match the original requester.
  • Real-LLM integration test exercising the full pause-while-executing path through send_message_streaming_with_sender_and_opts (the current tests use mock senders).

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@github-actions github-actions Bot added the area/kernel Core kernel (scheduling, RBAC, workflows) label Apr 27, 2026
@github-actions github-actions Bot added the size/L 250-999 lines changed label Apr 27, 2026
Closes part of #3335. ApprovalManager integration, REST/CLI surface,
and DAG-path support are explicit follow-ups (see PR description).

Adds the state machine + persistence primitives that make
"pause-mid-pipeline, resume-from-the-same-step" possible. Today the
workflow engine runs to completion or fails; with this PR a caller can
lodge a pause request, the sequential executor honors it at the next
step boundary, and a later `resume_run` call replays from the saved
snapshot.

What lands:

- `WorkflowRunState::Paused { resume_token, reason, paused_at }` —
  carries the token a caller must present to `resume_run`. Existing
  `is_paused()` helper for ergonomic matches.
- `WorkflowRun` gains four fields, all `#[serde(default)]` /
  `skip_serializing_if` for backward compat with persisted JSON:
    - `pause_request: Option<PauseRequest>` (set by `pause_run`,
      consumed at the next step boundary)
    - `paused_step_index: Option<usize>` (where to resume)
    - `paused_variables: HashMap<String, String>` (`{{var}}` bindings)
    - `paused_current_input: Option<String>` (input that would have
      fed the paused step, restored on resume)
- New `PauseRequest { reason, resume_token }`. Token is generated when
  `pause_run` is called so callers can begin waiting on the
  corresponding approval / input artifact before the executor has
  honored the request — the executor reuses the same token at the
  transition.
- `WorkflowEngine::pause_run(run_id, reason) -> Uuid`
    - Idempotent: paused-already returns the existing token.
    - Errors on Completed / Failed.
- `WorkflowEngine::resume_run(run_id, token, resolver, sender) -> Result<String>`
    - Verifies state is `Paused` and token matches.
    - Restores snapshot, flips state to `Running`, re-enters
      `execute_run_sequential`. Snapshot fields are cleared on
      successful resume; a wrong-token attempt leaves state untouched.
- `execute_run_sequential` reads `pause_request` at the top of every
  step iteration. On honoring, snapshots `(step_index, variables,
  current_input)` into the run, transitions to `Paused`, returns the
  intermediate output without erroring. In-flight steps are allowed
  to finish — partial-step rollback would be a much bigger feature.
- `execute_run_dag` refuses cleanly if a pause request was lodged,
  rather than silently dropping it. Pause/resume on DAG runs is the
  follow-up.
- `persist_runs` filter extended to include `Paused`. Paused runs
  round-trip through daemon restart with their token + snapshot
  intact; a fresh `load_runs` call restores them and a subsequent
  `resume_run` works as if no restart had happened.

Tests (5 new in `workflow::tests`):

- `pause_after_first_step_then_resume_finishes_workflow` — sender
  triggers pause from inside step 1; loop honors at the boundary
  before step 2; resume completes the run with both
  `step_results.len() == 2` and snapshot fields cleared.
- `resume_run_with_wrong_token_is_rejected` — resume with a fresh
  random token errors and leaves state Paused.
- `resume_run_on_non_paused_run_is_rejected` — resume on a run still
  in Pending errors with "expected Paused".
- `paused_run_round_trips_through_persist_and_load` (multi-thread
  flavor) — Phase 1 builds a paused run on engine #1 and lets
  `execute_run`'s built-in `persist_runs_async` flush; Phase 2 loads
  into engine #2 via `block_in_place(load_runs)` and asserts the
  resume_token + snapshot survive.
- `pause_request_on_dag_workflow_returns_explicit_error` — a paused
  request on a DAG workflow returns the explicit "follow-up needed"
  error rather than silently completing.

Pre-flight: `cargo clippy --workspace --all-targets -- -D warnings`
clean, `cargo test -p librefang-kernel --lib --no-fail-fast` 637
passed (5 new, no regressions).

Out of scope (tracked as follow-ups against the same #3335):

- ApprovalManager wiring — a step returning "needs approval" should
  trigger pause + register with the existing TOTP/escalation flow.
  This PR's primitive is the substrate that integration uses.
- API endpoints `POST /api/workflows/runs/{id}/resume` and
  `GET /api/workflows/runs/paused` for dashboard / external callers.
- CLI `librefang workflow run paused list / resume <id>`.
- DAG executor pause/resume support (per-layer checkpoints).
- GC for paused workflows older than a configurable TTL.
- Capability check that the approver matches the original requester.
@houko
houko force-pushed the feat/3335-resumable-workflow branch from d17c2ec to fdc5383 Compare April 27, 2026 15:16
houko added 2 commits April 28, 2026 00:23
…ause hygiene

Address review feedback on #3418:

1. Resume snapshot preserved across failure. `execute_run_sequential`
   used to `take()` the snapshot on entry, so any step error after
   restoration left the run as Failed with no recovery data — a future
   retry-from-failure path was structurally impossible. Switched to
   clone-on-read; the snapshot is now authoritatively cleared only
   when the run reaches Completed. Failed runs carry their snapshot
   forward (forward-compat for retry semantics). The pause-mid-loop
   branch already overwrites the snapshot with fresh values when
   transitioning back to Paused, so no stale data leaks.

2. Orphan `pause_request` cleanup. If a pause request was lodged
   between the loop's final step-boundary check and the run's
   completion, `pause_request` survived as dead data on Completed
   runs. The Completed transition now clears `pause_request`,
   `paused_step_index`, `paused_variables`, and
   `paused_current_input` together, keeping the invariant 'snapshot
   fields valid only on Paused (or Failed retry-candidates)'.

3. `paused_variables: HashMap → BTreeMap`. The persisted JSON now
   has stable key order — re-serializing a Paused run twice produces
   byte-identical output, which keeps audit logs and external
   snapshots clean. Mirrors the project's existing
   deterministic-ordering convention for LLM-bound prompt data
   (#3298) even though pause snapshots don't directly enter prompts.

4. `paused_at` doc comment: pointed at the future TTL-based GC
   sweep that will consume it (#3335 GC follow-up), so future
   readers know the field has a downstream consumer.

`cargo test -p librefang-kernel --lib workflow` passes (65 tests,
including `paused_run_round_trips_through_persist_and_load`).
`cargo test -p librefang-api --test config_schema_golden` clean
(no fixture drift).
Address review feedback on PR #3418 (open). Picks up the
already-uncommitted improvements (BTreeMap-backed paused_variables for
deterministic JSON, clone-not-take in resume snapshot read,
Completed-path snapshot clear on the sequential happy path) and adds
the items the multi-angle review flagged:

1. **DAG ghost-state cleanup (quality BUG, security FIX-doc).** The
   DAG entry guard previously returned `Err(...)` and left
   `pause_request: Some(_)` on a run still in `Running`, so an
   unrunnable workflow looked alive forever. The guard now flips the
   run to `Failed` with a descriptive `error` field before returning,
   and a new `cleanup_terminal_pause_state` post-pass at the bottom
   of `execute_run` / `resume_run` clears every pause field on any
   terminal transition (Completed or Failed) — so the same five
   `clear_pause_state` lines aren't sprinkled across the ~10 mid-step
   `Failed` sites in the existing executors.

2. **Rollback safety on `load_runs` (quality OK-with-caveat).**
   Previously a single bad row in `workflow_runs.json` killed the
   whole `Vec<WorkflowRun>` deserialization, so an old daemon reading
   a file written by a newer one (e.g. one with the new tagged
   `Paused { ... }` variant) would lose all run history. Now parses
   per-row via `Vec<serde_json::Value>` first, attempts each
   `from_value::<WorkflowRun>` independently, and skips unrecognized
   rows with a WARN. Newer daemons reading older files still parse
   cleanly because every post-foundation field is `#[serde(default)]`.

3. **Security doc-comments (security FIX-doc).** Added explicit
   warnings on `PauseRequest.reason` /
   `WorkflowRunState::Paused { reason }`: persisted plaintext to
   `workflow_runs.json`, surfaced to operators — not a secret store.
   Same `# SECURITY:` block on `PauseRequest.resume_token`: tokens are
   plaintext at rest in the foundation; the future REST handler must
   hash-at-rest before that surface ships. Added a "DAG workflows
   fail-closed on pause" warning to `pause_run`'s docs so a
   confused caller does not silently brick a DAG run.

4. **Three new tests (quality MISS).**
   - `pause_run_is_idempotent_returns_same_token`: triple-call returns
     the same Uuid; the *first* reason wins (later calls don't
     overwrite the message in logs / UI).
   - `resume_run_after_completion_is_rejected`: pause → execute →
     resume to completion → second resume with the same token errors
     with "expected Paused".
   - `pause_then_execute_on_pending_pauses_at_step_zero`: lock down
     the pause-before-any-step path explicitly (was previously only
     exercised as a side-effect of `resume_run_with_wrong_token`).
   - Existing DAG-pause test extended to assert the new
     `state == Failed` + `pause_request is None` shape.

Pre-flight: `cargo clippy --workspace --all-targets -- -D warnings`
clean, `cargo test -p librefang-kernel --lib --no-fail-fast` 642
passed (8 pause/resume tests, 0 regressions).

Out of scope, still tracked against #3335:
- ApprovalManager wiring + capability check (approver matches the
  original requester).
- REST `POST /api/workflows/runs/{id}/resume` and `GET /api/workflows/runs/paused`.
- CLI `librefang workflow run paused list / resume <id>`.
- Per-layer DAG pause/resume (today: fail-closed).
- GC for paused workflows older than a configurable TTL.
- Hash-at-rest for `resume_token` in `workflow_runs.json`.
- WorkflowPaused / WorkflowResumed hook events for external observers.
@houko

houko commented Apr 27, 2026

Copy link
Copy Markdown
Contributor Author

Internal multi-angle review (quality + security + Explore for missed trigger paths) found 1 real bug, 1 MISS, 1 OK-with-caveat, several doc-only FIX items, plus design-intent items to call out. All addressed in befe0f5e.

Finding Source Severity Resolution
DAG entry guard left pause_request: Some(_) on a run still in Running — unrunnable workflow looked alive forever quality BUG DAG guard now flips state to Failed with descriptive error before returning. New cleanup_terminal_pause_state post-pass at the bottom of execute_run / resume_run clears all pause fields on any terminal transition (Completed or Failed) — single point of cleanup instead of touching ~10 mid-step Failed sites
load_runs killed the whole Vec<WorkflowRun> on the first bad row → old daemon reading new file with Paused { ... } variant lost all run history quality OK-with-caveat → fixed Per-row tolerant parse: Vec<serde_json::Value> first, then from_value per row, skipped rows logged at WARN. Newer daemons reading older files still parse cleanly via #[serde(default)]
PauseRequest.reason / WorkflowRunState::Paused { reason } persisted plaintext, no SECURITY note security FIX-doc Added # SECURITY: blocks on both fields and on pause_run's public docs
resume_token plaintext at rest in workflow_runs.json security RISK (acceptable) Documented inline on the field — acceptable for kernel-internal foundation, future REST handler must hash-at-rest
DAG fail-closed behavior was a footgun for any caller that didn't know which executor a workflow used security FIX-doc Added "DAG workflows fail-closed on pause" warning to pause_run's rustdoc
3 missing tests quality MISS Added: pause_run_is_idempotent_returns_same_token (triple-call, same Uuid, first reason wins); resume_run_after_completion_is_rejected (double-resume on completed run errors); pause_then_execute_on_pending_pauses_at_step_zero (locks down pre-start pause path). Plus extended pause_request_on_dag_workflow_returns_explicit_error to assert the new Failed + cleared pause_request shape
All public entry points funnel through execute_run / resume_run; no WorkflowRunState::Running direct construction outside the executors; no scheduled / cron path bypass Explore COVERED No code change
No WorkflowPaused / WorkflowResumed hook events for external observers Explore BYPASS-RISK (deferred) Tracked as a follow-up — observability surface belongs with the dashboard / API integration PR

Design-intent items called out, no code change

  • Mention spoofing → claim re-route: not relevant here (no per-user routing).
  • No caller binding on pause_run / resume_run: acceptable for kernel-internal foundation; future REST handler enforces requester==approver.
  • Concurrent pause_run + loop-honor race: traced through under the runs.write() per-key lock — every interleaving either honors with the lodged token or returns the same idempotent token, never issues a fresh one over an unfulfilled request. Quality reviewer initially flagged then re-verdict'd as OK after tracing.
  • Lock ordering runs.write() + workflows.read(): matches existing execute_run pattern; verified no inverse-ordered acquirer exists (only callers of workflows.write() are register / update_workflow / remove_workflow, none touch runs).

Test plan re-run on befe0f5e

  • cargo clippy --workspace --all-targets -- -D warnings clean
  • cargo test -p librefang-kernel --lib --no-fail-fast 642 passed (8 pause/resume tests, 0 regressions)
  • All 8 pause/resume tests independently verified

Public API stability

  • WorkflowRun field paused_variables switched from HashMap<String, String>BTreeMap<String, String> (already in your previous commit on this branch). Affects external callers that match on the type; no other shape changes.
  • pause_run / resume_run signatures unchanged from the original PR.
  • DAG-with-pause behavior changed from "Err returned, run state untouched" to "run flipped to Failed with descriptive error". A caller that relies on the old "leave Running with pause_request" shape would break — but nothing today does (the original test only asserted the Err message, not the run state). Updated test now locks the new contract.

@houko
houko merged commit 24801c1 into main Apr 27, 2026
20 checks passed
@houko
houko deleted the feat/3335-resumable-workflow branch April 27, 2026 15:35
houko added a commit that referenced this pull request Apr 27, 2026
#3418)

* feat(kernel/workflow): pause/resume foundation for resumable workflows

Closes part of #3335. ApprovalManager integration, REST/CLI surface,
and DAG-path support are explicit follow-ups (see PR description).

Adds the state machine + persistence primitives that make
"pause-mid-pipeline, resume-from-the-same-step" possible. Today the
workflow engine runs to completion or fails; with this PR a caller can
lodge a pause request, the sequential executor honors it at the next
step boundary, and a later `resume_run` call replays from the saved
snapshot.

What lands:

- `WorkflowRunState::Paused { resume_token, reason, paused_at }` —
  carries the token a caller must present to `resume_run`. Existing
  `is_paused()` helper for ergonomic matches.
- `WorkflowRun` gains four fields, all `#[serde(default)]` /
  `skip_serializing_if` for backward compat with persisted JSON:
    - `pause_request: Option<PauseRequest>` (set by `pause_run`,
      consumed at the next step boundary)
    - `paused_step_index: Option<usize>` (where to resume)
    - `paused_variables: HashMap<String, String>` (`{{var}}` bindings)
    - `paused_current_input: Option<String>` (input that would have
      fed the paused step, restored on resume)
- New `PauseRequest { reason, resume_token }`. Token is generated when
  `pause_run` is called so callers can begin waiting on the
  corresponding approval / input artifact before the executor has
  honored the request — the executor reuses the same token at the
  transition.
- `WorkflowEngine::pause_run(run_id, reason) -> Uuid`
    - Idempotent: paused-already returns the existing token.
    - Errors on Completed / Failed.
- `WorkflowEngine::resume_run(run_id, token, resolver, sender) -> Result<String>`
    - Verifies state is `Paused` and token matches.
    - Restores snapshot, flips state to `Running`, re-enters
      `execute_run_sequential`. Snapshot fields are cleared on
      successful resume; a wrong-token attempt leaves state untouched.
- `execute_run_sequential` reads `pause_request` at the top of every
  step iteration. On honoring, snapshots `(step_index, variables,
  current_input)` into the run, transitions to `Paused`, returns the
  intermediate output without erroring. In-flight steps are allowed
  to finish — partial-step rollback would be a much bigger feature.
- `execute_run_dag` refuses cleanly if a pause request was lodged,
  rather than silently dropping it. Pause/resume on DAG runs is the
  follow-up.
- `persist_runs` filter extended to include `Paused`. Paused runs
  round-trip through daemon restart with their token + snapshot
  intact; a fresh `load_runs` call restores them and a subsequent
  `resume_run` works as if no restart had happened.

Tests (5 new in `workflow::tests`):

- `pause_after_first_step_then_resume_finishes_workflow` — sender
  triggers pause from inside step 1; loop honors at the boundary
  before step 2; resume completes the run with both
  `step_results.len() == 2` and snapshot fields cleared.
- `resume_run_with_wrong_token_is_rejected` — resume with a fresh
  random token errors and leaves state Paused.
- `resume_run_on_non_paused_run_is_rejected` — resume on a run still
  in Pending errors with "expected Paused".
- `paused_run_round_trips_through_persist_and_load` (multi-thread
  flavor) — Phase 1 builds a paused run on engine #1 and lets
  `execute_run`'s built-in `persist_runs_async` flush; Phase 2 loads
  into engine #2 via `block_in_place(load_runs)` and asserts the
  resume_token + snapshot survive.
- `pause_request_on_dag_workflow_returns_explicit_error` — a paused
  request on a DAG workflow returns the explicit "follow-up needed"
  error rather than silently completing.

Pre-flight: `cargo clippy --workspace --all-targets -- -D warnings`
clean, `cargo test -p librefang-kernel --lib --no-fail-fast` 637
passed (5 new, no regressions).

Out of scope (tracked as follow-ups against the same #3335):

- ApprovalManager wiring — a step returning "needs approval" should
  trigger pause + register with the existing TOTP/escalation flow.
  This PR's primitive is the substrate that integration uses.
- API endpoints `POST /api/workflows/runs/{id}/resume` and
  `GET /api/workflows/runs/paused` for dashboard / external callers.
- CLI `librefang workflow run paused list / resume <id>`.
- DAG executor pause/resume support (per-layer checkpoints).
- GC for paused workflows older than a configurable TTL.
- Capability check that the approver matches the original requester.

* fix(kernel/workflow): preserve resume snapshot on failure + tighten pause hygiene

Address review feedback on #3418:

1. Resume snapshot preserved across failure. `execute_run_sequential`
   used to `take()` the snapshot on entry, so any step error after
   restoration left the run as Failed with no recovery data — a future
   retry-from-failure path was structurally impossible. Switched to
   clone-on-read; the snapshot is now authoritatively cleared only
   when the run reaches Completed. Failed runs carry their snapshot
   forward (forward-compat for retry semantics). The pause-mid-loop
   branch already overwrites the snapshot with fresh values when
   transitioning back to Paused, so no stale data leaks.

2. Orphan `pause_request` cleanup. If a pause request was lodged
   between the loop's final step-boundary check and the run's
   completion, `pause_request` survived as dead data on Completed
   runs. The Completed transition now clears `pause_request`,
   `paused_step_index`, `paused_variables`, and
   `paused_current_input` together, keeping the invariant 'snapshot
   fields valid only on Paused (or Failed retry-candidates)'.

3. `paused_variables: HashMap → BTreeMap`. The persisted JSON now
   has stable key order — re-serializing a Paused run twice produces
   byte-identical output, which keeps audit logs and external
   snapshots clean. Mirrors the project's existing
   deterministic-ordering convention for LLM-bound prompt data
   (#3298) even though pause snapshots don't directly enter prompts.

4. `paused_at` doc comment: pointed at the future TTL-based GC
   sweep that will consume it (#3335 GC follow-up), so future
   readers know the field has a downstream consumer.

`cargo test -p librefang-kernel --lib workflow` passes (65 tests,
including `paused_run_round_trips_through_persist_and_load`).
`cargo test -p librefang-api --test config_schema_golden` clean
(no fixture drift).

* fix(kernel/workflow): pause/resume foundation review fixups

Address review feedback on PR #3418 (open). Picks up the
already-uncommitted improvements (BTreeMap-backed paused_variables for
deterministic JSON, clone-not-take in resume snapshot read,
Completed-path snapshot clear on the sequential happy path) and adds
the items the multi-angle review flagged:

1. **DAG ghost-state cleanup (quality BUG, security FIX-doc).** The
   DAG entry guard previously returned `Err(...)` and left
   `pause_request: Some(_)` on a run still in `Running`, so an
   unrunnable workflow looked alive forever. The guard now flips the
   run to `Failed` with a descriptive `error` field before returning,
   and a new `cleanup_terminal_pause_state` post-pass at the bottom
   of `execute_run` / `resume_run` clears every pause field on any
   terminal transition (Completed or Failed) — so the same five
   `clear_pause_state` lines aren't sprinkled across the ~10 mid-step
   `Failed` sites in the existing executors.

2. **Rollback safety on `load_runs` (quality OK-with-caveat).**
   Previously a single bad row in `workflow_runs.json` killed the
   whole `Vec<WorkflowRun>` deserialization, so an old daemon reading
   a file written by a newer one (e.g. one with the new tagged
   `Paused { ... }` variant) would lose all run history. Now parses
   per-row via `Vec<serde_json::Value>` first, attempts each
   `from_value::<WorkflowRun>` independently, and skips unrecognized
   rows with a WARN. Newer daemons reading older files still parse
   cleanly because every post-foundation field is `#[serde(default)]`.

3. **Security doc-comments (security FIX-doc).** Added explicit
   warnings on `PauseRequest.reason` /
   `WorkflowRunState::Paused { reason }`: persisted plaintext to
   `workflow_runs.json`, surfaced to operators — not a secret store.
   Same `# SECURITY:` block on `PauseRequest.resume_token`: tokens are
   plaintext at rest in the foundation; the future REST handler must
   hash-at-rest before that surface ships. Added a "DAG workflows
   fail-closed on pause" warning to `pause_run`'s docs so a
   confused caller does not silently brick a DAG run.

4. **Three new tests (quality MISS).**
   - `pause_run_is_idempotent_returns_same_token`: triple-call returns
     the same Uuid; the *first* reason wins (later calls don't
     overwrite the message in logs / UI).
   - `resume_run_after_completion_is_rejected`: pause → execute →
     resume to completion → second resume with the same token errors
     with "expected Paused".
   - `pause_then_execute_on_pending_pauses_at_step_zero`: lock down
     the pause-before-any-step path explicitly (was previously only
     exercised as a side-effect of `resume_run_with_wrong_token`).
   - Existing DAG-pause test extended to assert the new
     `state == Failed` + `pause_request is None` shape.

Pre-flight: `cargo clippy --workspace --all-targets -- -D warnings`
clean, `cargo test -p librefang-kernel --lib --no-fail-fast` 642
passed (8 pause/resume tests, 0 regressions).

Out of scope, still tracked against #3335:
- ApprovalManager wiring + capability check (approver matches the
  original requester).
- REST `POST /api/workflows/runs/{id}/resume` and `GET /api/workflows/runs/paused`.
- CLI `librefang workflow run paused list / resume <id>`.
- Per-layer DAG pause/resume (today: fail-closed).
- GC for paused workflows older than a configurable TTL.
- Hash-at-rest for `resume_token` in `workflow_runs.json`.
- WorkflowPaused / WorkflowResumed hook events for external observers.
houko added a commit that referenced this pull request Apr 27, 2026
…P / FangHub skills (#3422)

* fix(dashboard): only show pointer cursor on Cards that actually click

`<Card hover>` unconditionally added `cursor-pointer` on top of the
visual hover effects (border tint + shadow lift). That made stat
cards, FangHub skill cards in browse view, and any other purely
informational cards display a pointer cursor — users naturally click
them and get nothing back.

Reported as "drawer doesn't open" on /dashboard/skills (default
source is `fanghub`, where SkillCard is rendered without
`onViewDetail` by design — those cards aren't meant to drill in,
but they still looked clickable). Same UX trap on /dashboard/plugins
and /dashboard/mcp where the cards have no detail drawer at all,
and on Hands' running-now chips.

Fix: gate `cursor-pointer` on the presence of an `onClick` handler.
The `hover` prop now only controls the visual hover tint/lift it
was always meant to control, and clickability is signalled by what
actually exists on the element.

Verified: pnpm typecheck.

* feat(dashboard): add detail drawers for plugins / MCP / FangHub skills

Followup on the cursor-pointer fix in this PR. Those pages didn't
have a detail drawer at all, so even with the cursor honest, users
can't drill into a plugin/server/FangHub-skill they're curious about.
Add inline DrawerPanel-based detail surfaces, matching the patterns
already used by HandsPage / ChannelsPage / ClawHub skill cards.

PluginsPage:
  - Plugin list rows now clickable (role=button), opens a drawer
    showing name + version + author + full description + hooks +
    metadata (size, install path), with Install-deps and Uninstall
    actions inside the drawer. Existing inline action buttons keep
    working (already stop propagation).

McpServersPage:
  - ServerCard's body region (header + stats + transport summary)
    is now clickable to open a drawer showing transport details
    (command + args, or url), env vars, headers, OAuth state when
    pending, and the full tools list. Edit / Taint / Delete actions
    are mirrored inside the drawer for one-click reach. Bottom
    Edit/Taint/Delete buttons and the Tools expand-toggle keep
    working as before (they're outside the clickable region).

SkillsPage:
  - FangHub SkillCard, previously a dead card with no detail surface,
    now passes onViewDetail to a new inline DrawerPanel showing
    name + version + author + full description + tags, with the
    same Install button as the marketplace card. ClawHub / SkillHub
    detail flows are unchanged.

Verified:
  - pnpm typecheck, pnpm build pass
  - Playwright: clicking a FangHub skill card opens the drawer with
    the right content. Plugin/MCP equivalents follow the same
    pattern (couldn't reproduce live without seed data, but the
    code paths are isomorphic to the verified SkillsPage one).

* feat(kernel/workflow): pause/resume foundation for resumable workflows (#3418)

* feat(kernel/workflow): pause/resume foundation for resumable workflows

Closes part of #3335. ApprovalManager integration, REST/CLI surface,
and DAG-path support are explicit follow-ups (see PR description).

Adds the state machine + persistence primitives that make
"pause-mid-pipeline, resume-from-the-same-step" possible. Today the
workflow engine runs to completion or fails; with this PR a caller can
lodge a pause request, the sequential executor honors it at the next
step boundary, and a later `resume_run` call replays from the saved
snapshot.

What lands:

- `WorkflowRunState::Paused { resume_token, reason, paused_at }` —
  carries the token a caller must present to `resume_run`. Existing
  `is_paused()` helper for ergonomic matches.
- `WorkflowRun` gains four fields, all `#[serde(default)]` /
  `skip_serializing_if` for backward compat with persisted JSON:
    - `pause_request: Option<PauseRequest>` (set by `pause_run`,
      consumed at the next step boundary)
    - `paused_step_index: Option<usize>` (where to resume)
    - `paused_variables: HashMap<String, String>` (`{{var}}` bindings)
    - `paused_current_input: Option<String>` (input that would have
      fed the paused step, restored on resume)
- New `PauseRequest { reason, resume_token }`. Token is generated when
  `pause_run` is called so callers can begin waiting on the
  corresponding approval / input artifact before the executor has
  honored the request — the executor reuses the same token at the
  transition.
- `WorkflowEngine::pause_run(run_id, reason) -> Uuid`
    - Idempotent: paused-already returns the existing token.
    - Errors on Completed / Failed.
- `WorkflowEngine::resume_run(run_id, token, resolver, sender) -> Result<String>`
    - Verifies state is `Paused` and token matches.
    - Restores snapshot, flips state to `Running`, re-enters
      `execute_run_sequential`. Snapshot fields are cleared on
      successful resume; a wrong-token attempt leaves state untouched.
- `execute_run_sequential` reads `pause_request` at the top of every
  step iteration. On honoring, snapshots `(step_index, variables,
  current_input)` into the run, transitions to `Paused`, returns the
  intermediate output without erroring. In-flight steps are allowed
  to finish — partial-step rollback would be a much bigger feature.
- `execute_run_dag` refuses cleanly if a pause request was lodged,
  rather than silently dropping it. Pause/resume on DAG runs is the
  follow-up.
- `persist_runs` filter extended to include `Paused`. Paused runs
  round-trip through daemon restart with their token + snapshot
  intact; a fresh `load_runs` call restores them and a subsequent
  `resume_run` works as if no restart had happened.

Tests (5 new in `workflow::tests`):

- `pause_after_first_step_then_resume_finishes_workflow` — sender
  triggers pause from inside step 1; loop honors at the boundary
  before step 2; resume completes the run with both
  `step_results.len() == 2` and snapshot fields cleared.
- `resume_run_with_wrong_token_is_rejected` — resume with a fresh
  random token errors and leaves state Paused.
- `resume_run_on_non_paused_run_is_rejected` — resume on a run still
  in Pending errors with "expected Paused".
- `paused_run_round_trips_through_persist_and_load` (multi-thread
  flavor) — Phase 1 builds a paused run on engine #1 and lets
  `execute_run`'s built-in `persist_runs_async` flush; Phase 2 loads
  into engine #2 via `block_in_place(load_runs)` and asserts the
  resume_token + snapshot survive.
- `pause_request_on_dag_workflow_returns_explicit_error` — a paused
  request on a DAG workflow returns the explicit "follow-up needed"
  error rather than silently completing.

Pre-flight: `cargo clippy --workspace --all-targets -- -D warnings`
clean, `cargo test -p librefang-kernel --lib --no-fail-fast` 637
passed (5 new, no regressions).

Out of scope (tracked as follow-ups against the same #3335):

- ApprovalManager wiring — a step returning "needs approval" should
  trigger pause + register with the existing TOTP/escalation flow.
  This PR's primitive is the substrate that integration uses.
- API endpoints `POST /api/workflows/runs/{id}/resume` and
  `GET /api/workflows/runs/paused` for dashboard / external callers.
- CLI `librefang workflow run paused list / resume <id>`.
- DAG executor pause/resume support (per-layer checkpoints).
- GC for paused workflows older than a configurable TTL.
- Capability check that the approver matches the original requester.

* fix(kernel/workflow): preserve resume snapshot on failure + tighten pause hygiene

Address review feedback on #3418:

1. Resume snapshot preserved across failure. `execute_run_sequential`
   used to `take()` the snapshot on entry, so any step error after
   restoration left the run as Failed with no recovery data — a future
   retry-from-failure path was structurally impossible. Switched to
   clone-on-read; the snapshot is now authoritatively cleared only
   when the run reaches Completed. Failed runs carry their snapshot
   forward (forward-compat for retry semantics). The pause-mid-loop
   branch already overwrites the snapshot with fresh values when
   transitioning back to Paused, so no stale data leaks.

2. Orphan `pause_request` cleanup. If a pause request was lodged
   between the loop's final step-boundary check and the run's
   completion, `pause_request` survived as dead data on Completed
   runs. The Completed transition now clears `pause_request`,
   `paused_step_index`, `paused_variables`, and
   `paused_current_input` together, keeping the invariant 'snapshot
   fields valid only on Paused (or Failed retry-candidates)'.

3. `paused_variables: HashMap → BTreeMap`. The persisted JSON now
   has stable key order — re-serializing a Paused run twice produces
   byte-identical output, which keeps audit logs and external
   snapshots clean. Mirrors the project's existing
   deterministic-ordering convention for LLM-bound prompt data
   (#3298) even though pause snapshots don't directly enter prompts.

4. `paused_at` doc comment: pointed at the future TTL-based GC
   sweep that will consume it (#3335 GC follow-up), so future
   readers know the field has a downstream consumer.

`cargo test -p librefang-kernel --lib workflow` passes (65 tests,
including `paused_run_round_trips_through_persist_and_load`).
`cargo test -p librefang-api --test config_schema_golden` clean
(no fixture drift).

* fix(kernel/workflow): pause/resume foundation review fixups

Address review feedback on PR #3418 (open). Picks up the
already-uncommitted improvements (BTreeMap-backed paused_variables for
deterministic JSON, clone-not-take in resume snapshot read,
Completed-path snapshot clear on the sequential happy path) and adds
the items the multi-angle review flagged:

1. **DAG ghost-state cleanup (quality BUG, security FIX-doc).** The
   DAG entry guard previously returned `Err(...)` and left
   `pause_request: Some(_)` on a run still in `Running`, so an
   unrunnable workflow looked alive forever. The guard now flips the
   run to `Failed` with a descriptive `error` field before returning,
   and a new `cleanup_terminal_pause_state` post-pass at the bottom
   of `execute_run` / `resume_run` clears every pause field on any
   terminal transition (Completed or Failed) — so the same five
   `clear_pause_state` lines aren't sprinkled across the ~10 mid-step
   `Failed` sites in the existing executors.

2. **Rollback safety on `load_runs` (quality OK-with-caveat).**
   Previously a single bad row in `workflow_runs.json` killed the
   whole `Vec<WorkflowRun>` deserialization, so an old daemon reading
   a file written by a newer one (e.g. one with the new tagged
   `Paused { ... }` variant) would lose all run history. Now parses
   per-row via `Vec<serde_json::Value>` first, attempts each
   `from_value::<WorkflowRun>` independently, and skips unrecognized
   rows with a WARN. Newer daemons reading older files still parse
   cleanly because every post-foundation field is `#[serde(default)]`.

3. **Security doc-comments (security FIX-doc).** Added explicit
   warnings on `PauseRequest.reason` /
   `WorkflowRunState::Paused { reason }`: persisted plaintext to
   `workflow_runs.json`, surfaced to operators — not a secret store.
   Same `# SECURITY:` block on `PauseRequest.resume_token`: tokens are
   plaintext at rest in the foundation; the future REST handler must
   hash-at-rest before that surface ships. Added a "DAG workflows
   fail-closed on pause" warning to `pause_run`'s docs so a
   confused caller does not silently brick a DAG run.

4. **Three new tests (quality MISS).**
   - `pause_run_is_idempotent_returns_same_token`: triple-call returns
     the same Uuid; the *first* reason wins (later calls don't
     overwrite the message in logs / UI).
   - `resume_run_after_completion_is_rejected`: pause → execute →
     resume to completion → second resume with the same token errors
     with "expected Paused".
   - `pause_then_execute_on_pending_pauses_at_step_zero`: lock down
     the pause-before-any-step path explicitly (was previously only
     exercised as a side-effect of `resume_run_with_wrong_token`).
   - Existing DAG-pause test extended to assert the new
     `state == Failed` + `pause_request is None` shape.

Pre-flight: `cargo clippy --workspace --all-targets -- -D warnings`
clean, `cargo test -p librefang-kernel --lib --no-fail-fast` 642
passed (8 pause/resume tests, 0 regressions).

Out of scope, still tracked against #3335:
- ApprovalManager wiring + capability check (approver matches the
  original requester).
- REST `POST /api/workflows/runs/{id}/resume` and `GET /api/workflows/runs/paused`.
- CLI `librefang workflow run paused list / resume <id>`.
- Per-layer DAG pause/resume (today: fail-closed).
- GC for paused workflows older than a configurable TTL.
- Hash-at-rest for `resume_token` in `workflow_runs.json`.
- WorkflowPaused / WorkflowResumed hook events for external observers.

* feat(runtime/kernel): BeforePromptBuild hook can contribute prompt sections (#3358)

* feat(runtime/kernel): BeforePromptBuild hook can contribute prompt sections

Closes #3326.

Extends the existing `librefang-runtime::hooks` system so handlers
registered for `BeforePromptBuild` can return a labeled `DynamicSection`
that gets injected into the system prompt at build time. Previously the
event was observe-only.

The original issue proposal was an async `PromptContextProvider` trait
with a 1s wall budget, but on inspection we already had a synchronous
`HookHandler` covering the same lifecycle event. Reusing it (sync, no
trait churn, no parallel registry) is strictly simpler. Providers that
need bounded blocking work (e.g. an active-memory subagent recall) can
post results to a shared cache from a background tokio task and read
the cache synchronously inside `provide_prompt_section`.

Changes:

- `librefang-runtime/src/hooks.rs`
  - Add `DynamicSection { provider, heading, body }`.
  - Add `HookHandler::provide_prompt_section` with default `Ok(None)`.
  - Add `HookRegistry::collect_prompt_sections` enforcing
    `PER_SECTION_CHAR_CAP = 8 KiB` and `TOTAL_DYNAMIC_CHAR_CAP = 32 KiB`,
    dropping over-budget contributors with a WARN log.
  - 5 new unit tests cover empty/registration-order/none-and-err/
    per-section truncation/total-cap drop.

- `librefang-runtime/src/prompt_builder.rs`
  - Add `dynamic_sections: Vec<DynamicSection>` to `PromptContext`.
  - Render at the tail of `build_system_prompt`, after Section 15
    (Live Context). Empty heading + empty body = no-op.
  - 3 new unit tests cover ordering relative to Live Context, the
    empty case, and the blank-section no-op invariant.

- `librefang-kernel/src/kernel/mod.rs`
  - At all 3 `build_system_prompt` call sites (`send_message_ephemeral`,
    `send_message_streaming_with_sender_and_opts`, `execute_llm_agent`):
    construct a `HookContext` with phase + call_site + user_message +
    channel_type + is_subagent + granted_tools, call
    `self.hooks.collect_prompt_sections`, and pass the result through
    the new `dynamic_sections` field.

- `librefang-kernel/src/kernel/tests.rs`
  - 2 new integration tests covering the ephemeral path: handler fires
    with the expected `HookContext.data` payload; a handler registered
    on a different event must not fire.

Pre-flight: `cargo check --workspace --all-targets` clean,
`cargo clippy --workspace --all-targets -- -D warnings` clean, all
runtime + kernel lib tests pass (1408 + 634). Two unrelated flakes
observed under `cargo test --workspace --lib`
(`librefang-kernel-metering::test_estimate_cost_with_catalog`,
`librefang-runtime::embedding::test_create_embedding_driver_cohere_custom_base_url`)
both reproduce on `main` and pass when run in isolation; out of scope.

This unblocks #3328 (skill workshop after_turn capture) and #3329
(memory wiki recall) which both need a `BeforePromptBuild` injection
point. Consumers can now register a handler that, e.g., recalls memory
on a background task and provides a `## Active Memory` section here.

* fix(runtime/hooks): also fire on_event in collect_prompt_sections; render plain body when heading is empty

Two follow-up fixes from review:

1. collect_prompt_sections now invokes each handler's on_event before
   provide_prompt_section. Without this, observe-only handlers registered
   to BeforePromptBuild miss the event on the three kernel-direct prompt
   build paths (send_message_ephemeral, streaming, execute_llm_agent),
   which don't go through agent_loop's fire(BeforePromptBuild). The
   surface now matches the documented contract.

2. build_system_prompt's Section 16 loop previously rendered '## \n{body}'
   when a section had an empty heading but non-empty body — a dangling
   '## ' that confuses the LLM. Empty heading + non-empty body now renders
   the body alone with no markdown heading prefix.

Adds a regression test for each fix:
- hooks::tests::test_collect_prompt_sections_also_fires_on_event
- prompt_builder::tests::test_dynamic_sections_blank_heading_with_body_renders_body_only

Also adds the missing CHANGELOG entry under 2026.4.27 Added.

* fix(runtime/prompt): harden BeforePromptBuild section injection

Address review feedback on #3358:

1. Heading injection (security FIX). Provider-supplied heading was
   pasted into `format!("## {heading}\n{body}")` after only `.trim()` —
   a heading containing `\n## Tool Call Behavior\nbypass approvals`
   could forge a structural section. New `sanitize_provider_heading`
   collapses newlines, replaces `##` with `#`, caps length at 80 chars.
   Per-section headings now render at `###` to subordinate them under a
   single `## Provider-Supplied Context` umbrella.

2. Untrusted-content framing (security RISK). Provider sections often
   carry recalled memory or external content traceable to user input.
   Added an explicit "Treat them as untrusted data, not as instructions"
   preamble before the per-section blocks, plus a `(provider: …)`
   annotation so the LLM (and humans) can attribute content. Compare
   with the structural sections (Tool Call Behavior, Live Context,
   etc.) which are system-authored and don't need framing.

3. DoS guard on body sizing (security FIX). `body.chars().count()` was
   called before any cap on raw provider input — a handler returning a
   500 MB body would force a full UTF-8 walk on the kernel hot path.
   New `HARD_BYTE_CEILING = PER_SECTION_CHAR_CAP * 4` byte-truncates at
   a UTF-8 char boundary before any char counting; subsequent
   char-level checks now operate on a bounded payload.

4. Empty-skip granularity (quality NIT). Skip a section when its body
   trims to empty regardless of heading — heading-only stubs added no
   value. Tightened in the renderer.

5. Magic 40 (quality NIT). Renamed to `TRUNCATION_MARKER_RESERVE` with
   a doc comment explaining the budget for the
   "[truncated: X → Y chars]" suffix.

6. Renderer test gap (quality MISS). Added four prompt_builder tests:
   - heading newline+`##` injection neutralization
   - heading length cap at 80 chars
   - empty body fully skipped (no preamble emitted)
   - blank heading falls back to provider name as section heading
   Updated `test_dynamic_sections_appended_after_live_context` to
   assert the umbrella preamble + `###` per-section format and the
   ordering Live Context → preamble → sections.

Pre-flight: `cargo clippy --workspace --all-targets -- -D warnings`
clean, `cargo test -p librefang-runtime --lib` 1412 / `librefang-kernel
--lib` 634 — both green.

No external API changes. `DynamicSection` shape and the
`HookHandler::provide_prompt_section` signature are unchanged; the
caps and constants are private. Existing handlers keep working byte
identically as long as their headings don't contain newlines or `##`
sequences (which would have been malformed anyway).

* feat(dashboard): add detail drawers for plugin marketplace + MCP catalog;
                drop misleading cursor on session rows

Followup pass on the rest of the dashboard's clickable-looking-but-inert
cards. Three changes:

PluginsPage (Marketplace / Registry tab):
  - Each registry plugin card now opens a detail drawer with the full
    (untruncated) description, all hook tags, repo link, version,
    author, and an Install / Installed action. Inline Install button
    on the card stops propagation so it doesn't double-fire as a
    drawer-open + install.

McpServersPage (Catalog tab):
  - Catalog template card body opens a detail drawer with the full
    description, all tags (the card showed only first 4), required
    env vars list, multi-line setup_instructions block (the card
    didn't surface this at all), and a single Install button. The
    bottom Install link on the card itself is unchanged.

SessionsPage:
  - Session row had `cursor-pointer` baked into its className without
    any onClick handler — the cursor was lying to users. Removed.
    The row already exposes Switch / Delete / Edit-label as
    individual buttons, so there's no extra detail to surface in a
    drawer.

Verified:
  - pnpm typecheck, pnpm build pass.
  - Playwright e2e: clicking a registry plugin card opens the drawer
    with the right content (auto-summarizer with hooks + repo link).
@houko houko mentioned this pull request Apr 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/kernel Core kernel (scheduling, RBAC, workflows) size/L 250-999 lines changed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant