Summary

The workflow engine in crates/librefang-kernel/src/workflow.rs (~5097 LOC) has a competent multi-modal step executor — Sequential / FanOut / Collect / Conditional / Loop, plus DAG via depends_on, pause/resume, retry, persist+recover. The HTTP API, the agent tool surface, the trigger/webhook integration, and a few UI affordances around it have not kept up. The result is that the engine is more capable than the surface that exposes it, and several "this should just be a workflow" use cases hit avoidable walls.

This issue collects the concrete gaps from a read-through of the kernel, the routes, the canvas page, and the runtime tool surface. It's intended as an umbrella; happy to split into one-issue-per-bullet if maintainers prefer.

What's solid today (so we don't re-litigate it)

• Workflow / WorkflowStep data model — crates/librefang-kernel/src/workflow.rs:71, :91
• StepMode: Sequential / FanOut / Collect / Conditional / Loop — workflow.rs:140
• ErrorMode: Fail / Skip / Retry{max_retries} — workflow.rs:157
• DAG via depends_on — workflow.rs:120
• WorkflowRunState::Paused { resume_token, reason, paused_at } + on-disk snapshot of step index, variables, current input — workflow.rs:170, :184
• Persistence + crash recovery via ~/.librefang/workflow_runs.json and recover_stale_running_runs (gated by KernelConfig.workflow_stale_timeout_minutes)
• Per-run sharding through DashMap (refs WorkflowEngine runs RwLock<HashMap> serializes all step writes — O(N×M) lock contention #3717) — workflow.rs:343
• HTTP CRUD and run-history routes — crates/librefang-api/src/routes/workflows.rs:34–:60
• Workflow templates (list / get / instantiate / save-as-template) — routes/workflows.rs:62–:77
• Native tool workflow_run reachable from any agent — crates/librefang-runtime/src/tool_runner.rs:2589, impl at :4212
• Canvas editor (/canvas, crates/librefang-api/dashboard/src/pages/CanvasPage.tsx, ~2600 LOC) — round-trips real step fields (agent_id, agent_name, prompt_template, condition, max_iterations / until, error_mode + max_retries, output_var, depends_on); maps node types to StepMode (agent/respond/channel → Sequential, condition → Conditional, loop → Loop, parallel → FanOut, collect → Collect — CanvasPage.tsx:1564); persists via useCreateWorkflow / useUpdateWorkflow
• Schedule → workflow link via useCreateSchedule({ workflow_id, cron, tz }) — CanvasPage.tsx:2590

Gaps

A. HTTP API surface (crates/librefang-api/src/routes/workflows.rs)

1. No cancel/stop endpoint for a workflow run. POST /api/agents/{id}/stop only stops the active agent loop, not the workflow that dispatched it. A stuck step (e.g. an MCP call hanging on auth) cannot be aborted from the outside. Suggested: POST /api/workflows/runs/{run_id}/cancel.
2. No SSE / WS endpoint for live run progress. The kernel pushes step results into WorkflowRun.step_results between steps, but clients have to poll GET /api/workflows/runs/{run_id} to see them. Mid-step progress (a long-running agent loop inside a step) is invisible. Suggested: GET /api/workflows/runs/{run_id}/stream (SSE) or a WS topic mirroring the existing agent WS shape.
3. Pause / resume exist in the kernel but are not surfaced over HTTP. WorkflowEngine::pause_run (workflow.rs:1214) and resume_run (workflow.rs:1258) work, with full pause-snapshot persistence. The kernel author left an explicit follow-up note (workflow.rs:289–:294) that the REST handler must hash resume tokens at rest before shipping. Today plaintext tokens sit in ~/.librefang/workflow_runs.json next to the run, which blocks exposing the surface as-is. Suggested follow-up: add the token-hashing step, then ship POST /api/workflows/runs/{run_id}/pause and POST /api/workflows/runs/{run_id}/resume.
4. No retry-from-failed-step endpoint. A failed run can only be restarted from step 0 (re-POST /run), even though the engine has every prior step's StepResult on disk.
5. POST /api/workflows/{id}/run is synchronous and unbounded. The HTTP request stays open for the entire workflow duration; with timeout_secs = 120 per step (default) and N steps, requests routinely exceed the idle timeout of any reverse proxy in front of librefang. Suggested: split into POST /run (async, returns {run_id}) + the SSE endpoint above for progress.

B. Trigger / webhook ingress

6. Trigger and CronJob cannot fire a workflow directly. Schedules carry a workflow_id (good), but the older event-trigger model (crates/librefang-kernel/src/triggers.rs, crates/librefang-kernel/src/kernel/cron.rs) does not — rg workflow over those files returns zero hits. Workflows triggered by an event still need a degenerate single-prompt agent that calls workflow_run, which adds an unnecessary indirection and breaks per-run telemetry attribution.
7. No webhook → workflow ingress. routes/webhooks.rs exists but has no workflow target. The canvas exposes a webhook node type (CanvasPage.tsx:195) that has no backend resolver — drawing one does nothing.

C. Executor

8. No workflow-wide span / structured tracing. tracing::info!/debug! are sprinkled through the engine, but there is no dedicated span per workflow run / per step with workflow_id, run_id, step_index. Sub-agent LLM calls inside a step also don't carry a workflow_run_id correlation header, so external observability cannot stitch step-level cost / latency back to the run that produced it. (rg "x-librefang" crates/librefang-kernel/src/workflow.rs returns zero hits.)
9. No workflow-wide timeout. Per-step timeout_secs exists; total-run cap doesn't. A workflow that loops for hours has no overall budget.
10. No backoff on Retry mode. ErrorMode::Retry { max_retries } retries immediately. No jitter or exponential backoff, which makes retry-on-rate-limit unsafe.
11. FanOut concurrency interacts badly with max_concurrent_invocations. Truly-parallel FanOut steps are silently re-serialized at the agent's per-agent semaphore (default 1) when one agent is the target of multiple fanned-out steps. This is correct for safety but invisible from the workflow JSON — the user thinks they're getting parallelism. Worth documenting at minimum, ideally surfaced in the dry-run preview.
12. No step-level idempotency key. Re-running a workflow re-executes side-effecting steps; no dedup or per-step result cache.
13. Run persistence is a single JSON file. ~/.librefang/workflow_runs.json is appended to via copy-and-rewrite under a process-wide mutex (workflow.rs:354). There is no queryable history, no retention policy, no GC of completed or paused runs (the TTL-GC for paused runs is also called out as a follow-up at workflow.rs:188). For long-lived deployments the file grows monotonically. Suggested: move workflow runs to the existing SQLite substrate.
14. Disk-load shape ≠ API shape. .workflow.json files at <librefang_home>/workflows/ use the internal struct shape (nested agent: { name | id }, snake_case mode, prompt_template); POST /api/workflows is parsed by parse_step_mode (routes/workflows.rs:159) and accepts a flatter shape. The same logical workflow has two incompatible JSON encodings depending on how it was loaded. Workflows loaded from disk also need an explicit id pinned or the id rolls on every restart.

D. Step kinds

15. All steps are agent calls. No HTTP-call step, no MCP-call step, no shell-exec step, no wait/timer step, no manual-approval step (despite the Pause infrastructure being almost exactly what an approval step needs), no switch / case branching, no foreach / map-over-list step. The visual canvas exposes a webhook and a channel node type that don't map to runtime concepts (CanvasPage.tsx:194–:196).
16. Variable language is substring-only. evaluate_condition (workflow.rs:367) supports && / || / ! over case-insensitive substring match. There is no JSON path, no numeric comparison, no typed expressions. A condition like error_count > 5 cannot be expressed.
17. No typed I/O. Step inputs and outputs are String. With no JSON schema on workflow input or step output, the canvas cannot validate connections, and downstream steps have to grep for keywords.

E. Agent tool surface (crates/librefang-runtime/src/tool_runner.rs)

The native tool list contains exactly one workflow primitive: workflow_run. That tool is blocking — the calling agent loop sits idle until the workflow completes. Missing primitives that an agent realistically needs:

• workflow_start — fire-and-forget variant returning { run_id }.
• workflow_status / workflow_get_run — poll a run by id.
• workflow_cancel / workflow_pause / workflow_resume.
• workflow_list — agent discovers what's available without hard-coding ids in its prompt.

Sub-workflow composition today only works because a step invokes an agent that happens to call workflow_run, which means nested workflow telemetry is fragmented across an agent boundary.

F. UI (crates/librefang-api/dashboard/src/pages/CanvasPage.tsx)

18. No live run visualization on the canvas. Nodes don't light up as the run progresses; the run is observed in the side panel as a step-result list. Combined with the missing SSE endpoint above, mid-step progress is invisible from the UI.
19. Pause / resume are not reachable from the canvas. Even when the kernel can pause a run, the operator has no UI affordance to do it.
20. No workflow-definition history / diff. Save is in-place; previous shape is gone.
21. crates/librefang-api/dashboard/src/components/WorkflowEditor.tsx (157 LOC) appears to be unused — the live editor is CanvasPage.tsx. Worth deleting to avoid confusing future readers.

Suggested split

If maintainers want this carved into shippable units:

1. Cancel + retry-from-failed-step endpoints (smallest, high payoff).
2. SSE stream for run progress + async POST /run.
3. Token-hashing follow-up on workflow.rs:289 + REST surface for pause / resume.
4. Trigger / CronJob / webhook → workflow direct integration.
5. Step kinds: HTTP, manual-approval (reuses Pause), foreach.
6. Per-run / per-step tracing spans + x-librefang-workflow_run_id propagation on sub-agent calls.
7. Move workflow runs from JSON file to the SQLite substrate.
8. Tool surface: workflow_start, workflow_status, workflow_cancel, workflow_list as native tools.
9. Canvas: live run visualization + pause/resume affordance + delete the unused WorkflowEditor.tsx.

Happy to take a swing at any subset (1, 2, 6, and 8 are the ones I'd most like to land first). Which split, if any, would maintainers prefer?