Skip to content

Commit 3708adf

Browse files
committed
feat(workflows): add continue_on_error step field
Closes github#2591. Adds an optional `continue_on_error: bool` field on every step. When set to `true` and the step fails, the engine records the result (exit_code, stderr, status) into `steps.<id>.output` and continues to the next sibling step instead of halting the run. Downstream `if`, `switch`, or `gate` steps can then branch on `{{ steps.<id>.output.exit_code }}` to route the recovery path. This composes with primitives that already exist (the exit code is already captured, the expression engine already resolves it, and `if`/`switch`/`gate` are already available) — the only gap was that a non-zero exit hard-stopped the pipeline before any downstream step could evaluate it. ### Engine `WorkflowEngine._execute_steps` now consults the step config when a step returns `StepStatus.FAILED`: - Gate aborts (`output.aborted`) always halt the run — operator decisions take precedence over the flag. - Otherwise, if `continue_on_error: true`, log a `step_continue_on_error` event and proceed to the next sibling. - Otherwise, behave as before: set `RunStatus.FAILED` and return. ### Validation `_validate_steps` rejects non-bool values for `continue_on_error`. Coerced strings like `"true"` are not accepted so authoring mistakes surface at validation time rather than silently changing run semantics. ### Default behaviour preserved When `continue_on_error` is omitted, every code path is byte-equivalent to before this change. Existing workflows see no difference. ### Tests New `TestContinueOnError` class in `tests/test_workflows.py` covers all four scenarios from the issue's acceptance criteria plus two extras: - undeclared (default) failure halts the run. - declared-and-fired continues past the failure. - declared-but-step-succeeded is a no-op (flag only matters on FAILED). - if-branch end-to-end exercising the canonical recovery pattern from the issue discussion. - gate abort still halts even with `continue_on_error: true` set. - validation rejects non-bool values; accepts both `true` and `false` cleanly. ### Docs Adds an "Error Handling" section to `workflows/README.md` documenting the field, the gate-abort precedence rule, and the canonical recovery pattern. ### Follow-on Auto-retry-on-transient (e.g. retry a 429 at 3 AM without operator attendance) is intentionally out of scope. The current proposal covers the **skip** and **abort** verdicts from the original discussion; the **retry** verdict still pauses for an operator at the gate step. A future loop/retry-count primitive or an auto-approving gate could close that gap on top of this mechanism without further engine changes.
1 parent 3227b96 commit 3708adf

3 files changed

Lines changed: 330 additions & 4 deletions

File tree

‎src/specify_cli/workflows/engine.py‎

Lines changed: 41 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -231,6 +231,20 @@ def _validate_steps(
231231
step_errors = step_impl.validate(step_config)
232232
errors.extend(step_errors)
233233

234+
# Validate optional `continue_on_error` field. The engine honours
235+
# this on any step that returns FAILED so the pipeline can route
236+
# around the failure via downstream `if`/`switch`/`gate`. The
237+
# field must be a literal boolean — coercion from truthy strings
238+
# is deliberately not supported so authoring mistakes surface
239+
# at validation time rather than silently changing run semantics.
240+
if "continue_on_error" in step_config:
241+
coe = step_config["continue_on_error"]
242+
if not isinstance(coe, bool):
243+
errors.append(
244+
f"Step {step_id!r}: 'continue_on_error' must be a "
245+
f"boolean, got {type(coe).__name__}."
246+
)
247+
234248
# Recursively validate nested steps
235249
for nested_key in ("then", "else", "steps"):
236250
nested = step_config.get(nested_key)
@@ -622,7 +636,10 @@ def _execute_steps(
622636

623637
# Handle failures
624638
if result.status == StepStatus.FAILED:
625-
# Gate abort (output.aborted) maps to ABORTED status
639+
# Gate abort (output.aborted) maps to ABORTED status.
640+
# Aborts are deliberate operator decisions, so
641+
# `continue_on_error` does NOT override them — that flag
642+
# is for transient/expected step failures only.
626643
if result.output.get("aborted"):
627644
state.status = RunStatus.ABORTED
628645
state.append_log(
@@ -631,15 +648,35 @@ def _execute_steps(
631648
"step_id": step_id,
632649
}
633650
)
634-
else:
635-
state.status = RunStatus.FAILED
651+
state.save()
652+
return
653+
654+
# `continue_on_error: true` lets the pipeline route
655+
# around the failure instead of halting. The step
656+
# result (including exit_code, stderr, status) is
657+
# still recorded so downstream `if`/`switch`/`gate`
658+
# steps can branch on it. Log a single, unambiguous
659+
# event per failure resolution — either the run
660+
# continued past it, or it halted.
661+
if step_config.get("continue_on_error"):
636662
state.append_log(
637663
{
638-
"event": "step_failed",
664+
"event": "step_continue_on_error",
639665
"step_id": step_id,
640666
"error": result.error,
641667
}
642668
)
669+
state.save()
670+
continue
671+
672+
state.status = RunStatus.FAILED
673+
state.append_log(
674+
{
675+
"event": "step_failed",
676+
"step_id": step_id,
677+
"error": result.error,
678+
}
679+
)
643680
state.save()
644681
return
645682

‎tests/test_workflows.py‎

Lines changed: 250 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2154,6 +2154,256 @@ def test_while_loop_multi_step_body_inter_step_refs(self, project_dir):
21542154
assert "retry-loop:step-b:2" in state.step_results
21552155

21562156

2157+
# ===== continue_on_error Tests =====
2158+
#
2159+
# Locks the contract documented in workflows/README.md "Error Handling"
2160+
# section: when an executable step fails and `continue_on_error: true`
2161+
# is declared, the engine records the result (exit_code, stderr, status)
2162+
# and continues to the next sibling step instead of halting the run.
2163+
# Gate aborts (`output.aborted`) still halt regardless of the flag.
2164+
2165+
2166+
class TestContinueOnError:
2167+
"""Test the `continue_on_error` step-level field."""
2168+
2169+
def test_undeclared_failure_halts_run(self, project_dir):
2170+
"""Default behaviour (no `continue_on_error`): a failing step
2171+
halts the workflow run with `status == FAILED`.
2172+
2173+
Locks the byte-equivalent default — workflows that do not
2174+
declare the flag must behave exactly as before this feature.
2175+
"""
2176+
from specify_cli.workflows.engine import WorkflowDefinition, WorkflowEngine
2177+
from specify_cli.workflows.base import RunStatus
2178+
2179+
definition = WorkflowDefinition.from_string("""
2180+
schema_version: "1.0"
2181+
workflow:
2182+
id: "halt-on-fail"
2183+
name: "Halt On Fail"
2184+
version: "1.0.0"
2185+
steps:
2186+
- id: fail-step
2187+
type: shell
2188+
run: "exit 7"
2189+
- id: after
2190+
type: shell
2191+
run: "echo should-not-run"
2192+
""")
2193+
engine = WorkflowEngine(project_dir)
2194+
state = engine.execute(definition)
2195+
2196+
assert state.status == RunStatus.FAILED
2197+
assert "fail-step" in state.step_results
2198+
assert state.step_results["fail-step"]["output"]["exit_code"] == 7
2199+
# Subsequent step never executes when the flag is absent.
2200+
assert "after" not in state.step_results
2201+
2202+
def test_declared_and_fired_continues_run(self, project_dir):
2203+
"""`continue_on_error: true` + failing step: the run keeps
2204+
going, the failed step's result is recorded, and the
2205+
downstream step runs.
2206+
"""
2207+
from specify_cli.workflows.engine import WorkflowDefinition, WorkflowEngine
2208+
from specify_cli.workflows.base import RunStatus
2209+
2210+
definition = WorkflowDefinition.from_string("""
2211+
schema_version: "1.0"
2212+
workflow:
2213+
id: "continue-past-fail"
2214+
name: "Continue Past Fail"
2215+
version: "1.0.0"
2216+
steps:
2217+
- id: flaky-step
2218+
type: shell
2219+
run: "exit 42"
2220+
continue_on_error: true
2221+
- id: after
2222+
type: shell
2223+
run: "echo did-run"
2224+
""")
2225+
engine = WorkflowEngine(project_dir)
2226+
state = engine.execute(definition)
2227+
2228+
assert state.status == RunStatus.COMPLETED
2229+
# Failed step's exit_code is preserved so downstream branching
2230+
# can inspect it.
2231+
assert state.step_results["flaky-step"]["output"]["exit_code"] == 42
2232+
assert state.step_results["flaky-step"]["status"] == "failed"
2233+
# Downstream step ran successfully.
2234+
assert state.step_results["after"]["output"]["exit_code"] == 0
2235+
2236+
def test_declared_but_step_succeeded_is_noop(self, project_dir):
2237+
"""`continue_on_error: true` on a step that succeeds is a
2238+
no-op — the flag only changes behaviour on FAILED status.
2239+
"""
2240+
from specify_cli.workflows.engine import WorkflowDefinition, WorkflowEngine
2241+
from specify_cli.workflows.base import RunStatus
2242+
2243+
definition = WorkflowDefinition.from_string("""
2244+
schema_version: "1.0"
2245+
workflow:
2246+
id: "flag-but-success"
2247+
name: "Flag But Success"
2248+
version: "1.0.0"
2249+
steps:
2250+
- id: ok-step
2251+
type: shell
2252+
run: "echo ok"
2253+
continue_on_error: true
2254+
- id: after
2255+
type: shell
2256+
run: "echo done"
2257+
""")
2258+
engine = WorkflowEngine(project_dir)
2259+
state = engine.execute(definition)
2260+
2261+
assert state.status == RunStatus.COMPLETED
2262+
assert state.step_results["ok-step"]["status"] == "completed"
2263+
assert state.step_results["ok-step"]["output"]["exit_code"] == 0
2264+
assert state.step_results["after"]["output"]["exit_code"] == 0
2265+
2266+
def test_if_branch_routes_around_failure(self, project_dir):
2267+
"""End-to-end: `continue_on_error` + `if` cleanly routes around
2268+
a failure. The recovery branch runs; the success branch does
2269+
not.
2270+
2271+
Mirrors the canonical usage pattern from the original feature
2272+
discussion in issue #2591.
2273+
"""
2274+
from specify_cli.workflows.engine import WorkflowDefinition, WorkflowEngine
2275+
from specify_cli.workflows.base import RunStatus
2276+
2277+
definition = WorkflowDefinition.from_string("""
2278+
schema_version: "1.0"
2279+
workflow:
2280+
id: "route-around"
2281+
name: "Route Around Failure"
2282+
version: "1.0.0"
2283+
steps:
2284+
- id: heavy-thing
2285+
type: shell
2286+
run: "exit 1"
2287+
continue_on_error: true
2288+
- id: check-result
2289+
type: if
2290+
condition: "{{ steps.heavy-thing.output.exit_code != 0 }}"
2291+
then:
2292+
- id: recovery
2293+
type: shell
2294+
run: "echo recovery-ran"
2295+
else:
2296+
- id: happy-path
2297+
type: shell
2298+
run: "echo happy-path-ran"
2299+
""")
2300+
engine = WorkflowEngine(project_dir)
2301+
state = engine.execute(definition)
2302+
2303+
assert state.status == RunStatus.COMPLETED
2304+
assert "recovery" in state.step_results
2305+
assert "happy-path" not in state.step_results
2306+
2307+
def test_gate_abort_still_halts_with_continue_on_error(
2308+
self, project_dir, monkeypatch
2309+
):
2310+
"""`continue_on_error` does NOT override a deliberate gate
2311+
abort. `output.aborted` always halts the run with
2312+
`status == ABORTED`.
2313+
2314+
Aborts are explicit operator decisions; continue_on_error
2315+
is for transient/expected step failures only.
2316+
"""
2317+
from specify_cli.workflows.engine import WorkflowDefinition, WorkflowEngine
2318+
from specify_cli.workflows.base import RunStatus
2319+
from specify_cli.workflows.steps.gate import GateStep
2320+
from specify_cli.workflows.steps import gate as gate_module
2321+
2322+
# Force the gate step into interactive mode and feed a "reject"
2323+
# choice so the abort path actually runs in the test env
2324+
# (default behaviour returns PAUSED when stdin is not a TTY).
2325+
monkeypatch.setattr(gate_module.sys.stdin, "isatty", lambda: True)
2326+
monkeypatch.setattr(
2327+
GateStep, "_prompt", staticmethod(lambda _msg, _opts: "reject")
2328+
)
2329+
2330+
definition = WorkflowDefinition.from_string("""
2331+
schema_version: "1.0"
2332+
workflow:
2333+
id: "gate-abort-halts"
2334+
name: "Gate Abort Halts"
2335+
version: "1.0.0"
2336+
steps:
2337+
- id: gate-step
2338+
type: gate
2339+
message: "Approve?"
2340+
options: [approve, reject]
2341+
on_reject: abort
2342+
continue_on_error: true
2343+
- id: should-not-run
2344+
type: shell
2345+
run: "echo nope"
2346+
""")
2347+
engine = WorkflowEngine(project_dir)
2348+
state = engine.execute(definition)
2349+
2350+
assert state.status == RunStatus.ABORTED
2351+
assert "should-not-run" not in state.step_results
2352+
2353+
def test_validation_rejects_non_bool_continue_on_error(self):
2354+
"""`continue_on_error` must be a literal boolean; coerced
2355+
strings like `"true"` are rejected at validation time so
2356+
authoring mistakes surface before execution.
2357+
"""
2358+
from specify_cli.workflows.engine import (
2359+
WorkflowDefinition,
2360+
validate_workflow,
2361+
)
2362+
2363+
definition = WorkflowDefinition.from_string("""
2364+
schema_version: "1.0"
2365+
workflow:
2366+
id: "bad-coe"
2367+
name: "Bad COE"
2368+
version: "1.0.0"
2369+
steps:
2370+
- id: step-one
2371+
type: shell
2372+
run: "true"
2373+
continue_on_error: "true"
2374+
""")
2375+
errors = validate_workflow(definition)
2376+
assert any(
2377+
"continue_on_error" in e and "boolean" in e for e in errors
2378+
), errors
2379+
2380+
def test_validation_accepts_bool_continue_on_error(self):
2381+
"""Boolean values pass validation cleanly."""
2382+
from specify_cli.workflows.engine import (
2383+
WorkflowDefinition,
2384+
validate_workflow,
2385+
)
2386+
2387+
for value in (True, False):
2388+
yaml_value = "true" if value else "false"
2389+
definition = WorkflowDefinition.from_string(f"""
2390+
schema_version: "1.0"
2391+
workflow:
2392+
id: "good-coe"
2393+
name: "Good COE"
2394+
version: "1.0.0"
2395+
steps:
2396+
- id: step-one
2397+
type: shell
2398+
run: "true"
2399+
continue_on_error: {yaml_value}
2400+
""")
2401+
errors = validate_workflow(definition)
2402+
assert not any(
2403+
"continue_on_error" in e for e in errors
2404+
), errors
2405+
2406+
21572407
# ===== State Persistence Tests =====
21582408

21592409
class TestRunState:

‎workflows/README.md‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -219,6 +219,45 @@ Aggregate results from fan-out steps:
219219
output: {}
220220
```
221221

222+
## Error Handling
223+
224+
By default, a non-zero exit code from any step halts the entire run.
225+
Set `continue_on_error: true` on a step to record its result and
226+
continue to the next sibling step instead. The exit code remains
227+
available on `steps.<id>.output.exit_code` so downstream `if`,
228+
`switch`, or `gate` steps can branch on it:
229+
230+
```yaml
231+
- id: heavy-thing
232+
type: command
233+
integration: claude
234+
command: speckit.heavy-thing
235+
continue_on_error: true
236+
237+
- id: check-result
238+
type: if
239+
condition: "{{ steps.heavy-thing.output.exit_code != 0 }}"
240+
then:
241+
- id: review
242+
type: gate
243+
message: "Step failed (exit {{ steps.heavy-thing.output.exit_code }}). Retry or skip?"
244+
on_reject: skip
245+
else:
246+
- id: next-thing
247+
command: speckit.next-thing
248+
```
249+
250+
**Notes:**
251+
252+
- The field must be a literal boolean (`true` / `false`); coerced
253+
strings like `"true"` are rejected at validation time.
254+
- Gate aborts (`on_reject: abort` chosen by the operator) always halt
255+
the run — `continue_on_error` does not override them. The flag is
256+
for transient/expected step failures, not for overriding deliberate
257+
operator decisions.
258+
- When the flag is omitted, behaviour is byte-equivalent to before
259+
this feature.
260+
222261
## Expressions
223262

224263
Workflow definitions use `{{ expression }}` syntax for dynamic values:

0 commit comments

Comments
 (0)