Skip to content

impl/continuation#265

Merged
mbakgun merged 22 commits into
mainfrom
impl/continuation
Jun 28, 2026
Merged

impl/continuation#265
mbakgun merged 22 commits into
mainfrom
impl/continuation

Conversation

@mbakgun

@mbakgun mbakgun commented Jun 28, 2026

Copy link
Copy Markdown
Contributor
  • Implement crash recovery features and update documentation

If the server restarts while a workflow is running, the run used to get stuck as "running" and was eventually dropped with no record. Now those runs are picked up automatically and re-run from scratch with the same inputs, so they finish and show up in history.

How it works

Each running execution now stores what it needs to replay (inputs, trigger source, actor).
On startup the leader process finds interrupted runs and re-runs them in parallel. Recovery is instant on a graceful restart, or within about 60s after a hard crash.
A run is retried once. If it fails again, or the workflow was deleted, it is recorded as failed.
A per-workflow "Auto-recover runs" toggle (on by default) lives in the Properties panel when no node is selected. Turn it off to mark interrupted runs as skipped instead of re-running them.
Works the same on a single container and on multi-worker deployments. Only the leader recovers, so a run is never recovered twice.

UI

History shows skipped and failed status badges, plus a "Recovered" marker (icon in the list, full label on the detail panel).
Screenshot 2026-06-28 at 6 00 44 PM

Notes

Sub-workflows are not recovered on their own, the parent re-drives them.
Triggers with their own redelivery (RabbitMQ, cron, IMAP) can run a recovered job twice. This was a known, accepted tradeoff.

Screenshot 2026-06-28 at 6 01 11 PM Screenshot 2026-06-28 at 6 08 36 PM

mbakgun added 22 commits June 28, 2026 16:19
@mbakgun mbakgun enabled auto-merge (squash) June 28, 2026 16:28
@mbakgun mbakgun merged commit 7ac4c44 into main Jun 28, 2026
2 checks passed
@mbakgun mbakgun deleted the impl/continuation branch June 28, 2026 16:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant