Repository navigation
Python: FHA LRA workflow checkpoint gap investigation #7809
Description
In FHA LRA with MAF workflows, there are a few known gaps that require more in-depth investigation:
- A workflow that doesn't produce any output until the very end will never be recover from a crash, because a responses checkpoint can only be persisted when the workflow yields output. Upon recovery, the workflow will be restarted from the very beginning, which is not desired.
- There is always a gap between when the workflow stores its checkpoint and when the responses checkpoint is persisted. Say a superstep that yields two outputs, the first output gets to the client while the second output is still on the wire, the connection drops (client never gets the second output) and the workflow moves the next superstep after storing a checkpoint, and now the process crashes, the checkpoint gets stored but is never persisted with the response stream so upon recovery the workflow resumes from the checkpoint, which will not rerun the previous superstep. The client will lose the second update forever.
Activity
[-]FHA LRA workflow checkpoint gap investigation[/-][+]Python: FHA LRA workflow checkpoint gap investigation[/+]Handle updates through one tracker entry point and pair snapshots wit…
Handle updates through one tracker entry point and pair snapshots wit…
Python: mark hosted /responses incomplete on content_filter and lengt…
Python: mark hosted /responses incomplete on content_filter and lengt…
eavanvalkenburg commentedon Nov 3, 2026
/plan Plan a bounded resolution for the remaining legacy
agent=workflow.as_agent()recovery gap, separately from nativeworkflow=hosting.Internal validation against current
mainb2c4de3 changes the original investigation:first-2andsecond-2, advanced its core checkpoint, encountered one injected persistence failure, and was interrupted in the next superstep. Recovery returnedstatus=completedwith onlydone; both earlier outputs were missing. Without that injected failure, recovery replayed the unpaired superstep and returned all three outputs.The likely boundary is
_responses.py:2308-2319: with no persisted_last_checkpoint_id, legacy recovery chooses the latest workflow checkpoint, although that checkpoint can be newer than durably stored response output. Existing production-time stamping protects an established pairing but cannot recover a pairing that never persisted. Preserve the now-working silent-prefix case when addressing this.Validation: seven selected maintained tests passed. A session-only six-case characterization matrix had five passing cases and this one failing legacy case, reproduced again with a fresh response-store instance on restart. These were local real-SDK/ASGI shutdown-and-restart tests with file persistence and injected failure, not a process-SIGKILL, network-disconnect, live Foundry, or exactly-once external-tool test. No repository files were changed.
Plan focused regressions for missing first pairing, an existing older pairing, two outputs per superstep, silent prefixes, and native-path parity. Determine the smallest compatible legacy repair or an explicitly agreed fail-closed/migration boundary; the legacy wrapper is deprecated but remains available in this beta. Do not redesign native hosting or promise exactly-once external effects. Scope and compatibility are not yet clear enough for
ready-for-implementation.eavanvalkenburg commentedon Nov 9, 2026
Python hosting release-scope clarification
Future checkpoint/output contract redesign and expanded recovery guarantees are excluded from the current RC/stable plan and will likely coincide with Foundry Hosting 2.0. Keep them out of the mandatory API-freeze investigation queue.
Preserve the distinction already recorded between working native/silent-prefix paths and the independently tracked legacy pairing-persistence gap. A compatible fix for current supported behavior remains valid work; this deferral neither declares that gap fixed nor requires redesigning future checkpoints before a scoped correction can proceed.