Repository navigation
fix(log): make crash-recovered sample.messages deterministic under multi-agent solvers - #4417
Merged
craig-meridian merged 4 commits intoJul 13, 2026
Conversation
…lti-agent solvers
craig-meridian
requested changes
Jul 8, 2026
craig-meridian
left a comment
Collaborator
There was a problem hiding this comment.
Thanks very much for this contribution! Pinning accumulation to the first ModelEvent's role makes the model-event path deterministic, and the regression tests are clear. One residual issue in the compaction path that I think is worth addressing (or at least documenting) before merge.
The recovery heuristic used the most recent ModelEvent's role to decide which conversation a CompactionEvent belonged to. When a foreign role's event landed between a conversation's last message and its own compaction, that heuristic attributed the compaction to the wrong conversation and the compaction handling was skipped entirely, dropping the pre-compaction messages. CompactionEvent now carries the role of the conversation it compacted, set at the single emission site in the compaction handler. Recovery uses that role directly when present and falls back to the previous last-event heuristic for older logs that predate the field.
…age-determinism # Conflicts: # CHANGELOG.md
craig-meridian
added a commit
to meridianlabs-ai/ts-mono
that referenced
this pull request
Jul 13, 2026
Regenerate generated.ts from inspect-openapi.json to add the new optional `role` field on CompactionEvent. Accompanies inspect_ai PR UKGovernmentBEIS/inspect_ai#4417, which pins crash-recovery message reconstruction to a compaction's own conversation role. Co-authored-by: Claude Opus 4.8 <[email protected]>
craig-meridian
approved these changes
Jul 13, 2026
Collaborator
There was a problem hiding this comment.
This is great, thanks Pablo. I'll take care of merging it from here (coordinating with ts-mono update).
…LOG entry Bump the ts-mono submodule to main (52a6f58), which includes the regenerated generated.ts with CompactionEvent.role (meridianlabs-ai/ts-mono#413), so check-schema-and-types and submodule-on-main pass. Move the UKGovernmentBEIS#4414 CHANGELOG entry back under ## Unreleased after the merge from main relocated it under 0.3.245. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Crash recovery reconstructed
sample.messagesnondeterministically when a solver ran multiple agents concurrently.MessageAccumulatorfollowed whichever agent'sModelEventhappened to fire last, so recovering the same log twice could produce different messages, and none of them necessarily matched the conversation the solver was actually driving.This pins accumulation to the primary role, the role of the first
ModelEvent, which matches the conversation the solver started with, and skips interleaved events from other roles.CompactionEventcarries no role, so it is attributed to the most recentModelEvent's role.Added a regression test covering the interleaved multi-agent case, which fails on main and passes with this change, plus a CHANGELOG entry.
make checkpasses andtests/log/test_recover_reconstruct.pyis 22/22.Fixes #4414 (claimed on the issue earlier this week).