Skip to content

fix(log): make crash-recovered sample.messages deterministic under multi-agent solvers - #4417

Merged
craig-meridian merged 4 commits into
UKGovernmentBEIS:mainfrom
pablopupo:fix/4414-recover-message-determinism
Jul 13, 2026
Merged

craig-meridian merged 4 commits into
UKGovernmentBEIS:mainfrom
pablopupo:fix/4414-recover-message-determinism

Conversation

@pablopupo

Copy link
Copy Markdown
Contributor

Crash recovery reconstructed sample.messages nondeterministically when a solver ran multiple agents concurrently. MessageAccumulator followed whichever agent's ModelEvent happened to fire last, so recovering the same log twice could produce different messages, and none of them necessarily matched the conversation the solver was actually driving.

This pins accumulation to the primary role, the role of the first ModelEvent, which matches the conversation the solver started with, and skips interleaved events from other roles. CompactionEvent carries no role, so it is attributed to the most recent ModelEvent's role.

Added a regression test covering the interleaved multi-agent case, which fails on main and passes with this change, plus a CHANGELOG entry.

make check passes and tests/log/test_recover_reconstruct.py is 22/22.

Fixes #4414 (claimed on the issue earlier this week).

@craig-meridian craig-meridian left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks very much for this contribution! Pinning accumulation to the first ModelEvent's role makes the model-event path deterministic, and the regression tests are clear. One residual issue in the compaction path that I think is worth addressing (or at least documenting) before merge.

The recovery heuristic used the most recent ModelEvent's role to decide
which conversation a CompactionEvent belonged to. When a foreign role's
event landed between a conversation's last message and its own
compaction, that heuristic attributed the compaction to the wrong
conversation and the compaction handling was skipped entirely, dropping
the pre-compaction messages.

CompactionEvent now carries the role of the conversation it compacted,
set at the single emission site in the compaction handler. Recovery
uses that role directly when present and falls back to the previous
last-event heuristic for older logs that predate the field.
craig-meridian added a commit to meridianlabs-ai/ts-mono that referenced this pull request Jul 13, 2026
Regenerate generated.ts from inspect-openapi.json to add the new
optional `role` field on CompactionEvent. Accompanies inspect_ai PR
UKGovernmentBEIS/inspect_ai#4417, which pins crash-recovery message
reconstruction to a compaction's own conversation role.

Co-authored-by: Claude Opus 4.8 <[email protected]>

@craig-meridian craig-meridian left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great, thanks Pablo. I'll take care of merging it from here (coordinating with ts-mono update).

…LOG entry

Bump the ts-mono submodule to main (52a6f58), which includes the
regenerated generated.ts with CompactionEvent.role
(meridianlabs-ai/ts-mono#413), so check-schema-and-types and
submodule-on-main pass. Move the UKGovernmentBEIS#4414 CHANGELOG entry back under
## Unreleased after the merge from main relocated it under 0.3.245.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
@craig-meridian
craig-meridian merged commit eb52027 into UKGovernmentBEIS:main Jul 13, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Crash-recovered sample.messages is nondeterministic under multi-agent solvers (whichever agent's ModelEvent fired last)

3 participants