Skip to content

Preserve cumulative closed-tool resume checkpoints and accounting - #16

Merged
eyuansu62 merged 2 commits into
eyuansu62:mainfrom
adenzhou1350:codex/fix-cumulative-resume
Oct 27, 2026
Merged

eyuansu62 merged 2 commits into
eyuansu62:mainfrom
adenzhou1350:codex/fix-cumulative-resume

Conversation

@adenzhou1350

@adenzhou1350 adenzhou1350 commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #14.

Keep closed-tool details cumulative across subset/empty resumes while computing each invocation's metrics from its current selection. All three evaluators validate policy/provider/budget/harness and sample/corpus fingerprints before dispatch or output mutation. Incompatible or legacy checkpoints require a separate details path or explicit --force; reporting options do not erase failed attempts and their recorded costs.

Compaction writes a sibling temporary file and replaces the checkpoint atomically. Resume can recover an incomplete unterminated final append, but rejects interior corruption. Budget-prefix reports use the latest record per sample and report observed attempt costs separately.

Index accounting reports only selected snapshots, but resolves construction records from stable cumulative checkpoint history, including unselected samples sharing a snapshot. The first available cold build in persisted sample order (with attempt order within a sample) wins; available cached and failed-preparation records remain fallbacks. Snapshot output is sorted, so a completed input reorder or narrower selection cannot change the chosen construction record.

A failed Codex invocation carries its execution record through the error boundary. The action client appends that record before propagating the error; nonzero exits and timeouts therefore retain invocation counts, status/timeout markers, event counts, protocol violations and bounded output excerpts. Successful retries preserve that audit in prior_attempts, without adding it again on completed resumes.

Regressions cover cumulative checkpoint selection, atomic compaction/torn-tail recovery, incompatible inputs, retry-journal accounting, shared-snapshot cold/cached resumes, multiple cold records with reordered inputs, selected-snapshot filtering, and the real evaluator/action-client boundary with injected nonzero/timeout subprocess results, including partial stdout with no output file.

Validation of 5457266a75008dd34c803879575950f6b41e1784 (offline synthetic policies/providers; no paid model calls or published experiment reruns):

  • Isolated Linux/Python 3.12: PYTHONPATH=src python3 -m unittest discover -s tests -q ran 389 tests: 388 passed, 1 skipped. The tested source tree and changed-file hashes match the pushed commit. ResourceWarnings about two existing fixture file handles were observed.
  • Native Windows/Python 3.14: related resume/index modules 42 passed, 45 subtests passed. The new accounting/process-error regressions reproduce 5 failing cases on the previous d98b972 implementation.
  • Windows full-suite attempt: 363 passed, 11 skipped, 14 failed in the shell/path tests in test_trace_repro.py and test_v1_1.py; Windows full-suite success is not claimed. The complete Linux suite above is green.
  • git diff --check passes.

The documentation specifies cumulative-file reporting, construction-record ordering, retained process-failure audits, legacy resets, provider/configuration fingerprints, and single-writer scope. Fingerprints cannot attest remote model weights or undeclared ambient provider state; atomic checkpoint replacement is not a transaction across report files.

@eyuansu62

Copy link
Copy Markdown
Owner

Thanks @adenzhou1350. We reviewed d98b972b4ee6c4aae0d719e08ca239ade16747b3 and independently ran the full suite: 384 passed, 1 skipped. The cumulative checkpoint, selection handling, atomic compaction, torn-tail recovery, and retry-journal deduplication fixes passed our additional offline checks.

Two P2 accounting issues remain. Please fix these and add the regressions below; we will re-review the updated commit afterward. We are keeping this PR unmerged for now.

1. Index construction accounting still changes with the selected samples or their order

Relevant code: evaluate_closed_tool_openai() and index_builds().

We used two synthetic samples, s1 and s2, sharing the same (repo, base_commit), with a stable provider description and unchanged policy settings:

  1. Run only s1: the provider reports an available cold build of 12 seconds.
  2. Resume only s2: the provider reports an available cache hit of 0 seconds.
  3. Both records are correctly preserved in cumulative details, but the s2-only summary reports the cached 0-second record. A subsequent completed resume selecting both samples reports the original 12-second build instead.

In a second fixture, the same snapshot had two recorded cold builds, 12 and 20 seconds. Once both samples were complete, simply reversing their order in the samples input file, with zero provider or policy calls, changed summary.index_builds from 12 seconds to 20 seconds.

The caller passes only the current selection's details, and index_builds() chooses the first qualifying cold record in that selection's order. This leaves the original-construction accounting dependent on which sample happens to be selected first. It is a remaining gap in the cumulative-resume behavior, not a failure of the new available-versus-unavailable preference itself.

Please scope the reported snapshots to the current selection, but resolve their construction records from stable cumulative history (or a persisted construction ledger). Preserve an explicit, stable ordering/accounting rule for multiple cold builds; a no-op resume or input reorder must not change the chosen construction record.

Regression tests should cover:

  • Shared snapshot: cold build on s1 -> cached s2 -> s2-only completed resume -> full resume, retaining the original construction record once.
  • Two recorded cold builds for one snapshot -> completed resume -> reverse input order, with no new dispatch and unchanged construction accounting.
  • Keep the existing failed-prepare -> successful cold retry and cold-build -> failed policy -> cached retry tests passing.

2. A failed Codex process is still missing from the retained execution records

Relevant code: the new failure-recording path, CodexActionClient.create(), and run_codex_exec_action().

We exercised the actual evaluator/client/subprocess path with a local fake Codex executable. It emitted a JSON protocol-violation event and exited with code 3; no model or network call was made. Although one process had run, the retained failed trajectory contained:

{
  "codex_exec_calls": 0,
  "codex_exec_returncodes": [],
  "codex_exec_events": [],
  "protocol_violations": []
}

After a successful retry, prior_attempts still contained those empty fields. The error message includes a truncated output excerpt, but the structured execution and protocol-audit records are lost.

run_codex_exec_action() constructs the record and then raises on a nonzero exit before returning it. CodexActionClient.create() only appends records that were returned successfully, so record_codex_calls() cannot recover the failed record. The added test pre-populates client.call_records with a failed call, which bypasses this actual failure boundary.

Please carry or append the execution record before propagating the process error, including any available partial output for timeouts. Add regressions through the real action-client/process-result path, using a fake executable or injected subprocess results, for nonzero exit and timeout. Verify both the initially retained failure and its prior_attempts after a successful retry: actual invocation count, exit/timeout information, captured event metadata, and detected protocol violations must survive without double-counting on later resumes.

All of this can be validated offline. No rerun of published experiments or paid-policy calls is requested. Please include the focused regression results and the full-suite result with the update.

@adenzhou1350

Copy link
Copy Markdown
Contributor Author

Addressed both P2 findings in 5457266.

  1. Construction accounting: selection now limits the reported snapshot keys, while their records come from cumulative checkpoint history. The stable rule is the first available cold record in persisted sample/attempt order, with cached/failure fallbacks. Regressions cover cold s1 -> cached s2 -> completed s2-only/full resumes, two cold builds with reversed completed inputs and no new dispatch, and exclusion of unselected snapshots. Existing failed-prepare/cold-retry and cold-build/failed-policy/cached-retry tests still pass.
  2. Failed Codex executions: a process error now carries its record to CodexActionClient, which retains it before re-raising. Tests go through the real evaluator/client with injected subprocess results for exit 3, timeout with an output file, and timeout with partial stdout but no output file. Invocation count, exit/timeout information, event count, output excerpts and protocol violations survive the initial failure and prior_attempts after a successful retry; a completed resume adds no calls or history.

Focused Windows results: 42 tests and 45 subtests pass. The new cases reproduce five failures on d98b972. Complete isolated Linux/Python 3.12 suite: 388 passed, 1 skipped (389 tests); source hashes match this commit. The separate Windows full-suite run has 14 shell/path failures, so I am not claiming a green Windows full suite. No published experiment or paid-policy call was rerun. The description and accounting documentation are updated for re-review.

@eyuansu62 eyuansu62 left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved after re-review of 5457266. Both requested P2 fixes are verified; no blocking findings remain from this review.

  • Index construction records are resolved from cumulative history for selected snapshots and remain stable across cached subset resumes and reordered completed inputs, following the documented persisted sample/attempt ordering.
  • Real local fake-executable checks confirm that nonzero exits and timeouts retain invocation counts, exit/timeout information, available output and protocol-audit events. Successful retries preserve the failed record in prior_attempts; later no-op resumes neither duplicate history nor add calls.
  • Regression controls fail on the previous implementation and pass on this head.
  • Full offline unittest suite: 389 tests, 388 passed, 1 skipped. git diff --check passes.

The earlier cumulative-selection, compatibility-check, atomic-compaction, torn-tail and journal-deduplication checks also passed. Validation used offline synthetic policies/providers and local fake executables; no published experiment or paid-policy call was rerun.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Should subset resumes preserve existing closed-tool details?

2 participants