Repository navigation
Preserve cumulative closed-tool resume checkpoints and accounting - #16
Conversation
|
Thanks @adenzhou1350. We reviewed Two P2 accounting issues remain. Please fix these and add the regressions below; we will re-review the updated commit afterward. We are keeping this PR unmerged for now. 1. Index construction accounting still changes with the selected samples or their orderRelevant code: We used two synthetic samples,
In a second fixture, the same snapshot had two recorded cold builds, 12 and 20 seconds. Once both samples were complete, simply reversing their order in the samples input file, with zero provider or policy calls, changed The caller passes only the current selection's Please scope the reported snapshots to the current selection, but resolve their construction records from stable cumulative history (or a persisted construction ledger). Preserve an explicit, stable ordering/accounting rule for multiple cold builds; a no-op resume or input reorder must not change the chosen construction record. Regression tests should cover:
2. A failed Codex process is still missing from the retained execution recordsRelevant code: the new failure-recording path, We exercised the actual evaluator/client/subprocess path with a local fake Codex executable. It emitted a JSON protocol-violation event and exited with code {
"codex_exec_calls": 0,
"codex_exec_returncodes": [],
"codex_exec_events": [],
"protocol_violations": []
}After a successful retry,
Please carry or append the execution record before propagating the process error, including any available partial output for timeouts. Add regressions through the real action-client/process-result path, using a fake executable or injected subprocess results, for nonzero exit and timeout. Verify both the initially retained failure and its All of this can be validated offline. No rerun of published experiments or paid-policy calls is requested. Please include the focused regression results and the full-suite result with the update. |
Signed-off-by: Xucheng Zhou <[email protected]>
|
Addressed both P2 findings in 5457266.
Focused Windows results: 42 tests and 45 subtests pass. The new cases reproduce five failures on d98b972. Complete isolated Linux/Python 3.12 suite: 388 passed, 1 skipped (389 tests); source hashes match this commit. The separate Windows full-suite run has 14 shell/path failures, so I am not claiming a green Windows full suite. No published experiment or paid-policy call was rerun. The description and accounting documentation are updated for re-review. |
eyuansu62
left a comment
There was a problem hiding this comment.
Approved after re-review of 5457266. Both requested P2 fixes are verified; no blocking findings remain from this review.
- Index construction records are resolved from cumulative history for selected snapshots and remain stable across cached subset resumes and reordered completed inputs, following the documented persisted sample/attempt ordering.
- Real local fake-executable checks confirm that nonzero exits and timeouts retain invocation counts, exit/timeout information, available output and protocol-audit events. Successful retries preserve the failed record in prior_attempts; later no-op resumes neither duplicate history nor add calls.
- Regression controls fail on the previous implementation and pass on this head.
- Full offline unittest suite: 389 tests, 388 passed, 1 skipped. git diff --check passes.
The earlier cumulative-selection, compatibility-check, atomic-compaction, torn-tail and journal-deduplication checks also passed. Validation used offline synthetic policies/providers and local fake executables; no published experiment or paid-policy call was rerun.
Fixes #14.
Keep closed-tool details cumulative across subset/empty resumes while computing each invocation's metrics from its current selection. All three evaluators validate policy/provider/budget/harness and sample/corpus fingerprints before dispatch or output mutation. Incompatible or legacy checkpoints require a separate details path or explicit
--force; reporting options do not erase failed attempts and their recorded costs.Compaction writes a sibling temporary file and replaces the checkpoint atomically. Resume can recover an incomplete unterminated final append, but rejects interior corruption. Budget-prefix reports use the latest record per sample and report observed attempt costs separately.
Index accounting reports only selected snapshots, but resolves construction records from stable cumulative checkpoint history, including unselected samples sharing a snapshot. The first available cold build in persisted sample order (with attempt order within a sample) wins; available cached and failed-preparation records remain fallbacks. Snapshot output is sorted, so a completed input reorder or narrower selection cannot change the chosen construction record.
A failed Codex invocation carries its execution record through the error boundary. The action client appends that record before propagating the error; nonzero exits and timeouts therefore retain invocation counts, status/timeout markers, event counts, protocol violations and bounded output excerpts. Successful retries preserve that audit in
prior_attempts, without adding it again on completed resumes.Regressions cover cumulative checkpoint selection, atomic compaction/torn-tail recovery, incompatible inputs, retry-journal accounting, shared-snapshot cold/cached resumes, multiple cold records with reordered inputs, selected-snapshot filtering, and the real evaluator/action-client boundary with injected nonzero/timeout subprocess results, including partial stdout with no output file.
Validation of
5457266a75008dd34c803879575950f6b41e1784(offline synthetic policies/providers; no paid model calls or published experiment reruns):PYTHONPATH=src python3 -m unittest discover -s tests -qran 389 tests: 388 passed, 1 skipped. The tested source tree and changed-file hashes match the pushed commit. ResourceWarnings about two existing fixture file handles were observed.d98b972implementation.test_trace_repro.pyandtest_v1_1.py; Windows full-suite success is not claimed. The complete Linux suite above is green.git diff --checkpasses.The documentation specifies cumulative-file reporting, construction-record ordering, retained process-failure audits, legacy resets, provider/configuration fingerprints, and single-writer scope. Fingerprints cannot attest remote model weights or undeclared ambient provider state; atomic checkpoint replacement is not a transaction across report files.