Repository navigation
Return ETags from write_eval_log - #4713
Conversation
ransomr
left a comment
There was a problem hiding this comment.
Thanks for this — the design is sound: capturing the ETag from the exact PutObject/CompleteMultipartUpload response eliminates the HEAD-after-write race the viewer edit path previously had, and the test coverage (chained conditional writes, race simulations, trio variants) is genuinely good.
A handful of items need fixing before merge — see the inline comments. One more is on a line outside the diff:
[blocking] src/inspect_ai/log/_recorders/eval.py:573 — stale docstring on _rewrite_eval_zip_via_filesystem. It still references _write_s3_conditional (renamed _write_s3) and says it serves "the unconditional header_only write path" — after this PR, S3 unconditional header-only writes route through _write_log_s3, so this helper only serves non-S3 remote filesystems (GCS, abfs). Please update it to match.
This review was AI-generated, and reviewed by a maintainer before posting.
| from boto3.s3.transfer import TransferConfig, create_transfer_manager | ||
|
|
||
| capture = _SyncS3ETagCapture(client) | ||
| with create_transfer_manager( |
There was a problem hiding this comment.
[blocking] create_transfer_manager returns a CRTTransferManager when awscrt is installed and awscrt.s3.is_optimized_for_system() is true (p4d/p5/trn1-class EC2 hosts — exactly the machines large evals run on). The CRT manager serializes requests through the native CRT client and never calls client.put_object/client.complete_multipart_upload, so capture.etag stays None and require_etag() raises RuntimeError("S3 upload completed without returning an ETag") even though the upload succeeded. The pre-PR code used the same manager selection (inside client.upload_fileobj) but didn't need the ETag, so this is a new failure mode: it breaks .eval log finalization on affected hosts. Scope is narrow (this sync helper only runs on the trio backend), but the failure is total when it hits.
Please instantiate the classic manager directly — s3transfer.manager.TransferManager(capture, config) — instead of going through create_transfer_manager, so the capture proxy is always in the request path.
| else: | ||
| with file(location, "wb") as f: | ||
| f.write(log_bytes) | ||
| etag = await async_fs.write_file(location, log_bytes) |
There was a problem hiding this comment.
[blocking] Unconditional S3 JSON writes lost their trace_action("Log Write"): previously every standard write was wrapped in the trace action, but this new S3 branch returns before reaching it. Conditional writes are still traced inside _write_s3, but plain S3 JSON writes now emit nothing — please restore the trace for this path.
| await cls._write_log_s3_conditional( | ||
| ) -> str | None: | ||
| fs = filesystem(location) | ||
| if fs.is_s3() and (if_match_etag is not None or header_only): |
There was a problem hiding this comment.
[blocking] Routing unconditional header_only S3 writes through _write_log_s3 changes the error type when the target object doesn't exist: the get_object call (line 473) surfaces a botocore ClientError (NoSuchKey), where the old fsspec path raised FileNotFoundError. asyncfiles._map_missing_s3_object exists for exactly this normalization — please apply it here so callers keep seeing FileNotFoundError.
|
|
||
| def require_etag(self) -> str: | ||
| if self.etag is None: | ||
| raise RuntimeError("S3 upload completed without returning an ETag") |
There was a problem hiding this comment.
[non-blocking] This same RuntimeError is raised in three layers: here in require_etag, in eval.py (_write_log_s3), and in json.py (_write_log_impl). The caller-side checks exist only to narrow write_file's str | None return; an S3-specific helper typed to return str would remove both duplicates. Optional cleanup — fine to leave if you prefer.
| ) | ||
| return None | ||
|
|
||
| from aioboto3.s3.inject import upload_fileobj |
There was a problem hiding this comment.
[blocking] This reaches into aioboto3 private internals (aioboto3.s3.inject.upload_fileobj, called unbound against the proxy) with only a >=13.0.0 floor in requirements.txt, so a future aioboto3 release could change these internals silently. Please avoid the private import if there's a reasonable route through public API. If there isn't, then two things: (1) make sure a test exercises this real code path end-to-end (e.g. against moto, not a monkeypatched upload) so an aioboto3 upgrade breaks loudly rather than silently — if such a test already exists, point at it; and (2) add a brief comment here noting the coupling and what it was verified against.
|
Thanks for the detailed review. I’ve addressed each blocking point and brought the branch up to current main:
I also added focused regressions for the trace and missing-object cases. The full local suite completed with 9,641 passed and 3,873 skipped. Its two failures are unchanged The new exact-head Build, Changelog Lint, and Validate Embedded Viewer workflows are currently awaiting maintainer approval to run. When you have a chance, could you take another look? |
ransomr
left a comment
There was a problem hiding this comment.
Thanks for the thorough round-2 updates. All five blocking items from my earlier review are verified fixed at head 24d779b:
- CRT bypass —
_s3_upload_fileobj_syncnow constructs the classicTransferManagerdirectly (nevercreate_transfer_manager), with a trio-variant test asserting CRT selection cannot occur. - Lost "Log Write" trace on unconditional S3 JSON writes — restored, covered by
test_unconditional_s3_json_write_is_traced. - Missing-object error type —
_map_missing_s3_objectapplied, covered bytest_s3_eval_header_only_missing_object_raises_file_not_found. - aioboto3 private-API coupling — comment + moto-backed end-to-end tests are in place. I additionally downloaded aioboto3 13.0.0 (the requirements floor) and confirmed
aioboto3.s3.inject.upload_fileobjhas the same signature and callsself.put_object/self.complete_multipart_upload, so the capture proxy works across the whole supported range (13.0.0 through the 15.5.0 you tested). - Stale
_rewrite_eval_zip_via_filesystemdocstring — updated.
Verification performed: ran targeted tests in a sandboxed venv (moto server via loopback): tests/log/test_s3_conditional_writes.py + tests/util/test_asyncfiles.py — 91 passed on asyncio, 50 passed with --runtrio (trio variants); tests/_view/test_view_server.py -k "etag or edit" — 17 passed, 1 failure that is environment-only (read-only home dir, unrelated to the diff). ruff check and mypy clean on all changed files.
One remaining blocking item, inline below: the unconditional header-only S3 write path regresses under the trio backend. The fix is one line and also a simplification.
Regression notes I'm accepting as-is (informational, no action needed):
- Classic TransferManager only on the sync/trio upload path: hosts with
awscrtinstalled lose potential CRT acceleration there. Required for correct ETag capture (CRT bypasses the client proxy) and was requested in the prior review; the default asyncio path never used CRT. Accepted. - JSON S3 writes (including periodic
fsync=Falsesnapshots) move from fsspec/s3fs to boto3/aioboto3 clients: gains stale-signature retry and the GH-3858 checksum workaround, and matches how the .eval recorder already writes to S3. Users with s3fs-specific (non-AWS-standard) configuration could in principle see credential/endpoint resolution differences, but .eval logs already behave this way. Accepted. - Public return type
None→WriteEvalLogResult: additive; existing callers that ignore the return are unaffected; CHANGELOG entry present under Unreleased.
This review was AI-generated, and reviewed by a maintainer before posting.
| async with AsyncFilesystem() as async_fs: | ||
| s3_client = await async_fs.s3_client_async() | ||
| response = await s3_client.get_object(Bucket=bucket, Key=key) | ||
| body = await response["Body"].read() | ||
| with _map_missing_s3_object(location): | ||
| response = await s3_client.get_object(Bucket=bucket, Key=key) | ||
| body = await response["Body"].read() |
There was a problem hiding this comment.
[blocking] Unconditional header-only S3 writes now break under the trio backend.
Behavior that changes: pre-PR, write_eval_log_async(log, s3_loc, header_only=True) with no if_match_etag routed through _rewrite_eval_zip_via_filesystem (sync fsspec), which works under trio. Post-PR it routes through _write_log_s3, whose header-only branch calls async_fs.s3_client_async() + raw get_object; aiobotocore requires an asyncio loop, so under trio this fails with RuntimeError: no running event loop raised from aiobotocore/endpoint.py. Reproduced empirically against moto: the new path fails under trio; calling the old _rewrite_eval_zip_via_filesystem on the same S3 object under trio succeeds.
Who depends: trio-backend users (INSPECT_ASYNC_BACKEND=trio or trio-based tooling) editing .eval headers on S3 via the public write_eval_log_async. The sync write_eval_log forces asyncio and the view server runs asyncio, so both are unaffected — the trigger is narrow, but it silently breaks a previously working public-API path with no migration note.
Suggested fix, which is one line and also a simplification: replace this block with body = await async_fs.read_file(location). AsyncFilesystem.read_file already branches on backend (trio uses the sync client in a worker thread), already wraps with _map_missing_s3_object (so the new FileNotFoundError test still holds), and the change removes this file's import of the private _map_missing_s3_object helper entirely. The subsequent PUT via async_fs.write_file is already trio-safe. Note the conditional if_match_etag path was asyncio-only before this PR and stays that way — pre-existing, not something this PR must fix.
There was a problem hiding this comment.
Thanks — you are right. The unconditional S3 .eval header-only path was still calling the aioboto3 client directly under Trio.
Fixed in 41d1eea0a by reading the existing object through AsyncFilesystem.read_file(location), so asyncio retains the async client path while Trio uses the supported backend-aware path. I also added test_write_eval_log_async_s3_header_only_supports_trio, which verifies both persisted header metadata and the returned ETag under --runtrio.
The branch is now integrated with current main (9fa31ccbc). uv run make check passes; the relevant default selection is 92 passed / 56 skipped, the Trio selection is 51 passed / 97 skipped, the viewer selection is 18 passed, and the S3 header-only format selection is 3 passed. The full suite completed with 9,965 passed / 4,347 skipped and the same two checkout-path file:// failures; both pass from a no-space temporary copy. A fresh independent review found no actionable issues.
The new exact-head workflows are awaiting maintainer approval to run. When you have a chance, could you take another look?
… clarify write_eval_log Returns docstring Co-Authored-By: Claude Fable 5 <[email protected]>
|
Thanks heaps - I spend a lot of time trying to figure out systems to determine if a conclusion from AI is actually substantiated. All the best! |
|
@TheDarkniteFalls Thank you for the contribution |
s3_write_file_streaming: combine this branch's close-shielded source (s3transfer's non-multipart path closes the fileobj) with main's ETag-returning _s3_upload_fileobj_sync (UKGovernmentBEIS#4713); the streaming S3 test merges both sides' assertions (source stays open + ETag round-trip). CHANGELOG entry relocated under Unreleased (conflict sat in the released 0.3.261 section). Co-Authored-By: Claude Fable 5 <[email protected]>
This PR contains:
What is the current behavior? (You can also link to an open issue here)
write_eval_log()andwrite_eval_log_async()write evaluation logs but do not return the ETag of the newly written S3 object. A caller that wants to make a subsequent conditional write must read the object again, creating an avoidable race with another writer.Fixes #4699.
What is the new behavior?
Both write APIs return a
WriteEvalLogResult. Itsetagfield contains the normalized ETag returned by the exact S3 write, orNonefor non-S3 storage.Regular and conditional JSON and Eval log writes return writer-specific ETags without an intervening read. Existing managed streaming and multipart upload behavior is preserved. Tests cover local and S3 storage, regular and conditional writes, stale and chained writes, viewer edits, multipart uploads, and synchronous, asyncio, and Trio paths.
Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)
No intentional breaking change. Existing callers that ignore the return value are unaffected. Code that explicitly asserted these functions returned
Nonewill need to accept the result object.Other information:
Latest maintainer-review follow-up:
.evalheader-only async rewrites now load the existing object throughAsyncFilesystem.read_file(), which selects the active asyncio or Trio backend and preserves missing-object normalization.mainat9fa31ccbccd0157bab73b1caa8b03c522470f83d.Validation:
uv run make check— passed (Ruff, formatting, and mypy across 1,418 source files)uv run make test— 9,965 passed, 4,347 skipped, 2 failed, 1,087 warningsfile://path-handling tests triggered by the checkout path containing spaces and are outside this diff.Agent review
AI tooling disclosure: OpenAI Codex assisted with implementation, review, and validation. The final diff and validation results were reviewed before publication.