Skip to content

Return ETags from write_eval_log - #4713

Merged
ransomr merged 6 commits into
UKGovernmentBEIS:mainfrom
TheDarkniteFalls:codex/inspect-ai-4699-etag
Aug 27, 2026
Merged

ransomr merged 6 commits into
UKGovernmentBEIS:mainfrom
TheDarkniteFalls:codex/inspect-ai-4699-etag

Conversation

@TheDarkniteFalls

@TheDarkniteFalls TheDarkniteFalls commented Jul 31, 2026 •

Copy link
Copy Markdown
Contributor

This PR contains:

  • New features
  • Changes to dev-tools e.g. CI config / github tooling
  • Docs
  • Bug fixes
  • Code refactor

What is the current behavior? (You can also link to an open issue here)

write_eval_log() and write_eval_log_async() write evaluation logs but do not return the ETag of the newly written S3 object. A caller that wants to make a subsequent conditional write must read the object again, creating an avoidable race with another writer.

Fixes #4699.

What is the new behavior?

Both write APIs return a WriteEvalLogResult. Its etag field contains the normalized ETag returned by the exact S3 write, or None for non-S3 storage.

Regular and conditional JSON and Eval log writes return writer-specific ETags without an intervening read. Existing managed streaming and multipart upload behavior is preserved. Tests cover local and S3 storage, regular and conditional writes, stale and chained writes, viewer edits, multipart uploads, and synchronous, asyncio, and Trio paths.

Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)

No intentional breaking change. Existing callers that ignore the return value are unaffected. Code that explicitly asserted these functions returned None will need to accept the result object.

Other information:

Latest maintainer-review follow-up:

  • S3 .eval header-only async rewrites now load the existing object through AsyncFilesystem.read_file(), which selects the active asyncio or Trio backend and preserves missing-object normalization.
  • A dedicated unconditional S3 header-only regression runs under Trio and verifies both persisted header metadata and the returned ETag.
  • The branch is integrated with current upstream main at 9fa31ccbccd0157bab73b1caa8b03c522470f83d.

Validation:

  • uv run make check — passed (Ruff, formatting, and mypy across 1,418 source files)
  • S3 conditional-write and async-files selection — 92 passed, 56 skipped
  • Same selection under Trio — 51 passed, 97 skipped
  • Viewer ETag/edit selection — 18 passed, 108 deselected
  • S3 header-only log-format selection — 3 passed, 17 deselected
  • uv run make test — 9,965 passed, 4,347 skipped, 2 failed, 1,087 warnings
    • The two failures are unchanged file:// path-handling tests triggered by the checkout path containing spaces and are outside this diff.
    • Both tests pass from an otherwise identical no-space temporary copy (2 passed), confirming the failures are checkout-path-only.

Agent review

  • Reviewer: OpenAI Codex (fresh independent contexts), 5 passes
  • Findings: 4 from pre-PR review — all fixed; later post-maintainer-review passes verified the five earlier blockers and the current Trio follow-up, with no additional actionable issues.

AI tooling disclosure: OpenAI Codex assisted with implementation, review, and validation. The final diff and validation results were reviewed before publication.

@ransomr ransomr left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this — the design is sound: capturing the ETag from the exact PutObject/CompleteMultipartUpload response eliminates the HEAD-after-write race the viewer edit path previously had, and the test coverage (chained conditional writes, race simulations, trio variants) is genuinely good.

A handful of items need fixing before merge — see the inline comments. One more is on a line outside the diff:

[blocking] src/inspect_ai/log/_recorders/eval.py:573 — stale docstring on _rewrite_eval_zip_via_filesystem. It still references _write_s3_conditional (renamed _write_s3) and says it serves "the unconditional header_only write path" — after this PR, S3 unconditional header-only writes route through _write_log_s3, so this helper only serves non-S3 remote filesystems (GCS, abfs). Please update it to match.


This review was AI-generated, and reviewed by a maintainer before posting.

Comment thread src/inspect_ai/_util/asyncfiles.py Outdated
from boto3.s3.transfer import TransferConfig, create_transfer_manager

capture = _SyncS3ETagCapture(client)
with create_transfer_manager(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[blocking] create_transfer_manager returns a CRTTransferManager when awscrt is installed and awscrt.s3.is_optimized_for_system() is true (p4d/p5/trn1-class EC2 hosts — exactly the machines large evals run on). The CRT manager serializes requests through the native CRT client and never calls client.put_object/client.complete_multipart_upload, so capture.etag stays None and require_etag() raises RuntimeError("S3 upload completed without returning an ETag") even though the upload succeeded. The pre-PR code used the same manager selection (inside client.upload_fileobj) but didn't need the ETag, so this is a new failure mode: it breaks .eval log finalization on affected hosts. Scope is narrow (this sync helper only runs on the trio backend), but the failure is total when it hits.

Please instantiate the classic manager directly — s3transfer.manager.TransferManager(capture, config) — instead of going through create_transfer_manager, so the capture proxy is always in the request path.

Comment thread src/inspect_ai/log/_recorders/json.py Outdated
else:
with file(location, "wb") as f:
f.write(log_bytes)
etag = await async_fs.write_file(location, log_bytes)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[blocking] Unconditional S3 JSON writes lost their trace_action("Log Write"): previously every standard write was wrapped in the trace action, but this new S3 branch returns before reaching it. Conditional writes are still traced inside _write_s3, but plain S3 JSON writes now emit nothing — please restore the trace for this path.

await cls._write_log_s3_conditional(
) -> str | None:
fs = filesystem(location)
if fs.is_s3() and (if_match_etag is not None or header_only):

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[blocking] Routing unconditional header_only S3 writes through _write_log_s3 changes the error type when the target object doesn't exist: the get_object call (line 473) surfaces a botocore ClientError (NoSuchKey), where the old fsspec path raised FileNotFoundError. asyncfiles._map_missing_s3_object exists for exactly this normalization — please apply it here so callers keep seeing FileNotFoundError.


def require_etag(self) -> str:
if self.etag is None:
raise RuntimeError("S3 upload completed without returning an ETag")

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[non-blocking] This same RuntimeError is raised in three layers: here in require_etag, in eval.py (_write_log_s3), and in json.py (_write_log_impl). The caller-side checks exist only to narrow write_file's str | None return; an S3-specific helper typed to return str would remove both duplicates. Optional cleanup — fine to leave if you prefer.

)
return None

from aioboto3.s3.inject import upload_fileobj

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[blocking] This reaches into aioboto3 private internals (aioboto3.s3.inject.upload_fileobj, called unbound against the proxy) with only a >=13.0.0 floor in requirements.txt, so a future aioboto3 release could change these internals silently. Please avoid the private import if there's a reasonable route through public API. If there isn't, then two things: (1) make sure a test exercises this real code path end-to-end (e.g. against moto, not a monkeypatched upload) so an aioboto3 upgrade breaks loudly rather than silently — if such a test already exists, point at it; and (2) add a brief comment here noting the coupling and what it was verified against.

@TheDarkniteFalls

Copy link
Copy Markdown
Contributor Author

Thanks for the detailed review. I’ve addressed each blocking point and brought the branch up to current main:

  • documented the private aioboto3 upload coupling and version guard;
  • forced the classic s3transfer manager so CRT cannot bypass ETag capture, with Trio/multipart regression coverage;
  • restored the Log Write trace around unconditional JSON S3 writes;
  • preserved FileNotFoundError for missing header-only S3 targets; and
  • corrected the non-S3 filesystem fallback documentation.

I also added focused regressions for the trace and missing-object cases. make check and the focused asyncio/Trio suites pass, and a fresh independent review found no remaining actionable issues.

The full local suite completed with 9,641 passed and 3,873 skipped. Its two failures are unchanged file:// path-handling tests triggered by the checkout path containing spaces and are outside this diff.

The new exact-head Build, Changelog Lint, and Validate Embedded Viewer workflows are currently awaiting maintainer approval to run.

When you have a chance, could you take another look?

@ransomr ransomr left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the thorough round-2 updates. All five blocking items from my earlier review are verified fixed at head 24d779b:

  1. CRT bypass — _s3_upload_fileobj_sync now constructs the classic TransferManager directly (never create_transfer_manager), with a trio-variant test asserting CRT selection cannot occur.
  2. Lost "Log Write" trace on unconditional S3 JSON writes — restored, covered by test_unconditional_s3_json_write_is_traced.
  3. Missing-object error type — _map_missing_s3_object applied, covered by test_s3_eval_header_only_missing_object_raises_file_not_found.
  4. aioboto3 private-API coupling — comment + moto-backed end-to-end tests are in place. I additionally downloaded aioboto3 13.0.0 (the requirements floor) and confirmed aioboto3.s3.inject.upload_fileobj has the same signature and calls self.put_object / self.complete_multipart_upload, so the capture proxy works across the whole supported range (13.0.0 through the 15.5.0 you tested).
  5. Stale _rewrite_eval_zip_via_filesystem docstring — updated.

Verification performed: ran targeted tests in a sandboxed venv (moto server via loopback): tests/log/test_s3_conditional_writes.py + tests/util/test_asyncfiles.py — 91 passed on asyncio, 50 passed with --runtrio (trio variants); tests/_view/test_view_server.py -k "etag or edit" — 17 passed, 1 failure that is environment-only (read-only home dir, unrelated to the diff). ruff check and mypy clean on all changed files.

One remaining blocking item, inline below: the unconditional header-only S3 write path regresses under the trio backend. The fix is one line and also a simplification.

Regression notes I'm accepting as-is (informational, no action needed):

  • Classic TransferManager only on the sync/trio upload path: hosts with awscrt installed lose potential CRT acceleration there. Required for correct ETag capture (CRT bypasses the client proxy) and was requested in the prior review; the default asyncio path never used CRT. Accepted.
  • JSON S3 writes (including periodic fsync=False snapshots) move from fsspec/s3fs to boto3/aioboto3 clients: gains stale-signature retry and the GH-3858 checksum workaround, and matches how the .eval recorder already writes to S3. Users with s3fs-specific (non-AWS-standard) configuration could in principle see credential/endpoint resolution differences, but .eval logs already behave this way. Accepted.
  • Public return type None → WriteEvalLogResult: additive; existing callers that ignore the return are unaffected; CHANGELOG entry present under Unreleased.

This review was AI-generated, and reviewed by a maintainer before posting.

Comment thread src/inspect_ai/log/_recorders/eval.py Outdated
Comment on lines +473 to +477
async with AsyncFilesystem() as async_fs:
s3_client = await async_fs.s3_client_async()
response = await s3_client.get_object(Bucket=bucket, Key=key)
body = await response["Body"].read()
with _map_missing_s3_object(location):
response = await s3_client.get_object(Bucket=bucket, Key=key)
body = await response["Body"].read()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[blocking] Unconditional header-only S3 writes now break under the trio backend.

Behavior that changes: pre-PR, write_eval_log_async(log, s3_loc, header_only=True) with no if_match_etag routed through _rewrite_eval_zip_via_filesystem (sync fsspec), which works under trio. Post-PR it routes through _write_log_s3, whose header-only branch calls async_fs.s3_client_async() + raw get_object; aiobotocore requires an asyncio loop, so under trio this fails with RuntimeError: no running event loop raised from aiobotocore/endpoint.py. Reproduced empirically against moto: the new path fails under trio; calling the old _rewrite_eval_zip_via_filesystem on the same S3 object under trio succeeds.

Who depends: trio-backend users (INSPECT_ASYNC_BACKEND=trio or trio-based tooling) editing .eval headers on S3 via the public write_eval_log_async. The sync write_eval_log forces asyncio and the view server runs asyncio, so both are unaffected — the trigger is narrow, but it silently breaks a previously working public-API path with no migration note.

Suggested fix, which is one line and also a simplification: replace this block with body = await async_fs.read_file(location). AsyncFilesystem.read_file already branches on backend (trio uses the sync client in a worker thread), already wraps with _map_missing_s3_object (so the new FileNotFoundError test still holds), and the change removes this file's import of the private _map_missing_s3_object helper entirely. The subsequent PUT via async_fs.write_file is already trio-safe. Note the conditional if_match_etag path was asyncio-only before this PR and stays that way — pre-existing, not something this PR must fix.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — you are right. The unconditional S3 .eval header-only path was still calling the aioboto3 client directly under Trio.

Fixed in 41d1eea0a by reading the existing object through AsyncFilesystem.read_file(location), so asyncio retains the async client path while Trio uses the supported backend-aware path. I also added test_write_eval_log_async_s3_header_only_supports_trio, which verifies both persisted header metadata and the returned ETag under --runtrio.

The branch is now integrated with current main (9fa31ccbc). uv run make check passes; the relevant default selection is 92 passed / 56 skipped, the Trio selection is 51 passed / 97 skipped, the viewer selection is 18 passed, and the S3 header-only format selection is 3 passed. The full suite completed with 9,965 passed / 4,347 skipped and the same two checkout-path file:// failures; both pass from a no-space temporary copy. A fresh independent review found no actionable issues.

The new exact-head workflows are awaiting maintainer approval to run. When you have a chance, could you take another look?

@TheDarkniteFalls

Copy link
Copy Markdown
Contributor Author

Thanks heaps - I spend a lot of time trying to figure out systems to determine if a conclusion from AI is actually substantiated. All the best!

@ransomr

ransomr commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

@TheDarkniteFalls Thank you for the contribution

ransomr added a commit to meridianlabs-ai/inspect_ai that referenced this pull request Aug 31, 2026
s3_write_file_streaming: combine this branch's close-shielded source
(s3transfer's non-multipart path closes the fileobj) with main's
ETag-returning _s3_upload_fileobj_sync (UKGovernmentBEIS#4713); the streaming S3 test
merges both sides' assertions (source stays open + ETag round-trip).
CHANGELOG entry relocated under Unreleased (conflict sat in the
released 0.3.261 section).

Co-Authored-By: Claude Fable 5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Review upstream #4713: Return ETags from write_eval_log Return ETag from write_eval_log

2 participants