Repository navigation
fix(capture): an undelivered capture leaves no artifact behind (#737, slice 1) - #907
Conversation
… slice 1) post_tool_capture wrote the full-output artifact before handing the memory to dispatch. When dispatch declined the payload (it returns False on OSError/ValueError/CaptureTransportError) or raised, the artifact stayed on disk with no memory pointing at it: recall cannot surface it and forget cannot reach it. write_artifact now publishes the file exclusively and reports whether this call created it; the hook removes only an artifact it created when the memory was not handed over. A file that already existed is kept, since content addressing may share it with another memory (ADR-0503). Refs #737 Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_018kyZYtm1c2azpFFMFMaRsy
…stats run start The `unit` check of run 42187994087 failed in the mutation gate (ADR-1122): - post_tool_capture.py: "failed to collect stats. runner returned 1". test_cascade_counter_processes starts a `python -S` child from the mutants/ copy, whose mutated modules import mutmut at load time, which a -S child cannot reach. As #889 did for the hook subprocesses, the child drops MUTANT_UNDER_TEST and runs the original checkout; the in-process tests judge the mutants. - artifact_store.py: 6 survivors in write_artifact. Tests now cover the exclusive-write race (another writer publishes first) and the UTC month of the shard. The changed functions of post_tool_capture then had 32 survivors, most in lines the split of process_event moved: the new wiring tests pin what the event hands on (tool name, input, cwd, tags, pointer) and each log line. Five mutants that cannot change behaviour are registered as equivalents. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_018kyZYtm1c2azpFFMFMaRsy
|
CI autopilot : correction du check Les tests passaient (11 851). C'est le gate de mutation (ADR-1122) qui a échoué, sur les deux fichiers du diff.
Reproduit puis vérifié localement : Non vérifié localement : les tests qui exigent PostgreSQL, qui tournent dans le gate CI et pas ici. Le CI de Generated by Claude Code |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Claude finished @cdeust's task in 1m 57s —— View job PR #907 review (1 of at most 3)
No Blockers. The race where two captures of the same output run at once is already listed in the PR body as a known limit, so it is a Note, not a Blocker. Two smaller Notes cover a log assertion that is too loose and the narrow Verdict: approve |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e89a58554d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if not delivered: | ||
| _discard_undelivered_artifact(created) |
There was a problem hiding this comment.
Preserve artifacts when dispatch has already durably spooled the payload
On platforms using the spool fallback, capture_dispatch._spool() writes the payload before calling spawn_drainer(). If spawning raises OSError, dispatch() returns False even though the complete payload remains pending for a later drainer. This cleanup then deletes its artifact, so the recovered memory contains a broken pointer and the full output is lost. Executing the reviewed versions with a failing drainer spawn reproduced one pending payload and zero artifacts. Distinguish failure before durable handoff from failure to launch the drainer, and retain the artifact after the payload has been spooled.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed and fixed in e4245d4. _spool writes the payload before it starts the drainer, so a spawn failure made dispatch return False for a capture that is stored later. The same was true of a lost acknowledgement on the worker path.
dispatch now returns a Handoff: ADMITTED, NOT_HANDED_OVER or UNCERTAIN. capture_transport.send raises CaptureHandoffUncertainError once sending may have started; a refusal stays certain. A drainer that does not start after the payload is spooled is reported as capture_drainer_spawn_failed and the capture counts as admitted. The hook removes the artifact only on NOT_HANDED_OVER, and keeps it when _store_memory raises. Your scenario is now test_a_spooled_capture_whose_drainer_fails_keeps_its_artifact: one pending payload, one artifact.
Generated by Claude Code
| Pre: ``path`` was returned with ``created`` True by this process, and the | ||
| memory that would have referred to it was never handed over (issue #737). | ||
| An already absent file is not an error.""" | ||
| path.unlink(missing_ok=True) |
There was a problem hiding this comment.
Coordinate cleanup with concurrent users of the same artifact
When two overlapping captures have identical output, the first creates the artifact and the second reuses it with created=False. If the second successfully dispatches before the first dispatch fails, this unconditional unlink removes the successful capture's full output. Exclusive publication prevents two creators but does not establish exclusive ownership for deletion. Interleaving these two captures against the reviewed code reproduced a delivered memory with no remaining artifact. Coordinate publication and cleanup with other users of the content-addressed path, or defer removal until outstanding and persisted references can be checked.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Not fixed in this PR. This is the race the PR body lists under "Known limit", and the Claude review raised it as a Note. e4245d4 states it in the write_artifact docstring: created does not make a removal safe on its own.
Closing it needs the referrer count, which only the store can give (artifact_gc.count_artifact_references). The hook does not open the store: that import is what its latency boundary avoids. The sweep of unreferenced artifacts, the next slice listed on #737, removes files only after checking that no memory refers to them, and is meant to replace removal at capture. The race needs two captures of byte-identical large output in the same month, overlapping, with the creator's dispatch failing. I leave the call to the owner: keep this slice with the limit stated, or hold it for the sweep.
Generated by Claude Code
Review 1/3: PR #907 (
|
Review of #907 (Codex, P2): dispatch() returned False for any failure, and the hook took False to mean "not handed over". On the spool path the payload is written before the drainer starts, so a drainer spawn failure returned False while the spooled memory, with its artifact pointer, is stored later: the hook deleted the artifact and left that memory pointing nowhere. A lost acknowledgement on the worker path was the same case. dispatch now returns a Handoff: ADMITTED, NOT_HANDED_OVER or UNCERTAIN. capture_transport.send raises CaptureHandoffUncertainError for a failure after sending may have started; a refusal stays certain. A drainer that does not start after the payload is spooled is reported as capture_drainer_spawn_failed, not as a skipped capture. The hook removes the artifact only on NOT_HANDED_OVER, and keeps it when _store_memory raises. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_018kyZYtm1c2azpFFMFMaRsy
|
Claude finished @cdeust's task in 2m 7s —— View job PR #907 review (2 of at most 3)
No Blockers. Codex's P2 is fixed: when the payload is in the spool but its drainer fails to start,
I did not change any code, and I did not run the test suite. Verdict: approve |
Review 2/3: PR #907 (
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e4245d4369
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if isinstance(exc, CaptureHandoffUncertainError): | ||
| return Handoff.UNCERTAIN | ||
| return Handoff.NOT_HANDED_OVER |
There was a problem hiding this comment.
Make worker refusals prove the payload was not queued
If the worker queues a capture but then fails while preparing or sending its acknowledgement, CaptureServer._receive() catches that failure and attempts to send a refusal without removing the queued payload (capture_server.py:56–67). If that refusal reaches the hook, this branch reports NOT_HANDED_OVER, causing _capture() to delete the artifact even though the worker can still persist its pointer. Exercising the server with a deadline failure immediately after enqueue reproduced a plain CaptureTransportError on the client with the payload still pending. Ensure the server cannot send a refusal after enqueue, or treat that response as uncertain until the protocol guarantees it means no admission.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed and fixed in a52deec. CaptureServer._receive sends a refusal from its except even when the failure (deadline or acknowledgement) comes after self.pending.put, so a refusal does not prove the payload was dropped. send now raises CaptureHandoffUncertainError for a refusal as well. dispatch returns UNCERTAIN and the hook keeps the artifact. On the worker path, only a failure before the first byte (unreachable worker, deadline already past, oversize frame, invalid payload) is NOT_HANDED_OVER now.
The server-side cause, a refusal sent after enqueue that is also counted as capture_skipped, predates this PR and is left out of it. A follow-up can make the server refuse only before put, so that a refusal becomes a proof again.
Generated by Claude Code
|
Claude finished @cdeust's task in 2m 17s —— View job PR #907 review (3 of at most 3)
No Blockers. The review lists three Notes:
Accepted: the whole PR, all 4 commits. No file or commit carries an open Blocker. I did not run the tests or ruff: installing the environment needs approval in this job, so CI will run them. I did not change any code. Verdict: approve |
Review 3/3: PR #907 (
|
…ture-artifact # Conflicts: # memory/mutation-equivalents.json
Summary
First slice of #737 (proposal 2, "no orphan artifacts").
post_tool_capturewrites the full-output artifact before it hands the memory tocapture_dispatch.dispatch. Whendispatchdeclines the payload (it returnsFalseafter anOSError,ValueErrororCaptureTransportError) or raises, the artifact stayed under~/.claude/methodology/artifacts/with no memory pointing at it: recall cannot surface it andforgetcannot reach it.Root cause: the hook did not know whether the memory was handed over, and the artifact store did not say whether the file was new. Fix:
artifact_store.write_artifactpublishes the file exclusively (gated_files.write_text(..., exclusive=True)) and returns(path, created).store_artifactkeeps its signature and delegates to it._store_memoryreturns the result ofdispatch. When the memory is not handed over (declined or raised), the hook removes the artifact only if this capture created it. A file that already existed is kept, because content addressing may share it with another memory (ADR-0503).Refs #737 (slice 1 of several; the remaining slices are listed in the issue comment).
Type of change
Test plan
test_declined_dispatch_removes_the_artifact_it_created,test_failing_dispatch_removes_the_artifact_it_created,test_declined_dispatch_keeps_an_artifact_it_did_not_create(tests_py/hooks/test_post_tool_capture.py);test_write_artifact_reports_whether_it_created_the_file,test_discard_created_artifact_removes_it_and_tolerates_absence(tests_py/infrastructure/test_artifact_store.py). Mutations caught: dropping the cleanup, cleaning up on success, or deleting a pre-existing file.test_artifact_store_failure_falls_back_to_full_contentnow patcheswrite_artifact, the function the hook calls.pytest tests_py/hooks/ tests_py/architecture tests_py/infrastructure/test_artifact_store.py test_private_storage_modes.py test_redaction_gate_storage.py tests_py/handlers/test_forget.py test_redaction_once_per_path.py test_redaction_gate_entry_points.py test_ble001_sweep_handlers.py: 976 passed, 40 skipped, 2 failed. The 2 failures (test_entry_auto_dispatch.py::test_engine_selected_auto_backend_reaches_hook[False-*]) fail identically onorigin/mainwithout this change in this environment.ruff checkandruff format --checkon the 4 changed files: clean.python scripts/check_craftsmanship.py: OK.Audit notes
Coding-standards compliance
process_eventwas split (_capture).post_tool_capture.pywas already over the 300-line cap (baselined); it grows by about 30 lines.Breaking changes
None.
store_artifactkeeps its signature and behaviour.Generated by Claude Code