Skip to content

fix(event_store): treat concurrent eviction as idempotent - #4171

Closed
SarthakB11 wants to merge 1 commit into
PrefectHQ:mainfrom
SarthakB11:fix/issue-4143
Closed

SarthakB11 wants to merge 1 commit into
PrefectHQ:mainfrom
SarthakB11:fix/issue-4143

Conversation

@SarthakB11

Copy link
Copy Markdown
Contributor

Description

EventStore.store_event keeps a per-stream ring buffer by deleting old event keys when the list exceeds max_events_per_stream. The caller treats AsyncKeyValue.delete as idempotent, but some backends (notably py-key-value-aio's FileTreeStore) implement delete as an unguarded exists() + unlink(). When two concurrent store_event calls for the same session (the sse_writer and message_router tasks both run eviction) decide to evict the same event_id, the loser hits FileNotFoundError after the winner has already unlinked the file. That exception propagates out of store_event and produces ERROR-level tracebacks on every collision. The events that did get stored are fine, but on a busy server this is constant log noise that obscures real issues.

This wraps the per-key eviction delete() in try/except FileNotFoundError and logs at DEBUG. Eviction is idempotent by intent: a missing key is the desired post-condition, not an error. Catching FileNotFoundError specifically (rather than a broad Exception) preserves visibility for genuine backend failures like permission errors or Redis disconnects.

if len(event_ids) > self._max_events_per_stream:
    for old_id in event_ids[: -self._max_events_per_stream]:
        try:
            await self._event_store.delete(key=old_id)
        except FileNotFoundError:
            logger.debug("event_store: %s already evicted", old_id)
    event_ids = event_ids[-self._max_events_per_stream :]

The matching FileTreeStore.delete_entry race in py-key-value-aio is tracked separately upstream. The defensive guard here is the right place regardless, since any backend with non-atomic delete semantics can surface the same race.

Closes #4143

Contribution type

  • Bug fix (simple, well-scoped fix for a clearly broken behavior)
  • Documentation improvement
  • Enhancement (maintainers typically implement enhancements — see CONTRIBUTING.md)

Checklist

  • This PR addresses an existing issue (or fixes a self-evident bug)
  • I have read CONTRIBUTING.md
  • I have added tests that cover my changes
  • I have run uv run prek run --all-files and all checks pass
  • I have self-reviewed my changes
  • If I used an LLM, it followed the repo's contributing conventions (not generic output)

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@marvin-context-protocol marvin-context-protocol Bot added bug Something isn't working. Reports of errors, unexpected behavior, or broken functionality. server Related to FastMCP server implementation or server-side functionality. labels May 17, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fb62a14a51

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

await self._event_store.delete(key=old_id)
try:
await self._event_store.delete(key=old_id)
except FileNotFoundError:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reconcile the stream list after eviction conflicts

When this catches FileNotFoundError in the intended concurrent-eviction case, this store_event call has already read an old event_ids snapshot before the other task deleted the same old event and wrote its own new event ID. Continuing on to _stream_store.put(...) with that stale snapshot can overwrite the other successful call's stream list entry, so both store_event calls return successfully but one event is no longer replayable. The previous behavior aborted before writing the stale index; after swallowing the conflict, the code should refetch/merge or otherwise avoid overwriting a concurrent update.

Useful? React with 👍 / 👎.

@marvin-context-protocol

Copy link
Copy Markdown
Contributor

tl;dr: ty check failed in static_analysis — the # type: ignore[method-assign] on the new test is mypy syntax, but FastMCP's type checker is ty. Replace it with # ty: ignore[invalid-assignment] (or annotate flaky_delete to match the real delete signature) and re-push.

Root Cause: At tests/server/test_event_store.py:222, the new test assigns flaky_delete (signature (*args, **kwargs) -> CoroutineType[Any, Any, Unknown]) to event_store._event_store.delete, which ty infers as (key: str, *, collection: str | None = None) -> CoroutineType[Any, Any, bool]. The trailing # type: ignore[method-assign] comment is mypy's error code — ty doesn't honor it, and the actual rule it raises is invalid-assignment. The rest of the pipeline (prettier, ruff check, ruff format, codespell, loq) passed.

Fix: Pick one in tests/server/test_event_store.py:

  1. Swap the comment to ty's syntax — matches the convention already in the repo (see tests/conftest.py:83, tests/client/auth/test_oauth_client.py:230):
    event_store._event_store.delete = flaky_delete  # ty: ignore[invalid-assignment]
  2. Or make the assignment well-typed by matching the real signature (no ignore needed):
    async def flaky_delete(key: str, *, collection: str | None = None) -> bool:
        call_count[\"n\"] += 1
        if call_count[\"n\"] == 1:
            raise FileNotFoundError(2, \"No such file or directory\", \"/tmp/missing.json\")
        return await original_delete(key, collection=collection)
  3. Or use monkeypatch.setattr(event_store._event_store, \"delete\", flaky_delete) (requires adding the monkeypatch fixture), which sidesteps the assignment-typing issue and auto-restores at teardown.

Option 2 is the most type-safe and matches the surrounding code's preference for precise typing over suppressions.

Log excerpt

```
ty check.................................................................Failed
error[invalid-assignment]: Object of type `def flaky_delete(...) -> CoroutineType[Any, Any, Unknown]` is not assignable to attribute `delete` of type `def delete(self, key: str, *, collection: str | None = None) -> CoroutineType[Any, Any, bool]`
--> tests/server/test_event_store.py:222:9
|
220 | return await original_delete(*args, **kwargs)
221 |
222 | event_store._event_store.delete = flaky_delete # type: ignore[method-assign]
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
info: Implicit shadowing of function `delete`, add an annotation to make it explicit if this is intentional
info: rule `invalid-assignment` is enabled by default
Found 1 diagnostic
```

Related files
  • `tests/server/test_event_store.py` (the new `TestEventStoreEvictionRace.test_eviction_swallows_file_not_found` test added in this PR)
  • `tests/conftest.py:83` — example of correct `# ty: ignore[...]` usage already in the repo

@strawgate

Copy link
Copy Markdown
Collaborator

Deleting should not raise exceptions for keys that dont exist. Please follow the contributing guide next time and do not open PRs for issues without discussing the solution first.

@strawgate strawgate closed this May 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working. Reports of errors, unexpected behavior, or broken functionality. server Related to FastMCP server implementation or server-side functionality.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

event_store.store_event eviction can crash on concurrent delete (FileNotFoundError)

2 participants