Skip to content

Make the conversation-log shrink rewrite atomic (prevent history loss on a mid-write crash) - #70

Merged
rohitprasad15 merged 1 commit into
andrewyng:mainfrom
lifrary:fix/atomic-conversation-shrink
Aug 30, 2026
Merged

rohitprasad15 merged 1 commit into
andrewyng:mainfrom
lifrary:fix/atomic-conversation-shrink

Conversation

@lifrary

@lifrary lifrary commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Problem

ConversationStore.save() appends new messages on the common path, but when a turn reduces the message count (context compaction / summarization) it rewrites the whole .jsonl in place:

elif len(record.messages) < existing:  # rare; not append-only
    with open(self._file(sid), "w", encoding="utf-8") as f:
        for m in record.messages:
            f.write(json.dumps(m) + "\n")

open(..., "w") truncates the file at open(), so a crash (or disk-full) partway through the rewrite leaves a truncated or empty log — permanently losing conversation history, the most valuable data the app stores. The append path is safe; only this shrink branch is non-atomic.

This is the write-side companion to #56 (tolerate a corrupt line on read). #56's per-line skip can't recover a whole-file truncation from this path — the messages are simply gone — so preventing the truncation closes the remaining gap. (#56's own note already observes that an interrupted mid-write corrupts the file.)

Fix

Write the reduced log to a temp file and Path.replace() it in one atomic step — the same tmp-then-replace pattern subscriptions.ChannelBuffer._save() already uses in this repo. os.replace is atomic on POSIX and Windows when the temp file is on the same filesystem (it is — same directory), so a concurrent reader or a crash sees either the old complete log or the new one, never a partial.

Kept intentionally minimal: the atomic rename is what fixes the truncation. I left out an fsync (power-loss durability) to match the existing ChannelBuffer idiom — happy to add it if you'd prefer.

Tests

New tests/test_conversation_atomicity.py:

  • test_shrink_rewrite_preserves_history_when_write_crashes — persists a 5-message history, then simulates a crash partway through a shrink rewrite, and asserts the history survives.
    • Before the fix: FAILED — AssertionError: assert 1 == 5 (history truncated to a single message).
    • After the fix: passes.
  • test_shrink_rewrite_persists_reduced_history — the normal (non-crash) shrink still writes exactly the reduced set, with no leftover temp file.

The surrounding conversation tests (autotitle, durable-resume, session-persona, attachments, memory) still pass.

This is a backend fix with no UI surface, so the "before / after" is the test result above (red FAILED assert 1 == 5 → green 2 passed) rather than a screenshot — glad to attach a terminal capture if you'd like one.

ConversationStore.save() appends new messages on the common path, but when a
turn reduces the message count (context compaction / summarization) it rewrites
the whole .jsonl with open(..., "w"), which truncates the file at open(). A crash
mid-rewrite then leaves a truncated or empty log, permanently losing the
conversation history.

Write the reduced log to a temp file and replace() it in one atomic step -- the
same tmp-then-replace pattern subscriptions.ChannelBuffer._save() already uses.

Add tests/test_conversation_atomicity.py covering the crash path (history
preserved when the write fails partway) and the happy path (reduced set
persisted, no leftover temp file).

@rajpratham1 rajpratham1 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This pull request improves the durability of conversation persistence by replacing the in-place rewrite during conversation shrink operations with an atomic temporary-file-and-replace workflow. This prevents conversation history loss if a crash occurs while rewriting the log. The accompanying tests verify both the crash scenario, ensuring the original history remains intact when the rewrite fails before replacement, and the successful shrink path, confirming the rewritten log contains the expected reduced history without leaving temporary files behind. The implementation is focused, follows an established atomic write pattern, and no blocking issues are visible in the current diff.

@rohitprasad15
rohitprasad15 self-requested a review August 29, 2026 23:59

@rohitprasad15 rohitprasad15 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for handling this case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants