Skip to content

fix(memory): reject a second base-table pair in one AdvancedSQLiteSession database - #4441

Closed
ayaangazali wants to merge 4 commits into
openai:mainfrom
ayaangazali:fix/advanced-sqlite-structure-table-scope
Closed

ayaangazali wants to merge 4 commits into
openai:mainfrom
ayaangazali:fix/advanced-sqlite-structure-table-scope

Conversation

@ayaangazali

@ayaangazali ayaangazali commented Aug 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

AdvancedSQLiteSession inherits the configurable sessions_table / messages_table from SQLiteSession, but its four structure tables (message_structure, turn_usage, branch_reservations, session_clear_generations) are hardcoded names created with CREATE TABLE IF NOT EXISTS (advanced_sqlite_session.py:150, :170). The first base-table pair to touch a database file therefore owns them, and every later session on that file silently reuses message_structure with the first pair's foreign keys regardless of how it was configured.

get_items() joins message_structure against the configured messages table (advanced_sqlite_session.py:279, :297, :318), so once two pairs share one file each session's join picks up structure rows written by the other one:

a = AdvancedSQLiteSession(session_id="s1", db_path=p, create_tables=True,
                          sessions_table="a_sessions", messages_table="a_messages")
b = AdvancedSQLiteSession(session_id="s1", db_path=p, create_tables=True,
                          sessions_table="b_sessions", messages_table="b_messages")

await a.add_items([{"role": "user", "content": "from A"}])
await b.add_items([{"role": "user", "content": "from B"}])

await a.get_items()  # [{... 'from A'}, {... 'from A'}]  <- only one item was added
await b.get_items()  # [{... 'from B'}, {... 'from B'}]

Two message_structure rows now exist for session s1, both with message_id = 1, and each session matches both of them against its own messages table. Nothing errors, the history is just wrong. turn_usage collides the same way through its UNIQUE(session_id, branch_id, user_turn_number) constraint.

Namespacing the structure tables per base-table pair would mean renaming them in over 90 places in the module and would orphan the rows in every database file that already exists, so this takes the other direction: the layout really does only support one pair per file, so __init__ now checks that instead of corrupting data. It reads PRAGMA foreign_key_list(message_structure) and raises a ValueError naming both pairs when the structure tables already belong to a different one.

Supported configurations are untouched, since the check only fires on a mismatch that is already broken today. Verified by hand: default table names with two sessions on one file, a custom pair on its own file, reopening that file with create_tables=True and with create_tables=False, a fresh file with no structure tables yet, and :memory:.

Test plan

  • Added test_structure_tables_reject_a_second_base_table_pair. Reverting the source change makes it fail with DID NOT RAISE <class 'ValueError'>, so it covers the fix rather than restating it.
  • uv run pytest tests/extensions/memory/ tests/memory/: 613 passed, 11 skipped.
  • Ran .agents/skills/code-change-verification/scripts/run.sh: format and lint clean, pyright 0 errors, 8587 passed / 28 skipped.
  • CI here waits on a maintainer to approve the workflow for a first-time contributor, so the checks above are the local run, not a green CI badge.

Issue number

N/A, found while reading through the session backends.

Checks

  • I've added new tests, if relevant
  • I've run .agents/skills/code-change-verification/scripts/run.sh
  • I've confirmed all verification steps pass
  • If using Codex, I've run /review before submitting this PR

sorry in advance if I got any of this wrong. I worked through the logic myself and used Claude Code as a sounding board for the design call (validate vs rename the tables), and I confirmed the duplication with the snippet above before writing anything. happy to be corrected on the approach, still a freshman in college trying to be useful here :)

…sion file

The structure tables are hardcoded names created with CREATE TABLE IF NOT
EXISTS, so the first sessions_table/messages_table pair to touch a database
file owns them. A later session configured with different base table names
reused message_structure and joined it against its own messages table, so
both sessions read back duplicated history.

Validate the scope in __init__ instead: read the existing message_structure
foreign keys and raise when they belong to a different pair.
Copilot AI lite review requested due to automatic review settings August 15, 2026 18:12

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b6f2b1a92f

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

…eck both

Address the automated review on this PR.

SQLite resolves table identifiers case-insensitively while PRAGMA
foreign_key_list reports the CREATE-time spelling, so reopening a file with a
recased base-table pair raised ValueError for a configuration that is in fact
the same pair.

Verification now covers turn_usage as well as message_structure, and runs
inside the BEGIN IMMEDIATE transaction that creates the layout, so a file whose
structure tables were created for two different pairs is rejected instead of
being accepted by whichever table happens to match.
@ayaangazali

Copy link
Copy Markdown
Contributor Author

Both Codex points were real. Fixed in f68b409.

Identifier casing. This was a regression my patch introduced. SQLite resolves table names case-insensitively, but PRAGMA foreign_key_list reports the CREATE-time spelling, so reopening a file as foosessions after creating it as FooSessions hit the mismatch branch and raised on a configuration that is the same pair. Comparison is case-folded now, and test_structure_tables_accept_equivalent_identifier_casing covers it. That test fails with ValueError against the previous commit.

Split layout. Verification now checks turn_usage as well as message_structure, so a file whose two structure tables were created for different pairs is rejected instead of being accepted by whichever one happens to match. It also runs inside the transaction that creates the layout: _init_structure_tables now opens with BEGIN IMMEDIATE, which takes the database write lock before the first CREATE, and Python's sqlite3 keeps DDL inside the open transaction, so create-and-verify commits as one unit and two processes cannot each create half the layout for a different pair.

Checked by hand on top of the existing test: recased pair accepted with correct history, a file where only turn_usage carries the owner rejected through the turn_usage check, and the original two-pair case still rejected with the first session's history intact.

.agents/skills/code-change-verification/scripts/run.sh passes: format, lint, typecheck and the full test run.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f68b40940c

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@seratch seratch left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution. The underlying issue is real: in the released implementation, two custom base-table pairs sharing one database can silently duplicate or mix persisted history. Enforcing one pair per database is the right direction and is preferable to renaming every structure table.

Before another review, please reset the validation around one contract: create_tables=True must atomically create and claim a complete structure-table layout, while create_tables=False must accept only an already-existing complete layout owned by the configured pair. Please delegate identifier equality to SQLite's ASCII-only NOCASE semantics instead of Python casefold().

Please add regression tests for a no-create instance opened before ownership exists, ASCII recasing versus distinct non-ASCII identifiers, and controlled concurrent initialization with final foreign-key coherence. The current head still reproduces silent duplicated history in the first two cases.

…onfigured pair

Address review feedback.

Validation is now one contract. create_tables=True creates and claims the
layout inside a single BEGIN IMMEDIATE transaction, and create_tables=False
accepts only an already-existing layout owned by the configured pair. An
incomplete or unclaimed layout is rejected, so a no-create session can no
longer open a file before any pair owns it and then have another pair claim
the structure tables underneath it.

Identifier equality is delegated to SQLite's ASCII-only NOCASE collation
instead of Python casefold(), which equated names SQLite keeps distinct such
as 'sssessions' and the sharp-s spelling.
@ayaangazali

Copy link
Copy Markdown
Contributor Author

Thanks, that framing helped. Reset around the single contract in 6517db9.

One contract. _claim_structure_tables is now the only validation and both constructor paths run it:

  • create_tables=True creates and claims inside one BEGIN IMMEDIATE transaction, so the layout is created, checked and committed as a unit.
  • create_tables=False accepts only an already-existing complete layout owned by the configured pair. An incomplete or unclaimed layout is rejected, which closes the case you flagged: a no-create session could previously open a file before any pair owned it and then have another pair claim the structure tables underneath it.

Both owner-bearing tables, message_structure and turn_usage, must be present and agree. branch_reservations and session_clear_generations carry no foreign keys, but once one pair owns the file they cannot be shared across pairs either.

Identifier equality. Now delegated to SQLite via SELECT ? = ? COLLATE NOCASE rather than Python casefold(). You were right that this mattered: casefold() equates ßsessions and sssessions, which SQLite keeps as two distinct tables, so the second pair got through and the corruption came back.

Tests, all three cases:

  • test_no_create_session_rejects_a_database_without_an_owner
  • test_structure_tables_accept_equivalent_identifier_casing for ASCII recasing, and test_structure_tables_reject_distinct_non_ascii_identifiers for the non-ASCII pair, which asserts the casefold() collision up front so the reason the test exists is visible
  • test_concurrent_structure_table_claims_leave_one_coherent_owner, two processes claiming a fresh file with different pairs, asserting exactly one claim and that both owner-bearing tables end up naming that winner

Four of the five fail against main. The fifth is the ASCII recasing test, which guards against over-rejection rather than against main.

One note on how that concurrency test is written, since it looks over-specified otherwise. It pre-creates the file with WAL enabled before the race. Switching journal mode needs an exclusive lock, so two processes opening a brand-new file collide in SQLiteSession.__init__ at sqlite_session.py:107 (PRAGMA journal_mode=WAL) before either reaches the structure tables. On main, with the same base-table pair and none of this validation involved, that reproduces 6 out of 6 times with OperationalError: database is locked. That is pre-existing and outside this change, so I settled WAL up front to keep the test about the claim. Happy to open a separate issue for it if it is worth tracking.

Behavior change worth confirming: AdvancedSQLiteSession(session_id=..., create_tables=False) against a database with no structure tables now raises at construction instead of at the first add_items(). On main that configuration already fails, with OperationalError: no such table: message_structure, and every usage in docs/sessions/ and examples/memory/ passes create_tables=True, so I do not think anything working breaks. Say the word if you would rather it stayed lazy.

.agents/skills/code-change-verification/scripts/run.sh passes.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6517db95f5

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +74 to +75
with self._locked_connection() as conn:
self._claim_structure_tables(conn)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Validate ownership before creating the rejected base pair

When an existing file is owned by pair A and this constructor is called with pair B, super().__init__() has already created and committed B's sessions table, messages table, and index before this validation raises. The rejected construction therefore leaves persistent schema artifacts behind, potentially occupying names or conflicting with later migrations even though pair B was never accepted; preflight ownership before base-schema initialization or otherwise ensure the rejected initialization is rolled back.

AGENTS.md reference: AGENTS.md:L98-L98

Useful? React with 👍 / 👎.

Address the automated review on this PR.

Rejecting a second base-table pair happens after SQLiteSession.__init__ has
already created that pair's sessions table, messages table and index, so a
refused construction left those artifacts in a file it was never allowed to
use. Snapshot sqlite_master before the base schema is created and, when the
claim is rejected, drop only the names this construction added.
@ayaangazali

Copy link
Copy Markdown
Contributor Author

Fair point, and my change is what made that state reachable, so it should clean up after itself. Fixed in the latest commit.

I took the rollback option rather than preflighting before super().__init__(). Preflighting needs the effective sessions_table/messages_table before the base class has resolved them, which would mean duplicating its default names here as a second source of truth. Instead __init__ snapshots sqlite_master before the base schema is created, and a rejected claim drops only the names that snapshot did not contain. A pair that already owned tables in the file keeps them.

test_rejected_construction_leaves_no_schema_behind asserts the file's schema is byte-for-byte the same set before and after a rejected construction, and that the owner's history still reads correctly. It fails against the previous commit.

.agents/skills/code-change-verification/scripts/run.sh passes.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5151119c1f

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

self.messages_table,
self.sessions_table,
]
added = [name for name in candidates if name not in preexisting]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve recased preexisting tables during cleanup

When one configured table name is a case-only variant of the existing owner and the other name is different (for example, existing FooSessions/FooMessages versus configured foosessions/other_messages), ownership validation correctly rejects the pair, but this exact, case-sensitive membership check classifies foosessions as newly created. Cleanup then executes DROP TABLE foosessions, which SQLite resolves to the preexisting FooSessions table, deleting the accepted owner's schema and data. Although validation now follows SQLite identifier casing, the new rollback cleanup does not; reject before base-schema creation or use SQLite-equivalent identity when determining what this construction actually created.

AGENTS.md reference: AGENTS.md:L98-L98

Useful? React with 👍 / 👎.

Comment on lines +203 to +204
kind = "INDEX" if name.startswith("idx_") else "TABLE"
conn.execute(f"DROP {kind} IF EXISTS {name}")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve object types when cleaning custom tables

When a rejected configuration uses a valid custom sessions_table or messages_table name beginning with idx_, this prefix test treats that table as an index. DROP INDEX IF EXISTS then fails because the schema object is a table; the outer exception handler suppresses the cleanup error, leaving the rejected pair's base tables behind. Store each candidate together with its known schema-object type rather than inferring the type from its user-controlled name.

Useful? React with 👍 / 👎.

@seratch

seratch commented Aug 16, 2026

Copy link
Copy Markdown
Member

Resolved by #4454, which includes you as a co-author.

@seratch seratch closed this Aug 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants