Skip to content

feat: add OpenCode SQLite session database support#23

Open
JakobSachs wants to merge 2 commits intoMemPalace:developfrom
JakobSachs:feat/opencode-support
Open

feat: add OpenCode SQLite session database support#23
JakobSachs wants to merge 2 commits intoMemPalace:developfrom
JakobSachs:feat/opencode-support

Conversation

@JakobSachs
Copy link
Copy Markdown

@JakobSachs JakobSachs commented Apr 7, 2026

Played around with the project a little, but also wanted to carry over my Opencode sessions. The patch should hopefully be as minimal as possible.

Had to do this tmp-dir workaround to get the same directory-tree behaviour as with the other tools, trying not to touch too much of the existing logic. So it uses the dir-column in the DB to create a tmp-dir structure.
The table:

   session table (SQLite)
   ┌──────────┬─────────────────────┐
   │ id       │ directory           │
   ├──────────┼─────────────────────┤
   │ ses_a1b2 │ /home/user/frontend │
   │ ses_c3d4 │ /home/user/frontend │
   │ ses_e5f6 │ /home/user/backend  │
   │ ses_g7h8 │ /home/user/my-infra │
   └──────────┴─────────────────────┘

becomes

mempalace_oc_*/
├── frontend/
│   ├── frontend_ses_a1b2.txt
│   └── frontend_ses_c3d4.txt
├── backend/
│   └── backend_ses_e5f6.txt
└── my_infra/
    └── my_infra_ses_g7h8.txt

which is then handled the same way how claude-code/chatgpt/etc. ingestion works.


Ingest should work via:

mempalace mine ~/.local/share/opencode/opencode.db --mode convos

Tests + Lint should be clean

@thecodenomad
Copy link
Copy Markdown

Just curious, what version of opencode are you running? For me, this change couldn't find anything with opencode v1.3.10

@JakobSachs
Copy link
Copy Markdown
Author

Just curious, what version of opencode are you running? For me, this change couldn't find anything with opencode v1.3.10

Using v1.3.13 on macOS, (mind you im not sure where opencode saves the db on other platforms..)

Did you get any error or just no-output/nothing found ?

IgorTavcar added a commit to IgorTavcar/mempalace that referenced this pull request Apr 7, 2026
Cherry-picked from upstream PR MemPalace#23 (JakobSachs).
- Support mining OpenCode CLI sessions from their SQLite DB
- New normalize_opencode_sessions() in normalize.py
- Usage: `mempalace mine ~/.../opencode.db --mode convos`
- Reuses existing conversation mining pipeline
- 30 new tests

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>
@thecodenomad
Copy link
Copy Markdown

Just curious, what version of opencode are you running? For me, this change couldn't find anything with opencode v1.3.10

Using v1.3.13 on macOS, (mind you im not sure where opencode saves the db on other platforms..)

Did you get any error or just no-output/nothing found ?

No errors, it just didn't find any data, or at least it reported it found 0 files.

I'll update just to be sure and give it another go.

NOTE: On Linux I do see the db at ~/.local/share/opencode/opencode.db

.local/share/opencode
❯ ls
auth.json  log          opencode.db-shm  snapshot  tool-output
bin        opencode.db  opencode.db-wal  storage

@myInstagramAlternative
Copy link
Copy Markdown

opencode db path

@bensig
Copy link
Copy Markdown
Collaborator

bensig commented Apr 7, 2026

Cool feature but conflicts with main. Can you rebase if you'd like to continue? We'd be interested in OpenCode support.

@JakobSachs JakobSachs force-pushed the feat/opencode-support branch from 74e9c01 to 91a3c5e Compare April 7, 2026 21:26
@JakobSachs
Copy link
Copy Markdown
Author

JakobSachs commented Apr 7, 2026

Cool feature but conflicts with main. Can you rebase if you'd like to continue? We'd be interested in OpenCode support.

done, rebased 👍

@JakobSachs
Copy link
Copy Markdown
Author

JakobSachs commented Apr 7, 2026

Just curious, what version of opencode are you running? For me, this change couldn't find anything with opencode v1.3.10

Using v1.3.13 on macOS, (mind you im not sure where opencode saves the db on other platforms..)
Did you get any error or just no-output/nothing found ?

No errors, it just didn't find any data, or at least it reported it found 0 files.

I'll update just to be sure and give it another go.

NOTE: On Linux I do see the db at ~/.local/share/opencode/opencode.db

.local/share/opencode
❯ ls
auth.json  log          opencode.db-shm  snapshot  tool-output
bin        opencode.db  opencode.db-wal  storage

Actually, found the issue with this: For some reason on my one Mac OpenCode has a different db path, on my private one it also uses the XDG data dir path (like on linux).

pushed a small fix to search both places so we catch all of it now 👍

@thecodenomad
Copy link
Copy Markdown

thecodenomad commented Apr 7, 2026

Just curious, what version of opencode are you running? For me, this change couldn't find anything with opencode v1.3.10

Using v1.3.13 on macOS, (mind you im not sure where opencode saves the db on other platforms..)
Did you get any error or just no-output/nothing found ?

No errors, it just didn't find any data, or at least it reported it found 0 files.
I'll update just to be sure and give it another go.
NOTE: On Linux I do see the db at ~/.local/share/opencode/opencode.db

.local/share/opencode
❯ ls
auth.json  log          opencode.db-shm  snapshot  tool-output
bin        opencode.db  opencode.db-wal  storage

Actually, found the issue with this: For some reason on my one Mac OpenCode has a different db path, on my private one it also uses the XDG data dir path (like on linux).

pushed a small fix to search both places so we catch all of it now 👍

Thank you! Just tried again, it worked for me! I did noticed that there was an unrelated failure:

Failed to send telemetry event CollectionGetEvent: capture() takes 1 positional argument but 3 were given

This appears to be due to a breaking change with posthog, but pinning it posthog = ">=2.4.0,<6.0.0" fixes it *(Opened #163 for that issue)

Nice work!

mining opencode convo

=======================================================
  MemPalace Mine — Conversations
=======================================================
  Wing:    step1
  Source:  /tmp/mempalace_oc_eszqwao7/step1
  Files:   4
  Palace:  /var/home/user/.local/distrobox_homes/cppdev/.mempalace/palace
-------------------------------------------------------

Before establishing client
Before client.get_collection
  ✓ [   1/4] step1_ses_330da6dbcffed2cb9JowsSAxJK.txt           +29
  ✓ [   2/4] step1_ses_335b2b72bffeS76Nj6gG2702Kw.txt           +11
  ✓ [   3/4] step1_ses_335bdf907ffeYLloh40zvEWBIl.txt           +4
  ✓ [   4/4] step1_ses_335dc12b1ffeZ1KP1hY8UDtjfv.txt           +8

=======================================================
  Done.
  Files processed: 4
  Files skipped (already filed): 0
  Drawers filed: 52

  By room:
    technical            3 files
    decisions            1 files

  Next: mempalace search "what you're looking for"
=======================================================

@bgauryy
Copy link
Copy Markdown

bgauryy commented Apr 8, 2026

PR Review: feat: add OpenCode SQLite session database support

Executive Summary

Aspect Value
PR Goal Add OpenCode SQLite session database as a 6th supported conversation format for mining
Files Changed 7
Risk Level 🟡 MEDIUM - New I/O path with SQLite, has a Python 3.9 compat crash and a resource leak
Review Effort 3/5 - Moderate: new data source integration, SQL queries, temp-dir recursion pattern
Recommendation 🔄 REQUEST_CHANGES

Affected Areas: mempalace/normalize.py (core), mempalace/convo_miner.py (integration), tests/test_normalize.py, mempalace/cli.py, READMEs, uv.lock

Business Impact: Enables OpenCode users to mine their AI coding sessions into MemPalace — expands supported tool ecosystem.

Flow Changes: Adds an early-return path in mine_convos() for .db files that bypasses the normal scan_convos()normalize() loop, creating temp directories grouped by project before recursing back into mine_convos().

Ratings

Aspect Score
Correctness 3/5
Security 5/5
Performance 4/5
Maintainability 3/5

PR Health

  • Has clear description
  • References ticket/issue (not applicable — community feature contribution)
  • Appropriate size (201 additions is reasonable)
  • Has relevant tests (single happy-path test only)

High Priority Issues

🐛 #1: str | None union syntax crashes on Python 3.9

Location: mempalace/normalize.py_resolve_opencode_db() and normalize_opencode_sessions() | Confidence: ✅ HIGH

The project declares requires-python = ">=3.9" in pyproject.toml and lists Python 3.9 in classifiers. The str | None union syntax for type hints is only available at runtime from Python 3.10+. On Python 3.9 this raises TypeError: unsupported operand type(s) for |: 'type' and 'NoneType' at import time, breaking the entire normalize module.

- def _resolve_opencode_db(db_path: str | None = None) -> str:
+ def _resolve_opencode_db(db_path: Optional[str] = None) -> str:
- def normalize_opencode_sessions(db_path: str | None = None) -> List[dict]:
+ def normalize_opencode_sessions(db_path: Optional[str] = None) -> List[dict]:

Optional is already imported in the file — this is a straightforward fix.


🐛 #2: SQLite connection leak on exception in main processing loop

Location: mempalace/normalize.pynormalize_opencode_sessions() | Confidence: ✅ HIGH

The SQLite connection is opened with conn = sqlite3.connect(resolved) and manually closed in two specific error branches (missing tables, no json_extract). However, if the main session-processing loop throws any exception (e.g., malformed JSON in json_extract, unexpected None in unpacking), conn.close() is never called. This leaks the connection and can leave the database file locked.

- def normalize_opencode_sessions(db_path: Optional[str] = None) -> List[dict]:
-     resolved = _resolve_opencode_db(db_path)
-     try:
-         conn = sqlite3.connect(resolved)
-         tables = { ... }
-     except Exception as e:
-         raise IOError(...)
-
-     if not {"session", "message", "part"}.issubset(tables):
-         conn.close()
-         raise IOError(...)
-     try:
-         conn.execute("SELECT json_extract('{}', '$')")
-     except Exception:
-         conn.close()
-         raise IOError(...)
-     sessions = conn.execute(...).fetchall()
-     results = []
-     for sid, title, directory, created in sessions:
-         ...  # ← exception here = connection leaked
-     conn.close()
-     return results
+ def normalize_opencode_sessions(db_path: Optional[str] = None) -> List[dict]:
+     resolved = _resolve_opencode_db(db_path)
+     try:
+         conn = sqlite3.connect(resolved)
+     except Exception as e:
+         raise IOError(f"Could not read SQLite database: {resolved}: {e}")
+
+     try:
+         tables = {
+             r[0] for r in conn.execute(
+                 "SELECT name FROM sqlite_master WHERE type='table'"
+             ).fetchall()
+         }
+         if not {"session", "message", "part"}.issubset(tables):
+             raise IOError(f"Not a recognized OpenCode database (missing tables): {resolved}")
+         conn.execute("SELECT json_extract('{}', '$')")
+
+         sessions = conn.execute(...).fetchall()
+         results = []
+         for sid, title, directory, created in sessions:
+             ...
+         return results
+     except IOError:
+         raise
+     except Exception as e:
+         raise IOError(f"Error reading OpenCode database: {resolved}: {e}")
+     finally:
+         conn.close()

Medium Priority Issues

🔄 #3: Dual processing path — .db files behave differently when found via directory scan

Location: mempalace/convo_miner.py:276 + mempalace/normalize.py:44 | Confidence: ✅ HIGH

Adding .db/.sqlite3/.sqlite to CONVO_EXTENSIONS means scan_convos() will discover these files during directory walks. However, the two discovery paths produce very different results:

Path Trigger Behavior
Direct .db file mempalace mine opencode.db --mode convos Early return → per-project temp dirs → grouped mining
.db in directory mempalace mine ~/chats/ --mode convos scan_convos() picks it up → normalize()_normalize_opencode_sqlite() → ALL sessions concatenated into one blob → single wing/room

The directory-scan path loses project grouping and dumps all sessions into one flat transcript. Two options:

Option A — Remove .db/.sqlite3/.sqlite from CONVO_EXTENSIONS so only explicit file paths trigger OpenCode mining:

  CONVO_EXTENSIONS = {
      ".txt",
      ".md",
      ".json",
      ".jsonl",
-     ".db",
-     ".sqlite3",
-     ".sqlite",
  }

Option B — Handle .db files specially in the file-processing loop (check inside the for i, filepath in enumerate(files, 1) loop and delegate to the same temp-dir approach).

Option A is simpler and avoids surprising behavior. Users would need to point directly at the .db file, which is clearer.


🚨 #4: _extract_opencode_messages type hint mismatch

Location: mempalace/normalize.py_extract_opencode_messages() | Confidence: ⚠️ MED

The parameter session_id: str = None declares type str but defaults to None. Should be Optional[str] = None for consistency with the type system and to avoid mypy/pyright warnings.

- def _extract_opencode_messages(
-     conn: sqlite3.Connection, session_id: str = None
- ) -> List[Tuple[str, str]]:
+ def _extract_opencode_messages(
+     conn: sqlite3.Connection, session_id: Optional[str] = None
+ ) -> List[Tuple[str, str]]:

🚨 #5: Error message uses input db_path instead of resolved path

Location: mempalace/normalize.pynormalize_opencode_sessions() | Confidence: ✅ HIGH

When the database is missing required tables, the error message prints the original db_path parameter (which could be None if called without arguments) instead of the resolved path:

- raise IOError(f"Not a recognized OpenCode database (missing tables): {db_path}")
+ raise IOError(f"Not a recognized OpenCode database (missing tables): {resolved}")

Low Priority Issues

🎨 #6: Missing space after # in comment

Location: mempalace/normalize.py — above OPENCODE_DB_PATHS | Confidence: ✅ HIGH

PEP 8 requires a space after # in inline comments.

- #List of known opencode db paths
+ # List of known opencode db paths

🎨 #7: Test coverage is thin — only happy path

Location: tests/test_normalize.py | Confidence: ⚠️ MED

The single test test_opencode_sqlite covers only the happy path (valid DB with one session). Missing test scenarios:

  • Database without required tables (should raise IOError)
  • Empty sessions (< 2 messages, should be skipped)
  • Multiple sessions across different project directories
  • The convo_miner.py temp-dir recursive mining flow
  • json_extract unavailability error path
  • Tool-call filtering (messages starting with "Called the ")

Not blocking, but the lack of error-path coverage means the resource leak (#2) could go unnoticed.


Flow Impact Analysis

User passes .db directly          User passes directory containing .db
          │                                       │
          ▼                                       ▼
  mine_convos()                            mine_convos()
  convo_path.is_file() ─── True            convo_path.is_file() ─── False
          │                                       │
          ▼                                       ▼
  normalize_opencode_sessions()             scan_convos() finds .db
  returns List[dict] per session            adds to files list
          │                                       │
          ▼                                       ▼
  Write to temp dirs by project             normalize() per file
  ┌── project_a/                                  │
  │   ├── session1.txt                            ▼
  │   └── session2.txt              _normalize_opencode_sqlite()
  └── project_b/                    concatenates ALL sessions
      └── session3.txt              into ONE transcript blob
          │                                       │
          ▼                                       ▼
  Recursive mine_convos()            chunk_exchanges() on blob
  per project subdir                 → single wing, single room
  → proper wing per project          → project grouping LOST

Created by Octocode MCP https://octocode.ai 🔍🐙

@bensig
Copy link
Copy Markdown
Collaborator

bensig commented Apr 8, 2026

@JakobSachs pls see robot review above

Copy link
Copy Markdown

@web3guru888 web3guru888 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of #23feat: add OpenCode SQLite session database support

Scope: +201/−7 · 7 file(s)

  • README.md (modified: +2/−2)
  • mempalace/README.md (modified: +1/−1)
  • mempalace/cli.py (modified: +1/−1)
  • mempalace/convo_miner.py (modified: +34/−1)
  • mempalace/normalize.py (modified: +123/−2)
  • tests/test_normalize.py (modified: +30/−0)
  • uv.lock (modified: +10/−0)

Issues

  • ⚠️ Hardcoded filesystem path — breaks portability

Suggestions

  • Magic number(s) 1000 — consider extracting to named constant(s)

Strengths

  • ✅ Includes test coverage

🟡 Minor items — good work overall, a few things to address.


🏛️ Reviewed by MemPalace-AGI · Autonomous research system with perfect memory · Showcase: Truth Palace of Atlantis

@bensig bensig changed the base branch from main to develop April 11, 2026 22:23
@igorls igorls added enhancement New feature or request area/mining File and conversation mining labels Apr 14, 2026
bensig added a commit that referenced this pull request Apr 18, 2026
Draft plugin specification for source adapters, mirroring RFC 001's
role for storage backends. Formalizes the contract six community
ingester PRs (#274, #23, #169, #232, #567, #98, #702) plus #981's
metadata-only mode have been reinventing ad-hoc, so adapter authors
can build to a stable surface.

Key decisions:
- Single ingest() method; lazy adapters yield SourceItemMetadata
  ahead of drawers, eager adapters interleave
- Declared-transformation model (§1.4) replaces informal verbatim
  promise with a verifiable one; byte_preserving adapters declare
  the empty set, declared_lossy adapters enumerate. Existing
  miner.py and the convo_miner+normalize pipeline map cleanly
- Palace is the incremental cursor via is_current(item, metadata);
  no sidecar persistence
- Routing is adapter-owned; detect_room/detect_hall move into the
  filesystem adapter
- Flat metadata per ChromaDB (RFC 001 §1.4) — entity hints as
  json_string field, KG triples route to SQLite knowledge graph
- Closets stay core-built as a post-step; adapters may emit flat
  closet_hints. Closes existing gap where convo drawers get no
  closets
- No per-drawer field renames: source_file, filed_at, source_mtime,
  added_by, normalize_version, entities, ingest_mode all preserved.
  Spec adds adapter_name, adapter_version, privacy_class

§9 enumerates the cleanup PR prerequisites (mempalace/sources/
module, PalaceContext facade, KnowledgeGraph.add_triple gaining
backwards-compatible source_drawer_id + adapter_name params).

Tracking issue: #989
jphein pushed a commit to jphein/mempalace that referenced this pull request Apr 30, 2026
…Code, MemPalace#274/MemPalace#232 Cursor, MemPalace#169 Pi, MemPalace#702 Cursor+factory.ai)

Updates the multi-agent-support bullet to cite the actual upstream
work instead of just gesturing at it. RFC 002 itself is PR MemPalace#990
(tracking issue MemPalace#989). Existing third-party prototypes already
proposed against the spec:

* OpenCode SQLite — PR MemPalace#23
* Cursor SQLite — issue MemPalace#274
* Cursor JSONL (earlier variant) — PR MemPalace#232
* Pi agent JSONL — PR MemPalace#169
* Combined Cursor + factory.ai — PR MemPalace#702

Each becomes a mempalace-source-<agent> package once RFC 002 lands.
Names the path explicitly: fork unblocks the pattern by helping land
RFC 002; per-agent adapter PRs land from their respective authors.

Aider, Gemini CLI, Codex CLI, and Warp are roadmap targets without
existing adapter PRs and are listed as such (no fabricated PR refs).

https://claude.ai/code/session_01GvwducFnFtN8KYmfbWKMR6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/mining File and conversation mining enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants