Skip to content

fix(memory-lancedb): skip processed auto-capture messages safely#72663

Merged
vincentkoc merged 1 commit into
mainfrom
clownfish/ghcrawl-156626-autonomous-smoke
Apr 27, 2026
Merged

fix(memory-lancedb): skip processed auto-capture messages safely#72663
vincentkoc merged 1 commit into
mainfrom
clownfish/ghcrawl-156626-autonomous-smoke

Conversation

@vincentkoc

Copy link
Copy Markdown
Member

Summary

Credit

Thanks @as775116191 for the original implementation in #42083. This ProjectClownfish repair keeps that source PR attribution and addresses the Codex/Greptile review findings before validation.

Validation

  • pnpm -s vitest run extensions/memory-lancedb/index.test.ts
  • pnpm check:changed

ProjectClownfish replacement details:

@aisle-research-bot

aisle-research-bot Bot commented Apr 27, 2026

Copy link
Copy Markdown

🔒 Aisle Security Analysis

We found 5 potential security issue(s) in this PR:

# Severity Title
1 🟡 Medium Auto-capture cursor fallback can reset to index 0 causing repeated embedding work (cost-amplification DoS)
2 🟡 Medium Algorithmic CPU/memory DoS via JSON.stringify message fingerprinting in auto-capture cursor logic
3 🟡 Medium Sensitive user message content retained in memory via autoCaptureCursors fingerprint
4 🟡 Medium Unbounded per-session cursor Map can cause memory exhaustion (autoCaptureCursors)
5 🟡 Medium Cross-session cursor state collision via un-namespaced sessionKey/sessionId in memory-lancedb auto-capture cursor map
1. 🟡 Auto-capture cursor fallback can reset to index 0 causing repeated embedding work (cost-amplification DoS)
Property Value
Severity Medium
CWE CWE-400
Location extensions/memory-lancedb/index.ts:95-109

Description

The auto-capture cursor logic can fall back to processing from the beginning of the conversation when the previously processed message fingerprint is no longer present in event.messages.

  • Cursor state is tracked per session using { nextIndex, lastMessageFingerprint }.
  • On subsequent agent_end events, resolveAutoCaptureStartIndex() tries to find lastMessageFingerprint in the new messages array.
  • If it is not found (e.g., history compaction/truncation, message rewriting, or attacker-influenced message sequences), the function returns 0.
  • The agent_end handler then iterates from startIndex and performs up to 3 embedding requests (and DB searches/writes) again.
  • The duplicate check happens after await embeddings.embed(text), so even when a memory already exists, the expensive embedding call is repeated.

This creates a cost-amplification/denial-of-service vector where an attacker (or normal compaction behavior) can repeatedly trigger full-history rescans and repeated embedding calls.

Vulnerable code:

if (cursor.lastMessageFingerprint && cursor.nextIndex > 0) {
  for (let index = messages.length - 1; index >= 0; index--) {
    if (messageFingerprint(messages[index]) === cursor.lastMessageFingerprint) {
      return index + 1;
    }
  }
  return 0;
}

Recommendation

Avoid resetting to the beginning of the conversation when the cursor cannot be reconciled.

Recommended mitigations (pick one or combine):

  1. Bound the fallback scan window to only the most recent N messages (e.g., last 20) to limit worst-case work.
  2. Use stable message identifiers (message IDs / monotonic sequence numbers) instead of content fingerprints.
  3. On fingerprint miss, fall back to a safe recent index rather than 0.

Example bounded fallback:

function resolveAutoCaptureStartIndex(messages: unknown[], cursor?: AutoCaptureCursor): number {
  const RECENT_FALLBACK = 20;
  if (!cursor) return Math.max(0, messages.length - RECENT_FALLBACK);

  if (cursor.lastMessageFingerprint && cursor.nextIndex > 0) {
    for (let i = messages.length - 1; i >= 0; i--) {
      if (messageFingerprint(messages[i]) === cursor.lastMessageFingerprint) return i + 1;
    }// fingerprint not found -> only process the most recent messages
    return Math.max(0, messages.length - RECENT_FALLBACK);
  }

  return Math.min(cursor.nextIndex, messages.length);
}

Also consider adding a cheap dedupe key (e.g., hash of text) before calling embed() to avoid repeated embedding calls for identical text.

2. 🟡 Algorithmic CPU/memory DoS via JSON.stringify message fingerprinting in auto-capture cursor logic
Property Value
Severity Medium
CWE CWE-400
Location extensions/memory-lancedb/index.ts:80-107

Description

The auto-capture cursor logic computes and compares message fingerprints by calling JSON.stringify({ role, content }) on each message while scanning backwards through event.messages.

  • Attacker-controlled input: event.messages[*].content can contain very large strings/arrays (e.g., long user content blocks).
  • Expensive operation in a loop: resolveAutoCaptureStartIndex() performs a reverse scan over the full messages array and calls messageFingerprint() for each element until a match is found.
  • Worst-case complexity: O(n * size(message.content)) per agent_end hook invocation, potentially repeatedly for the same session; this can cause significant CPU and memory pressure.
  • Additional hazard: if content contains cyclic/hostile objects, JSON.stringify throws and the code falls back to String(msgObj.content); while caught, the repeated exception path can still be costly.

Vulnerable code:

for (let index = messages.length - 1; index >= 0; index--) {
  if (messageFingerprint(messages[index]) === cursor.lastMessageFingerprint) {
    return index + 1;
  }
}// ...
return JSON.stringify({ role: msgObj.role, content: msgObj.content });

Recommendation

Avoid serializing full content for cursoring.

Recommended options (prefer 1):

  1. Use stable message IDs / monotonic indices from the runtime (if available) and store the last processed message ID (or index) rather than deep content.
  2. If IDs are unavailable, compute a bounded fingerprint:
    • only include small, fixed-size fields (e.g., role + first N chars of text)
    • and hash it (e.g., SHA-256) to keep comparisons O(1) in memory.
  3. Avoid backward scans:
    • trust nextIndex when messages.length >= nextIndex, and only fall back to scanning when the array has been truncated/rotated.

Example bounded hash approach:

import { createHash } from "node:crypto";

function boundedMessageFingerprint(message: unknown): string {
  const msgObj = asRecord(message);
  const role = String(msgObj?.role ?? "");
  const content = msgObj?.content;
  const preview = typeof content === "string"
    ? content.slice(0, 256)
    : Array.isArray(content)
      ? JSON.stringify(content.slice(0, 3)).slice(0, 256)
      : "";
  return createHash("sha256").update(role + ":" + preview).digest("hex");
}

Also consider enforcing maximum event.messages length (or max total characters) processed per hook invocation to cap work.

3. 🟡 Sensitive user message content retained in memory via autoCaptureCursors fingerprint
Property Value
Severity Medium
CWE CWE-359
Location extensions/memory-lancedb/index.ts:80-89

Description

The auto-capture cursor mechanism stores a lastMessageFingerprint derived from JSON stringifying the full message {role, content}, which can include entire user prompts (PII, secrets, credentials).

This increases the exposure window for sensitive data because:

  • autoCaptureCursors is a long-lived in-memory Map keyed by session identifiers.
  • messageFingerprint() returns JSON.stringify({ role, content }), retaining full content.
  • If session_end is not emitted for any termination path (crash, forced shutdown, unexpected disconnect), these fingerprints may persist indefinitely in the process heap and be recoverable via heap dumps, diagnostics, crash reports, or memory disclosure bugs.

Vulnerable code:

return JSON.stringify({ role: msgObj.role, content: msgObj.content });
...
autoCaptureCursors.set(cursorKey, {
  nextIndex: index + 1,
  lastMessageFingerprint: messageFingerprint(message),
});

Recommendation

Avoid storing raw message content in cursors. Store a non-reversible digest (or other minimal metadata) instead, and consider adding eviction/TTL as defense-in-depth.

Example (hashing canonicalized content):

import { createHash } from "node:crypto";

function messageFingerprint(message: unknown): string {
  const msgObj = asRecord(message);
  if (!msgObj) return `${typeof message}:${String(message)}`;// Only include minimal, stable fields; avoid retaining full content in memory
  const canonical = JSON.stringify({ role: msgObj.role, content: msgObj.content });
  return createHash("sha256").update(canonical).digest("hex");
}

Additionally:

  • Add a max size/TTL for autoCaptureCursors entries.
  • Ensure cursor cleanup runs on all shutdown paths (plugin/service stop, process exit), not only session_end.
4. 🟡 Unbounded per-session cursor Map can cause memory exhaustion (autoCaptureCursors)
Property Value
Severity Medium
CWE CWE-400
Location extensions/memory-lancedb/index.ts:381-736

Description

The memory-lancedb plugin introduces a per-session autoCaptureCursors map that is updated during every agent_end hook and only cleared on session_end.

  • State is retained per session key/id: cursorKey is derived from ctx.sessionKey ?? ctx.sessionId and used as the Map key.
  • Unbounded growth: Entries are only removed when a session_end hook fires. If sessions are long-lived, abandoned, or if session_end is not reliably emitted (crashes, malicious clients creating many sessions, etc.), the Map can grow without bound.
  • Large/PII retention & extra overhead: Each entry stores lastMessageFingerprint, computed as JSON.stringify({role, content}), which can retain large user message content (and potential PII) in memory and adds CPU/GC pressure when called repeatedly.

Vulnerable code:

const autoCaptureCursors = new Map<string, AutoCaptureCursor>();
...
autoCaptureCursors.set(cursorKey, {
  nextIndex: index + 1,
  lastMessageFingerprint: messageFingerprint(message),
});
return JSON.stringify({ role: msgObj.role, content: msgObj.content });

This creates a memory-exhaustion/DoS risk in the plugin host process when an attacker (or buggy client) can cause many unique sessions and/or very large messages.

Recommendation

Add explicit bounds to in-memory per-session cursor tracking and avoid storing raw message content.

Suggested mitigations:

  • Implement an LRU/TTL cache for cursors (e.g., max N sessions, expire after M minutes of inactivity).
  • Store a fixed-size hash of the message instead of JSON.stringify of the entire content.
  • Optionally store only nextIndex if messages are guaranteed stable, or use a stable message id if available.

Example (bounded + hashed fingerprint):

import { createHash } from "node:crypto";

const MAX_CURSORS = 1000;
const cursorTimestamps = new Map<string, number>();

function messageFingerprintHash(message: unknown): string {
  const s = JSON.stringify(message, (_k, v) => (typeof v === "string" && v.length > 1024 ? v.slice(0, 1024) : v));
  return createHash("sha256").update(s).digest("hex");
}

function setCursor(key: string, cursor: AutoCaptureCursor) {
  if (autoCaptureCursors.size >= MAX_CURSORS && !autoCaptureCursors.has(key)) {// evict oldest
    let oldestKey: string | undefined;
    let oldestTs = Infinity;
    for (const [k, ts] of cursorTimestamps) {
      if (ts < oldestTs) { oldestTs = ts; oldestKey = k; }
    }
    if (oldestKey) { autoCaptureCursors.delete(oldestKey); cursorTimestamps.delete(oldestKey); }
  }
  autoCaptureCursors.set(key, cursor);
  cursorTimestamps.set(key, Date.now());
}

Also consider clearing cursors on additional lifecycle events (e.g., agent errors / gateway stop) to reduce leak risk if session_end is missed.

5. 🟡 Cross-session cursor state collision via un-namespaced sessionKey/sessionId in memory-lancedb auto-capture cursor map
Property Value
Severity Medium
CWE CWE-639
Location extensions/memory-lancedb/index.ts:691-754

Description

The memory-lancedb plugin maintains an in-memory autoCaptureCursors map to track which messages have already been processed for auto-capture.

The map key is derived from ctx.sessionKey ?? ctx.sessionId in agent_end, but the deletion logic in session_end uses a broader fallback chain (ctx.sessionKey ?? event.sessionKey ?? ctx.sessionId ?? event.sessionId). If sessionKey is user-controlled/guessable (it is supplied by clients as key in the gateway protocol) or not globally unique across trust boundaries (agent/account/tenant), another session can intentionally or accidentally reuse the same cursorKey.

Impact:

  • Cross-session interference: an attacker (or another user/session) that can cause a session to use the same sessionKey could advance or reset another session’s cursor.
  • This can cause skipped or duplicated memory capture, and can keep lastMessageFingerprint state for another session in memory longer than intended.

Vulnerable code:

// agent_end
const cursorKey = ctx.sessionKey ?? ctx.sessionId;
...
autoCaptureCursors.set(cursorKey, { ... });// session_end
const cursorKey = ctx.sessionKey ?? event.sessionKey ?? ctx.sessionId ?? event.sessionId;
autoCaptureCursors.delete(cursorKey);

Because autoCaptureCursors is global to the plugin instance, keys must be collision-resistant and scoped to the correct isolation boundary (agent/account/tenant/workspace).

Recommendation

Use a stable, non-user-controlled, uniquely-scoped cursor key.

Options (pick one):

  1. Prefer sessionId (if it is guaranteed unique and not user-controlled) and never fall back to event.sessionKey:
const cursorKey = ctx.sessionId; // required by PluginHookSessionContext
  1. If you must use sessionKey, namespace it with other authenticated context (e.g., agentId, workspaceDir, account id if available):
const cursorKey = `${ctx.agentId ?? "unknown"}:${ctx.sessionId}`;

Also make session_end deletion use the exact same derivation logic as agent_end (single source of truth), rather than mixing ctx and event values, to avoid deleting the wrong session’s cursor state.


Analyzed PR: #72663 at commit 4a4c590

Last updated on: 2026-04-27T06:53:14Z

@openclaw-barnacle openclaw-barnacle Bot added extensions: memory-lancedb Extension: memory-lancedb size: M maintainer Maintainer-authored PR labels Apr 27, 2026
@vincentkoc
vincentkoc merged commit 69c30e3 into main Apr 27, 2026
11 of 13 checks passed
@vincentkoc
vincentkoc deleted the clownfish/ghcrawl-156626-autonomous-smoke branch April 27, 2026 06:51
@vincentkoc vincentkoc added the clawsweeper Tracked by ClawSweeper automation label Apr 27, 2026
@greptile-apps

greptile-apps Bot commented Apr 27, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR introduces per-session cursor tracking in the memory-lancedb autoCapture flow so that already-processed messages are skipped on subsequent agent_end calls, failed messages are retried, compacted histories fall back to full rescanning, and cursor state is evicted on session end. The implementation uses fingerprint-based cursor anchoring to handle history rewriting, and the four new unit tests cover all advertised scenarios.

Confidence Score: 4/5

Safe to merge; the logic is correct and well-covered by the new tests, with one minor defensive-coding concern.

No P0 or P1 issues found. The cursor-advance-on-success / retry-on-failure semantics are implemented correctly via try/finally. The fingerprint-based compaction fallback is sound. The only concern is the proactive deletion of nextCursorKey in session_end, which is harmless but unexplained and could cause extra embedding calls on session transitions with intact histories — a P2 style/performance note.

extensions/memory-lancedb/index.ts — specifically the session_end handler's nextCursorKey deletion logic.

Prompt To Fix All With AI
This is a comment left during a code review.
Path: extensions/memory-lancedb/index.ts
Line: 751-754

Comment:
**Proactive deletion of `nextCursorKey` may cause redundant re-embedding**

When a session transitions (e.g. daily compaction, `reason: "new"`), deleting the cursor for `nextCursorKey` forces the incoming session to re-scan all messages from index 0 on its first `agent_end`. In practice `resolveAutoCaptureStartIndex` would already return 0 for a compacted history (fingerprint not found), so this delete is mostly defensive. However, for transitions where the history is *not* compacted and the next session key had pre-built cursor state, this causes extra redundant embedding API calls — all caught by the dedup similarity check, but at unnecessary cost. A clarifying comment here would help future readers understand the intent.

How can I resolve this? If you propose a fix, please make it concise.

Reviews (1): Last reviewed commit: "fix(memory-lancedb): skip processed auto..." | Re-trigger Greptile

Comment on lines +751 to +754
const nextCursorKey = event.nextSessionKey ?? event.nextSessionId;
if (nextCursorKey) {
autoCaptureCursors.delete(nextCursorKey);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Proactive deletion of nextCursorKey may cause redundant re-embedding

When a session transitions (e.g. daily compaction, reason: "new"), deleting the cursor for nextCursorKey forces the incoming session to re-scan all messages from index 0 on its first agent_end. In practice resolveAutoCaptureStartIndex would already return 0 for a compacted history (fingerprint not found), so this delete is mostly defensive. However, for transitions where the history is not compacted and the next session key had pre-built cursor state, this causes extra redundant embedding API calls — all caught by the dedup similarity check, but at unnecessary cost. A clarifying comment here would help future readers understand the intent.

Prompt To Fix With AI
This is a comment left during a code review.
Path: extensions/memory-lancedb/index.ts
Line: 751-754

Comment:
**Proactive deletion of `nextCursorKey` may cause redundant re-embedding**

When a session transitions (e.g. daily compaction, `reason: "new"`), deleting the cursor for `nextCursorKey` forces the incoming session to re-scan all messages from index 0 on its first `agent_end`. In practice `resolveAutoCaptureStartIndex` would already return 0 for a compacted history (fingerprint not found), so this delete is mostly defensive. However, for transitions where the history is *not* compacted and the next session key had pre-built cursor state, this causes extra redundant embedding API calls — all caught by the dedup similarity check, but at unnecessary cost. A clarifying comment here would help future readers understand the intent.

How can I resolve this? If you propose a fix, please make it concise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

clawsweeper Tracked by ClawSweeper automation extensions: memory-lancedb Extension: memory-lancedb maintainer Maintainer-authored PR size: M

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant