Skip to content

fix(memory): use explicit qmd snippet line metadata#58181

Merged
vincentkoc merged 3 commits into
mainfrom
fix/qmd-snippet-lines
Mar 31, 2026
Merged

fix(memory): use explicit qmd snippet line metadata#58181
vincentkoc merged 3 commits into
mainfrom
fix/qmd-snippet-lines

Conversation

@vincentkoc

Copy link
Copy Markdown
Member

Summary

  • Problem: QMD/mcporter memory_search hits dropped explicit start_line / end_line metadata and reconstructed offsets from the snippet header instead.
  • Why it matters: returned hits can point at the wrong lines even when QMD already supplied the exact snippet bounds.
  • What changed: preserved explicit line metadata in the QMD parser and preferred it in the QMD manager result mapping.
  • What did NOT change (scope boundary): this does not alter snippet extraction when QMD does not provide explicit line numbers.

Change Type (select all)

  • Bug fix
  • Feature
  • Refactor required for the fix
  • Docs
  • Security hardening
  • Chore/infra

Scope (select all touched areas)

  • Gateway / orchestration
  • Skills / tool execution
  • Auth / tokens
  • Memory / storage
  • Integrations
  • API / contracts
  • UI / DX
  • CI/CD / infra

Linked Issue/PR

Root Cause / Regression History (if applicable)

  • Root cause: OpenClaw parsed mcporter results into an internal shape that discarded explicit line metadata, then fell back to parsing snippet headers.
  • Missing detection / guardrail: parser and manager tests did not cover the explicit start_line / end_line path.
  • Prior context (git blame, prior PR, issue, or refactor if known): contributor PR fix(memory): use start_line/end_line from QMD/mcporter response #47960 identified the right fix direction; this branch reapplies it on current main.
  • Why this regressed now: mcporter/QMD already exposes the precise line bounds, so dropping them was entirely an integration-side loss of fidelity.
  • If unknown, what was ruled out: ruled out DB path resolution issues; the bug occurs before path lookup, when the result payload is normalized.

Regression Test Plan (if applicable)

  • Coverage level that should have caught this:
    • Unit test
    • Seam / integration test
    • End-to-end test
    • Existing coverage already sufficient
  • Target test or file: packages/memory-host-sdk/src/host/qmd-query-parser.test.ts, extensions/memory-core/src/memory/qmd-manager.test.ts
  • Scenario the test should lock in: preserve explicit QMD line metadata and prefer it over snippet-header fallback when returning search hits.
  • Why this is the smallest reliable guardrail: one parser test locks the payload shape, and one manager test locks the user-visible hit mapping.
  • Existing test that already covers this (if any): existing tests only covered snippet-header fallback.
  • If no new test is added, why not: N/A

User-visible / Behavior Changes

  • QMD/mcporter search hits now return the real snippet line numbers when QMD provides them explicitly.

Diagram (if applicable)

Before:
[qmd/mcporter result with start_line/end_line] -> [OpenClaw drops fields] -> [reconstruct from snippet header]

After:
[qmd/mcporter result with start_line/end_line] -> [OpenClaw preserves fields] -> [return exact line offsets]

Security Impact (required)

  • New permissions/capabilities? (Yes/No) No
  • Secrets/tokens handling changed? (Yes/No) No
  • New/changed network calls? (Yes/No) No
  • Command/tool execution surface changed? (Yes/No) No
  • Data access scope changed? (Yes/No) No
  • If any Yes, explain risk + mitigation:

Repro + Verification

Environment

  • OS: macOS
  • Runtime/container: Node 22 / pnpm
  • Model/provider: N/A
  • Integration/channel (if any): QMD via mcporter
  • Relevant config (redacted): memory.backend=qmd, memory.qmd.mcporter.enabled=true

Steps

  1. Run a QMD/mcporter search that returns explicit start_line / end_line.
  2. Inspect the mapped memory_search result.
  3. Compare before/after.

Expected

  • OpenClaw should preserve the explicit line numbers from QMD.

Actual

  • Before this PR, OpenClaw reconstructed offsets from the snippet header and could point at the wrong lines.

Evidence

  • Failing test/log before + passing after
  • Trace/log snippets
  • Screenshot/recording
  • Perf numbers (if relevant)

Human Verification (required)

  • Verified scenarios: focused parser and QMD manager tests for explicit line metadata.
  • Edge cases checked: kept snippet-header fallback for results without explicit line numbers.
  • What you did not verify: full repo build; local build gate was behaving pathologically across concurrent worktrees, so verification stayed scoped to the touched seam.

Review Conversations

  • I replied to or resolved every bot review conversation I addressed in this PR.
  • I left unresolved only the conversations that still need reviewer or maintainer judgment.

Compatibility / Migration

  • Backward compatible? (Yes/No) Yes
  • Config/env changes? (Yes/No) No
  • Migration needed? (Yes/No) No
  • If yes, exact upgrade steps:

Risks and Mitigations

  • Risk: malformed line metadata could produce bad offsets.
    • Mitigation: only finite positive numbers are accepted, and the old snippet-header fallback remains in place when explicit metadata is absent.

@greptile-apps

greptile-apps Bot commented Mar 31, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR fixes a bug in the QMD/mcporter memory search integration where explicit start_line/end_line metadata was being dropped during result normalization, causing memory_search hits to reconstruct (potentially incorrect) line offsets from the snippet header instead of using the precise bounds already supplied by QMD.

Changes:

  • qmd-query-parser.ts: replaces a bare as QmdQueryResult[] cast with an explicit field mapping that captures start_line/end_line from raw JSON and stores them as startLine/endLine on QmdQueryResult.
  • qmd-manager.ts: adds resolveSnippetLines() (prefers explicit metadata, falls back to snippet-header parsing) and normalizeSnippetLine() (rejects non-positive/non-finite values); also mirrors the same start_line ?? startLine extraction in the direct mcporter JSON path.
  • Two new focused tests — one in the parser, one in the manager — lock in the end-to-end behavior.

The fix is minimal and surgical. Fallback behavior (snippet-header parsing) is preserved for results that don't carry explicit line metadata, and the value > 0 guard prevents malformed zero or negative line numbers from being accepted.

Confidence Score: 5/5

Safe to merge — the change is narrowly scoped to result normalization, is non-breaking, and is covered by new targeted tests.

All changed paths are defensive (strict type checks, positive-integer guard, swap-if-inverted), the old fallback is preserved when explicit metadata is absent, and the two new tests cover the newly added code path end-to-end. No data-integrity, security, or build concerns were found.

No files require special attention.

Important Files Changed

Filename Overview
packages/memory-host-sdk/src/host/qmd-query-parser.ts Replaces bare type cast with explicit field mapping that preserves start_line/end_line as startLine/endLine on QmdQueryResult; all other fields correctly preserved.
extensions/memory-core/src/memory/qmd-manager.ts Adds resolveSnippetLines() preferring explicit metadata over snippet-header fallback, normalizeSnippetLine() with positive-integer guard, and mirrors start_line extraction in the direct mcporter JSON path.
packages/memory-host-sdk/src/host/qmd-query-parser.test.ts Adds test covering explicit start_line/end_line preservation through the parser.
extensions/memory-core/src/memory/qmd-manager.test.ts Adds integration-style test verifying that explicit mcporter line metadata flows through to the final search hit result.
CHANGELOG.md Adds changelog entry for the QMD explicit line metadata fix.

Reviews (1): Last reviewed commit: "fix(memory): preserve qmd snippet line m..." | Re-trigger Greptile

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 86d909acb1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread extensions/memory-core/src/memory/qmd-manager.ts Outdated
@aisle-research-bot

aisle-research-bot Bot commented Mar 31, 2026

Copy link
Copy Markdown

🔒 Aisle Security Analysis

We found 1 potential security issue(s) in this PR:

# Severity Title
1 🟡 Medium Improper input validation allows non-object entries (including null) in QMD query results, causing runtime crashes
1. 🟡 Improper input validation allows non-object entries (including null) in QMD query results, causing runtime crashes
Property Value
Severity Medium
CWE CWE-20
Location packages/memory-host-sdk/src/host/qmd-query-parser.ts:78-81

Description

parseQmdQueryResultArray attempts to “sanitize” parsed JSON items, but for non-object entries it returns the value unchanged (cast to QmdQueryResult).

  • If the JSON array contains null, the function returns null as an element.
  • Downstream callers (e.g., extensions/memory-core/src/memory/qmd-manager.ts) iterate results and access properties like entry.collection, which will throw a TypeError when entry is null.
  • This enables a denial-of-service condition if an attacker can influence the QMD output/response (or any upstream source that feeds into parseQmdQueryJson) to include null or other unexpected entries.

Vulnerable code:

return parsed.map((item) => {
  if (typeof item !== "object" || item === null) {
    return item as QmdQueryResult;
  }// ...
});

Recommendation

Reject or filter out non-object entries instead of returning them.

Option A (filter non-objects out):

return parsed
  .filter((item): item is Record<string, unknown> => typeof item === "object" && item !== null)
  .map((record) => {
    const docid = typeof record.docid === "string" ? record.docid : undefined;// ...
    return {
      docid,// ...
      startLine: parseQmdLineNumber(record.start_line ?? record.startLine),
      endLine: parseQmdLineNumber(record.end_line ?? record.endLine),
    } satisfies QmdQueryResult;
  });

Option B (hard-fail on invalid array contents):

for (const item of parsed) {
  if (typeof item !== "object" || item === null) {
    throw new Error("qmd query JSON response contained non-object items");
  }
}

Either approach prevents null/primitives from propagating to callers that assume an object shape.


Analyzed PR: #58181 at commit 004b62e

Last updated on: 2026-03-31T08:08:53Z

@vincentkoc
vincentkoc merged commit 075645f into main Mar 31, 2026
10 checks passed
@vincentkoc
vincentkoc deleted the fix/qmd-snippet-lines branch March 31, 2026 08:05
pgondhi987 pushed a commit to pgondhi987/openclaw that referenced this pull request Mar 31, 2026
* fix(memory): preserve qmd snippet line metadata

* Memory/QMD: preserve snippet span with partial line metadata
pgondhi987 pushed a commit to pgondhi987/openclaw that referenced this pull request Mar 31, 2026
* fix(memory): preserve qmd snippet line metadata

* Memory/QMD: preserve snippet span with partial line metadata
lovewanwan pushed a commit to lovewanwan/openclaw that referenced this pull request Apr 28, 2026
* fix(memory): preserve qmd snippet line metadata

* Memory/QMD: preserve snippet span with partial line metadata
ogt-redknie pushed a commit to ogt-redknie/OPENX that referenced this pull request May 2, 2026
* fix(memory): preserve qmd snippet line metadata

* Memory/QMD: preserve snippet span with partial line metadata
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request May 9, 2026
* fix(memory): preserve qmd snippet line metadata

* Memory/QMD: preserve snippet span with partial line metadata
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request May 24, 2026
* fix(memory): preserve qmd snippet line metadata

* Memory/QMD: preserve snippet span with partial line metadata
Nachx639 pushed a commit to Nachx639/clawdbot that referenced this pull request Jun 17, 2026
* fix(memory): preserve qmd snippet line metadata

* Memory/QMD: preserve snippet span with partial line metadata
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

extensions: memory-core Extension: memory-core maintainer Maintainer-authored PR size: M

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant