Skip to content

fix(llm-drivers,runtime,skills): enforce prompt-cache key determinism (#5143)#5177

Merged
houko merged 1 commit into
mainfrom
fix/5143-prompt-cache-determinism
May 17, 2026
Merged

fix(llm-drivers,runtime,skills): enforce prompt-cache key determinism (#5143)#5177
houko merged 1 commit into
mainfrom
fix/5143-prompt-cache-determinism

Conversation

@houko

@houko houko commented May 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Closes #5143. Audit follow-up to #3298: the three remaining prompt-cache
determinism soft spots were sustained-by-convention rather than
enforced-by-type. None exploitable today, but each was one careless
change from silently invalidating the Anthropic/OpenAI prompt cache
(~75% input-token savings). Each is now enforced at its boundary with a
determinism regression test mirroring
mcp_summary_is_byte_identical_across_input_orders.

Soft-spots addressed

  • serde_json/preserve_order implicitly load-bearing for openai
    request bodies
    crates/librefang-llm-drivers/src/drivers/openai.rs
    (non-streaming 1083-1087, streaming 1502-1509). The
    extra_body: HashMap was merged into the wire body via obj.insert,
    byte-stable ONLY because the workspace does not enable
    serde_json/preserve_order (so serde_json::Map is BTreeMap).
    Extracted merge_extra_body() which collects + sorts keys before
    insertion, so the merged body is deterministic regardless of which
    serde_json::Map backend is active. The existing
    test_oai_request_extra_body_merged_overrides_standard_field now
    calls the production helper instead of a hand-copied merge loop.

  • HookRegistry::collect_prompt_sections ordering depends on
    registration order
    crates/librefang-runtime/src/hooks.rs
    (200-205). BeforePromptBuild handlers feed Section 16 of the
    system prompt; they live in a Vec inside a DashMap, appended in
    registration order. No production handler registers BeforePromptBuild
    today, but a future filesystem/plugin-discovered handler would leak
    its order into the prompt. Now sorts the collected sections by the
    stable (provider, heading) key at the boundary before returning.
    Renamed test_collect_prompt_sections_returns_in_registration_order
    _returns_sorted_by_provider to match the new contract.

  • SkillRegistry::list() returned unsorted (HashMap order)
    crates/librefang-skills/src/registry.rs (360-362). Prompt-bound
    callers (sorted_enabled_skills, all_tool_definitions) sorted
    downstream, but the bare list() is also called from the API
    (routes/commands.rs:63,102) and CLI (main.rs:7157), and any future
    prompt-side caller picking up .list() directly would silently
    reorder. list() now sorts by manifest.skill.name; the existing
    downstream sorts remain (defense-in-depth; all_tool_definitions
    reads self.skills.values() directly, not via list()).
    Order-insensitive API/CLI consumers are unaffected.

Not changed

  • Anthropic / Gemini drivers: confirmed they have NO HashMap-merge-into-
    wire-body pattern (grep for as_object_mut / obj.insert over a
    HashMap returns nothing; only OaiRequest in openai.rs does this).
    Scope is correctly limited to openai.rs.
  • deny.toml rule pinning the serde_json feature set: optional per the
    issue. The explicit sort makes the merge correct regardless of the
    feature, which is a stronger guarantee than a lint that only fires if
    someone touches Cargo.toml; a deny.toml rule would be a separate
    tooling-domain change and is not required to close the issue.

Tests (real assertions, byte-equality across two input orders)

  • librefang_skills::registry::tests::list_is_deterministic_across_insertion_orders
  • drivers::openai::tests::extra_body_merge_is_byte_identical_across_insertion_orders
  • hooks::tests::collect_prompt_sections_is_deterministic_across_registration_orders

Verification

  • cargo check --lib: librefang-skills, librefang-llm-drivers,
    librefang-runtime, plus downstream librefang-kernel,
    librefang-api, librefang-cli — all clean.
  • cargo clippy --all-targets -- -D warnings: skills, llm-drivers,
    runtime — zero warnings.
  • cargo test: skills (1 new), llm-drivers (3, incl. 1 new), runtime
    hooks (7, incl. 1 new + 1 renamed), kernel skill regression suite
    (80 passed) — all green.

…#5143)

Three soft spots where the #3298 prompt-cache determinism invariant was
sustained-by-convention rather than enforced-by-type. None exploitable
today, but each was one careless change from silently invalidating the
Anthropic/OpenAI prompt cache.

- openai.rs extra_body merge (1083-1087, 1502-1509): the HashMap was
  merged into the wire body via obj.insert; byte-stable ONLY because the
  workspace does not enable serde_json/preserve_order (Map = BTreeMap).
  Extracted merge_extra_body() which sorts keys before insertion, so the
  body is deterministic regardless of the serde_json::Map backend.
  Existing test now exercises the production helper instead of a copy.

- hooks.rs collect_prompt_sections (200-205): BeforePromptBuild handlers
  feed Section 16 of the system prompt in DashMap<Vec> registration
  order. Sort the collected sections by (provider, heading) at the
  boundary so a future filesystem/plugin-discovered handler cannot leak
  registration order into the prompt.

- registry.rs SkillRegistry::list() (360-362): returned HashMap order.
  Prompt-bound callers sorted downstream, but API/CLI callers and any
  future prompt-side caller did not. Sort by skill name inside list()
  so the invariant holds at the source; order-insensitive consumers are
  unaffected.

Each boundary gets a determinism regression test asserting byte-identical
output across two input insertion/registration orders, mirroring
mcp_summary_is_byte_identical_across_input_orders.

Verification:
- cargo check --lib: librefang-skills, librefang-llm-drivers,
  librefang-runtime, librefang-kernel, librefang-api, librefang-cli
- cargo clippy --all-targets -D warnings: skills, llm-drivers, runtime
- cargo test: skills (list_is_deterministic_across_insertion_orders),
  llm-drivers (extra_body_merge_is_byte_identical_across_insertion_orders
  + 2 existing), runtime hooks (7 incl.
  collect_prompt_sections_is_deterministic_across_registration_orders),
  kernel skill regression (80 passed)

Closes #5143

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b358c8d19a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +291 to +295
sections.sort_by(|a, b| {
a.provider
.cmp(&b.provider)
.then_with(|| a.heading.cmp(&b.heading))
});

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Sort sections before enforcing the total budget

When multiple BeforePromptBuild handlers produce more than TOTAL_DYNAMIC_CHAR_CAP bytes in aggregate, the drop decision has already been made in registration order before this sort runs, so two nondeterministic discovery orders can retain different section sets and still produce different system prompts after sorting. Please collect/truncate sections, apply the deterministic order, and only then enforce the total cap so the prompt-cache key is stable under the overflow scenario this change is trying to harden.

Useful? React with 👍 / 👎.

@github-actions github-actions Bot added area/skills Skill system and FangHub marketplace area/runtime Agent loop, LLM drivers, WASM sandbox size/L 250-999 lines changed labels May 17, 2026
@houko
houko merged commit df389e4 into main May 17, 2026
32 checks passed
@houko
houko deleted the fix/5143-prompt-cache-determinism branch May 17, 2026 10:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/runtime Agent loop, LLM drivers, WASM sandbox area/skills Skill system and FangHub marketplace size/L 250-999 lines changed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[audit] Prompt-cache determinism follow-ups (#3298 area)

1 participant