fix(llm-drivers,runtime,skills): enforce prompt-cache key determinism (#5143)#5177
Conversation
…#5143) Three soft spots where the #3298 prompt-cache determinism invariant was sustained-by-convention rather than enforced-by-type. None exploitable today, but each was one careless change from silently invalidating the Anthropic/OpenAI prompt cache. - openai.rs extra_body merge (1083-1087, 1502-1509): the HashMap was merged into the wire body via obj.insert; byte-stable ONLY because the workspace does not enable serde_json/preserve_order (Map = BTreeMap). Extracted merge_extra_body() which sorts keys before insertion, so the body is deterministic regardless of the serde_json::Map backend. Existing test now exercises the production helper instead of a copy. - hooks.rs collect_prompt_sections (200-205): BeforePromptBuild handlers feed Section 16 of the system prompt in DashMap<Vec> registration order. Sort the collected sections by (provider, heading) at the boundary so a future filesystem/plugin-discovered handler cannot leak registration order into the prompt. - registry.rs SkillRegistry::list() (360-362): returned HashMap order. Prompt-bound callers sorted downstream, but API/CLI callers and any future prompt-side caller did not. Sort by skill name inside list() so the invariant holds at the source; order-insensitive consumers are unaffected. Each boundary gets a determinism regression test asserting byte-identical output across two input insertion/registration orders, mirroring mcp_summary_is_byte_identical_across_input_orders. Verification: - cargo check --lib: librefang-skills, librefang-llm-drivers, librefang-runtime, librefang-kernel, librefang-api, librefang-cli - cargo clippy --all-targets -D warnings: skills, llm-drivers, runtime - cargo test: skills (list_is_deterministic_across_insertion_orders), llm-drivers (extra_body_merge_is_byte_identical_across_insertion_orders + 2 existing), runtime hooks (7 incl. collect_prompt_sections_is_deterministic_across_registration_orders), kernel skill regression (80 passed) Closes #5143
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b358c8d19a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| sections.sort_by(|a, b| { | ||
| a.provider | ||
| .cmp(&b.provider) | ||
| .then_with(|| a.heading.cmp(&b.heading)) | ||
| }); |
There was a problem hiding this comment.
Sort sections before enforcing the total budget
When multiple BeforePromptBuild handlers produce more than TOTAL_DYNAMIC_CHAR_CAP bytes in aggregate, the drop decision has already been made in registration order before this sort runs, so two nondeterministic discovery orders can retain different section sets and still produce different system prompts after sorting. Please collect/truncate sections, apply the deterministic order, and only then enforce the total cap so the prompt-cache key is stable under the overflow scenario this change is trying to harden.
Useful? React with 👍 / 👎.
Summary
Closes #5143. Audit follow-up to #3298: the three remaining prompt-cache
determinism soft spots were sustained-by-convention rather than
enforced-by-type. None exploitable today, but each was one careless
change from silently invalidating the Anthropic/OpenAI prompt cache
(~75% input-token savings). Each is now enforced at its boundary with a
determinism regression test mirroring
mcp_summary_is_byte_identical_across_input_orders.Soft-spots addressed
serde_json/preserve_orderimplicitly load-bearing for openairequest bodies —
crates/librefang-llm-drivers/src/drivers/openai.rs(non-streaming
1083-1087, streaming1502-1509). Theextra_body: HashMapwas merged into the wire body viaobj.insert,byte-stable ONLY because the workspace does not enable
serde_json/preserve_order(soserde_json::MapisBTreeMap).Extracted
merge_extra_body()which collects + sorts keys beforeinsertion, so the merged body is deterministic regardless of which
serde_json::Mapbackend is active. The existingtest_oai_request_extra_body_merged_overrides_standard_fieldnowcalls the production helper instead of a hand-copied merge loop.
HookRegistry::collect_prompt_sectionsordering depends onregistration order —
crates/librefang-runtime/src/hooks.rs(
200-205).BeforePromptBuildhandlers feed Section 16 of thesystem prompt; they live in a
Vecinside aDashMap, appended inregistration order. No production handler registers
BeforePromptBuildtoday, but a future filesystem/plugin-discovered handler would leak
its order into the prompt. Now sorts the collected sections by the
stable
(provider, heading)key at the boundary before returning.Renamed
test_collect_prompt_sections_returns_in_registration_order→
_returns_sorted_by_providerto match the new contract.SkillRegistry::list()returned unsorted (HashMap order) —crates/librefang-skills/src/registry.rs(360-362). Prompt-boundcallers (
sorted_enabled_skills,all_tool_definitions) sorteddownstream, but the bare
list()is also called from the API(
routes/commands.rs:63,102) and CLI (main.rs:7157), and any futureprompt-side caller picking up
.list()directly would silentlyreorder.
list()now sorts bymanifest.skill.name; the existingdownstream sorts remain (defense-in-depth;
all_tool_definitionsreads
self.skills.values()directly, not vialist()).Order-insensitive API/CLI consumers are unaffected.
Not changed
wire-body pattern (
grepforas_object_mut/obj.insertover aHashMapreturns nothing; onlyOaiRequestinopenai.rsdoes this).Scope is correctly limited to
openai.rs.deny.tomlrule pinning theserde_jsonfeature set: optional per theissue. The explicit sort makes the merge correct regardless of the
feature, which is a stronger guarantee than a lint that only fires if
someone touches
Cargo.toml; adeny.tomlrule would be a separatetooling-domain change and is not required to close the issue.
Tests (real assertions, byte-equality across two input orders)
librefang_skills::registry::tests::list_is_deterministic_across_insertion_ordersdrivers::openai::tests::extra_body_merge_is_byte_identical_across_insertion_ordershooks::tests::collect_prompt_sections_is_deterministic_across_registration_ordersVerification
cargo check --lib:librefang-skills,librefang-llm-drivers,librefang-runtime, plus downstreamlibrefang-kernel,librefang-api,librefang-cli— all clean.cargo clippy --all-targets -- -D warnings: skills, llm-drivers,runtime — zero warnings.
cargo test: skills (1 new), llm-drivers (3, incl. 1 new), runtimehooks (7, incl. 1 new + 1 renamed), kernel skill regression suite
(80 passed) — all green.