Your AI Agent Is Being Sabotaged
by Its Own Search Results

This is the key Hornet insight: align retrieval configuration with how coding agents already work. An agent that can edit tsconfig.json can edit a Hive config. No new abstractions. The entire workspace is files on disk — versioned, diffable, reviewable.

Why this matters: LLM agents are fundamentally code-writing tools. They understand files, directories, and JSON. By making retrieval config a file, we turn search optimization into a coding task — which is exactly what these agents are good at.

Fields marked filterable: true can be used in config filters. Filters are applied as a WHERE clause before scoring (pre-filter), not after. This means distractors in excluded categories never even enter the ranking pipeline.

{
  "name": "knowledge-base",
  "fields": {
    "title":    { "type": "text" },
    "category": { "type": "keyword", "filterable": true },
    "content":  { "type": "text" }
  },
  "chunking": {
    "strategy": "by_heading",
    "max_tokens": 512,
    "heading_level": 2
  }
}

The agent can read this file to discover: "category is filterable, and the available values are api-docs, tutorials, faqs, changelogs." Then it can write a config with "filters": {"category": ["api-docs", "tutorials"]} to exclude FAQ distractors entirely.

This is the Verifiable APIs principle from Hornet: the system's interface should be checkable at multiple levels before anything gets deployed. The agent gets immediate, actionable feedback — not vague errors, but specific "here's what's wrong and how to fix it" messages.

The behavioral gate is the critical innovation: even if a config passes syntactic and semantic checks, it must prove it's better by running against the golden evaluation set. This prevents an agent from "improving" a config that actually makes retrieval worse.

Traditional retrieval always returns a fixed number of results (e.g., 10). But what if only 5 are good and the rest are noise? Dynamic-k detects the "quality cliff" where results transition from strong to weak, and stops there.

Why not score-ratio cutoff? RRF scores are compressed into a narrow band (all values between ~0.018 and ~0.033 for k=60). A result at BM25 rank 20 / vector rank 30 still has a ratio of ~0.72 to the best result. Score-ratio cutoffs would almost never trigger with RRF. Instead, we measure relative gaps.

The algorithm: After ranking by RRF, compute the gap between each consecutive pair. Track a running mean of gaps seen so far. If gap_i > gap_threshold_factor × mean_gap_so_far, stop — cliff detected.

Returns 6 results instead of 10. Ranks 1–6 have gradually increasing but consistent gaps (~0.001), then rank 7 drops sharply — the cliff. This is interpretable: "stop when the next result is dramatically worse than the trend so far." The approach works with compressed RRF scores because it measures relative gaps, not absolute score levels.

The naive approach to hybrid search is to normalize BM25 scores (range 0–15) and cosine similarity scores (range 0–1) to the same scale and take a weighted average. This is fragile — the normalization depends on the query, the corpus size, and the score distributions. RRF sidesteps all of this by operating on ranks instead of scores.

For keyword-only mode, only BM25 ranks are used. For vector-only, only cosine ranks. For hybrid, both contribute to the final score.

Distractors share keywords with the query but mean something different. BM25 ranks them high (keyword overlap), but vector search ranks them low (different semantics). This disagreement between scoring methods is a detectable signal at query time — no golden labels needed.

Doc B matches on keywords but means something different — a classic FAQ distractor. The flag appears in query output so the agent can see it, but it does NOT auto-filter the result. The agent decides what to do. At evaluation time (not query time), golden labels provide exact distractor counts as ground truth.

Each result's contribution is weighted by its rank position — a distractor at rank 1 costs more than one at rank 5. The result is normalized by the ideal score (all relevant docs ranked first) to get nUDCG in the range [-1, +1].

Document-level dedup: Golden labels are per-document but retrieval returns per-chunk results. A document may produce multiple chunks. To avoid double-counting: for each document, only the highest-ranked chunk contributes to UDCG. All subsequent chunks from the same document are skipped (utility = 0). This prevents distractor penalties from being inflated by multi-chunk documents.

Traditional search evaluation treats results as either relevant (1) or irrelevant (0). This made sense when humans scanned a list — an irrelevant result was just skipped. But when an LLM reads search results as context, a distractor (a result that looks right but is wrong) actively corrupts the answer. The LLM can't distinguish between "API authentication via bearer tokens" and "authentication FAQ about password resets" — both mention authentication, but one answers the wrong question.

This is the MAD insight from Hornet: in LLM-powered retrieval, irrelevant is not the worst outcome — misleading is. An FAQ document about password resets doesn't just fail to help with an API auth question; it pulls the LLM's response toward password reset instructions, contaminating the final answer.

Every primitive in Hive exists to address this: UDCG makes the cost visible (-1 per distractor), score disagreement detects them at query time, dynamic-k prevents over-retrieving them, category filters exclude them structurally, and deploy gates prevent configs that increase them. The entire system is oriented around one question: "are we injecting harmful context into the LLM?"

Hive implements the entire search engine from first principles: BM25 scoring with an inverted index (~100 lines), vector search via cosine similarity on numpy arrays, and RRF fusion — all stored in a single SQLite database. For the prototype's corpus of ~60 chunks, this runs in milliseconds with no external services.