Summary

• Replace BM25 with SQLite FTS5 for endpoint search — drops bm25s and snowballstemmer dependencies in favor of SQLite's built-in full-text search with porter tokenizer. Benchmarked against 93 golden queries covering all 72 endpoint titles: MRR
  +0.023, Recall@1 +0.032, Recall@5 +0.022 vs master.
• Consolidate 4 tools into 3 — merge get_endpoint_docs into search_endpoints via a new detail parameter ("default", "more", "verbose"). Adds max_results parameter. Reduces LLM tool calls by ~15% in testing.
• Single-request doc fetching — fetch all endpoint documentation from llms-full.txt in one request instead of ~50+ individual page fetches.
• Structured endpoint data model — replace compressed_doc string with typed QueryParam and ResponseAttribute lists.
• Search result deduplication — endpoints shared across markets (e.g. "Unified Snapshot" x5) are deduplicated by title, keeping the highest-scoring variant.
• Nested JSON flattening — extract_records now expands single-record responses containing a list-of-dicts (gainers/snapshots tickers array, technical indicator values array) into proper rows instead of stringifying them.
• New SQL functions — TO_TIMESTAMP(epoch) and TO_DATE(epoch) auto-detect seconds/ms/ns magnitude. CORR(x, y) Pearson correlation aggregate. Enables cross-asset JOINs via to_date() where crypto and equity timestamps differ by hours.
• Large response + store_as fix — the 50MB response size limit no longer blocks storage when store_as is set, since the data goes into an in-memory table, not text output.
• Empty response warnings — invalid tickers and empty date ranges now return Warning [EMPTY] with guidance instead of silent empty data.
• Leaner tool docstrings — ~50% shorter, removing duplication with Field descriptions while keeping behavioral directives.
• Fix manifest.json — query_data description was a copy-paste of call_api.

Test plan

• 615 unit/integration tests pass (22 new)
• Search benchmark: 93 endpoint queries + 10 function queries, FTS5 wins on all metrics
• LLM tool-use benchmark: tested with Gemma 4 26B (local) and Claude Code (live MCP) — branch scores higher with fewer tool calls
• Nested flattening verified with gainers, RSI, MACD, normal aggs, and tagged-record edge cases
• TO_TIMESTAMP/TO_DATE verified with epoch ms, ns, seconds, ISO-8601 strings, and NULL
• CORR verified with perfect positive/negative, zero correlation, NULLs, and insufficient data
• Large response + store_as bypass verified with mocked size limit