capitalcite

An evaluated, auditable RAG and agent system that answers questions about U.S. bank filings with citations, and fails closed when it cannot back an answer.

Banks cannot use an LLM that guesses. This repo is built around three rules.

Every claim has a source. Answers cite the filing, section, and span they came from.
Every change is measured. Retrieval quality is span-overlap recall@k against a golden set. CI fails when a tracked rate drops.
Every action goes through a gate. Agent and MCP tool calls hit a default-deny policy and an append-only audit log.

Numbers in the tables below come from JSON under evals/results/ written in this run. A cell this run could not produce is UNVERIFIED plus a reason, not a guessed value. See evals/results/7388ac3eea43-unverified.json.

Demo

Fake-model path on the fixture 10-K excerpts: chunk, index, /health, /search, /ask, agent, gate deny. Local bge-small, embedded Qdrant, ASK_MODEL=fake. No API key. No Docker.

Source cast: docs/demo/quickstart.cast. Replay with asciinema play docs/demo/quickstart.cast. Re-record with docs/demo/record.sh, which runs docs/demo/quickstart.sh.

Architecture

flowchart TB
  subgraph ingestPath [Ingest]
    EDGAR[SEC EDGAR 10-K / 10-Q] --> ingestCmd[capitalcite ingest]
    ingestCmd --> rawStore[(raw/edgar + metadata.json)]
    rawStore --> chunkCmd[capitalcite chunk<br/>section or fixed]
    chunkCmd --> chunkStore[(chunks.jsonl + document.md)]
    chunkStore --> indexCmd[capitalcite index<br/>bge-small dense + BM25]
    indexCmd --> qdrant[(Qdrant collection<br/>hybrid RRF)]
  end

  subgraph servePath [FastAPI]
    question[User question] --> searchEp[POST /search]
    question --> askRag[POST /ask path=rag]
    question --> askAgent[POST /ask path=agent<br/>LangGraph plan]
    qdrant --> searchEp
    qdrant --> askRag
    askRag --> ground[ask / ground / guard]
    askAgent --> gate{Gate.execute<br/>default deny}
    gate -->|allow| tools[search_filings, xbrl_fact,<br/>fdic_bankfind, calculator, graph_query]
    gate -->|deny| refuse[Refuse]
    tools --> ground
    gate --> audit[(hash-chained JSONL + .tip)]
    ground --> response[Answered or Refused]
  end

  mcp[capitalcite mcp stdio] --> gate
  mcp --> ground
  ground --> traces[Langfuse SDK or CAPITALCITE_TRACE_FILE]
  graphTool[optional Neo4j / MemoryGraphStore] --> tools

  subgraph evalPath [Eval and CI]
    golden[evals/golden] --> evalCmd[capitalcite eval / check]
    evalCmd --> results[evals/results + evals/baselines]
    results --> gha[GitHub Actions PR pack + nightly]
  end

  iac[infra Bicep] --> aca[Azure Container Apps IaC<br/>no live apply in CI]

MCP search_filings and ask use the same gate, audit log, and ground() path as HTTP. graph_query is a tool. Blank NEO4J_URI leaves retrieve() unchanged. There is no reranker and no PydanticAI on this path. RAGAS is not wired.

Evaluation

Golden set: 112 questions (v1). Handwritten items over the twelve ingested 10-Ks carry start/end spans in parsed document.md. FinanceBench and FinQA tails are answer-scoring only and stay out of recall@k. See evals/golden/README.md and ADR 0004.

Retrieval (per configuration)

Twelve 10-Ks, FastEmbed bge-small, span-overlap on 77 handwritten items. --small on the fixture HTML is a different condition and is not this table (7388ac3eea43-small.json, all Recall@5 = 0.000 because those excerpts do not contain the gold spans).

Chunking	Embedding	Retrieval	n	Recall@5	MRR@5	nDCG@5	Recall@10	MRR@10	nDCG@10
section	fastembed:BAAI/bge-small-en-v1.5	dense	77	0.078	0.043	0.047	0.156	0.053	0.070
section	fastembed:BAAI/bge-small-en-v1.5	bm25	77	0.182	0.089	0.103	0.286	0.101	0.133
section	fastembed:BAAI/bge-small-en-v1.5	hybrid	77	0.169	0.096	0.098	0.299	0.114	0.140
fixed	fastembed:BAAI/bge-small-en-v1.5	dense	77	0.169	0.118	0.103	0.208	0.124	0.118
fixed	fastembed:BAAI/bge-small-en-v1.5	bm25	77	0.130	0.071	0.080	0.195	0.079	0.097
fixed	fastembed:BAAI/bge-small-en-v1.5	hybrid	77	0.195	0.114	0.114	0.234	0.120	0.127
section	-	hybrid + rerank	-	UNVERIFIED	UNVERIFIED	UNVERIFIED	UNVERIFIED	UNVERIFIED	UNVERIFIED

Generated from evals/results/7388ac3eea43.json (sha 7388ac3eea43, embedder fastembed:BAAI/bge-small-en-v1.5, sparse fastembed:Qdrant/bm25:k1=1.2:b=0.75:avgdl=256.0, fastembed 0.8.1, qdrant-client 1.19.1). Noise floor: n=77, one item is 0.013 recall. RAGAS context precision/recall is UNVERIFIED (no LLM key). minilm is UNVERIFIED (a second full embed is not cheap on this corpus).

Every retrieval cell in that table comes from evals/results/7388ac3eea43.json.

Answer quality

System	Faithfulness	Answer relevancy	Numeric exact-match	Citation accuracy	Correct refusals
Plain RAG	UNVERIFIED	UNVERIFIED	UNVERIFIED	UNVERIFIED	UNVERIFIED
Agent (plan-then-act)	UNVERIFIED	UNVERIFIED	UNVERIFIED	UNVERIFIED	UNVERIFIED

OPENAI_API_KEY was empty. RAGAS and DeepEval did not run. capitalcite check --live skipped. The fake-model contract in 7388ac3eea43-check.json answered cc-ci-002 and refused cc-u-001. That is a cite/refuse contract, not a faithfulness rate.

Agent vs RAG (retrieval)

Comparison and multi-hop handwritten items. The agent row is the fake planner that searches each named ticker and year. It is not a live LLM.

System	n	Recall@5	Recall@10	Planner
Plain RAG (section/hybrid)	15	0.267	0.333	-
Agent	15	0.400	0.600	fake/heuristic planner

Comparison and multi-hop handwritten items only. Answer quality stays UNVERIFIED.

Source: evals/results/7388ac3eea43.json. --small on the fixture HTML is not this table.

Entity graph vs fixture-corpus substring

Questions that need a BHC → subsidiary or RSSD join. Not in the 77-item filing-span matrix.

System	n	Hit rate
Fixture-corpus substring	6	0.000
graph_query (memory snapshot)	6	1.000

Delta (graph minus fixture substring) = 1.000. Generated from evals/results/7388ac3eea43-graph.json. Fixture-corpus substring is 1 only when every expected token is in the --small HTML. Those files do not contain the RSSDs. These items need entity traversal. They are not in the 77-item filing-span matrix.

Metric exact match

One question, one governed metric, one quarter. The figure comes from the fixture row the pointer names. Exact match is that figure as render_figure prints it. A null cell is a refusal, scored on its own denominator.

Path	Model	n	Exact match	Correct refusals
agent	fake	18	1.000	1.000
rag	fake	18	0.000	0.333

Mart answers 0 of 56 handwritten numeric v1 items. Generated from evals/results/4447164efb28-metrics.json.

Gate and safety

Test suite	Cases	Pass rate
Disallowed tool / arg injection attempts	3	1.00
Prompt injection inside retrieved docs	2	1.00
Audit log integrity (hash chain verify)	3	1.00
CI contract pack (`capitalcite check`)	2 cite/refuse + 5 red-team	1.00

capitalcite check also gates quote-index sanity (36/77), fixture span-overlap (3/3), and the metrics suite (agent exact 15/15, agent refusal 3/3, RAG exact 0/15, RAG refusal 1/3, mart coverage 0/56) against evals/baselines/ci.json at zero item drop. Live OpenAI is --live and UNVERIFIED without OPENAI_API_KEY. Nightly live-corpus section/hybrid Recall@5 vs evals/baselines/nightly.json allows one item of rank-boundary noise. Baseline file edits need the eval-baseline label.

Agent vs RAG bench (phases and cost)

95 items: 77 handwritten v1 recall rows from evals/golden/v1.jsonl plus 18 pointer-gold rows from evals/golden/metrics.jsonl. ASK_MODEL=fake, seed 20261103, temperature 0, prompt hash 5d0ae638231240c9b774ec7d7a5f2f81e7f9ae4230ca9cca713f0e35973d1b52, tokenizer o200k_base. Fixture index, not the twelve-filing corpus. Recall@5 is not measured on that index (gold spans sit at offsets the fixture HTML does not contain). --corpus full is the twelve-filing path; it was not run here.

Arm	n	Recall@5	Accuracy	p50 ms	p95 ms
rag	95	not measured	0.053	1.2	1.6
graph	95	not measured	0.232	6.9	37.8
loop	95	not measured	0.232	4.8	36.1

graph-loop Recall@5 delta=not measured (fixture index has no gold spans) graph-loop accuracy delta=0.000 (95% CI 0.000 to 0.000). Bottleneck on the graph arm is tool. Generated from evals/results/17f203a9bcf6-bench.json.

Accuracy is 22/95 on graph and loop, 5/95 on rag. Cost per correct is 0 on the fake model. The measured bottleneck phase is tool (gate plus execute, exclusive of nested retrieve): graph mean tool 6.2 ms, retrieve 0.9 ms, model 0.1 ms, compile 0.0 ms.

The optimization in this change is compile-once. Before, run_agent compiled a LangGraph StateGraph on every question (17f203a9bcf6-bench-before.json): graph compile 2.6 ms, p50 9.7 ms. After (17f203a9bcf6-bench.json): compile 0.0 ms, p50 6.9 ms. Those compile-mean and p50 drops are the claim. The p95 change (40.6 ms to 37.8 ms) is noise unless repeated runs hold it. Answers did not change (identity_digest b4da1bafb70374e50eaf28c05e15a5e3c6cea66869b8c963174f4843e05533a1 on both files). Tool remains the largest phase. Fake-model generate time is not live gpt-4o-mini latency. Estimated tokens per graph item from this fake run: 1374.968 in, 47.905 out. The one recorded paid run is in Real-model bench (fixture corpus, one run) below. The pin is evals/bench/pin.json.

Real-model bench (fixture corpus, one run)

One hosted gpt-4o-mini run on 2026-11-10 used the fixture corpus and the 12-chunk fixture index. Results are in evals/results/aca24c13e81d-bench.json. The pin is evals/bench/pin.json. A twelve-filing corpus paid run is still UNVERIFIED.

Accuracy is not a measure of model quality. The retrieval index has 12 fixture chunks, so most items refuse. 276 of 285 arm-queries refused. 174 were no passages were retrieved. 79 were question names an entity that is not in the retrieved passages. Each arm is 9/95 (0.095). Every accurate row is a refusal. Recall@5 is not measured. The graph-loop accuracy delta is 0.000, with a 95% CI of 0.000 to 0.000.

OpenAI billed 206,221 input tokens and 10,725 output tokens across 222 chat-completion calls at $0.15 per 1M input tokens and $0.60 per 1M output tokens. That bill is $0.0374. The JSON cost_usd fields sum to $0.0690. They price reconstructed o200k_base prompt counts and the local answer or refusal string (tokens_in, tokens_out). They do not price billed usage (tokens_in_api, tokens_out_api). Those API fields in the file sum to the billed 206,221 and 10,725.

The table is the eval CLI --from-results rendering. The cost/query column is the reconstructed-token list price per query. It is not billed spend divided by 95. On rag, p50 is 1.7 ms and p95 is 1307.8 ms. On graph, p50 is 1240.6 ms and p95 is 1974.5 ms. On loop, p50 is 1228.9 ms and p95 is 2359.3 ms. Reconstructed cost per query is 0.000486 on rag, 0.000120 on graph, and 0.000120 on loop.

Arm	n	Recall@5	Accuracy	p50 ms	p95 ms	cost/query
rag	95	not measured	0.095	1.7	1307.8	0.000486
graph	95	not measured	0.095	1240.6	1974.5	0.000120
loop	95	not measured	0.095	1228.9	2359.3	0.000120

graph-loop Recall@5 delta=not measured (fixture index has no gold spans) graph-loop accuracy delta=0.000 (95% CI 0.000 to 0.000). Bottleneck on the graph arm is model. Generated from evals/results/aca24c13e81d-bench.json. This run used provider hosted, host class openai, model gpt-4o-mini, and pin pin.json.

Nightly gates the same-run graph p95 / rag p95 ratio and cost per query against evals/baselines/bench.json at a 2x band on that ratio and a 0 absolute cost budget. When compile_once is true it also fails if mean graph compile_ms is 0.5 or higher. Absolute millisecond p95 is host-dependent.

To run the same 95 items against a local Chat Completions server, start Ollama and pass the local pin. Do not pass --paid or --baseline. This command is UNVERIFIED against a live lfm3:latest server. The results file is <sha12>-bench-localhost.json. It records provider local, host_class localhost, the model id, and the pin file name. It does not store the base URL. Latency is the machine that served the model and is not comparable to hosted gpt-4o-mini or to the fake-model table above.

ollama serve
ASK_MODEL=openai \
  ASK_OPENAI_BASE_URL=http://localhost:11434/v1 \
  ASK_OPENAI_MODEL=lfm3:latest \
  uv run capitalcite eval --suite bench \
  --pin evals/bench/pin-lfm3-local.json \
  --out evals/results

The pin is evals/bench/pin-lfm3-local.json. Rates in evals/bench/prices/lfm3-local.json are zero. A non-loopback, non-OpenAI host is a third-party Chat Completions server: POST /ask can call it, the bench refuses it because the cost is unknown, and the result is not $0.

Latency and cost

Path	p50 latency	p95 latency
RAG (`ASK_MODEL=fake`)	14 ms	16 ms
Agent (`ASK_MODEL=fake`)	21 ms	21 ms
Gate `decide()`	0.018 ms	0.020 ms

RAG and agent are 10 timed POST /ask calls after one warmup on the 12-point fixture index, ASK_MODEL=fake, this 4-core VM. Median / 95th percentile. One RAG sample was 75 ms (max, not p95). The limiter is retrieve plus JSON on one process, not the gate. Live OpenAI latency and cost stay UNVERIFIED. Gate numbers are capitalcite gate bench --n 2000.

OTLP on the same fake model, measured separately, is 2.1 ms p50 and 2.5 ms p95 with tracing off, and 2.4 ms p50 and 2.8 ms p95 with an in-process OTLP receiver. The difference is 0.3 ms at p50 and 0.2 ms at p95. Per-round p50 differences were 0.29 ms and 0.38 ms. Tracing was slower in both rounds.

40 timed POST /ask calls per side after two warmup calls, two alternating rounds, one in-memory chunk, FakeEmbedder, TestClient, this 4-core VM. The timer wraps the POST. Collector flush is outside it. scripts/measure_ask_trace_overhead.py wrote the file. Hottest self time sat in anyio call with tracing off and deepcopy with OTLP on, not in retrieve.

Rewrite the retrieval and agent-vs-RAG blocks from the twelve-filing JSON:

make eval-report RESULTS=evals/results/7388ac3eea43.json

make eval-report uses evals/results/<sha12>.json for this commit when that file exists. If it is missing, the target reads evals/results/. load_reports then picks the report whose recorded git_sha matches HEAD, or the newest ancestor of HEAD. It prefers the full matrix over *-small.json. Set RESULTS= to force a file. A directory with two unrelated matrices errors. The last filename does not win.

Tradeoffs and what I'd do next

This section is the design argument. The ADRs hold the longer form.

Chunking: section vs fixed

A 10-K is a book with labeled chapters. Item 7 is MD&A. Item 8 is the financials. A CET1 question is almost always answered inside one Item, often next to a table.

A fixed window of 512 tokens with 64 tokens of overlap is easy to reproduce. It is also blind. It will cut a markdown table in half. It will start in Item 7 and end in Item 8. The overlap saves a sentence on the cut. It does not restore the Item boundary.

Section-aware splitting cuts on Item headings first. An Item that fits the budget stays one chunk. An Item that does not is split with the recursive splitter, and every piece keeps that Item name. Tables are atomic. A table that overflows is split on row boundaries with the header repeated so each piece is still valid markdown.

Both chunkers are scored on the same 77 spanned items. The label is span overlap on document.md, not the section string. n=77 means one item is 0.013 recall. A 0.02 gap is noise. This run: section/hybrid Recall@5 = 0.169, fixed/hybrid = 0.195 (7388ac3eea43.json). That is two items. Do not rank the chunkers on it. Section BM25 at 0.182 vs section dense at 0.078 is eight items and is the one gap that survives the noise floor.

Recursive splitting exists in the CLI. It is not a third default arm. That would be another re-chunk and another embed of 12k-plus points.

Next. Measure late chunking (embed bigger windows, split only the hits you show the model) against this section/fixed matrix. Keep tables atomic. Do not add a chunker that is not an eval arm.

Hybrid search: BM25, dense, RRF, no reranker

Dense search places a chunk near paraphrases. It misses a rare token the question names exactly when the HNSW walk never visits that point. BM25 does the opposite. It scores term frequency, rarity, and length. It will not find a paraphrase that shares no tokens.

Those scores are not in the same unit. A weighted sum needs a weight we have not measured. Reciprocal rank fusion ignores the raw scores and adds 1 / (k + rank) per list. RRF_K defaults to 60, the Cormack constant. Qdrant's rank index starts at 0, so a first-place hit on one leg scores 1/k. We pass Qdrant's k, not k+1.

A cross-encoder could repair a fusion miss. It also downloads another model and adds latency on every search. This repo has not measured that gain. retrieve() returns Qdrant's order truncated to k. The hybrid+rerank row stays UNVERIFIED until an eval writes it. A setting with one legal value would be a second way to say the same thing.

The first BM25 matrix in this project was hashed term counts plus IDF (TF-IDF). Section BM25 Recall@5 was 0. Production now uses FastEmbed Qdrant/bm25 (k1=1.2, b=0.75, avgdl=256). sparse_id sits inside IndexConfig so a hashed-count collection cannot be reused as BM25.

Next. One rerank arm on the same 77 items, with p95 latency next to Recall@5. If the lift is inside the 0.013 noise floor, leave rerank off.

Gate: default-deny, extra keys, audit hash chain

The planner is untrusted. ToolCall.name is a string. The policy file is the allow-list, not TOOL_NAMES in Python. Default is deny. An unknown name, extra keys on a known tool, a broken policy file, or a failed audit write never calls the runner.

Extra keys matter for MCP. The stock MCP SDK drops unknown fields before the handler runs. That would turn {"query":"x","cmd":"ls"} into an allow. This server overrides call_tool so the raw object reaches Gate.decide. invalid_args is the deny.

The audit log is a hash-chained JSONL plus a sibling .tip. Each record stores prev_hash. Genesis is 64 zero hex characters. A mutated field fails the hash. A deleted last line fails the tip. The wrapper writes the gate decision before the tool runs. A failed write is audit unavailable, not a silent skip. The chain is not HMAC-keyed. Anyone with write access can rebuild it. That is a stated limit, not an accident.

This run: 5/5 red-team cases passed (7388ac3eea43-gate.json). Intact / tamper / truncated-last verify was 3/3 (7388ac3eea43-audit.json). decide() p95 was 0.020 ms on 2000 samples.

Next. Key the chain (HMAC) or put the log in a store the API identity cannot truncate. Measure that against the current JSONL, which needs no Docker.

Agent vs RAG, and graph as a tool

path=rag is one retrieve then ask() / ground() / guard. path=agent is LangGraph plan → gated tools → the same synthesizer. The agent does not get a second way to emit an answer. A hallucinated figure after four successful searches is still a refusal.

On comparison and multi-hop items the fake planner searches each named ticker and year. That is more retrieves, not a smarter generator. This run, n=15, section/hybrid: RAG Recall@5 = 0.267, agent = 0.400 (7388ac3eea43.json). The table is retrieval only. Answer quality stays UNVERIFIED without a live LLM.

Hybrid search ranks passages that are close in wording or embedding space. It loses on identity joins: parent to subsidiary, ticker to RSSD. Reciprocal rank fusion cannot walk PARENT_OF. graph_query is a fifth tool with a typed kind enum, not Cypher from the model. Blank NEO4J_URI returns graph not configured and leaves the 77-item baseline alone. This run's graph suite is 1.000 exact on the memory snapshot vs 0.000 substring hit on the fixture HTML (7388ac3eea43-graph.json).

Next. A live planner vs the fake planner on those 15 items, with the same ground() checks. Do not fuse graph neighbors into every retrieve().

Local quickstart (fake model)

This path does not call OpenAI and does not need Docker. It chunks the checked-in fixture 10-K excerpts, indexes them with local FastEmbed bge-small into embedded Qdrant, and answers with ASK_MODEL=fake. This run timed a fresh clone of this branch at 11 seconds on a 4-core VM with warm uv and Hugging Face caches (docs/demo/quickstart-timing.log). A cold FastEmbed download of bge-small on the same VM was 67 MB in about one second. The budget is 15 minutes.

git clone https://github.com/naveed949/capitalcite && cd capitalcite
# uv: curl -LsSf https://astral.sh/uv/install.sh | sh
uv sync --frozen
cp .env.example .env

In .env, set all of the following. Leave OPENAI_API_KEY empty. ASK_MODEL is already fake in .env.example.

SEC_USER_AGENT="Your Name [email protected]"
QDRANT_PATH=data/qdrant
LANGFUSE_TRACING_ENABLED=false

QDRANT_PATH wins over QDRANT_URL. Do not use QDRANT_URL=:memory: here. The CLI and uvicorn cannot share that store.

uv run capitalcite chunk --input-dir tests/fixtures/chunking/corpus --output-dir data --chunker section
uv run capitalcite index --input-dir data --chunker section --embedder bge-small
uv run uvicorn capitalcite.api:app --host 127.0.0.1 --port 8000

In another terminal:

curl -s localhost:8000/health
curl -s localhost:8000/search -H 'content-type: application/json' \
  -d '{"query":"What CET1 ratio did JPMorgan disclose?","mode":"hybrid","k":3,"ticker":"JPM"}'
curl -s localhost:8000/ask -H 'content-type: application/json' \
  -d '{"question":"What CET1 ratio did JPMorgan disclose?","ticker":"JPM","model":"fake"}'
curl -sN localhost:8000/ask/stream -H 'content-type: application/json' \
  -d '{"question":"What CET1 ratio did JPMorgan disclose?","ticker":"JPM","model":"fake"}'
curl -s localhost:8000/ask -H 'content-type: application/json' \
  -d '{"question":"What CET1 ratio did JPMorgan disclose for 2024?","path":"agent","model":"fake"}'
uv run capitalcite gate decide --tool shell --args '{"cmd":"ls"}'

The first index downloads FastEmbed bge-small (about 67 MB) and the BM25 vocab. Later runs reuse the cache. The same commands live in docs/demo/quickstart.sh.

GET /health does not talk to Qdrant. Search and ask need the index and the same QDRANT_PATH as uvicorn.

Web app

With that fixture index built and uvicorn still on 127.0.0.1:8000, the page in web/ asks the same API. It does not add CORS. The browser posts to Next. Next forwards the body to uvicorn.

pnpm --dir web install
CAPITALCITE_API_ORIGIN=http://127.0.0.1:8000 ASK_MODEL=fake pnpm --dir web build
CAPITALCITE_API_ORIGIN=http://127.0.0.1:8000 ASK_MODEL=fake pnpm --dir web start

Open http://127.0.0.1:3001. That is the same next start path the e2e job and verify.sh use. Local editing can use pnpm --dir web dev on the same port without a build. Both listen on port 3001 so a local Langfuse stack can keep port 3000. ASK_MODEL defaults to fake when it is unset. The Next proxy writes that value into the API body and ignores a client model. When it is fake, the page says deterministic demo model before you ask.

localhost and 127.0.0.1 are different browser origins. The quickstart binds uvicorn to 127.0.0.1. Set CAPITALCITE_API_ORIGIN to http://127.0.0.1:8000, and open the page on http://127.0.0.1:3001. Do not point the origin at localhost while uvicorn is bound to 127.0.0.1. The origin is not a NEXT_PUBLIC_ variable. The browser never reads it.

A retrieval question with ticker JPM, the agent question What CET1 ratio did JPMorgan disclose for 2024?, and the Wells Fargo 2026 refusal all run on this fixture index. Fixture-index shots: answered span, agent timeline, refusal. Regenerate the client types after an API model change.

pnpm --dir web generate:api
pnpm --dir web check:api

Optional paths

Live EDGAR corpus. uv run capitalcite ingest --tickers JPM,BAC,WFC,C --forms 10-K --years 2023-2025 then chunk / index without --input-dir tests/fixtures/.... Exit 2 is UNVERIFIED (EDGAR 403/429/5xx), not a pass.

Compose Qdrant and Langfuse. docker compose up -d then leave QDRANT_PATH empty so the client dials http://localhost:6333. Compose Qdrant must be 1.16 or newer for parameterized RRF. This VM had no Docker, so that path is UNVERIFIED.

Live OpenAI. Set OPENAI_API_KEY and ASK_MODEL=openai. Without a key, model openai returns HTTP 503.

Graph. uv run capitalcite graph load --backend memory then uv run capitalcite graph query --kind subsidiaries --ticker JPM. Live Bolt needs NEO4J_URI and docker compose --profile graph. UNVERIFIED here.

MCP. uv run capitalcite mcp speaks JSON-RPC on stdio. stdout is the wire. Logs go to stderr.

Eval and CI. uv run capitalcite eval --suite golden --small (FakeEmbedder, no download). WAREHOUSE_PATH=dbt/target/ci.duckdb METRIC_QUERY_BACKEND=catalog uv run capitalcite eval --suite metrics (fake model, catalog cells). uv run capitalcite check vs evals/baselines/ci.json. Full twelve-filing matrix: QDRANT_PATH=... uv run capitalcite eval --suite golden --embedder bge-small.

Azure. make infra-validate compiles Bicep. make deploy is UNVERIFIED without az login. Design: ADR 0012. Cost and roles: infra/README.md.

Ingest design: ADR 0002. Parse and chunk: ADR 0003. Golden set: ADR 0004. Hybrid retrieval: ADR 0005. Retrieval eval: ADR 0006. Structured answers: ADR 0007. Plan-then-act agent: ADR 0008. Fail-closed tool gate: ADR 0009. Langfuse tracing: ADR 0010. OTLP tracing: ADR 0028. Audit log: ADR 0011. Azure Container Apps: ADR 0012. Regression CI: ADR 0013. Optional entity graph: ADR 0014. MCP server: ADR 0015. This README: ADR 0016. Write-up number check: ADR 0017. Eval report pick and the fake-model default: ADR 0018. Streaming answers: ADR 0021. BankFind host and metrics extract: ADR 0022. Web UI: ADR 0020.

The running decision log is docs/DECISIONS.md.

Features

Area	What it does	Status
Ingestion	Pulls bank 10-K/10-Q filings from EDGAR with a declared User-Agent and rate limits	`capitalcite ingest`
Chunking	Fixed, recursive, or section-aware (10-K Item boundaries)	`capitalcite chunk --chunker fixed\|recursive\|section`
Embedding	Local FastEmbed `bge-small` (default) and `minilm`, plus `openai-small` when `OPENAI_API_KEY` is set	`bge-small` measured; minilm UNVERIFIED
Hybrid search	BM25 and dense vectors in one Qdrant collection, fused with RRF. `POST /search` returns scores and citations	`capitalcite index`, `POST /search`
Structured answers	`POST /ask` returns a citation copied from a retrieved chunk, or a refusal. `ASK_MODEL=fake` does not read `OPENAI_API_KEY`	`POST /ask`
Streaming answers	`POST /ask/stream` is SSE of the same run. Frames reuse `PlanStep`, `TraceStep`, `Citation`, and `AskResponse`	`POST /ask/stream`
Web app	Page on port 3001. Same-origin `POST /api/ask/stream` proxies to the API. The proxy pins `model` from `ASK_MODEL`. A claim renders only with its spans	`pnpm --dir web start`
Agent	`POST /ask` `path=agent` runs LangGraph plan → tools → the same `ask()` / `ground()` / `guard`	`POST /ask` `path=agent`
Tools	`search_filings`, `xbrl_fact`, `fdic_bankfind`, `calculator`, `graph_query`. Unknown names are denied	agent tools
Fail-closed gate	JSON allow-list. Unknown names, extra keys, and a policy-load failure are denies	`capitalcite gate`
Audit log	Append-only hash-chained JSONL plus a sibling tip. Write before the tool runs	`capitalcite audit verify`
Observability	Optional Langfuse trees, a JSONL file, and OTLP HTTP when `OTEL_EXPORTER_OTLP_ENDPOINT` is set. Off without that endpoint or Langfuse keys	`observation()`
Evals in CI	`capitalcite check` on every PR. Nightly twelve-filing section/hybrid vs `evals/baselines/nightly.json`	`capitalcite check`
Deploy	Azure Container Apps Bicep. Live `/health` is UNVERIFIED until someone opts in	IaC + image recipe
MCP server	`capitalcite mcp` stdio. `search_filings` and `ask` go through the same gate	`capitalcite mcp`
Entity graph	BHC → subsidiary → filing → metric. Memory store in tests. Live Bolt UNVERIFIED without Docker	`capitalcite graph`

Data sources

Everything below is public. Each link was checked on 2026-10-16.

Source	What	Format / access	Use in this repo
SEC EDGAR APIs	Filing history and XBRL facts for every filer (JPM, BAC, WFC, C…)	JSON REST on `data.sec.gov`, no key; nightly bulk ZIPs	Main corpus (10-K/10-Q), plus ground-truth numbers for golden answers via `companyfacts`
EDGAR access rules	Fair-access policy, index files, archive paths	Max 10 req/s, declared User-Agent required	The ingester follows these rules
EDGAR full-text search	Phrase search across filings since 2001	JSON endpoint	Finding filings with specific disclosures (CET1, CECL…)
FDIC BankFind Suite API	Institutions, financials, failures, branches, Summary of Deposits	JSON/CSV REST, OpenAPI spec, no key required now	Agent tool for structured bank facts
FFIEC CDR bulk data	Call Reports and UBPR for all commercial banks	Tab-delimited or XBRL, quarterly	Structured financials, cross-checks
FFIEC NIC	Bank holding company structure and identifiers (RSSD, LEI, FDIC cert)	Web search and data downloads	Entity resolution and the Neo4j graph
FR Y-9C form and instructions	BHC consolidated report forms and long instructions PDFs	PDF	Long regulatory text for chunking experiments
Fed stress test archive	DFAST results 2013–2026	PDF/HTML	Cross-bank comparison questions
Basel Framework (BIS)	Consolidated Basel standards (CAP, RBC, LCR, DIS…)	HTML	Definitions corpus (what CET1/LCR mean)
BofA Pillar 3	Example U.S. G-SIB Pillar 3 capital and liquidity disclosures	PDF	Regulatory-disclosure documents
OCC Comptroller's Handbook	Examiner handbooks (liquidity, credit, compliance)	PDF	Compliance and policy Q&A
CFPB regulations	Consumer finance rules and recent actions	HTML/PDF	Compliance Q&A
CFPB Complaint DB · API	Consumer complaints about financial firms	CSV ZIP / JSON API	Optional: complaint triage/classification demo
FinanceBench	150 open QA pairs with evidence over 10-K/10-Q filings	HF dataset (parquet)	Calibrating the eval harness against a published benchmark
FinQA · GitHub	About 8k numerical-reasoning QA pairs over financial reports (CC-BY-4.0)	JSON	Numeric-reasoning eval subset

Terms: U.S. federal government works (SEC, FDIC, FFIEC, Fed, OCC, CFPB) are generally public domain, but SEC requires following its fair-access rules. BIS and bank-published documents keep their own copyright, so this repo links to them and does not redistribute them. Check each dataset's license on its card before redistributing derived data.

Layout

src/capitalcite/     async FastAPI app, settings, CLI, ingest, parse, chunk, retrieval, ask, agent, gate, audit, kg, mcp
tests/               pytest (recorded fixtures; no live network)
docs/DECISIONS.md    append-only index of architecture decisions
docs/decisions/      architecture decision records
docs/writeup.md      project write-up with measured numbers
docs/demo/           fake-model terminal recording and the script that produced it
evals/               golden set, baselines, and results
data/                raw and parsed filings (gitignored contents)
docker-compose.yml   Qdrant + self-hosted Langfuse + optional Neo4j (`--profile graph`)
Dockerfile           multi-stage API image (`ASK_MODEL=fake` by default)
infra/               bootstrap.bicep (identities + RBAC) and main.bicep (apps)

Tooling: uv, ruff, mypy, pytest. CI runs all three on every PR, plus capitalcite check, eval --suite metrics, and eval --small. A second workflow compiles Bicep, builds the image, and runs gitleaks. It does not log in to Azure.

Why this exists

An analyst question over a 10-K sounds simple. The wrong answer is a fluent paragraph that names a neighboring bank's ratio, last year's ratio, or a figure the model invented. This repo exists so every answer points at a span in a parsed filing, and a missing span is a refusal rather than a guess.

License

MIT (code). Third-party documents and datasets keep their own licenses.

Navigation Menu

Repository navigation

About

Repository files navigation

capitalcite

Demo

Architecture

Evaluation

Retrieval (per configuration)

Answer quality

Agent vs RAG (retrieval)

Entity graph vs fixture-corpus substring

Metric exact match

Gate and safety

Agent vs RAG bench (phases and cost)

Real-model bench (fixture corpus, one run)

Latency and cost

Tradeoffs and what I'd do next

Chunking: section vs fixed

Hybrid search: BM25, dense, RRF, no reranker

Gate: default-deny, extra keys, audit hash chain

Agent vs RAG, and graph as a tool

Local quickstart (fake model)

Web app

Optional paths

Features

Data sources

Layout

Why this exists

License

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
.cursor		.cursor
.github/workflows		.github/workflows
data		data
dbt		dbt
docs		docs
evals		evals
infra		infra
scripts		scripts
src/capitalcite		src/capitalcite
tests		tests
web		web
.dockerignore		.dockerignore
.env.example		.env.example
.gitignore		.gitignore
.gitleaks.toml		.gitleaks.toml
.nvmrc		.nvmrc
.python-version		.python-version
Dockerfile		Dockerfile
LICENSE		LICENSE
MILESTONES.md		MILESTONES.md
Makefile		Makefile
README.md		README.md
docker-compose.yml		docker-compose.yml
pyproject.toml		pyproject.toml
uv.lock		uv.lock

Latest commit

History

Folders and files

Repository files navigation

capitalcite

Demo

Architecture

Evaluation

Retrieval (per configuration)

Answer quality

Agent vs RAG (retrieval)

Entity graph vs fixture-corpus substring

Metric exact match

Gate and safety

Agent vs RAG bench (phases and cost)

Real-model bench (fixture corpus, one run)

Latency and cost

Tradeoffs and what I'd do next

Chunking: section vs fixed

Hybrid search: BM25, dense, RRF, no reranker

Gate: default-deny, extra keys, audit hash chain

Agent vs RAG, and graph as a tool

Local quickstart (fake model)

Web app

Optional paths

Features

Data sources

Layout

Why this exists

License

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages