Skip to content

About

Evaluation harness for financial QA systems with scoring, citations, refusal checks, and reports

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Financial LLM Evaluation Harness

Library-first evaluation harness for financial QA systems. It validates curated financial QA cases, runs them against a target adapter, scores answer quality and citation behavior, and writes machine-readable plus human-readable reports.

Why This Matters

Financial QA systems fail in quiet ways: vague answers, stale knowledge, fake citations, unsupported precision, and confident investment advice. This harness makes those failures inspectable by pairing curated cases with citation rules, refusal checks, latency/cost tracking, and CI-style gates.

Evaluation workflow

  • 50-case source-grounded quality suite across factual extraction, cited summary, multi-document synthesis, company comparison, refusal, and adversarial traps.
  • Built-in mock target for deterministic smoke tests plus a copilot-api adapter for evaluating a live research assistant endpoint.
  • Generated run artifacts include results.json, summary.md, report.html, failures.csv, and config.json.
  • Report template included for turning eval results into a readable engineering or product review.

Start with the current quality suite and its source evidence. The plumbing suite and explicit fixture replay check execution and scoring; their results do not establish model quality.

Eval Suite

The core suite lives at evals/core.yaml and currently contains 50 cases:

Category Cases Purpose
factual_extraction 15 Pull specific metrics, drivers, risks, or management statements from a filing.
cited_summary 10 Summarize a filing section with source-grounded citations.
multi_document_synthesis 10 Compare annual, quarterly, and transcript-style sources.
company_comparison 5 Compare companies across growth, margin, macro, and supply-chain exposure.
refusal 5 Check that the system refuses unsupported or non-public questions.
adversarial 5 Catch investment advice, stale knowledge, fake citation, prompt injection, and unsupported precision traps.

Quality cases separate factual expected_answer_points from behavioral judge_rubric instructions. Refusal cases have no factual answer points. Each answer source has a pinned excerpt and URL in source_evidence; cross-document cases require citations to every necessary source. See evals/evidence/ for the provenance and curation notes. evals/legacy_core_v1.yaml preserves the original instruction-based suite for inspecting historical reports; it is not a quality benchmark.

evals/plumbing.yaml is the small generated-mock smoke suite. Running the quality suite with --target mock requires an explicit, complete fixture: no expected-answer echo fallback is allowed. Fixture replay verifies scoring and artifact plumbing; it is not independent evidence of model quality. To measure a target, run copilot-api against a corpus containing the pinned source versions.

Quality cases may add required_answer_patterns for explicit factual constraints (for example, accepting “rose” or “increased” while rejecting a contrary sales direction). Missing patterns zero answer-point credit and fail the case; the failed patterns are included in diagnostics. Invalid regexes are rejected during suite validation.

The deterministic scorer checks lexical/numeric answer coverage and citation metadata; it is not a semantic entailment judge and does not independently authenticate the text a target claims to cite. Human review of retrieved evidence and answer meaning remains necessary when interpreting quality scores.

fin-eval validate-suite loads the YAML/JSON suite through Pydantic models, rejects duplicate IDs, and catches missing questions or missing expected answer points for answerable cases. fin-eval schema emits JSON schemas for eval cases, top-level suites, and generated results.json artifacts.

Install

See v0.2.0 release notes for wheel installation, source-release contents, and a clean-environment smoke test.

Use an isolated environment, then install the package and CLI:

python -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[dev]"

This exposes the fin-eval command. You can also run commands as python -m fin_eval.cli from a checkout.

Commands

Run commands from the project root:

cd financial-llm-eval-harness

Validate the suite:

fin-eval validate-suite --suite evals/core.yaml
fin-eval schema --out schemas.json

Run the plumbing suite with the built-in mock target:

fin-eval run --suite evals/plumbing.yaml --target mock --out runs/latest

Replay selected curated quality fixtures (a scorer check, not a model evaluation):

fin-eval run --suite evals/core.yaml --target mock --fixture evals/fixtures/core_factual.json --category refusal,adversarial --out runs/traps
fin-eval run --suite evals/core.yaml --target mock --fixture evals/fixtures/core_factual.json --tag nvda --limit 5 --out runs/nvda_sample
fin-eval run --suite evals/core.yaml --target mock --fixture evals/fixtures/core_factual.json --case-id nvda_10k_datacenter_revenue_001 --out runs/single

Evaluate the copilot API:

fin-eval run \
  --suite evals/core.yaml \
  --target copilot-api \
  --base-url http://localhost:8001 \
  --endpoint /research/chat \
  --timeout 20 \
  --out runs/copilot

Override CI gate thresholds:

fin-eval run \
  --suite evals/core.yaml \
  --target mock \
  --fixture evals/fixtures/core_factual.json \
  --threshold-overall 0.85 \
  --threshold-citation-precision 0.90 \
  --threshold-citation-recall 0.80 \
  --threshold-refusal-accuracy 0.95 \
  --max-error-rate 0.02 \
  --max-median-latency-ms 8000 \
  --out runs/strict

Print generated reports:

fin-eval report --run runs/latest --format markdown
fin-eval report --run runs/latest --format html
fin-eval report --run runs/latest --format json

List failures:

fin-eval list-failures --run runs/latest
fin-eval list-failures --run runs/latest --json

Compare two runs:

fin-eval compare --baseline runs/baseline --candidate runs/latest
fin-eval compare --baseline runs/baseline --candidate runs/latest --format markdown --out runs/compare
fin-eval compare --baseline runs/baseline --candidate runs/latest --gate

Run tests:

pytest

Run Outputs

fin_eval.cli run writes these artifacts to the --out directory:

File Format Contents
results.json JSON Full run payload with metadata, summary metrics, category breakdown, gate result, and per-case results.
summary.md Markdown Human-readable run report with aggregate metrics, gate violations, category metrics, failures, severe hallucination cases, slowest cases, costliest cases, and recommendations.
report.html HTML Browser-friendly version of the run report.
failures.csv CSV Failed case rows with case ID, category, difficulty, score, error, and answer.
metrics.csv CSV Flat run/category metrics for dashboards and spreadsheet checks.
cases.jsonl JSONL One scored case result per line for diffing or downstream ingestion.
config.json JSON Suite path, target, endpoint, fixture, filters, and thresholds used for the run.
manifest.json JSON Artifact inventory with file names and byte sizes.

The CLI prints a compact JSON summary to stdout and exits with code 0 when the aggregate gate passes. It exits with code 1 when the gate fails.

Important top-level results.json fields:

  • metadata: suite, target, endpoint, run size, start time, and duration.
  • summary: aggregate quality, citation, refusal, format, latency, token, cost, and error metrics.
  • category_breakdown: aggregate metrics grouped by case category.
  • gate: thresholds and any violations.
  • results: per-case score details, answer text, citations, token usage, and raw response.

Report Template

Use report_templates/eval_report_template.md when drafting a manual report or PR summary around generated run artifacts. The runtime reporter is currently implemented in fin_eval/runner.py; the template is documentation-only.

Verified Copilot citation mapping

Copilot returns document UUIDs. Supply --document-map path/to/document-map.json, a JSON object of target UUID to canonical suite document ID, or its ingestion manifest. The map is validated before any target request and recorded with file/content hashes. Unknown, missing or explicitly wrong citation IDs cannot borrow validity from labels/excerpts. Raw responses preserve original UUIDs. See public ingestion and reproduction instructions for creating the map from actual verified uploads.

The API receives only question, company_ids and top_k; case IDs, expected documents, facts, rubric and fixtures remain with the evaluator. The supplied map checks identity consistency and is not independent proof of cited evidence.

Scoring and regression provenance

Scorer v3 adds quantity normalization, local metric/topic checks for opposite or negated quantities, quote-aware refusal recognition, and finite unsupported advice/target/guarantee/forecast phrase rules. Correct facts followed by recognized unsupported assertions fail. Currency word/symbol forms and scaled amounts are normalized consistently. These checks remain lexical heuristics with false positives and false negatives; contextual rubrics and semantic review are reported as not performed. See precise limitations.

Scorer v3 retains the v2 separation of execution status (error, empty, answered) from refusal behavior. Errors and empty answers fail the case and have behavior_evaluated=false, refused=null, and refusal_correct=null. They are not hallucinated claims. Refusal accuracy is correct behavior divided by nonempty, error-free responses; behavior_evaluated_cases and behavior_unavailable_cases expose that denominator. It is 0 when no response is evaluable, preventing an outage from passing the refusal gate. Error rate still uses all attempted cases. Other aggregate quality scores retain all attempted cases in their denominators.

Runs record scorer_version, the complete case definition and its SHA-256 fingerprint. compare --gate requires identical case IDs, matching case fingerprints, and matching nonempty scorer versions. Added, removed, and changed cases are reported separately; omitted or changed failures are never called fixed. Incompatible or legacy unversioned runs return comparable=false, no aggregate deltas, and a failed regression gate. An empty selection is rejected before an adapter runs.

Changing source targets, case wording, evidence, or scorer semantics requires a new baseline; retain the old artifacts. Replaying archived responses with a new scorer is a rescore, not a fresh model run, and should preserve execution timestamps alongside a separate rescoring timestamp.

Examples and history

The historical mock report excerpt and v1 mock run card preserve June 2026 output from the original instruction-based suite and scorer. They illustrate the earlier report format; their metrics are not results for the current factual quality suite.

A self-contained public comparison workflow and methodology cover exact source fetching, production ingestion, fresh local API/baseline captures and redacted export. New captures use scorer v3 and require a new baseline. Historical examples retain their original scores; this workflow does not regrade them.

About

Evaluation harness for financial QA systems with scoring, citations, refusal checks, and reports

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages