Interview-grade benchmarking / experimentation / evaluation harness inside the Sarvam AI Cookbook.
Closes the portfolio gap the VAD challenge does not cover: RAG retrieval quality, faithfulness, hallucination traps, latency/cost, prompt A/B, and multi-model comparison.
Yes — this cookbook is a valid home. Keep product demos under examples/ and put
systematic evaluation under labs/llm-eval/ so recruiters can find both:
| Path | Story |
|---|---|
examples/ |
Ship India-first apps |
labs/llm-eval/ |
Measure models like a research / applied-AI engineer |
examples/sarvam-vad-challenge/ |
Speech ML under interview constraints |
Classical ML fundamentals live in this monorepo under mlfs/.
Sarvam product demos: sarvam-ai-cookbook.
# from repo root
.venv/bin/pip install -q scikit-learn rank-bm25 pyyaml
make lab-llm-evalThis runs:
- Ingest FAQ corpus → TF-IDF (+ optional dense) index
- Retrieval metrics: Recall@k, MRR, nDCG
- Rerank fusion
- Faithfulness / hallucination heuristics on gold + adversarial sets
- Prompt A/B
- Latency + cost estimate table
- Model comparison leaderboard (configured providers; offline stubs if no keys)
labs/llm-eval/
├── configs/models.yaml pricing.yaml experiments/
├── data/corpus/ data/gold/qa.jsonl data/gold/adversarial.jsonl
├── src/ ingest retrieve rerank generate compare report + metrics/
├── tests/
├── notebooks/llm_eval_walkthrough.ipynb
├── ui/streamlit_app.py
└── reports/ # generated JSON/MD
| Metric | Measures |
|---|---|
| Recall@k | Fraction of queries where a relevant doc is in top-k |
| MRR | Mean reciprocal rank of first relevant hit |
| nDCG@k | Rank-discounted graded relevance |
| Faithfulness | Answer claims supported by retrieved context |
| Hallucination rate | Answers invented on unanswerable / adversarial queries |
| p50/p95 latency | End-to-end retrieve+generate wall time |
| $/1k queries | Token pricing × usage |
Configured in configs/models.yaml:
OpenAI · Claude · Gemini · Qwen · Gemma · DeepSeek · Sarvam
Runs offline with deterministic stub generators when API keys are absent so CI
and local demos always work. Plug real keys via env vars to swap stubs for live calls.
| Mode | How | When |
|---|---|---|
| Stub (default) | Leave EVAL_LIVE unset |
CI, local demos, make lab-llm-eval |
| Live | EVAL_LIVE=1 + API keys |
Optional A/B against real models |
# Offline (CI-safe)
make lab-llm-eval
# Live (optional; skips providers whose keys are missing)
export EVAL_LIVE=1
export OPENAI_API_KEY=... # optional
export ANTHROPIC_API_KEY=... # optional
export GOOGLE_API_KEY=... # optional
export SARVAM_API_KEY=... # optional
make lab-llm-eval-liveLive calls use lightweight HTTP (urllib; optional httpx if installed). Provider
failures fall back to stubs so the lab stays runnable. CI must not set
EVAL_LIVE.
Gold size and labeling: DATA_CARD.md (40 QA + 15 adversarial).