Measures how different documentation formats affect LLM token consumption and answer quality, using a real-world technical quickstart as the test document.
Four representations of the same page — Quickstart: Ingest custom metrics with EDOT — are evaluated side by side:
| Format | Description |
|---|---|
annotated_md |
Elastic's source Markdown with frontmatter and custom directives (<stepper>, <step>) |
doclang |
DocLang XML conversion — semantic tags, procedure steps, code blocks |
html |
Main-content HTML extracted from the live web page |
plain_text |
Stripped baseline — no markup, just the words |
- Token consumption — input tokens for each format (counted locally with
tiktoken) - Answer accuracy — 10 Q&A tasks scored 0–2 by an LLM judge
- Token efficiency — tokens per correct point (lower is better)
Tasks span five categories: factual retrieval, code extraction, troubleshooting, navigation, metadata, reasoning, and comprehension. Each answer is judged against a ground-truth string on a 0–2 rubric.
Trigger manually from the Actions tab → Doc Format Benchmark → Run workflow.
No secrets to configure — the workflow uses GITHUB_TOKEN with GitHub Models (Copilot). Requires Copilot access enabled for the organization.
Results are posted as a GitHub issue labelled benchmark.
pip install -r benchmark/requirements.txt
GITHUB_TOKEN=<your-pat> python benchmark/run.py --output results.json
python benchmark/report.py results.jsonEach run produces a results.json artifact and opens a GitHub issue like this:
[Benchmark] doc-format · gpt-4o-mini · 2026-06-16
The issue contains token counts, accuracy tables, per-task scores, and judge reasoning.
benchmark/
tasks.py 10 evaluation tasks with ground-truth answers
formats.py Fetchers and converters for all four formats
judge.py LLM answering + tiktoken counting + LLM judging
report.py Markdown report generator
run.py Main orchestrator
.github/workflows/
doc-format-benchmark.yml workflow_dispatch trigger