LSP token-cost eval

Single-trial probe for the M-AILANG-LSP-FOR-AI milestone: measures the input+output token delta when an AI agent performs a fixed cross-module AILANG editing task with the ailang-lsp plugin enabled vs. disabled.

What this is: a real number with caveats — proof that the LSP changes token consumption in the predicted direction. What this is NOT: statistically robust. N=1, single task, single model. Multi-trial / multi-task / multi-model is M-LSP-EVAL-FOLLOWUP.

Files

File	Purpose
task.md	The fixed prompt the agent receives (add subtract + use_subtract to the xref fixture).
run.sh	Driver: runs the task twice (lsp_on / lsp_off), captures token usage, writes results.json.
results.json	Output. Generated by run.sh. Not committed — re-run to regenerate.

Running

# Prerequisites:
make install                              # ailang on PATH
/plugin install ailang-lsp@ailang-tools    # in your Claude Code session

# Then:
bash bench/lsp_token_cost/run.sh

The script seeds a fresh temp workspace per trial (so neither trial sees the other's edits), runs claude --print --output-format json headlessly, and extracts input_tokens / output_tokens / duration_ms / num_turns from each trial's response. Final delta is written to results.json.

Interpreting results

The delta.pct_reduction field is the headline number. Positive values mean the LSP saved tokens (the predicted direction). Watch for:

• Negative or near-zero delta: the task may not have triggered the navigation patterns the LSP optimizes (grep / read-whole-file / re-run-check). Try a more navigation-heavy task.
• num_turns smaller with LSP-on: typical signal — the agent makes fewer round trips because it can answer "what's the type of X" in one structured request.
• duration_ms larger with LSP-on: typical too, especially on first start (LSP server boot + initial typecheck). The token saving usually outweighs the wall-clock cost.

Limitations

• Single task means we're measuring this task's particular pattern, not "AILANG navigation in general".
• Single trial means run-to-run variance is invisible. Re-run a few times to eyeball stability.
• The claude --print headless mode doesn't perfectly replicate an interactive Claude Code session — the actual savings in real use may differ.
• Both trials write to a fresh temp workspace, so the LSP's own indexing cost gets amortized over a single small task. On a large workspace the LSP startup is a one-time cost spread across many later requests.

Follow-up

M-LSP-EVAL-FOLLOWUP (filed in the design doc's "Out of Scope" section) tracks a proper N≥10 multi-task multi-model eval that produces a CI-checkable token-cost regression bound.