llm-compare

A CLI for comparing multiple LLM responses side by side in one terminal run.

llm-compare is a Node.js CLI for sending the same prompt to multiple LLM models in parallel and comparing their outputs, latency, token usage, and estimated cost in one terminal view.

Features

• ⚡ Run the same prompt across multiple providers in parallel from one CLI command.
• 🧩 Compare response text, latency, token usage, and estimated cost side by side.
• 🛡️ Isolate failures per model so one timeout or auth error does not break the full run.
• 🎯 Target all configured models or narrow execution with repeated -m provider/model selectors.
• 🔐 Keep API keys out of config files by referencing environment variable names instead.
• 🧾 Turn on --verbose when you want raw provider responses for debugging.

Install

npm install
npm run build
npm link

Then run:

llm-compare "Explain eventual consistency in one paragraph."

Usage

Use every configured model:

llm-compare "Summarize the CAP theorem."

Use specific models only:

llm-compare -m openai/gpt-5.1 -m anthropic/claude-sonnet-4-6 "Summarize the CAP theorem."

Print raw provider responses:

llm-compare --verbose "Give me three naming suggestions for a TypeScript utility library."

Example output:

openai/gpt-5.1
Latency | 1421ms
Tokens | 24 -> 118 (total 142)
Est. cost | $0.001210

Eventual consistency is a distributed systems model where replicas may temporarily disagree after a write, but converge to the same value over time.
================================================================================
anthropic/claude-sonnet-4-6
Latency | 1312ms
Tokens | 24 -> 96 (total 120)
Est. cost | $0.001512

Eventual consistency means different nodes can return different answers right after an update, but the system is designed so those differences disappear as replication catches up.
================================================================================
google/gemini-2.5-flash
Error | Rate limit exceeded: quota exhausted for the current project
Latency | 287ms

Architecture

Each provider implements the shared ProviderClient interface, so the CLI can treat OpenAI, Anthropic, and Google behind the same request contract.
runComparison dispatches all selected model calls concurrently with Promise.all, which keeps comparisons fast and aligned to the same prompt.
Every model result is captured independently, so one timeout, auth issue, or rate limit does not crash the whole run.
Pricing is loaded from src/pricing.json, which keeps model cost data outside the request logic and makes pricing updates straightforward.

Config

The config file lives at ~/.llm-compare/config.yaml.

On first run, the CLI creates a default config and prints its path. API keys are never stored in the config. Each model references the environment variable that contains its API key.

If you already have a config file at ~/.llm-compare/config.yaml from a previous version, it will NOT be overwritten. Delete it or update it manually to use the latest default models.

Example:

providers:
  - id: openai
    models:
      - id: gpt-5.1
        envKey: OPENAI_API_KEY
      - id: gpt-5-mini
        envKey: OPENAI_API_KEY
  - id: anthropic
    models:
      - id: claude-sonnet-4-6
        envKey: ANTHROPIC_API_KEY
      - id: claude-haiku-4-5
        envKey: ANTHROPIC_API_KEY
  - id: google
    models:
      - id: gemini-2.5-flash
        envKey: GOOGLE_API_KEY
      - id: gemini-2.5-flash-lite-preview-09-2025
        envKey: GOOGLE_API_KEY

Set the required environment variables before running the CLI:

export OPENAI_API_KEY=...
export ANTHROPIC_API_KEY=...
export GOOGLE_API_KEY=...

Tooling

npm run build
npm run lint
npm run check-format

Add A Provider

1. Add a new file under src/providers/ implementing the shared interface from src/providers/types.ts.
2. Register the provider in src/providers/index.ts.
3. Add the provider ID to the config validation list in src/config.ts.
4. Add pricing entries to src/pricing.json so estimated cost can be calculated.