There are a ton of large language models out there now. GPT-4, Claude, Gemini, LLaMA, Mistral… the list keeps growing.
And let’s be honest — they all sound pretty great in their announcements. But which one’s actually smart? Which one’s good at math? Or code? Or languages other than English?
That’s where benchmarks come in.
This post breaks down how LLMs are tested, which benchmarks matter, what the scores mean, and how you can use all this to figure out which model fits your needs.
Why We Even Need Benchmarks
Back when we had just GPT-3 or GPT-4, it was easy to know what the “best” model was.
Now? Everyone’s got a “state-of-the-art” model. So we need a way to compare them fairly.
Benchmarks are basically tests — sets of questions or tasks that we run every model through to see how they perform. Think of them like school exams for AIs.
Some focus on general knowledge. Others test math or code. Some are in English only, others are multilingual. A few even test how well models handle images or audio.
They’re not perfect, but they’re the best tools we’ve got to cut through the marketing and see what a model is actually good at.
Key Benchmarks for Text Models
Here are some of the most common benchmarks for text-based LLMs — what they test, what good scores tell us, and how it applies in the real world.
MMLU
- What it tests: Academic and professional knowledge across 57 subjects — from US history to electrical engineering.
- Why it matters: It shows how broadly a model “knows stuff.”
- Real-world use: If you’re building a study tool or internal knowledge assistant, high MMLU performance means the model might actually know what it’s talking about across topics.
- Good to know: Humans score around 90%. GPT-4 scores in the high 80s [1].
- Caveat: Models might see some of this data during training, so scores can be inflated unless carefully controlled.
GSM8K
- What it tests: Grade school-level math word problems.
- Why it matters: It shows if a model can reason through multi-step logic, not just memorize answers.
- Real-world use: Useful for things like budgeting tools, supply chain helpers, or any scenario where step-by-step math reasoning is needed.
- Good to know: GPT-4 crushes this. Earlier models like GPT-3.5 struggled. Chain-of-thought prompting helps [2].
ARC
- What it tests: Grade-school science and commonsense questions.
- Why it matters: Tests simple reasoning and basic science facts.
- Real-world use: If you’re building educational apps for younger users or need solid commonsense responses in your chatbot, this matters.
- Good to know: Not as famous as MMLU, but still useful. Strong models ace the “Easy” set and do well on “Challenge” [3].
HumanEval
- What it tests: Code generation. Can the model write correct Python functions from a description?
- Why it matters: If you’re building with LLMs for dev tools, this one’s a must.
- Real-world use: High scores mean your model can assist with bug fixing, automate code generation, or review pull requests.
- Good to know: GPT-4 scores around 68% pass@1. GPT-3.5 sits way lower. Some newer models claim to beat GPT-4 here [4].
How Models Handle Other Languages
Most benchmarks are in English. But the real world isn’t.
So: how well do these models perform in other languages?
Answer: depends on the model, the language, and the benchmark.
Top-tier models like GPT-4 do surprisingly well in many languages. For example, it scored basically the same in Polish as it did in English on a medical exam — almost 80% [5].
Lower-end or smaller models often fall apart once you leave English. In one benchmark, a few open models completely failed simple Hungarian questions. One got nearly 0% [6].
Real-world use: If your company operates in a non-English market — say, building a legal assistant for Hungarian lawyers — this kind of multilingual test is critical.
Also: training data matters more than model size here. A smaller model with good multilingual training can beat a larger English-only one.
Vision Benchmarks: How Image-Ready Are These Models?
Multimodal models like GPT-4V, Gemini, and Claude 3 can take images as input. That’s cool. But can they actually understand what they see?
Here’s how we test that.
VQAv2
- What it tests: Simple Q&A about images. “What’s the person doing in this photo?” etc.
- Real-world use: Customer support tools that let users upload images of a broken device. A good VQAv2 score means the model might actually help troubleshoot.
- Good to know: GPT-4V scores around 77% without fine-tuning [7]. Human-level is ~80%+.
MMMU
- What it tests: University-level questions that involve reading charts, diagrams, and other visuals.
- Real-world use: Think data analysis, business dashboards, or technical diagrams in product manuals.
- Good to know: GPT-4V scores around 56%. Tough benchmark. Shows how hard real visual reasoning still is [8].
MathVista
- What it tests: Visual math — geometry diagrams, plots, etc.
- Real-world use: Education tech, math tutoring, or any task where charts and numbers are shown together.
- Good to know: GPT-4V leads here too (~50%) but even that’s below human performance [9]. Most other models don’t come close.
Bottom line: vision is still a weak spot, especially for tasks that require reasoning. The models can “see,” but they’re not yet great at thinking through what they see.
Audio Benchmarks: Can They Listen?
Some models (like Whisper, or the new GPT-4o) handle audio. Here’s how we measure that.
WER (Word Error Rate)
- What it tests: Speech-to-text accuracy.
- Real-world use: Transcription, voice search, meeting notes — anything where people talk and the model has to understand.
- Good to know: Lower is better. Whisper hits 1.8% WER on clean English audio — better than most human transcribers [10].
Other metrics for audio:
- BLEU: Used when testing translation from speech (e.g., English audio → Spanish text).
- Intent accuracy: Used for voice assistants — did the model understand what the user meant?
Multilingual speech is still a challenge, especially in noisy or accented recordings. But top models are getting better fast.
Where to Compare Models
Benchmarks are great, but leaderboards make it easier to compare.
Chatbot Arena (LMSYS)
You chat with two models side by side. You vote. Rankings are based on Elo scores and win rates.
- Good to know: GPT-4 dominates here. But some open models are getting close [11].
Hugging Face Open LLM Leaderboard
Fully benchmarked scores on a fixed suite of tasks (MMLU, GSM8K, ARC, etc.).
- Good to know: Great for open models. Closed ones (like Claude or Gemini) don’t appear here [12].
HELM (Stanford)
More than just accuracy. Tracks calibration, robustness, fairness, toxicity, and multilingual ability.
- Good to know: Good for digging into how models succeed or fail, not just raw scores [13].
Common Metrics (And What They Mean)
Here’s a quick guide to LLM metrics you’ll see on leaderboards:
- Accuracy: % of correct answers on a task. Easy to understand. Higher = better.
- BLEU: Measures how close a generated sentence is to a reference (used in translation).
- WER: For audio. How many words were transcribed wrong. Lower = better.
- Win Rate: How often a model is preferred over another in a head-to-head.
- MT-Bench: A 0–10 chatbot quality score, often judged by GPT-4.
- Elo Rating: Chess-style score based on win/loss records across battles. More stable over time than win rate.
- Robustness: Does the model still perform when questions are paraphrased, or slightly altered?
- Hallucination Rate: How often the model makes stuff up. Lower is better. Some top models are now under 1% in summarization tasks [14].
Is the Model Actually Smart — Or Just Well-Trained?
This is the million-dollar question. If a model scores well on a benchmark, does that mean it’s “intelligent”? Or did it just memorize the answers?
In short: we don’t know for sure.
A model could absolutely get high scores by memorizing questions seen during training. That’s why some benchmarks rotate test sets or hold out certain questions. But even then, it’s hard to say if the model is solving problems or just pattern matching at a higher level.
What we can do is look for:
- Generalization: Can it answer new questions that weren’t in training data?
- Consistency: Does it still perform well when you reword or tweak the prompt?
- Reasoning steps: If it explains how it reached an answer, does the logic check out?
Benchmarks help, but they’re not the full story. For now, LLMs are great at seeming smart — and in many cases, that’s enough. But truly measuring intelligence? That’s still open research.
Final Thoughts
Benchmarks are the only reliable way to tell what an LLM is actually good at.
They’re not perfect, and they can be gamed. But until someone invents a universal IQ test for AI, this is the best we’ve got.
If you’re building something serious — especially where accuracy, code, or non-English support matters — dig into the benchmarks before you choose a model.
Sources
[1] https://arxiv.org/abs/2303.08774
[2] https://openai.com/research/gpt-4
[3] https://allenai.org/data/arc
[4] https://openai.com/blog/code-interpretation
[5] https://arxiv.org/abs/2303.07281
[6] https://huggingface.co/spaces/HUNGPT/hungarian-benchmark
[7] https://openai.com/index/gpt-4o
[8] https://mmmu.org
[9] https://mathvista.github.io
[10] https://openai.com/research/whisper
[11] https://chat.lmsys.org
[12] https://huggingface.co/spaces/HuggingFaceH4/open-llm-leaderboard
[13] https://crfm.stanford.edu/helm/latest/
[14] https://vectara.com/hallucination-leaderboard/
