Jump to content

Humanity's Last Exam

From Wikipedia, the free encyclopedia

Humanity's Last Exam (HLE) is a language model benchmark consisting of 2,500 questions across a broad range of subjects. It was created jointly by the Center for AI Safety and Scale AI and publicly released in January 2025.

Creation

[edit]

Stanford HAI's AI Index 2025 Annual Report cites Humanity's Last Exam as one of the "more challenging benchmarks" developed in response to the popular AI benchmarks having reached "saturation".[1] The test has been described as the brainchild of Dan Hendrycks, a machine learning researcher and the director of the Center for AI Safety, who stated that he was inspired to create the test after a conversation with Elon Musk, who thought the existing language model benchmarks, such as the MMLU, were too easy. Hendrycks worked with Scale AI to compile the questions.[2] The questions were crowdsourced from subject matter experts from various institutions across the world.[3][4] The questions were first filtered by the leading AI models; if the models failed to answer the question or did worse than random guessing on the multiple-choice questions, they were reviewed by human experts in two rounds and approved for inclusion in the dataset. The submitters of the top-rated questions were given prize money from a pool of 500,000 U.S. dollars—$5,000 for each of the top 50 questions and $500 for the next 500. After the initial release, a "community feedback bug bounty program" was opened to "identify and remove major errors in the dataset".[4]

Composition

[edit]

The benchmark consists of 2,500 questions in the publicly released set. The questions "typically require graduate-level expertise or test knowledge of highly specific topics." The paper classifies the questions into the following broad subjects: mathematics (41%), physics (9%), biology/medicine (11%), humanities/social science (9%), computer science/artificial intelligence (10%), engineering (4%), chemistry (7%), and other (9%). Around 14% of the questions require the ability to understand both text and images, i.e., multi-modality. 24% of the questions are multiple-choice; the rest are short-answer, exact-match questions. A private set is also maintained to test for benchmark overfitting.[4]

An example question:[2]

Hummingbirds within Apodiformes uniquely have a bilaterally paired oval bone, a sesamoid embedded in the caudolateral portion of the expanded, cruciate aponeurosis of insertion of m. depressor caudae. How many paired tendons are supported by this sesamoid bone? Answer with a number.

An independent investigation by FutureHouse, published in July 2025, suggested that around 30% of the HLE answers for text-only chemistry and biology questions could be incorrect; the benchmark's team partially replicated the findings, and said they hope to institute a continuous revision process.[5] The team subsequently launched HLE-Rolling, a version of the benchmark that is "regularly updated to address community feedback and integrate new questions".[4]

On 22 September 2026, HLE-Diamond, a "refined subset of HLE" consisting of 1,000 questions, was launched.[6]

Results

[edit]
Performance of various models on the HLE-Diamond subset
Organization Model Accuracy (%)
AnthropicClaude Fable 5.280.2
OpenAIGPT-6.2 Astra72.2
Google DeepMindGemini 4 Neon71.3
Alibaba CloudQwen4-Max44.9
Meta Superintelligence LabsMuse Wonder39.0
Moonshot AIKimi K3.1 Pro37.3
TencentHy432.7
Thinking Machines LabInkling 2 Small28.7
MiniMaxMiniMax-M3.1 Flash28.1
DeepSeekDeepSeek-V4.1-Flash-101725.3
xAIGrok 4.723.2
Z.aiGLM-5.317.9
XiaomiMiMo-V2.6-Pro15.2
NvidiaNemotron 3.5 Super VL8.8
Source: Artificial Analysis. 19 November 2026.

References

[edit]
  1. ↑ Maslej, Nestor; et al. (April 2025). The AI Index 2025 Annual Report (PDF) (Report). Institute for Human-Centered AI. pp. 141–142.
  2. 1 2 Roose, Kevin (23 January 2025). "When A.I. Passes This Test, Look Out". New York Times. Retrieved 24 January 2025.{{cite web}}: CS1 maint: deprecated archival service (link)
  3. ↑ Dastin, Jeffrey; Paul, Katie (16 September 2024). "AI experts ready 'Humanity's Last Exam' to stump powerful tech". Reuters. Retrieved 24 January 2025.{{cite web}}: CS1 maint: deprecated archival service (link)
  4. 1 2 3 4 Center for AI Safety; Scale AI; HLE Contributers Consortium (2026). "A benchmark of expert-level academic questions to assess AI capabilities". Nature. 649: 1139–1146. doi:10.1038/s41586-025-09962-4. PMC 12851929.
  5. ↑ Skarlinski, Michael; Laurent, Jon; Bou, Albert; White, Andrew (16 September 2025). "About 30% of Humanity's Last Exam chemistry/biology answers are likely wrong". FutureHouse. Retrieved 15 October 2025.
  6. ↑ Center for AI Safety; Scale AI (22 September 2026). "Introducing HLE-Diamond". Humanity's Last Exam. Retrieved 11 November 2026.
[edit]