METR (@METR_Evals) / X

METR

612 posts

METR

@METR_Evals

We work to scientifically measure whether and when AI systems might threaten catastrophic harm to society. Nonprofit.

Berkeley, CA

metr.org

Joined September 2023

Following

26.9K

Followers

Pinned
METR
@METR_Evals
May 19
Could an AI company lose control of its own agents? To find out, Anthropic, Google, Meta, and OpenAI let us (1) test their best internal models with CoT access, (2) review non-public info about capabilities, alignment, and control. The result: our first Frontier Risk Report.
364K
METR reposted
Parv Mahajan is at ICML!
@parvmahajan0
Jul 6
I‘ll be repping @METR_Evals at ICML, and we're hiring! Excited to talk to folks about: - AI R&D Evals - Measuring research taste - Saving time horizon - How to collect tens of thousands of hours of human data in a couple months DMs are open :)
5.8K
METR reposted
Bruce W. Lee ✈️ ICML Seoul
@BruceWLee2
Jul 2
I joined @METR_Evals to help build out their monitorability effort. Our mission is to measure and inform the world about AI covert capability. I'll be at ICML next week. If you work on monitoring or AI Control, let's chat in person. My DMs are open! I’m curious about …🧵
8.4K
METR reposted
Chris Painter
@ChrisPainterYup
Jul 1
My METR colleague @slimshetty_ has an interesting post exploring the nature of improvements to NanoGPT Speedruns over time. I find the history of speedruns in general fascinating. One thing that stands out is the cumulative effect of relatively shallow contributions.
8.1K
METR reposted
Michal Bravansky ✈️ ICML
@michalbravansky
Jun 30
I recently joined @METR_Evals. I'm studying AI R&D automation, with a current focus on measuring human uplift. If you're in the Bay Area or heading to ICML next week, I'd love to connect!
4.7K
METR reposted
david rein
@idavidrein
Jun 26
Excited to have this (fully) out! While the best models already do very well on it, I think it can still be a useful platform to study inference compute efficiency, and various kinds of generalization (e.g. across programming languages).
Epoch AI
@EpochAIResearch
Jun 26
What are the largest software engineering tasks AI can perform? To answer this, we built MirrorCode, our long-horizon SWE benchmark that lets AI code autonomously for days at a time. The best models complete some tasks we estimate would take human engineers several weeks.
00:00
6.4K
METR reposted
Charles Foster
@CFGeek
Jun 27
Replying to @fleetingbits
My understanding is that attempted cheating is now very common in both training and evaluations, but occurs most often in highly-autonomous, eval-like settings, especially on the hardest tasks.
3.1K
METR reposted
Mischa Spiegelmock
@spiegelmock
Jun 27
@METR_Evals and @EpochAIResearch ran complex agentic SWE evaluations spanning many days and gigabytes of compressed transcripts. This challenged us to significantly optimize many aspects of our eval platform data format, transcript ingestion, web viewer, and database.
Epoch AI
@EpochAIResearch
Jun 26
What are the largest software engineering tasks AI can perform? To answer this, we built MirrorCode, our long-horizon SWE benchmark that lets AI code autonomously for days at a time. The best models complete some tasks we estimate would take human engineers several weeks.
00:00
6.9K
METR
@METR_Evals
Jun 26
OpenAI gave METR early access to GPT-5.6 Sol for testing including raw chain-of-thought, a railfree version of the model, and internal information about the model. With this access, METR conducted a pre-deployment evaluation of GPT-5.6 Sol, including an attempted measurement of
OpenAI
@OpenAI
Jun 26
Introducing a limited preview of GPT-5.6 Sol, our next generation frontier model, as well as GPT-5.6 Terra, a balanced model for efficient, everyday work, and GPT-5.6 Luna, a fast and affordable model for high-volume work. openai.com/index/previewi…
581K
METR
@METR_Evals
Jun 26
Replying to @METR_Evals
If future models display much fewer undesirable propensities, we could become more concerned about catastrophic misalignment, as we’d be worried that models may have learnt to evade detection (for example, as a result of being trained not to produce misaligned reasoning).
25K
METR
@METR_Evals
Jun 26
You can find additional information about our pre-deployment evaluation of GPT-5.6 Sol on our website:
Summary of METR's predeployment evaluation of GPT-5.6 Sol
From metr.org
25K
METR reposted
tom cunningham
@testingham
Jun 25
Here's an updated survey of definitions of Recursive Self-Improvement (with visualization). Thanks to people who had useful suggestions of sources I was missing.
tom cunningham
@testingham
Jun 8
A list definitions of RSI and associated concepts (there are a lot!). It's mostly agent-written. Tell me if I'm misquoting or missing any: tecunningham.github.io/posts/2026-06-…
4.9K
METR reposted
tom cunningham
@testingham
Jun 8
A list definitions of RSI and associated concepts (there are a lot!). It's mostly agent-written. Tell me if I'm misquoting or missing any:
tecunningham.github.io
Definitions of Recursive Self-Improvement | Tom Cunningham
Tom Cunningham blog
14K
METR reposted
david rein
@idavidrein
Jun 7
To evaluate our monitoring systems, we typically rely on a pretty strong assumption that we've given agents a concrete, narrowly scoped task. But we're going to give agents increasingly abstract/high-level tasks as they get better. This means (to the extent they’re misaligned)
david rein
@idavidrein
May 9
Frontier AI labs like OpenAI and Anthropic have started using automated LLM monitoring systems to make sure their internal AI agents stay aligned [1, 2]. I think this is really important and good! Unfortunately, it's difficult to get perfect coverage with these systems, and there
11K
METR reposted
tom cunningham
@testingham
Jun 1
A very big picture of what's happening with AI (slides from a talk at UCSB):
13K