Log inSign up
Pareto AI
78 posts
user avatar
Pareto AI
@pareto_ai
People improve models, models lift people. We advance the signal that keeps the loop turning.
San Francisco, CA
pareto.ai
Joined October 2019
8
Following
3,064
Followers

New to X?

Sign up now to get your own personalized timeline!

Create account

By signing up, you agree to the Terms of Service and Privacy Policy, including Cookie Use.

Terms·Privacy·Cookies·Accessibility·Ads Info·© 2026 X Corp.
Don't miss what's happening
People on X are the first to know.
Log inSign up
  • Pareto AI reposted
    user avatar
    Phoebe Yao
    @phoebeyao
    Jun 24
    just ran GPT 5.5, Gemini 3.5 Flash, and GLM 5.2 through LEAPBench GLM 5.2 beat every frontier model evaluated except Opus 4.7 and 4.8
    GIF
    user avatar
    Phoebe Yao
    @phoebeyao
    Jun 23
    Today we’re releasing LEAPBench, a benchmark for how efficiently models learn from feedback over many rounds. Most evals only score whether a model eventually reaches a good solution. LEAPBench scores how many tries it took to get there. Two systems can reach the same optimum
    6.4K
  • Pareto AI reposted
    user avatar
    Phoebe Yao
    @phoebeyao
    Jun 23
    Today we’re releasing LEAPBench, a benchmark for how efficiently models learn from feedback over many rounds. Most evals only score whether a model eventually reaches a good solution. LEAPBench scores how many tries it took to get there. Two systems can reach the same optimum
    GIF
    24K
  • user avatar
    Pareto AI
    @pareto_ai
    Jun 4
    We added Claude Opus 4.8 to AttuneBench. It takes the top overall score (54.55), ahead of Opus 4.6 (54.30) and GPT-5.5 (53.71). It's also the strongest model we've tested at predicting users' emotion labels and overall conversation ratings.
    1.3K
    user avatar
    Pareto AI
    @pareto_ai
    Jun 4
    The Opus family now holds all three top spots for predicting which response a user would prefer. But the dissociation we reported in our paper remains. Opus 4.8 improved on predicting what users wanted from individual replies (76.7% → 78.3%), yet still ranks near the bottom of
    1K
    user avatar
    Pareto AI
    @pareto_ai
    Jun 4
    Opus 4.7, Opus 4.8, and Sonnet 4.6 all score effectively zero on identifying what specifically went wrong when a conversation missed a user's needs. Opus 4.6 and Haiku 4.5 do not show the same pattern. Predicting that a conversation failed and identifying why it failed appear to
    597
  • Pareto AI reposted
    user avatar
    faisal ⁂
    @faisal_sayed05
    May 28
    Opus 4.8 outperforms every other model on AttuneBench - best at picking the response humans actually preferred - biggest MSCEIT four-branch jump of any Opus generation - entire pairwise top-4 is now Anthropic models. non-Anthropic frontiers stall ~50%
    Benchmark table showing how Claude Opus 4.8 compares to its predecessor and to other models on tests of coding, agentic skills, reasoning, and practical knowledge work tasks.
    user avatar
    Claude
    Anthropic
    @claudeai
    May 28
    Introducing Claude Opus 4.8: it builds on Opus 4.7 with sharper judgment, more honesty about its own progress, and the ability to work independently for longer than its predecessors. Available today at the same price.
    7.6K
  • Pareto AI reposted
    user avatar
    Phoebe Yao
    @phoebeyao
    May 27
    1/ Today we're releasing AttuneBench, the first open EQ benchmark grounded in real multi-turn human-model conversations, scored against what the person actually felt and wanted at each turn. Built by the research team at @pareto_ai in collaboration with @thoughtfullab. Most
    22K
  • Pareto AI reposted
    user avatar
    Mark Whiting
    @MarkWhiting
    Apr 1
    Do LLMs have metacognition? It is complicated, but mostly no — check out work from my team at @pareto_ai
    user avatar
    Phoebe Yao
    @phoebeyao
    Apr 1
    model confidence tracks a shared model-agnostic signal for fact recall, not true self-knowledge. we tested metacognitive confidence across 19 frontier models on a closed-book SQuAD task. f1 scores look reasonable (0.6–0.8), but confidence and accuracy are nearly uncorrelated
    1.5K
  • Pareto AI reposted
    user avatar
    Phoebe Yao
    @phoebeyao
    Apr 1
    Article cover image
    Article
    LLM Metacognition: Shared and Shallow?
    Across 19 frontier models, metacognitive confidence on question and answer tasks tracks a shared difficulty heuristic with only a weak relationship to actual performance. Do models know what they...
    5.6K
  • user avatar
    Pareto AI
    @pareto_ai
    Mar 31
    There's a hidden layer in AI: the systems that decide who gets to shape the model. Not all contributors are equal but most pipelines treat them like they are. Who's feeding the model in the first place takes quietly a big part in determining everything.
    511
  • user avatar
    Pareto AI
    @pareto_ai
    Mar 30
    LLMs don't fail because they lack information. They fail because they learn from unverified judgment. That's a data problem, not a model problem.
    420
  • user avatar
    Pareto AI
    @pareto_ai
    Mar 27
    The next bottleneck in frontier AI is verification.
    499
  • user avatar
    Pareto AI
    @pareto_ai
    Mar 5
    Frontier models are trained to answer. But in high-stakes domains, the safer behavior is often to pause, express uncertainty, and ask for more context instead of acting like a doctor who never asks. Great work from @phoebeyao, Marilyn Zhang, and the team on a model capability
    user avatar
    Phoebe Yao
    @phoebeyao
    Mar 5
    Article cover image
    Article
    Confidence needs calibration
    Most people think the biggest risk in AI systems is hallucination. It isn’t. The more dangerous failure mode is answering confidently when the model shouldn’t answer at all. Frontier models do this...
    914
  • user avatar
    Pareto AI
    @pareto_ai
    Jan 28
    Our CEO @phoebeyao on the limits of prompt-based AI safety. “The future of AI isn't about replacing human expertise. It's about building better systems to capture expert insight at the edge of what's known and scale it into training data that actually moves model capabilities
    user avatar
    Phoebe Yao
    @phoebeyao
    Jan 28
    Article cover image
    Article
    You Can't Prompt Your Way to Safety
    AI models are giving medical and mental health advice to millions of people. Can you prevent harmful advice by adding safety instructions to the prompt? The UK's AI Security Institute (AISI) recently...
    825
  • user avatar
    Pareto AI
    @pareto_ai
    Mar 17, 2025
    An AI lab tried to cut costs by switching to a low-cost data vendor—only to face cascading quality failures. After months of setbacks, they came back to Pareto. Why? Because the best labs don’t gamble with data quality. Read more: pareto.ai/blog/pitfalls-…
    1K