Log inSign up
Scale Labs
123 posts
user avatar
Scale Labs
@ScaleAILabs
welcome to the lab. from the researchers at @scale_AI
labs.scale.com
Joined October 2025
109
Following
1,722
Followers
  • Scale Labs reposted
    user avatar
    MohammadHossein Rezaei
    @mhrezaeics
    Jun 12
    Rubrics are becoming the standard way to train/evaluate LLMs on open-ended tasks. But rubric-RL has a bottleneck: every rollout needs to be graded by an LLM verifier. That’s expensive, slow, and is prone to reward hacking. At the same time, the field is moving toward on-policy
    Rubrics have emerged as an alternative to RLVR in open-ended domains where a single groundtruth final answer is not available. Existing rubric-based training methods rely on an LLM verifier
that scores each rollout against rubrics. This introduces substantial training-time overhead, exposes optimization to verifier-specific biases, and reduces rubric feedback to a sparse end-of-trajectory
signal. We propose Rubric-Guided Self-Distillation (RGSD), a verifier-free training method in which the base policy, conditioned on the rubric, serves as the teacher for the unconditioned student.
RGSD distills the rubric-conditioned teacher distribution into the student token-by-token, replacing sparse trajectory-level rewards with dense per-token learning signals and removing the LLM judge
from the training loop entirely. Across Qwen-2.5 (3B, 7B) and Qwen3-Thinking (4B, 8B) models on medical and science domains, RGSD achieves rubric satisfaction comparable to judge-based
GRPO while using one on-poli
    28K
  • Scale Labs reposted
    user avatar
    Afra Feyza Akyürek
    @afeyzaakyurek
    Jun 5
    Excited to share a new @ScaleAILabs research in collaboration with @phylo_bio on coding agents for drug-discovery research! 💊 We ran Claude Code, Codex, and Gemini on 60+ expert-curated drug-discovery tasks inside a shared Biomni-powered biomedical research environment and the
    11K
  • Scale Labs reposted
    user avatar
    Akshay
    @akshay_manglik
    Jun 2
    How do you turn agent traces into an improvement flywheel? Excited to share Insights Generator (IG) — new @scale_AI / @ScaleAILabs research that finds behavioral patterns and bugs in agent traces. Engineers & coding agents using IG achieved 30+% gains on agent benchmarks. 🧵
    715
  • user avatar
    Scale Labs
    @ScaleAILabs
    Jun 1
    Today we're releasing HiL-Dynamics, the first open-source tool that measures how production agents actually collaborate with humans under uncertainty. Not just whether they got the answer. Now you can measure exactly when your agent asks for help, when it makes assumptions, and
    4.3K
    user avatar
    Scale Labs
    @ScaleAILabs
    Jun 1
    Replying to @ScaleAILabs
    Selective escalation remains one of the biggest challenges for reliable human-in-the-loop AI. We hope HiL-Dynamics helps users find the right setup for their workflows and gives model builders clearer signals for building agents that collaborate with humans more effectively.
    305
    user avatar
    Scale Labs
    @ScaleAILabs
    Jun 1
    HiL-Dynamics: github.com/melfeki-11/HiL… Blog: labs.scale.com/blog/hil-dynam…
    GitHub - melfeki-11/HiL-Dynamics: Does your coding agent know when it doesn't know? HiL-Dynamics...
    From github.com
    304
  • user avatar
    Scale Labs
    @ScaleAILabs
    May 28
    Claude Opus 4.8 just landed on our MCP Atlas Leaderboard! Opus 4.8’s performance places it in the top band of SOTA models for agentic tool calling. The Claude 4 family keeps getting better at long-horizon tool use. Check out the updated rankings:
    MCP Atlas
    From labs.scale.com
    697
  • user avatar
    Scale Labs
    @ScaleAILabs
    May 26
    Replying to @ScaleAILabs
    We built ASPI to isolate clarification-seeking as its own agent state. Each benchmark scenario compares: - Execution mode → the agent receives a fully specified task - Clarification mode → the agent must ask follow-up questions before acting This allows us to measure how
    626
    user avatar
    Scale Labs
    @ScaleAILabs
    May 26
    Replying to @ScaleAILabs
    The takeaway: standard security evaluations may be underestimating the attack surface of interactive AI agents. A model that appears secure on fully specified tasks may become significantly more vulnerable once it has to handle ambiguity and request additional user input.
    310
    user avatar
    Scale Labs
    @ScaleAILabs
    May 26
    Full paper:
    ASPI: Seeking Ambiguity Clarification Amplifies Prompt Injection Vulnerability in LLM Agents
    From labs.scale.com
    253
  • user avatar
    Scale Labs
    @ScaleAILabs
    May 26
    New @scale_AI research introduces ASPI: Ambiguous-State Prompt Injection. Good AI agents should ask clarifying questions when instructions are ambiguous, but our study shows that this behavior can also open the door to new security vulnerabilities. Across 728 attack scenarios
    2.1K
  • user avatar
    Scale Labs
    @ScaleAILabs
    May 22
    Rubric-based rewards are now standard for open-ended RL. But higher rubric scores don’t always mean better models. Our latest research shows models can learn to optimize the rubric-verifier setup itself, improving checklist coverage while broader quality declines. Robust
    user avatar
    Anas Mahmoud
    @nas_mahmoud_
    May 13
    1/ Using rubrics (a.k.a. checklists) in RL training is now standard for open-ended tasks without final verifiable result. However, rubric rewards are still proxy rewards that can get hacked during RL training. We study when rubric-based RL genuinely improves models vs. teaches
    6.1K
  • Scale Labs reposted
    user avatar
    Utkarsh Tyagi
    @utkarsh4430
    May 20
    1/ New from @ScaleAILabs: Rubrics (a.k.a. checklists) have become the default reward interface for RL on open-ended tasks without final verifiable answers. But most rubric RL still relies on static aggregation: fixed human weights over criteria, summed into one scalar reward.
    8.7K
  • user avatar
    Scale Labs
    @ScaleAILabs
    May 19
    Congrats to @GoogleDeepMind for releasing Gemini 3.5 Flash and topping our MCP Atlas leaderboard! 🥇
    2.5K
  • Scale Labs reposted
    user avatar
    jade
    @jadechoghari
    May 19
    At @ScaleAILabs, we’ve been exploring how to get models to accurately caption large-scale robot and human manipulation videos. More than 1,000 hours of new demonstrations hit our platform daily from factories, homes, and industrial sites and every episode needs precise action
    00:00
    6K

New to X?

Sign up now to get your own personalized timeline!

Create account

By signing up, you agree to the Terms of Service and Privacy Policy, including Cookie Use.

Terms of Service|Privacy Policy|Cookie Policy|Accessibility|Ads info|© 2026 X Corp.
Don't miss what's happening
People on X are the first to know.
Log inSign up