Rubrics are becoming the standard way to train/evaluate LLMs on open-ended tasks.
But rubric-RL has a bottleneck: every rollout needs to be graded by an LLM verifier. That’s expensive, slow, and is prone to reward hacking.
At the same time, the field is moving toward on-policy
Scale Labs
123 posts
welcome to the lab.
from the researchers at @scale_AI
Joined October 2025
- Scale Labs repostedExcited to share a new @ScaleAILabs research in collaboration with @phylo_bio on coding agents for drug-discovery research! 💊 We ran Claude Code, Codex, and Gemini on 60+ expert-curated drug-discovery tasks inside a shared Biomni-powered biomedical research environment and the
- Scale Labs repostedHow do you turn agent traces into an improvement flywheel? Excited to share Insights Generator (IG) — new @scale_AI / @ScaleAILabs research that finds behavioral patterns and bugs in agent traces. Engineers & coding agents using IG achieved 30+% gains on agent benchmarks. 🧵
- Today we're releasing HiL-Dynamics, the first open-source tool that measures how production agents actually collaborate with humans under uncertainty. Not just whether they got the answer. Now you can measure exactly when your agent asks for help, when it makes assumptions, andReplying to @ScaleAILabsSelective escalation remains one of the biggest challenges for reliable human-in-the-loop AI. We hope HiL-Dynamics helps users find the right setup for their workflows and gives model builders clearer signals for building agents that collaborate with humans more effectively.
- Claude Opus 4.8 just landed on our MCP Atlas Leaderboard! Opus 4.8’s performance places it in the top band of SOTA models for agentic tool calling. The Claude 4 family keeps getting better at long-horizon tool use. Check out the updated rankings:
- Replying to @ScaleAILabsWe built ASPI to isolate clarification-seeking as its own agent state. Each benchmark scenario compares: - Execution mode → the agent receives a fully specified task - Clarification mode → the agent must ask follow-up questions before acting This allows us to measure howReplying to @ScaleAILabsThe takeaway: standard security evaluations may be underestimating the attack surface of interactive AI agents. A model that appears secure on fully specified tasks may become significantly more vulnerable once it has to handle ambiguity and request additional user input.
- New @scale_AI research introduces ASPI: Ambiguous-State Prompt Injection. Good AI agents should ask clarifying questions when instructions are ambiguous, but our study shows that this behavior can also open the door to new security vulnerabilities. Across 728 attack scenarios
- Rubric-based rewards are now standard for open-ended RL. But higher rubric scores don’t always mean better models. Our latest research shows models can learn to optimize the rubric-verifier setup itself, improving checklist coverage while broader quality declines. Robust1/ Using rubrics (a.k.a. checklists) in RL training is now standard for open-ended tasks without final verifiable result. However, rubric rewards are still proxy rewards that can get hacked during RL training. We study when rubric-based RL genuinely improves models vs. teaches
- Scale Labs reposted1/ New from @ScaleAILabs: Rubrics (a.k.a. checklists) have become the default reward interface for RL on open-ended tasks without final verifiable answers. But most rubric RL still relies on static aggregation: fixed human weights over criteria, summed into one scalar reward.
- Scale Labs repostedAt @ScaleAILabs, we’ve been exploring how to get models to accurately caption large-scale robot and human manipulation videos. More than 1,000 hours of new demonstrations hit our platform daily from factories, homes, and industrial sites and every episode needs precise action
00:00
















