Log inSign up
Daria Soboleva
Cerebras
660 posts
user avatar
Daria Soboleva
Cerebras
@dmsobol
Scaling the world's largest MoE models @Cerebras | Creator of The MoE 101 Guide & SlimPajama | Ex-@Google @Yandex. Tweets are my own views.
Sunnyvale, California
soboleva-daria.github.io
Joined June 2022
697
Following
2,020
Followers

New to X?

Sign up now to get your own personalized timeline!

Create account

By signing up, you agree to the Terms of Service and Privacy Policy, including Cookie Use.

Terms·Privacy·Cookies·Accessibility·Ads Info·© 2026 X Corp.
Don't miss what's happening
People on X are the first to know.
Log inSign up
  • Pinned
    user avatar
    Daria Soboleva
    Cerebras
    @dmsobol
    Jul 22, 2025
    After more than a year of getting burned with MoE gotchas, I finally sat down and wrote the guide I wish existed. Every paper skips the messy production details. This fills those gaps. No theory without implementation. cerebras.ai/moe-guide
    This post is unavailable.
    38K
  • user avatar
    Daria Soboleva
    Cerebras
    @dmsobol
    Aug 25, 2025
    Router wasn't learning at first, we debugged it step-by-step and showed you how despite perfect load balancing, routing can be completely useless. We root caused it and fixed the problem. Papers skip the methodology, but you can find all details in our part 3 of MoE 101 series
    user avatar
    Cerebras
    @cerebras
    Aug 25, 2025
    MoE 101 - Episode 3: You don’t need Google-scale compute to train effective MoE models We proved it with a hands-on experiment that beats dense GPT-2 scale model. @dmsobol will show you how with methodology (and debugging secrets)
    00:00
    31K
  • user avatar
    Daria Soboleva
    Cerebras
    @dmsobol
    May 21, 2025
    Just dropped on arXiv: "Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training": arxiv.org/abs/2505.13738 We found precise power laws that predict optimal hyperparameters for any model/data combination, enabling efficient large-scale training. 1/n 🧵
    arXiv logo
    arxiv.org
    Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM...
    Efficient LLM pre-training requires well-tuned hyperparameters (HPs), including learning rate $η$ and weight decay $λ$. We study scaling laws for HPs: formulas for how to scale HPs as we...
    18K
  • user avatar
    Daria Soboleva
    Cerebras
    @dmsobol
    May 5, 2025
    Working on a "Mixture of Experts (MoE) 101" blog post after a year of deep-diving. What's your biggest "wish someone told me this" moment with MoEs?
    8.2K
  • user avatar
    Daria Soboleva
    Cerebras
    @dmsobol
    Jun 6, 2025
    Speaking next week about Qwen 3 models w @arankomatsuzaki & @grmcameron on @cerebras Discord! We're analyzing what makes these new models different and their implications for AI development🤖 discord.gg/ggfYtRE7S7 RT if your network would find this interesting!
    15K
  • user avatar
    Daria Soboleva
    Cerebras
    @dmsobol
    Aug 7, 2025
    Nothing like watching your MoE model die at 3am teaches you what actually matters... I keep getting asked "what routing should I use?" and "what's special about DeepSeek-V3, will it solve my problems?" Answered everything in part 2 cerebras.ai/blog/moe-guide…
    user avatar
    Cerebras
    @cerebras
    Aug 7, 2025
    Here’s what nobody tells you about MoE: the router can single-handedly destroy your model. You can have perfect expert network architecture, tuned hyperparameters, and unlimited compute, but if your router collapses, you’re back to dense model performance regardless of number of
    2.4K
  • user avatar
    Daria Soboleva
    Cerebras
    @dmsobol
    Apr 15, 2025
    Original MoE vision (Jacobs, 1991): experts should COMPETE, not cooperate. Yet modern LLMs ignore this, treating experts as interchangeable compute chunks. With hundreds of experts in trillion-parameter model scales, are we just creating massive redundancy? 1/n
    5K
  • user avatar
    Daria Soboleva
    Cerebras
    @dmsobol
    May 1, 2025
    Yesterday I presented on MoE (Mixture of Experts) models at @fdotinc! Recording coming soon! Thanks to my amazing team for their support: @hi_im_dev_, Kevin Taylor, @PearlHulbert, @SarahChieng Co-presented w/ @learnwdaniel Great catching up w/ @arankomatsuzaki #ML #LLM #MoE
    6K
  • user avatar
    Daria Soboleva
    Cerebras
    @dmsobol
    Dec 16, 2023
    🌟 Excited to present at the ENLSP Workshop! Join me for the poster session of “BTLM-3B-8K: 7B Parameter Performance in a 3B Parameter Model” (Paper ID 45). Let’s dive into the world of AI efficiency together! 📅 Today, 1-2 PM 📍 Room 206-207 #NeurIPS2023 #AIResearch
    5.4K
  • user avatar
    Daria Soboleva
    Cerebras
    @dmsobol
    Feb 27, 2025
    Excited to share our paper accepted at @iclr_conf! 🚀 "Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs" We found that simple linear decay-to-zero consistently outperforms other LR schedules, potentially saving up to 60% compute when
    3.6K
  • user avatar
    Daria Soboleva
    Cerebras
    @dmsobol
    Apr 8, 2025
    Llama 4's MoE implementation, oh so many questions… Meta chose just 16 experts when DeepSeek demonstrated clear improvements from 160 to 256. Feels like deliberately leaving performance on the table??!! 1/n
    ai.meta.com
    The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation
    We’re introducing Llama 4 Scout and Llama 4 Maverick, the first open-weight natively multimodal models with unprecedented context support and our first built using a mixture-of-experts (MoE) archit...
    3.2K
  • user avatar
    Daria Soboleva
    Cerebras
    @dmsobol
    Jul 29, 2025
    I am giving a talk at #TPC25 on compute-efficient MoE design choices! This time I will focus on the architectural and training dynamics strategies for these models (like which batch size to use?) 📅 July 31st at 9am PST 📍DoubleTree by Hilton San Jose Come say hi 👋
    4.5K
  • user avatar
    Daria Soboleva
    Cerebras
    @dmsobol
    Jun 9, 2023
    Library of methods for SlimPajama is available on GitHub: github.com/Cerebras/model… This efficient, multi-processed implementation enabled us to process trillion token dataset in just a matter of days 💪 Now, we’re excited to share these powerful tools with you! Enjoy! ✨🚀
    2.8K
  • user avatar
    Daria Soboleva
    Cerebras
    @dmsobol
    Jul 2, 2025
    Thanks to @aiDotEngineer for releasing the recording of our Mixture of Agents workshop! Watch it here: youtube.com/watch?v=tzRvcT… 🧵 with insights from it:
    user avatar
    Daria Soboleva
    Cerebras
    @dmsobol
    Jun 4, 2025
    The most annoying thing about working with LLMs isn't that they're wrong -- it's the endless refinement loop.   Here's how Mixture of Agents (MoA) eliminates the back-and-forth that kills productivity.  1/n 🧵
    4.1K