lol my research direction is fun-based architectures
When you know itās gonna be an interesting arch paper
Songlin Yang
2,456 posts
Songlin Yang
@SonglinYang4
- science is better when shared ā¤ļøToday Thinking Machines Lab is launching our research blog, Connectionism. Our first blog post is āDefeating Nondeterminism in LLM Inferenceā We believe that science is better when shared. Connectionism will cover topics as varied as our research is: from kernel numerics to
- Hi @JeffDean, whatās the plan for releasing the code for this line of work? None of these papers so far seem to have released any code
An exciting new approach for doing continual learning, using nested optimization for enhancing long context processing. - I've created slides for those curious about the recent rapid progress in linear attention: from linear attention to Lightning-Attention, Mamba2, DeltaNet, and TTT/Titans. Check it out here: sustcsonglin.github.io/assets/pdf/talā¦
- Flash Linear Attention (github.com/fla-org/flash-ā¦) will no longer maintain support for the RWKV series (existing code will remain available). Hereās why:
- Introducing the first open-source implementation of native sparse attention: github.com/fla-org/nativeā¦. Give it a spin and cook your NSA model! š³š³š³
- Check out log-linear attentionāour latest approach to overcoming the fundamental limitation of RNNsā constant state size, while preserving subquadratic time and space complexityWe know Attention and its linear-time variants, such as linear attention and State Space Models. But what lies in between? Introducing Log-Linear Attention with: - Log-linear time training - Log-time inference (in both time and memory) - Hardware-efficient Triton kernels
- š¢ (1/16) Introducing PaTH š£ļø ā a RoPE-free contextualized position encoding scheme, built for stronger state tracking, better extrapolation, and hardware-efficient training. PaTH outperforms RoPE across short and long language modeling benchmarks
- Excited to see Gated DeltaNet being adopted in the @Alibaba_Qwen series ! It has also previously demonstrated strong effectiveness in @nvidia's Jet-Nemotronš Introducing Qwen3-Next-80B-A3B ā the FUTURE of efficient LLMs is here! š¹ 80B params, but only 3B activated per token ā 10x cheaper training, 10x faster inference than Qwen3-32B.(esp. @ 32K+ context!) š¹HybridĀ Architecture:Ā GatedĀ DeltaNetĀ +Ā GatedĀ AttentionĀ āĀ bestĀ ofĀ speedĀ &
- hybrid is the future:)Qwen3-Next is hybrid GatedAttention (for outliers fix) GatedDelta net rnn for kv saving all new models will be either sink+swa hyprids like gpt oss or gated attn + linear rnn hybrids (mamba , gated deltanet etc) like qwen3-next age of pure attn for timemixing layer is over,
- Many people are confused by Minimaxās recent return to full attention - especially since it was the first large-scale pivot toward hybrid linear attention - and by Kimiās later adoption of hybrid linear variants (as well as earlier attempts by Qwen3-Next, or Qwen3.5). I actually
- math matters in scaling!Efficient training of neural networks is difficult. Our second Connectionism post introduces Modular Manifolds, a theoretical step toward more stable and performant training by co-designing neural net optimizers with manifold constraints on weight matrices.
- where is appendix and arxiv šIntroducing Nested Learning: A new ML paradigm for continual learning that views models as nested optimization problems to enhance long context processing. Our proof-of-concept model, Hope, shows improved performance in language modeling. Learn more: goo.gle/47LJrzI
- How can models learn to generate weight updates in token space?What if an LLM could update its own weights? Meet SEALš¦: a framework where LLMs generate their own training data (self-edits) to update their weights in response to new inputs. Self-editing is learned via RL, using the updated modelās downstream performance as reward.
