Omead Pooladzandi (@HessianFree) / X

Pinned

your spotify cache is bigger than our largest AI model. Bonsai: 1-bit weights. 1.7B to 8B params. 14x compression vs bf16. 8x faster on edge. 256 MB to 1.2GB. Based on Qwen 3. we just came out of stealth. intelligence belongs at the edge and we're going to put it there.

PrismML

@PrismML

Mar 31

Today, we are emerging from stealth and launching PrismML, an AI lab with Caltech origins that is centered on building the most concentrated form of intelligence. At PrismML, we believe that the next major leaps in AI will be driven by order-of-magnitude improvements in

We've been cooking. No this is not PSGD. Yes, I have trained a 3b parameter model with it.

This will probs be one of the most influential papers of the year

Momentum in optimization isn't always your best friend. In dynamic systems, even tiny momentum can be catastrophically destructive. 🐾 Here we're using PSGD's Dense Newton method as the base optimizer—an 'old school' method that's reliable, like a wise old dog. 🐕‍🦺 For static

Wait what how did I miss this one @aaron_defazio at it again

Replying to @janonacct

Unionically this would probably save them a lot of money for people's spam messages to gpt's

Alright let's put Muon and SOAP head-to-head with PSGD to solve a simple loss = (1 - (x^T * x))^2, such that x is initialized as a 2x2 random matrix. At high beta both all optimizers fail but, otherwise PSGD is able to reduce the loss down exactly to zero whereas Muon can

I'm reviewing for icml right now. Muon clones everywhere. It's actually so sad to see. This is a bad look for the community. If ur doing a Muon colne and u don't use modded nanoGPT. It's literally just an auto reject from me. Sheesh. @kellerjordan0

A fun plot today from a convo with @evaninwords @torchcompiled @leloykun and @rami_mmo. It shows Muon's Newton-Schulz iterations flips the sign of the gradient update ~25% of the time. The cosine similarity is conserved only ~63%. While update directions being flipped are not of

Since we were rejected from ICML here are some results we are proud of. We find that the Shampoo and CASPR preconditioners leave a lot of performance on the table when estimating the Hessian. We propose a much cheaper Hessian estimate. No expensive matrix inverse is required!