we've been building @GoodfireAI as a true 'age of research' company.
if you liked the VPD research on decoding model weights, there will be bangers all month, and we're only accelerating
Just wrote a piece on why I believe interpretability is AI’s most important frontier - we're building the most powerful technology in history, but still can't reliably engineer or understand our models. With rapidly improving model capabilities, interpretability is more urgent,
we're growing quickly at @GoodfireAI and need more senior engineers to join so we can understand and design neural networks!
i think most people underestimate the engineering lift necessary to run good experiments at scale, and our goal in AI interpretability is to develop
monitoring chain of thought is not going to lead to good understanding of how models think.
understanding the internal activations and parameters of the model is much more fundamental and necessary to deeply understand AI
my sense is that restricting reasoning to coherent
impressed by igor's conviction to move on to build in AI safety. as we get closer to AGI, i expect more and more people to realize that nothing matters more than making sure that we transition safely to a world with smarter than human level intelligences
Today was my last day at xAI, the company that I helped start with Elon Musk in 2023. I still remember the day I first met Elon, we talked for hours about AI and what the future might hold. We both felt that a new AI company with a different kind of mission was needed.
Building
Today, we're announcing our $50M Series A and sharing a preview of Ember - a universal neural programming platform that gives direct, programmable access to any AI model's internal thoughts.
Hear from our CTO, Dan Balsam, about Field Research at Goodfire. The Field Team deploys cutting-edge interpretability research in real-world AI settings -- helping customers uncover hidden insights in their models across bio, language, and other domains.
for the @GoodfireAI hackathon, i built a tool to visualize which experts activate the most in gpt-oss! found that certain experts tend to fire in interpretable contexts, like in business, poems, and code
"We are thus in a race between interpretability and model intelligence"
Important read from Dario - interpretability is a deeply urgent problem, and I hope that more people join the effort to understand AI models
Super cool result from @livgorton leveraging interpretability to explain a longstanding vulnerability in basically every AI model!
As we build out our interpretability toolkit, I think we’ll be able to debug and fix so many more problems with AI models
Adversarial examples - a vulnerability of every AI model, and a “mystery” of deep learning - may simply come from models cramming many features into the same neurons!
Less feature interference → more robust models.
New research from @livgorton 🧵 (1/4)
Incredibly excited for this to be out. We've been collaborating with the team at @arcinstitute to interpret Evo 2 and found that that the model has learned to recognize sophisticated biological concepts, from basic DNA elements to complex protein structures.
@pdhsu,
AI provides a universal framework that leverages data and compute at scale to uncover higher-order patterns
Today, @arcinstitute in collaboration with @nvidia releases Evo 2—a fully open source biological foundation model trained on genomes spanning the entire tree of life 🧵
I think we need new tools in the interp toolbox other than sparse dictionary learning techniques like SAEs, and SPD is probably one of them
now to scale!
(1/7) New research: how can we understand how an AI model actually works? Our method, SPD, decomposes the *parameters* of neural networks, rather than their activations - akin to understanding a program by reverse-engineering the source code vs. inspecting runtime behavior.
Incredibly excited to launch Ember to accelerate interpretability research! Looking forward to seeing what people think, and so proud of the team for how far we've come in such a short timeframe.
Scaling interpretability work for frontier AI alignment has never been more important. That’s why we’re launching Ember - the first hosted mechanistic interpretability API with fast inference support for generative models like Llama 3.3 70B.
increased investment in AI interpretability as part of the new AI action plan. more funding in fundamental interpretability research is one of the highest leverage ways we can make progress on safety and reliability
