Pinned
📷📷📷New paper! (with @OpenAI) 📷📷📷
We trained weight-sparse models (transformers with almost all of their weights set to zero) on code: we found that their circuits become naturally interpretable! Our models seem to learn extremely simple, disentangled, internal mechanisms!
