✨Excited to finally drop our new paper: SSMs “look like” RNNs, but we show their statefulness is an illusion🪄🐇
Current SSMs cannot express basic state tracking, but a minimal change fixes this! 👀
w/ @jowenpetty, @Ashish_S_AIarxiv.org/abs/2404.08819
📣 @Ashish_S_AI and I prove that transformers can be translated to sentences in first-order logic with majority-vote quantifiers (FOM).
FOM is a symbolic language that can capture computation inside transformers!
arxiv.org/abs/2210.02671
How does the depth of a transformer affect reasoning capabilities? New preprint by myself and @Ashish_S_AI shows that a little depth goes a long way to increase transformers’ expressive power
We take this as encouraging for further research on looped transformers!🧵
Excited to announce I'll be starting as an assistant professor at @TTIC_Connect for fall 2026!
In the meantime, I'll be graduating and hanging around Ai2 in Seattle🏔️
Padding a transformer’s input with blank tokens (...) is a simple form of test-time compute. Can it increase the computational power of LLMs? 👀
New work with @Ashish_S_AI addresses this with *exact characterizations* of the expressive power of transformers with padding 🧵
[1/6] Excited to share a year-long project re: theory of language understanding in LMs w/
@a_stadt, @tallinzen
TLDR:
Judging entailments (NLI) can be reduced to LMing over "Gricean data"*
∴ Learning distribution (perfectly) => learning semantics
Is it possible for GPT-n to "understand" the semantics of English? What about Python?
I'm excited to finally share work formalizing this question! We give formal languages that are *provably* un-understandable by LMs (within our setup, at least)
arxiv.org/abs/2104.10809
How do we understand logical reasoning in non-symbolic models like transformers?
📣New preprint w/ Ashish Sabharwal shows any transformer can be translated to a fixed-size first-order-logic formulae (with majority quantifiers)
📢 Preprint: We can predict entailment relations from LM sentence co-occurrence prob. scores
These results suggest predicting sentence co-occurrence may be one way that next-word prediction leads to (partial) semantic representations in LMs🧵
What inductive biases does training impose on transformers?
We find that T5, RoBERTa, etc. are well-approximated by saturated transformers (simplified attention patterns), and explain how this arises during training.
w/ @RamanujanVivek@yoavgo@royschwartzNLP@nlpnoah
