Karthik Narasimhan (@karthik_r

Karthik Narasimhan

330 posts

Karthik Narasimhan

@karthik_r_n

Professor@PrincetonCS, ex @OpenAI, @SierraPlatform, @MIT_CSAIL, @iitmadras Work: GPT, ReAct, Tree-of-Thoughts, SWE-Bench/Agent, TAU-bench, GEO

Princeton, NJ

karthiknarasimhan.com

Joined July 2015

Karthik Narasimhan
@karthik_r_n
Apr 2, 2024
SWE-agent is finally out. A few highlights: 1. Agent-Computer Interface (ACI) design will be critical for the success of AI agents, much like HCI is critical for how effective humans are with computers. 2. You can use SWE-agent out of the box on any github issue. (1/2)
John Yang
@jyangballin
Apr 2, 2024
SWE-agent is our new system for autonomously solving issues in GitHub repos. It gets similar accuracy to Devin on SWE-bench, takes 93 seconds on avg + it's open source! We designed a new agent-computer interface to make it easy for GPT-4 to edit+run code github.com/princeton-nlp/…
42K
Karthik Narasimhan
@karthik_r_n
May 19, 2023
LMs are not just about generating text in a linear fashion anymore. They can be used to explore unique points of attack, backtrack, search, and much more to solve complex problems and puzzles. Check out 🌲Tree-of-thought, our new framework for systematic reasoning with LLMs!
Shunyu Yao
@ShunyuYao12
May 19, 2023
Still use ⛓️Chain-of-Thought (CoT) for all your prompting? May be underutilizing LLM capabilities🤠 Introducing 🌲Tree-of-Thought (ToT), a framework to unleash complex & general problem solving with LLMs, through a deliberate ‘System 2’ tree search. arxiv.org/abs/2305.10601
23K
Karthik Narasimhan
@karthik_r_n
Jun 20, 2024
Excited to release 𝜏-bench (TAU for Tool-Agent-User ⚒️-🤖-🧑), a new benchmark to evaluate AI agents' performance and reliability in real-world settings with dynamic user and tool interaction. Paper: arxiv.org/abs/2406.12045, Blog: sierra.ai/blog/benchmark…
arxiv.org
$τ$-bench: A Benchmark for Tool-Agent-User Interaction in...
Existing benchmarks do not test language agents on their interaction with human users or ability to follow domain-specific rules, both of which are vital for deploying them in real world...
25K
Karthik Narasimhan
@karthik_r_n
Oct 10, 2022
We teach language models to both reason and act in the same breath. Exciting since it provides a natural mechanism for allowing LLMs to incorporate external knowledge (APIs, databases, web) vs black box internal reasoning, for tasks from QA to webpage navigation.
Shunyu Yao
@ShunyuYao12
Oct 10, 2022
Large Language Models (LLM) are 🔥in 2 ways: 1.🧠Reason via internal thoughts (explain jokes, math reasoning..) 2.💪Act in external worlds (SayCan, ADEPT ACT-1, WebGPT..) But so far 🧠and💪 remain distinct methods/tasks... Why not 🧠+💪? In our new work ReAct, we show 1+1>>2!
Karthik Narasimhan
@karthik_r_n
Jun 18, 2020
Thank you @AmazonScience for supporting our research! Hope to make some useful advances in conversational systems!
Amazon Science
@AmazonScience
Jun 17, 2020
Congratulations to the 51 award recipients of the 2019 Amazon Research Awards, who represent 39 universities in 10 countries. View the full list and find out how to be added to the 2020 Call For Proposal distribution list here: amazon.science/blog/recipient… #AmazonResearchAwards
Karthik Narasimhan
@karthik_r_n
Feb 21, 2022
Even though we're still at the tip of the iceberg here, I'm very excited about this direction and its future potential. As we continue to scale up our neural networks, we will need more methods to enable more efficient and cost-effective inference per input instance.
Vishvak Murahari
@VishvakM
Feb 21, 2022
Data Multiplexing for Neural Networks🔀 Can neural networks process multiple instances simultaneously as a single mixed input, similar to how radio channels can share bandwidth to carry multiple signals? Surprisingly, we find they can indeed!! arxiv.org/abs/2202.09318 📜 [1/6]
GIF
Karthik Narasimhan
@karthik_r_n
Apr 15, 2023
Very excited for this upcoming Center for Language and Intelligence @ Princeton! nlp.cs.princeton.edu/center-languag… We are actively recruiting!
Sanjeev Arora
@prfsanjeevarora
Apr 14, 2023
Princeton has a new Center for Language and Intelligence, researching LLMs + large AI models, as well as their interdisciplinary applications. Looking for postdocs/research scientists/engineers; attractive conditions. nlp.cs.princeton.edu/center-languag…
13K
Karthik Narasimhan
@karthik_r_n
Mar 13, 2024
Software engineering is so much more than just generating code. Exciting to see good progress on SWE-bench (swebench.com) - solving ~13% of real-world bugs is impressive. Still a long way to go though!
Cognition
@cognition
Mar 12, 2024
Today we're excited to introduce Devin, the first AI software engineer. Devin is the new state-of-the-art on the SWE-Bench coding benchmark, has successfully passed practical engineering interviews from leading AI companies, and has even completed real jobs on Upwork. Devin is
00:00
4.9K
Karthik Narasimhan
@karthik_r_n
Jul 11, 2022
The web (or even a simulation of it) is a great environment for building language-grounded RL agents! Has real-world content, is interactive and can be dynamically changing, is easily scalable and fast to run, and has direct practical applications in assisting humans with tasks.
Shunyu Yao
@ShunyuYao12
Jul 10, 2022
What if you had a bot you could just instruct in English to shop online for you? Check out our latest work 🛒WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents arxiv.org/abs/2207.01206 w/@__howardchen @jyangballin , @karthik_r_n @princeton_nlp
GIF
Karthik Narasimhan
@karthik_r_n
Jan 12, 2024
Thank you for inviting me! Much of the work I presented was led by @ShunyuYao12 (who is on the job market!), @jyangballin (who is applying to PhD programs this year!), @_carlosejimenez, and @tedsumers
Stanford NLP Group
@stanfordnlp
Jan 11, 2024
We were really excited to have @karthik_r_n from @princeton_nlp join us today to give us the good oil on the future of language agents in the age of Foundation Models! #NLProc
4.7K
Karthik Narasimhan
@karthik_r_n
Mar 26, 2021
If you work on RL agents for text-based games (or other interactive envs with language), this might be interesting! We show that agents may not be adequately leveraging the rich semantics present in the observations and there's lots of room for more semantic-centric approaches
Shunyu Yao
@ShunyuYao12
Mar 26, 2021
For autonomous tasks with language (e.g. text games), how much does an agent rely on language semantics vs. memorization? Our #NAACL2021 paper (arxiv.org/abs/2103.13552, joint w/ @karthik_r_n, @mhauskn) proposes ablation studies with surprising findings and useful insights! (1/3)
Karthik Narasimhan
@karthik_r_n
Sep 15, 2022
Training agents to operate autonomously over the web will be the next frontier for AI. If you are looking for an open-source benchmark to build and test your own web agents, check out webshop-pnlp.github.io (by @ShunyuYao12 , @__howardchen , @jyangballin )
This post is unavailable.
Karthik Narasimhan
@karthik_r_n
Jun 25, 2020
Seems like a very cool RL testbed for many different reasons!
Edward Grefenstette
@egrefen
Jun 25, 2020
Want to help push the boundaries of RL research? Need a rich, difficult, and procedurally-generated environment with loads of structure and intricacy? An astounding amount of human play data? Sophisticated strategies and documentation? We got you (and it's faster than ALE!) [1/6]
Karthik Narasimhan
@karthik_r_n
May 25, 2021
Check out our new #ACL2021 paper which shows how self-attention nets can learn languages with bounded hierarchical structure (which applies to most practical uses of human languages). A step towards understanding why Transformers are 🔥 for NLP!
Shunyu Yao
@ShunyuYao12
May 25, 2021
Hierarchical structure is a core aspect of language syntax. Recurrent networks can systematically process recursion by emulating stacks, but can self-attention networks? If so, how? Our #ACL2021 paper shed lights into this fundamental issue! arxiv.org/abs/2105.11115 (1/5)