We recently came across an interesting paper that helps LLMs be better at handling domain-specific languages like database queries or probabilistic programming languages, using an approach called "grammar prompting".
Link + brief thread below.
- ⚡️ Speed up LLM inference by 5x. ⚡️ We introduce a new framework, coalescence, that makes structured generation several times faster than standard generation. Coalescence is very flexible, and raises unexpected questions 🧐 blog.dottxt.co/coalescence.ht…
- 👉 Structured generation beats GPT-4 Using structured generation, phi-3 achieves 95.5% accuracy when it only achieves 86% without structured generation. More importantly, it beats GPT-4 (93.5%) by a whopping 2 percentage point. 🔥🔥🔥
- Highly recommend this post by @simonw about extracting structured data from text with LLMs. Simon puts it well: "the single most commercially valuable application of LLMs is turning unstructured content into structured data." Link below 👇
- 🚨 Outlines v1.0 is out 🚨 🌱 Simplified - Use any Python type to define the structure 🧨 More powerful - Expressive language to define complex structure and a library of built-in types. 🚀 Production-ready - Integration with inference servers (vLLM, Ollama, etc.) and APIs
- To celebrate the release of @huggingface's new SmolLM2 series of models we created a fun demo: the Bunny B1. The Bunny B1 shows how a small device using a SmolLM2 model + Outlines can consistently map natural language requests to the correct app! Check it out in this gif!
- A new paper, "Let Me Speak Freely" has been spreading rumors that structured generation hurts LLM evaluation performance. Well, we've taken a look and found serious issues in this paper, and shown, once again, that structured generation *improves* evaluation performance!
- Open models available TODAY can beat GPT-4 using structured generation 👇 While we are proud of this achievement, we wanted to talk about the communities and projects that made this possible 🧑🤝🧑 The reasons why, eventually, Open Source shall prevail 📖 blog.dottxt.co/oss-v-gpt4.html
- "Type-Constrained Code Generation with Language Models" is a relatively new paper that addresses a common challenge with LLM-generated code. The researchers are from ETH Zurich and UC Berkeley.
- DeepSeek's new reasoning model R1 "thinks" before it responds. With Outlines, you can set the number of characters the model is allowed to think before responding.
- There's often a lot of confusion about what structured generation actually is. Brief thread. Structured generation, also called structured output or constrained decoding, is always correctly formatted. The model literally cannot fail to output the format you specify.
- We’ve been cooking with @huggingface and just released a Rust port of Outlines’ structured generation. 👉 Faster compilation 👉 Lightweight library (poke @vllm_project) 👉 Bindings in many languages We can’t wait to see what you will build! Blog 👇
- We're often reaching for fancy things to explain how structured generation can speed up inference, and always forget the obvious one:
