Feature request

It would be nice to constrain the model output with a CFG directly when calling model.generate.

This is already done by llama.cpp grammars

An example is in this repo.

prompt = "ReLLM, the best way to get structured data out of LLMs, is an acronym for "
pattern = regex.compile(r'Re[a-z]+ L[a-z]+ L[a-z]+ M[a-z]+')
output = complete_re(model=model, 
                     prompt=prompt,
                     pattern=pattern)

> Realized Logistic Logistics Model

Is such a parameter on the roadmap for transformers?

Motivation

This can be super useful to make model output parseable within architectures that process the output of an LLM using classical methods. E.g. it can be used to make a model generate valid JSON in every case.

Your contribution

Happy to build this with CFGs if it helps! 😄