Repository navigation
[FEATURE]: Allow provider-native options in stateless generate.text #47366
Description
Feature hasn't been suggested before.
- I have verified this feature I'm about to request hasn't been suggested before.
Describe the enhancement you want to request
Following #39243, V2 now has a stateless generate.text API, but its input only exposes model and prompt. Plugins sometimes need per-request provider controls, such as an output token limit, without creating a session.
Allow callers to pass an optional provider-native JSON body:
await ctx.generate.text({
model: { providerID: "anthropic", id: "claude-sonnet-4-6" },
prompt: "Review this action.",
body: { max_tokens: 2_000 },
})The body could use the existing raw request overlay after protocol lowering. Keeping it generic avoids adding a new API field for every provider option, while calls that omit it retain the current behavior.
Metadata
Metadata
Assignees
Labels
No labels
Activity
opencodejr commentedon Oct 28, 2026
Thanks for the suggestion! You're asking for
generate.textto accept an optional provider-native JSONbody, applied after protocol lowering using the existing raw request overlay. That would let plugins set per-request options likemax_tokenswithout creating a session, and calls withoutbodywould behave as they do now.Today the endpoint only takes
promptandmodel, and the stateless generate path doesn't go through any plugin hook. So for now a plugin can't work around this; it needs a change to the API. The maintainers will consider it.Related:
generate.textAPI. It also asked for an output token limit.abogartz commentedon Nov 7, 2026
From a memory-plugin that mines transcripts with
generate.textafter each turn: the call costs ~2,430 tokens and ~25 s on a local seat, and 78% of the time is reasoning that gets thrown away — only a small JSON array reaches the store.The one field we need is a semantic way to say "do not reason" on a transient generate call. Measured on a llama.cpp-family seat with six spellings:
reasoning_effort: "none"chat_template_kwargs: {enable_thinking: false}reasoning: {effort: "none"}thinking_budget_tokens: 0think: falsereasoning_effort: "none"is the provider-neutral one and the seat honors it, so a semanticreasoningEfforton the generate call would cover llama.cpp, vLLM and OpenAI-compat endpoints without plugins guessing body fields.Two things we tried instead, on a live server with an instrumented plugin:
session.hook("generate", …)does not fire forgenerate.text(see Plugin request hooks don't run for experimental.generate.text #62096) — registration succeeds, the hook is never called.ctx.session.generatedoes fire the hook, andevent.optionsis mutable, but settingchat_template_kwargsthere did not reach the wire (docs: provider options are protocol-semantic names, not raw body fields). It also carries the session's agent and history, so the same mining prompt returned an in-character answer instead of JSON: 9,323 ms viagenerate.text, 15,036 ms viasession.generate, 23,574 ms with the rewrite attempted and 0 candidates.So neither workaround substitutes for options on the stateless call. Happy to test a branch.