You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Custom servers: Find models + Save silently resets a hand-set maxTokens even when the context is unchanged #1350
Re-detecting a custom server resets a user's max output for its models, even when nothing about the model changed:
Find models + Save. When the server's listing reports a context window (vLLM, sglang, llama.cpp, LM Studio, Ollama with num_ctx), the dialog drops the saved max output on any reported context, not only a changed one (apps/web/src/panels/customEndpointDraft.ts:95). Save then writes the default — ¼ of the context, capped at 16K — into models[] (packages/server/src/agent/customEndpoints.ts:431-435). A hand-set maxTokens: 32000 silently becomes 16384. apps/web/src/panels/SPEC.md says re-detection drops a saved max output only "once a new context is reported".
Plain Save with a user context window (feat(providers): let a custom endpoint's model set its own context window #1197).maxTokensFor fits max output against the detected context window instead of the effective one from modelOverrides.contextWindow, so a plain Save can shrink it (e.g. override 262144, maxTokens 65536, detected 32768 → maxTokens 8192).
Max output isn't shown in the dialog, so the user gets no hint that it changed; long reasoning or long answers are then cut at the lower limit. This hits exactly the users following the #1316 workaround of setting maxTokens by hand.
Steps to reproduce
Configure a custom server (Settings → Providers → Custom servers) whose /models listing reports a context window, e.g. vLLM.
In pi's models.json, set one of its models to "contextWindow": 131072, "maxTokens": 32000 (in models[] or as a modelOverrides entry).
Settings → Providers → Custom servers → Edit → Find models (the server still reports 131072) → Save.
Open models.json again.
Expected behavior
A saved max output is kept while the reported context window is unchanged, and max output is fitted against the effective context window (the user's override when present).
Actual behavior
maxTokens is 16384 after step 3. With a user context override larger than the detected window, even a plain Save lowers maxTokens (65536 → 8192).
Environment
OS: any
Bun version (bun --version): 1.4.4
Node version (node --version): n/a
ThinkRail version / commit: main 52ae54782 (reproduced with a test calling listCustomEndpoints → editEndpointDraft → probeCustomEndpoint → mergeProbe → draftParams → saveCustomEndpoint against a mock server)
Activity
fix(providers): keep custom server max output on Find models and Save (…