Skip to content

Custom servers: Find models + Save silently resets a hand-set maxTokens even when the context is unchanged #1350

Closed

Description

@danyaberezun

Description

Re-detecting a custom server resets a user's max output for its models, even when nothing about the model changed:

  • Find models + Save. When the server's listing reports a context window (vLLM, sglang, llama.cpp, LM Studio, Ollama with num_ctx), the dialog drops the saved max output on any reported context, not only a changed one (apps/web/src/panels/customEndpointDraft.ts:95). Save then writes the default — ¼ of the context, capped at 16K — into models[] (packages/server/src/agent/customEndpoints.ts:431-435). A hand-set maxTokens: 32000 silently becomes 16384. apps/web/src/panels/SPEC.md says re-detection drops a saved max output only "once a new context is reported".
  • Plain Save with a user context window (feat(providers): let a custom endpoint's model set its own context window #1197). maxTokensFor fits max output against the detected context window instead of the effective one from modelOverrides.contextWindow, so a plain Save can shrink it (e.g. override 262144, maxTokens 65536, detected 32768 → maxTokens 8192).

Max output isn't shown in the dialog, so the user gets no hint that it changed; long reasoning or long answers are then cut at the lower limit. This hits exactly the users following the #1316 workaround of setting maxTokens by hand.

Steps to reproduce

  1. Configure a custom server (Settings → Providers → Custom servers) whose /models listing reports a context window, e.g. vLLM.
  2. In pi's models.json, set one of its models to "contextWindow": 131072, "maxTokens": 32000 (in models[] or as a modelOverrides entry).
  3. Settings → Providers → Custom servers → Edit → Find models (the server still reports 131072) → Save.
  4. Open models.json again.

Expected behavior

A saved max output is kept while the reported context window is unchanged, and max output is fitted against the effective context window (the user's override when present).

Actual behavior

maxTokens is 16384 after step 3. With a user context override larger than the detected window, even a plain Save lowers maxTokens (65536 → 8192).

Environment

  • OS: any
  • Bun version (bun --version): 1.4.4
  • Node version (node --version): n/a
  • ThinkRail version / commit: main 52ae54782 (reproduced with a test calling listCustomEndpoints → editEndpointDraft → probeCustomEndpoint → mergeProbe → draftParams → saveCustomEndpoint against a mock server)

Activity

  1. added a commit that references this issue on Nov 14, 2026
    bf566cc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions