[Bug]: OpenAI-compatible serving issues: reverse-proxy Host validation, HTTP framing, reasoning streaming, context-error ordering, keepalives, and GLM turn termination #597

Summary

I managed to get GLM-5.2 running through Colibri's OpenAI-compatible API in a Kilo code / VSCodium client through Tailscale Serve. I eventually obtained working inference at about 0.33 tok/s , but only after a lot of hacky fixes in both the Python gateway and the native GLM engine. Here are the 6 issues I encountered and a summary of my local fixes:

1. c/glm.c: serve mode filters all non-EOS metadata stop tokens for tool-call safety, including GLM's <|user|> assistant-to-user turn boundary. The model then continues into an invented multi-turn transcript.
2. c/openai_server.py: the DNS-rebinding Host guard rejects legitimate reverse-proxy/MagicDNS hostnames and has no supported allowlist configuration.
3. c/openai_server.py: HTTP/1.1 connection reuse can become desynchronised, with a previous JSON body appearing before the next request line.
4. c/openai_server.py: GLM reasoning is emitted as ordinary content, rather than incrementally through reasoning_content.
5. c/openai_server.py: the prefill keepalive uses a visible . reasoning delta, producing hundreds of dots during very slow prefill.
6. c/openai_server.py + c/glm.c: a streaming HTTP 200 is committed before the engine validates context length, so a later CONTEXT_EXCEEDED cannot be returned as a clean HTTP 400.

Item 1 has a narrow native C patch that dynamically resolves <|user|> and applies it only in serve mode. Items 2-5 have working local Python changes. Item 6 is fully diagnosed below and includes a concrete engine-protocol design, but I have not implemented that protocol change locally.

Environment

• Colibri: v1.1.0, commit 6368e1a
• Model: GLM-5.2, int4 experts / int8 heads build
• OS: Linux Mint
• CPU: Intel i7-8700K, 6 cores / 12 threads
• RAM: 64 GB
• GPU: NVIDIA Titan Xp, 12 GB VRAM, sm_61
• API bind: 127.0.0.1:8000
• Reverse proxy: Tailscale Serve
• Client: remote OpenAI-compatible coding client
• KV slots: 1
• Context after diagnosis: CTX=32768

Representative launch command:

cd ~/colibri/c

CTX=32768 \
OMP_NUM_THREADS=10 \
OMP_PROC_BIND=spread \
OMP_PLACES=cores \
COLI_MODEL=/path/to/glm-5.2-int4-int8-mtp \
COLI_API_KEY="$COLIBRI_API_KEY" \
COLI_CUDA=1 \
COLI_GPU=0 \
CUDA_EXPERT_GB=12 \
COLI_CUDA_PIPE=0 \
PIN=auto \
PIN_FILL=1 \
PIPE=1 \
PIPE_WORKERS=4 \
DIRECT=1 \
URING=1 \
PILOT=0 \
I4S=0 \
MTP=0 \
THINK=0 \
KVSAVE=0 \
./coli serve \
  --host 127.0.0.1 \
  --port 8000 \
  --model-id glm-5.2-colibri \
  --allowed-host my-host.example.ts.net

1. GLM <|user|> does not terminate an assistant turn

Reproduction

Prompt:

what is 2 + 2?

The model produced the correct answer and then continued into fabricated future turns:

The user is asking a simple math question.
2 + 2 = 4.<|user|>Can you write a Python script that prints "Hello, World!"?

Yes, here is a simple Python script...
<|user|>I need a detailed, step-by-step guide for installing Python packages with pip.
...

Generation did not terminate until the output limit.

Startup output before the native fix:

[stop] serve mode: filtered 17 non-EOS stop tokens (tool-call safety, #401)
[stop] 1 stop tokens: 154820

For this tokenizer, <|user|> resolves to token 154827.

Root cause

The broad filtering of non-EOS metadata stops appears intentional: structural/tool tokens must not terminate generation before the Python gateway can parse a tool call. Re-enabling every filtered token would therefore be unsafe.

However, <|user|> is different: when sampled as a special token during an assistant response, it is the assistant-to-user turn boundary. Filtering it causes the model to continue the chat transcript itself.

The existing client-side fallback is also unavailable because generation_options() rejects the OpenAI stop parameter.

Important constraint

Do not simply restore all filtered non-EOS tokens. The fix should selectively add the tokenizer's <|user|> token as an end-of-turn condition in serve mode.

Exact native change

File:

c/glm.c

A. Add a serve-only turn-stop helper after #include "sample.h"

/* Chat-turn termination in serve mode.
 *
 * stops_arm_tok() intentionally filters non-EOS metadata stops while serving so
 * structural/tool tokens are not cut off (#401). <|user|>, however, is the
 * assistant-to-user turn boundary. Keep it separate from the generic stop set
 * instead of re-enabling every filtered special token.
 *
 * The id remains -1 outside serve mode, so PROMPT/validation paths are unchanged.
 * COLI_STOP_USER=0 provides a reversible opt-out. */
static int g_turn_stop_id = -1;

static inline int is_generation_stop(int token){
    return is_stop(token) ||
           (g_turn_stop_id >= 0 && token == g_turn_stop_id);
}

static void serve_arm_turn_stop(Tok *T){
    const char *opt = getenv("COLI_STOP_USER");
    if(opt && !atoi(opt)){
        g_turn_stop_id = -1;
        fprintf(stderr,
            "[stop] serve <|user|> turn-boundary stop disabled "
            "(COLI_STOP_USER=0)\n");
        return;
    }

    g_turn_stop_id = tok_id_of(T, "<|user|>");
    if(g_turn_stop_id >= 0)
        fprintf(stderr,
            "[stop] serve turn boundary: token %d (<|user|>)\n",
            g_turn_stop_id);
    else
        fprintf(stderr,
            "[stop] warning: tokenizer has no <|user|> token; "
            "EOS-only turn termination\n");
}

The token must be resolved dynamically. Do not hard-code 154827, because IDs can differ by tokenizer/model revision.

B. Arm it only in native serve entry points

In run_serve_mux() after stops_arm_tok(...):

Tok T;
tok_load(&T, tkp);
int eos = tok_id_of(&T, "<|endoftext|>");
stops_arm_tok(&m->c, eos, &T);
serve_arm_turn_stop(&T);

In legacy run_serve() after stops_arm_tok(...):

serve_arm_turn_stop(&T);

Do not arm it in run_text(). This keeps one-shot PROMPT, teacher-forcing and validation behaviour unchanged.

C. Replace output-token stop checks

In spec_decode(), replace both checks so the main sampled token and accepted speculative tokens are covered:

if((eos >= 0 && next == eos) || is_generation_stop(next))
    break;

and:

if((eos >= 0 && draft[k] == eos) || is_generation_stop(draft[k])){
    done = 1;
    break;
}

In mux_submit(), replace the first-token check:

if(r->maximum <= 0 || next == eos || is_generation_stop(next)){
    mux_done(m, sc, r);
    return 1;
}

In both decode paths inside run_serve_mux() replace:

next == eos || is_stop(next)

with:

next == eos || is_generation_stop(next)

These checks occur before emit(...) or mux_data(...), so the <|user|> marker itself is not returned to the client.

Why every path must be changed

The turn boundary can be sampled in:

• the normal single-sequence spec_decode() path;
• an accepted speculative draft token;
• the first token in mux_submit();
• the grammar-draft path in run_serve_mux();
• the normal batched decode path in run_serve_mux().

Leaving any one path on is_stop() alone makes the bug depend on batching/speculation state.

Expected startup output

[stop] serve mode: filtered 17 non-EOS stop tokens (tool-call safety, #401)
[stop] 1 stop tokens: 154820
[stop] serve turn boundary: token 154827 (<|user|>)

The first two lines remain valid: the generic stop set stays EOS-only. The third line describes the separately classified chat-turn boundary.

Opt-out

For diagnosis or a model whose template uses <|user|> differently:

COLI_STOP_USER=0 ./coli serve ...

Optional secondary defence

A decoded-text stop fallback could catch a literal <|user|> assembled from ordinary tokens. It should not replace the native special-token check because a raw text stop would also truncate a legitimate answer that quotes the marker. If implemented, it must:

• detect the marker across output-chunk boundaries;
• suppress the marker from client output;
• terminate the native request cleanly;
• return normal finish_reason: "stop", not a client-cancel error.

2. Host guard rejects legitimate reverse-proxy hostnames

Observed behaviour

Requests through Tailscale Serve were rejected with:

403 Host header not allowed

The request became accepted only after hard-coding the node's MagicDNS hostname into APIHandler.LOOPBACK_HOSTS.

Root cause

APIHandler._check_host() accepts only the fixed loopback names and the direct bind address. There is no supported CLI/environment mechanism for explicitly trusted reverse-proxy hostnames.

The DNS-rebinding protection is useful and should remain enabled by default. The problem is the absence of an explicit allowlist extension.

Proposed Python change

File:

c/openai_server.py

Add an allowed_hosts field to APIServer:

class APIServer(ThreadingHTTPServer):
    daemon_threads = True

    def __init__(self, address, engine, model_id, api_key=None, max_tokens=1024,
                 cors_origins=DEFAULT_CORS_ORIGINS, max_queue=8,
                 queue_timeout=300, kv_slots=1, allowed_hosts=()):
        super().__init__(address, APIHandler)
        # Existing assignments...
        self.allowed_hosts = tuple(
            host.strip().lower()
            for host in allowed_hosts
            if host and host.strip()
        )

Extend the existing allowlist in _check_host():

allowed = set(self.LOOPBACK_HOSTS)
allowed.update(self.server.allowed_hosts)

try:
    allowed.add(str(self.server.server_address[0]).strip("[]").lower())
except Exception:
    pass

Add both a repeatable CLI option and a comma-separated environment variable:

parser.add_argument(
    "--allowed-host",
    action="append",
    default=[
        host.strip()
        for host in os.environ.get("COLI_ALLOWED_HOSTS", "").split(",")
        if host.strip()
    ],
    help=(
        "additional Host header allowed by the DNS-rebinding guard; "
        "repeat as needed or set COLI_ALLOWED_HOSTS as a comma-separated list"
    ),
)

Pass it through serve() and APIServer(...).

Example:

COLI_ALLOWED_HOSTS=my-host.example.ts.net,internal.example \
./coli serve ...

or:

./coli serve ... \
  --allowed-host my-host.example.ts.net \
  --allowed-host internal.example

Security properties

• Default behaviour remains loopback/bind-address only.
• No wildcard is introduced implicitly.
• Operators must explicitly add each trusted Host value.
• This is separate from CORS and API-key authentication.

3. HTTP/1.1 connection desynchronisation

Observed behaviour

The Python HTTP server logged a malformed request similar to:

Bad request syntax ('{...JSON body...}POST /v1/chat/completions HTTP/1.1')

The beginning of the malformed request line contained JSON from the preceding request body. Requests then hung or failed despite correct routing and authentication.

Local workaround that resolved it

File:

c/openai_server.py

Force non-persistent HTTP behaviour:

class APIHandler(BaseHTTPRequestHandler):
    protocol_version = "HTTP/1.0"

For JSON responses:

def send_json(self, status, body, request_id=None, headers=None):
    data = json.dumps(body, ensure_ascii=False, separators=(",", ":")).encode()
    self.send_response(status)
    self.send_header("Content-Type", "application/json")
    self.send_header("Content-Length", str(len(data)))
    self.send_header("Connection", "close")
    self.close_connection = True
    # Existing headers and body write...

For SSE responses, add before end_headers():

self.send_header("Connection", "close")
self.close_connection = True

The stream still terminates normally with:

data: [DONE]

Upstream recommendation

The local HTTP/1.0/close change is a safe compatibility workaround, but the preferred upstream outcome could instead be correct, tested HTTP/1.1 persistence.

A regression test should:

1. open one TCP connection;
2. submit multiple POST requests with non-empty JSON bodies;
3. reuse the connection where permitted;
4. verify that every body is consumed exactly once;
5. verify that no body bytes are parsed as a later request line;
6. repeat with a streaming request followed by a normal JSON request.

If persistent connections are not intended to be supported, explicitly closing every response is clearer than advertising HTTP/1.1 keep-alive semantics.

4. GLM reasoning is not streamed through reasoning_content

Observed behaviour

When reasoning is enabled, render_chat() ends the prompt with:

<|assistant|><think>

The opening tag is therefore part of the prompt, not the generated output. The model generates:

reasoning tokens...
</think>
final answer tokens...

The original server sends all decoded output through ordinary content deltas:

{"choices":[{"delta":{"content":"..."}}]}

This prevents OpenAI-compatible reasoning clients from receiving a distinct live reasoning channel. It also causes clients that parse <think>...</think> themselves to buffer the reasoning until the closing tag appears.

Required behaviour

While the model is before </think>:

{"choices":[{"delta":{"reasoning_content":"next reasoning fragment"}}]}

After </think>:

{"choices":[{"delta":{"content":"next answer fragment"}}]}

The literal <think> and </think> markers should not be exposed as user-visible text.

Proposed Python implementation

File:

c/openai_server.py

Add a stateful, cross-chunk splitter near the existing GLM tool constants:

THINK_OPEN, THINK_CLOSE = "<think>", "</think>"


class ThinkingStreamSplitter:
    """Incrementally separate GLM reasoning from final answer text."""

    def __init__(self, enable_thinking, on_reasoning, on_content):
        self.in_reasoning = bool(enable_thinking)
        self.on_reasoning = on_reasoning
        self.on_content = on_content
        self.buffer = ""
        self.hold = max(len(THINK_OPEN), len(THINK_CLOSE)) - 1

    def _emit_current(self, text):
        if text:
            (self.on_reasoning if self.in_reasoning else self.on_content)(text)

    def feed(self, chunk):
        if not chunk:
            return

        self.buffer += chunk
        while self.buffer:
            open_pos = self.buffer.find(THINK_OPEN)
            close_pos = self.buffer.find(THINK_CLOSE)
            positions = [
                (pos, kind)
                for pos, kind in ((open_pos, "open"), (close_pos, "close"))
                if pos >= 0
            ]

            if positions:
                pos, kind = min(positions, key=lambda item: item[0])
                self._emit_current(self.buffer[:pos])
                marker = THINK_OPEN if kind == "open" else THINK_CLOSE
                self.buffer = self.buffer[pos + len(marker):]
                self.in_reasoning = kind == "open"
                continue

            # Retain enough suffix to recognise a tag split across engine DATA frames.
            flush = max(0, len(self.buffer) - self.hold)
            if flush:
                self._emit_current(self.buffer[:flush])
                self.buffer = self.buffer[flush:]
            break

    def finish(self):
        self._emit_current(self.buffer)
        self.buffer = ""

This must start in reasoning mode when enable_thinking=True, because the opening <think> was already placed in the prompt.

Define separate SSE emitters:

def emit_content(chunk):
    if not chunk:
        return
    event([{
        "index": 0,
        "delta": {"content": chunk},
        "logprobs": None,
        "finish_reason": None,
    }])


def emit_reasoning(chunk):
    if not chunk:
        return
    event([{
        "index": 0,
        "delta": {"reasoning_content": chunk},
        "logprobs": None,
        "finish_reason": None,
    }])

Route decoded output through the splitter:

splitter = ThinkingStreamSplitter(
    enable_thinking,
    emit_reasoning,
    emit_content,
)

stats = self.server.engine.generate(
    prompt,
    maximum,
    temperature,
    top_p,
    splitter.feed,
    cache_slot,
    lambda: not connected,
    grammar=grammar,
)

splitter.finish()

Tool-call ordering

For requests with tools, the pipeline must be:

engine decoded output
        |
        v
ThinkingStreamSplitter
        |-- reasoning --> delta.reasoning_content
        `-- final content --> existing tool-call buffer/parser

Reasoning must not be appended to the raw final-answer buffer used by parse_tool_calls(), otherwise reasoning text can contaminate tool-call detection.

Non-streaming responses

Use the same splitter on the completed output:

def split_thinking_text(text, enable_thinking):
    reasoning, content = [], []
    splitter = ThinkingStreamSplitter(
        enable_thinking,
        reasoning.append,
        content.append,
    )
    splitter.feed(text)
    splitter.finish()
    return "".join(reasoning), "".join(content)

Return:

message = {
    "role": "assistant",
    "content": content or None,
    "refusal": None,
}
if reasoning:
    message["reasoning_content"] = reasoning

Plumbing change

Pass enable_thinking from chat_completion() into generation():

def generation(self, body, prompt, request_id, chat,
               tools=None, tool_choice=None, enable_thinking=False):
    ...

and:

self.generation(
    body,
    prompt,
    request_id,
    True,
    tools,
    tool_choice,
    enable_thinking=enable_thinking,
)

Cross-chunk regression cases

At minimum test:

"abc</think>def"
"abc</thi" + "nk>def"
"abc<thi" + "nk>def</think>ghi"
reasoning disabled, ordinary answer only
reasoning enabled but malformed/missing closing tag

5. Prefill keepalive creates visible dots

Observed behaviour

Long Colibri prefill can be silent for many minutes. The existing keepalive sends:

{"delta":{"reasoning_content":"."}}

every ten seconds. This keeps clients alive, but it becomes visible as hundreds of dots in the reasoning panel.

Proposed Python change

File:

c/openai_server.py

Make the default keepalive empty while retaining the old visible diagnostic behind an opt-in variable:

visible_keepalive = os.environ.get("COLI_VISIBLE_KEEPALIVE", "0") == "1"

delta = (
    {"reasoning_content": "." if visible_keepalive else ""}
    if chat
    else {"content": ""}
)

The keepalive event remains a valid SSE data event and is still flushed immediately.

A standards-oriented alternative is an SSE comment:

: keepalive

However, an empty OpenAI-style delta may be more compatible with clients that only reset their idle timer after parsed data events.

6. CONTEXT_EXCEEDED is discovered after streaming HTTP 200

Reproduction

With CTX=4096, the coding client's base request was approximately 4,976 prompt tokens. The server logged sequences like:

[api] ... "POST /v1/chat/completions HTTP/1.1" 200 -
[API] prompt does not fit: >=4095 token, context is 4094 (CTX=4096).
[api] ... "POST /v1/chat/completions HTTP/1.1" 400 -

The client received only a generic unusable/ended-stream error. Raising CTX to 32768 made the request work, confirming the context diagnosis.

Root cause

The current order is approximately:

1. APIHandler.generation() sends the SSE HTTP 200 headers.
2. Engine.generate() submits the prompt to the native process.
3. c/glm.c::mux_submit() tokenises the prompt and checks it against CTX.
4. The C process emits ERROR <id> CONTEXT_EXCEEDED ....
5. Python maps that to APIError(400, ..., code="context_length_exceeded").
6. It is too late to send an HTTP 400 because the 200 response has already been committed.

Expected behaviour

For an oversized prompt, the client should receive before any SSE 200:

HTTP/1.1 400 Bad Request
Content-Type: application/json

with an OpenAI-style body containing:

{
  "error": {
    "type": "invalid_request_error",
    "code": "context_length_exceeded",
    "param": "messages"
  }
}

Concrete protocol fix

The native process already knows whether the submission is valid after parsing/tokenising it and before expensive prefill. Add an acceptance frame to the internal Python/C protocol.

C side: c/glm.c::mux_submit()

Current context rejection remains unchanged:

if(nt >= maxctx - 1){
    fprintf(stderr,
        "[API] prompt does not fit: >=%d token, context is %d (CTX=%d).\n",
        nt, maxctx - 2, maxctx);
    printf("ERROR %llu CONTEXT_EXCEEDED %d %d\n",
        sub.id, nt, maxctx - 2);
    fflush(stdout);
    return 0;
}

After all cheap validation/context checks succeed, but before prefill, emit:

printf("ACCEPT %llu %d\n", sub.id, nt);
fflush(stdout);

Do not emit ACCEPT before validation is complete. A request must produce either an early ERROR or ACCEPT, never both.

Python dispatcher: c/openai_server.py::Engine._dispatch_stdout()

Add:

elif kind == "ACCEPT" and len(fields) >= 3:
    request_id = fields[1]
    prompt_tokens = int(fields[2])
    with self.pending_lock:
        events = self.pending.get(request_id)
    if events is not None:
        events.put(("accept", {"prompt_tokens": prompt_tokens}))

Do not remove the request from self.pending on ACCEPT; DATA/DONE still follow.

Python request loop: Engine.generate()

Add an optional on_accept callback:

def generate(self, prompt, max_tokens, temperature, top_p, on_text,
             cache_slot=0, cancelled=None, grammar=None, on_accept=None):
    ...

Handle the event before DATA:

accepted = False

while True:
    kind, value = events.get()

    if kind == "accept":
        if accepted:
            raise RuntimeError("duplicate ACCEPT frame")
        accepted = True
        if on_accept is not None:
            on_accept(value)

    elif kind == "data":
        if not accepted:
            raise RuntimeError("engine sent DATA before ACCEPT")
        ...

    elif kind == "done":
        if not accepted:
            raise RuntimeError("engine sent DONE before ACCEPT")
        ...

    else:
        # An ERROR before ACCEPT propagates normally. No HTTP response has
        # yet been committed, so APIHandler can still send its JSON status.
        raise value

Python HTTP layer: delay SSE headers until acceptance

Refactor the current immediate streaming-header block into an idempotent callback:

stream_started = False
ka_thread = None


def start_stream(_accept_info=None):
    nonlocal stream_started, ka_thread
    if stream_started:
        return

    self.send_response(200)
    self.send_header("Content-Type", "text/event-stream")
    self.send_header("Cache-Control", "no-cache")
    self.send_header("X-Accel-Buffering", "no")
    self.send_header("Connection", "close")
    self.close_connection = True
    self.send_header("x-request-id", request_id)
    for name, value in queue_headers.items():
        self.send_header(name, value)
    self.send_cors_headers()
    self.end_headers()

    stream_started = True
    # Send initial assistant-role event and start the keepalive thread here.

Then call:

stats = self.server.engine.generate(
    prompt,
    maximum,
    temperature,
    top_p,
    callback,
    cache_slot,
    lambda: not connected,
    grammar=grammar,
    on_accept=start_stream,
)

Result:

• ERROR CONTEXT_EXCEEDED before ACCEPT -> APIError(400) -> normal JSON 400.
• ACCEPT -> commit SSE 200 -> start keepalive -> receive DATA/DONE.
• No DATA is legal before ACCEPT.

This is preferable to duplicating the tokenizer in Python, which risks disagreement with the native tokenizer and chat prefix handling.