Skip to content

[Bug]: Prompt-cache not used because OpenClaw is changing the prompt-prefix (inference costs x1000) #45110

Closed
Closed
[Bug]: Prompt-cache not used because OpenClaw is changing the prompt-prefix (inference costs x1000)#45110
Labels
bugSomething isn't workingregressionBehavior that previously worked and now fails

Description

@kripper
kripper
opened on Mar 13, 2026

Bug type

Regression (worked before, now fails)

Summary

OpenClaw is changing the prompt prefix.
This casues the prompt to be recomputed for every message, causing very high inference costs.

I'm seeing additional 40k computed tokens per message. The initial context is like 40k.
Thus, the price is like x times the number of multi-turn messages, and an agenting workflow can have 1000 events.

Steps to reproduce

  • Setup OpenClaw to use llama-server with qwen3-coder-next
  • Run a single OpenClaw agent and make sure to ignore logs during prompt compacting/condensation (prompt-prefix will change).
  • you will see that llama-server is constantly recomputing the prompt after aprox. 70k tokens:
    • Only the first 70k tokens of the prompt uses the existing prompt cache. The rest is recomputed every time.
    • llama-server logs "erased invalidated context checkpoint".
    • "selected slot by LCP similarity" with only "sim_best = 0.665" shows that the prompt prefix changed around token 70k.

llama-server logs

Mar 13 13:59:39 llama-server.sh[150217]: slot update_slots: id  1 | task 19720 | n_tokens = 94208, memory_seq_rm [94208, end)
Mar 13 13:59:39 llama-server.sh[150217]: slot update_slots: id  1 | task 19720 | prompt processing progress, n_tokens = 98304, batch.n_tokens = 4096, progress = 0.946168
Mar 13 13:59:55 llama-server.sh[150217]: srv          stop: cancel task, id_task = 19748
Mar 13 14:00:09 llama-server.sh[150217]: srv          stop: cancel task, id_task = 19750
Mar 13 14:00:12 llama-server.sh[150217]: slot update_slots: id  1 | task 19720 | n_tokens = 98304, memory_seq_rm [98304, end)
Mar 13 14:00:12 llama-server.sh[150217]: slot update_slots: id  1 | task 19720 | 8192 tokens since last checkpoint at 90112, creating new checkpoint during processing at position 99801
Mar 13 14:00:12 llama-server.sh[150217]: slot update_slots: id  1 | task 19720 | prompt processing progress, n_tokens = 99801, batch.n_tokens = 1497, progress = 0.960576
Mar 13 14:00:25 llama-server.sh[150217]: srv          stop: cancel task, id_task = 19753
Mar 13 14:00:28 llama-server.sh[150217]: slot update_slots: id  1 | task 19720 | created context checkpoint 12 of 128 (pos_min = 98303, pos_max = 98303, n_tokens = 98304, size = 75.376 MiB)
Mar 13 14:00:33 llama-server.sh[150217]: slot update_slots: id  1 | task 19720 | n_tokens = 99801, memory_seq_rm [99801, end)
Mar 13 14:00:33 llama-server.sh[150217]: slot update_slots: id  1 | task 19720 | prompt processing progress, n_tokens = 103893, batch.n_tokens = 4092, progress = 0.999961
Mar 13 14:00:37 llama-server.sh[150217]: slot update_slots: id  1 | task 19720 | created context checkpoint 13 of 128 (pos_min = 99800, pos_max = 99800, n_tokens = 99801, size = 75.376 MiB)
Mar 13 14:00:55 llama-server.sh[150217]: srv          stop: cancel task, id_task = 19757
Mar 13 14:00:56 llama-server.sh[150217]: slot update_slots: id  1 | task 19720 | n_tokens = 103893, memory_seq_rm [103893, end)
Mar 13 14:00:56 llama-server.sh[150217]: slot init_sampler: id  1 | task 19720 | init sampler, took 12.41 ms, tokens: text = 103897, total = 103897
Mar 13 14:00:56 llama-server.sh[150217]: slot update_slots: id  1 | task 19720 | prompt processing done, n_tokens = 103897, batch.n_tokens = 4
Mar 13 14:01:09 llama-server.sh[150217]: srv          stop: cancel task, id_task = 19760
Mar 13 14:01:13 llama-server.sh[150217]: slot update_slots: id  1 | task 19720 | created context checkpoint 14 of 128 (pos_min = 103892, pos_max = 103892, n_tokens = 103893, size = 75.376 MiB)
Mar 13 14:01:13 llama-server.sh[150217]: srv  log_server_r: done request: POST /v1/chat/completions 172.19.0.2 200
Mar 13 14:01:49 llama-server.sh[150217]: srv          stop: cancel task, id_task = 19720
Mar 13 14:01:49 llama-server.sh[150217]: slot      release: id  1 | task 19720 | stop processing: n_tokens = 104773, truncated = 0
Mar 13 14:32:05 llama-server.sh[150217]: srv  params_from_: Chat format: peg-native
Mar 13 14:32:05 llama-server.sh[150217]: slot get_availabl: id  1 | task -1 | selected slot by LCP similarity, sim_best = 0.665 (> 0.100 thold), f_keep = 0.675
Mar 13 14:32:05 llama-server.sh[150217]: slot launch_slot_: id  1 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> ?temp-ext -> dist
Mar 13 14:32:05 llama-server.sh[150217]: slot launch_slot_: id  1 | task 20766 | processing task, is_child = 0
Mar 13 14:32:05 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | new prompt, n_ctx_slot = 200192, n_keep = 0, task.n_tokens = 106359
Mar 13 14:32:05 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | n_past = 70731, slot.prompt.tokens.size() = 104773, seq_id = 1, pos_min = 104772, n_swa = 1
Mar 13 14:32:05 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | Checking checkpoint with [103892, 103892] against 70730...
Mar 13 14:32:05 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | Checking checkpoint with [99800, 99800] against 70730...
Mar 13 14:32:05 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | Checking checkpoint with [98303, 98303] against 70730...
Mar 13 14:32:05 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | Checking checkpoint with [90111, 90111] against 70730...
Mar 13 14:32:05 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | Checking checkpoint with [81919, 81919] against 70730...
Mar 13 14:32:05 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | Checking checkpoint with [73727, 73727] against 70730...
Mar 13 14:32:05 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | Checking checkpoint with [65535, 65535] against 70730...
Mar 13 14:32:05 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | restored context checkpoint (pos_min = 65535, pos_max = 65535, n_tokens = 65536, n_past = 65536, size = 75.376 MiB)
Mar 13 14:32:05 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | erased invalidated context checkpoint (pos_min = 73727, pos_max = 73727, n_tokens = 73728, n_swa = 1, size = 75.376 MiB)
Mar 13 14:32:05 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | erased invalidated context checkpoint (pos_min = 81919, pos_max = 81919, n_tokens = 81920, n_swa = 1, size = 75.376 MiB)
Mar 13 14:32:05 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | erased invalidated context checkpoint (pos_min = 90111, pos_max = 90111, n_tokens = 90112, n_swa = 1, size = 75.376 MiB)
Mar 13 14:32:05 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | erased invalidated context checkpoint (pos_min = 98303, pos_max = 98303, n_tokens = 98304, n_swa = 1, size = 75.376 MiB)
Mar 13 14:32:05 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | erased invalidated context checkpoint (pos_min = 99800, pos_max = 99800, n_tokens = 99801, n_swa = 1, size = 75.376 MiB)
Mar 13 14:32:05 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | erased invalidated context checkpoint (pos_min = 103892, pos_max = 103892, n_tokens = 103893, n_swa = 1, size = 75.376 MiB)
Mar 13 14:32:05 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | n_tokens = 65536, memory_seq_rm [65536, end)
Mar 13 14:32:05 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | prompt processing progress, n_tokens = 69632, batch.n_tokens = 4096, progress = 0.654688
Mar 13 14:32:17 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | n_tokens = 69632, memory_seq_rm [69632, end)
Mar 13 14:32:17 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | prompt processing progress, n_tokens = 73728, batch.n_tokens = 4096, progress = 0.693199
Mar 13 14:32:40 llama-server.sh[150217]: srv          stop: cancel task, id_task = 20770
Mar 13 14:32:41 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | n_tokens = 73728, memory_seq_rm [73728, end)
Mar 13 14:32:41 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | 8192 tokens since last checkpoint at 65536, creating new checkpoint during processing at position 77824
Mar 13 14:32:41 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | prompt processing progress, n_tokens = 77824, batch.n_tokens = 4096, progress = 0.731711
Mar 13 14:32:53 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | created context checkpoint 9 of 128 (pos_min = 73727, pos_max = 73727, n_tokens = 73728, size = 75.376 MiB)
Mar 13 14:32:54 llama-server.sh[150217]: srv          stop: cancel task, id_task = 20773
Mar 13 14:33:07 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | n_tokens = 77824, memory_seq_rm [77824, end)
Mar 13 14:33:07 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | prompt processing progress, n_tokens = 81920, batch.n_tokens = 4096, progress = 0.770222
Mar 13 14:33:24 llama-server.sh[150217]: srv          stop: cancel task, id_task = 20777
Mar 13 14:33:33 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | n_tokens = 81920, memory_seq_rm [81920, end)
Mar 13 14:33:33 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | 8192 tokens since last checkpoint at 73728, creating new checkpoint during processing at position 86016
Mar 13 14:33:33 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | prompt processing progress, n_tokens = 86016, batch.n_tokens = 4096, progress = 0.808733
Mar 13 14:33:47 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | created context checkpoint 10 of 128 (pos_min = 81919, pos_max = 81919, n_tokens = 81920, size = 75.376 MiB)
Mar 13 14:33:54 llama-server.sh[150217]: srv          stop: cancel task, id_task = 20781
Mar 13 14:34:02 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | n_tokens = 86016, memory_seq_rm [86016, end)
Mar 13 14:34:02 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | prompt processing progress, n_tokens = 90112, batch.n_tokens = 4096, progress = 0.847244
Mar 13 14:34:24 llama-server.sh[150217]: srv          stop: cancel task, id_task = 20785
Mar 13 14:34:32 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | n_tokens = 90112, memory_seq_rm [90112, end)
Mar 13 14:34:32 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | 8192 tokens since last checkpoint at 81920, creating new checkpoint during processing at position 94208
Mar 13 14:34:32 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | prompt processing progress, n_tokens = 94208, batch.n_tokens = 4096, progress = 0.885755
Mar 13 14:34:46 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | created context checkpoint 11 of 128 (pos_min = 90111, pos_max = 90111, n_tokens = 90112, size = 75.376 MiB)
Mar 13 14:34:54 llama-server.sh[150217]: srv          stop: cancel task, id_task = 20789
Mar 13 14:35:03 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | n_tokens = 94208, memory_seq_rm [94208, end)
Mar 13 14:35:03 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | prompt processing progress, n_tokens = 98304, batch.n_tokens = 4096, progress = 0.924266
Mar 13 14:35:25 llama-server.sh[150217]: srv          stop: cancel task, id_task = 20793
Mar 13 14:35:36 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | n_tokens = 98304, memory_seq_rm [98304, end)
Mar 13 14:35:36 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | 8192 tokens since last checkpoint at 90112, creating new checkpoint during processing at position 102263
Mar 13 14:35:36 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | prompt processing progress, n_tokens = 102263, batch.n_tokens = 3959, progress = 0.961489
Mar 13 14:35:52 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | created context checkpoint 12 of 128 (pos_min = 98303, pos_max = 98303, n_tokens = 98304, size = 75.376 MiB)
Mar 13 14:35:55 llama-server.sh[150217]: srv          stop: cancel task, id_task = 20797
Mar 13 14:36:09 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | n_tokens = 102263, memory_seq_rm [102263, end)
Mar 13 14:36:09 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | prompt processing progress, n_tokens = 106355, batch.n_tokens = 4092, progress = 0.999962
Mar 13 14:36:24 llama-server.sh[150217]: srv          stop: cancel task, id_task = 20801
Mar 13 14:36:25 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | created context checkpoint 13 of 128 (pos_min = 102262, pos_max = 102262, n_tokens = 102263, size = 75.376 MiB)
Mar 13 14:36:39 llama-server.sh[150217]: srv          stop: cancel task, id_task = 20803
Mar 13 14:36:44 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | n_tokens = 106355, memory_seq_rm [106355, end)
Mar 13 14:36:44 llama-server.sh[150217]: slot init_sampler: id  1 | task 20766 | init sampler, took 13.23 ms, tokens: text = 106359, total = 106359
Mar 13 14:36:44 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | prompt processing done, n_tokens = 106359, batch.n_tokens = 4
Mar 13 14:36:55 llama-server.sh[150217]: srv          stop: cancel task, id_task = 20806
Mar 13 14:37:02 llama-server.sh[150217]: slot update_slots: id  1 | task 20766 | created context checkpoint 14 of 128 (pos_min = 106354, pos_max = 106354, n_tokens = 106355, size = 75.376 MiB)
Mar 13 14:37:02 llama-server.sh[150217]: srv  log_server_r: done request: POST /v1/chat/completions 172.19.0.2 200
Mar 13 14:37:03 llama-server.sh[150217]: slot print_timing: id  1 | task 20766 |
Mar 13 14:37:03 llama-server.sh[150217]: prompt eval time =  297528.18 ms / 40823 tokens (    7.29 ms per token,   137.21 tokens per second)
Mar 13 14:37:03 llama-server.sh[150217]:        eval time =    1346.97 ms /    33 tokens (   40.82 ms per token,    24.50 tokens per second)
Mar 13 14:37:03 llama-server.sh[150217]:       total time =  298875.15 ms / 40856 tokens
Mar 13 14:37:03 llama-server.sh[150217]: slot      release: id  1 | task 20766 | stop processing: n_tokens = 106391, truncated = 0
Mar 13 14:37:04 llama-server.sh[150217]: srv  params_from_: Chat format: peg-native
Mar 13 14:37:04 llama-server.sh[150217]: slot get_availabl: id  1 | task -1 | selected slot by LCP similarity, sim_best = 0.999 (> 0.100 thold), f_keep = 1.000
Mar 13 14:37:04 llama-server.sh[150217]: slot launch_slot_: id  1 | task -1 | sampler chain: logits -> ?penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> min-p -> ?xtc -> ?temp-ext -> dist
Mar 13 14:37:04 llama-server.sh[150217]: slot launch_slot_: id  1 | task 20841 | processing task, is_child = 0
Mar 13 14:37:04 llama-server.sh[150217]: slot update_slots: id  1 | task 20841 | new prompt, n_ctx_slot = 200192, n_keep = 0, task.n_tokens = 106448
Mar 13 14:37:04 llama-server.sh[150217]: slot update_slots: id  1 | task 20841 | n_past = 106363, slot.prompt.tokens.size() = 106391, seq_id = 1, pos_min = 106390, n_swa = 1
Mar 13 14:37:04 llama-server.sh[150217]: slot update_slots: id  1 | task 20841 | Checking checkpoint with [106354, 106354] against 106362...
Mar 13 14:37:04 llama-server.sh[150217]: slot update_slots: id  1 | task 20841 | restored context checkpoint (pos_min = 106354, pos_max = 106354, n_tokens = 106355, n_past = 106355, size = 75.376 MiB)
Mar 13 14:37:04 llama-server.sh[150217]: slot update_slots: id  1 | task 20841 | n_tokens = 106355, memory_seq_rm [106355, end)
Mar 13 14:37:04 llama-server.sh[150217]: slot update_slots: id  1 | task 20841 | prompt processing progress, n_tokens = 106444, batch.n_tokens = 89, progress = 0.999962
Mar 13 14:37:04 llama-server.sh[150217]: slot update_slots: id  1 | task 20841 | n_tokens = 106444, memory_seq_rm [106444, end)
Mar 13 14:37:04 llama-server.sh[150217]: slot init_sampler: id  1 | task 20841 | init sampler, took 8.64 ms, tokens: text = 106448, total = 106448
Mar 13 14:37:04 llama-server.sh[150217]: slot update_slots: id  1 | task 20841 | prompt processing done, n_tokens = 106448, batch.n_tokens = 4
Mar 13 14:37:05 llama-server.sh[150217]: slot update_slots: id  1 | task 20841 | created context checkpoint 15 of 128 (pos_min = 106443, pos_max = 106443, n_tokens = 106444, size = 75.376 MiB)
Mar 13 14:37:05 llama-server.sh[150217]: srv  log_server_r: done request: POST /v1/chat/completions 172.19.0.2 200
Mar 13 14:37:06 llama-server.sh[150217]: slot print_timing: id  1 | task 20841 |
Mar 13 14:37:06 llama-server.sh[150217]: prompt eval time =     653.61 ms /    93 tokens (    7.03 ms per token,   142.29 tokens per second)
Mar 13 14:37:06 llama-server.sh[150217]:        eval time =    1891.10 ms /    46 tokens (   41.11 ms per token,    24.32 tokens per second)
Mar 13 14:37:06 llama-server.sh[150217]:       total time =    2544.71 ms /   139 tokens

Expected behavior

Keep prompt prefix.

Actual behavior

Aprox 50% of the whole prompt (140K) is recomputed for every message.

Technical proposal

Use a class to construct and maintain two versions of the full prompt:

  • Fixed version: A version where new events (messages) are simply appended at the end, while the prompt prefix remains unchanged. This is the version we will always sent to the LLM to not break the cache.
  • Optimal version: A version where events are arranged in the ideal order.

When prompt compaction or condensation is triggered, we copy the optimal version into the fixed version, and continue appending new events to the fixed version. Since the cache is invalidated anyway during this process, it’s a good opportunity to switch to the optimal ordering.

Without a framework like this, we will see this problems over and over again.

OpenClaw version

2026.3.8

Operating system

Linux

Model

qwen3-coder-next

Provider / routing chain

All. It's a problem in OpenClaw.
Use llama-server to test.

Impact and severity

HIGH environmental and economical costs.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingregressionBehavior that previously worked and now fails

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions