Skip to content

docs(prefix): document prefixMatchInfoProducerName on prefix-cache-scorer - #3682

Merged
ahg-g merged 1 commit into
llm-d:mainfrom
fjvbn2003:docs/prefix-scorer-producer-name
Oct 30, 2026
Merged

ahg-g merged 1 commit into
llm-d:mainfrom
fjvbn2003:docs/prefix-scorer-producer-name

Conversation

@fjvbn2003

Copy link
Copy Markdown

What type of PR is this?

/kind documentation

What this PR does / why we need it:

prefix-cache-scorer has a prefixMatchInfoProducerName parameter (Config in scorer/prefix/plugin.go), and the approx-prefix-cache-producer README uses it in its multi-instance example, but the scorer README's Configuration table does not list it.

Without it, a producer configured under a custom name is not read by the scorer. Running the v1.0.0 epp (file discovery, three llm-d-inference-sim v1.0.0 backends behind Envoy) with this config:

plugins:
  - name: prefix-producer
    type: approx-prefix-cache-producer
    parameters:
      autoTune: false
      blockSizeTokens: 128
  - name: prefix-scorer
    type: prefix-cache-scorer
  - name: max-score-picker
    type: max-score-picker

At startup the EPP logs:

auto-created default producer producer=approx-prefix-cache-producer/approx-prefix-cache-producer dataKey=PrefixCacheMatchInfoDataKey/approx-prefix-cache-producer consumer=prefix-scorer

After 10 requests, llm_d_epp_plugin_duration_seconds_count shows both producers running on every request (DataProducer and PreRequest, 10 each for prefix-producer and approx-prefix-cache-producer). The scorer reads the auto-created producer, which has default parameters. The Effective raw configuration log line lists only prefix-producer. With prefixMatchInfoProducerName: prefix-producer on the scorer, there is no auto-created prefix producer and only prefix-producer runs.

Changes:

  • scorer/prefix/README.md: add prefixMatchInfoProducerName to the Configuration table, and an operational note describing the auto-created default producer and the auto-created default producer log line.
  • dataproducer/approximateprefix/README.md, multi-instance example: blockSizeTokens: 16 is below the 64-token minimum (Enforce a minimum blockSizeTokens to prevent EPP OOMs from oversized prefix cache indexer #1158), so the EPP logs WARNING: configured blockSizeTokens is below the recommended minimum, overriding it. and both instances use 64. Also, with the default autoTune: true, GetBlockSize uses the endpoint's reported block size instead of blockSizeTokens when the endpoint metrics report one. The example now sets autoTune: false and uses 64 and 128, so the two instances use different block sizes.

Tests:

  • Scorer without prefixMatchInfoProducerName next to a custom-named producer: startup log shows the auto-created producer, and metrics show both producers running per request. With the parameter set, only the configured producer runs.
  • The current multi-instance example logs the blockSizeTokens warning at startup. The updated example starts without it and serves requests (5/5 returned 200).

The runs were made on 2026-10-26 (UTC) with the v1.0.0 release epp (commit b9393d1). Both READMEs and the scorer and producer code are unchanged on main at 6e5f3f0 (2026-10-27), and the patch applies to it as is. Autotuning was not checked against a real vLLM endpoint. The simulator reports block_size="0", so the autoTune part of the change is based on reading GetBlockSize.

Not changed here: prefix-cache-affinity-filter also accepts prefixMatchInfoProducerName (and latencyPredictionInfoProducerName, inFlightLoadProducerName), and its README Config table does not list them either. That can be a separate PR if it helps.

Which issue(s) this PR fixes:

None. This is a documentation correction. The missing warning for an unconsumed producer is tracked in #3680; this PR does not change code.

Release note (write NONE if no user-facing change):

NONE

AI disclosure: this change was prepared with the help of an AI coding assistant (Claude Code). Claude ran the checks above in a container in my home lab and drafted the diff and this description.

🤖 Generated with Claude Code

…orer

The scorer README did not list prefixMatchInfoProducerName. Without it,
a prefix producer configured under a custom name is not read by the
scorer: the EPP auto-creates a default approx-prefix-cache-producer and
the scorer uses that one, while the configured producer still runs on
every request.

The multi-instance example in the approx-prefix-cache-producer README
set blockSizeTokens: 16, which the producer raises to the 64-token
minimum, and left autoTune on, which replaces the configured block size
with the endpoint's when its metrics report one. Use values at or above
the minimum and turn autoTune off so the two producers differ.

Signed-off-by: YoungjuKim <[email protected]>
@fjvbn2003
fjvbn2003 requested review from a team, liu-cong and vMaroon as code owners October 27, 2026 09:47
@github-actions github-actions Bot added size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. kind/documentation Categorizes issue or PR as related to documentation. area/epp area/scheduling labels Oct 27, 2026
@ahg-g
ahg-g added this pull request to the merge queue Oct 30, 2026
Merged via the queue into llm-d:main with commit 17e4d90 Oct 30, 2026
46 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/epp area/scheduling kind/documentation Categorizes issue or PR as related to documentation. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants