Repository navigation
docs(prefix): document prefixMatchInfoProducerName on prefix-cache-scorer - #3682
Merged
Merged
Conversation
…orer The scorer README did not list prefixMatchInfoProducerName. Without it, a prefix producer configured under a custom name is not read by the scorer: the EPP auto-creates a default approx-prefix-cache-producer and the scorer uses that one, while the configured producer still runs on every request. The multi-instance example in the approx-prefix-cache-producer README set blockSizeTokens: 16, which the producer raises to the 64-token minimum, and left autoTune on, which replaces the configured block size with the endpoint's when its metrics report one. Use values at or above the minimum and turn autoTune off so the two producers differ. Signed-off-by: YoungjuKim <[email protected]>
ahg-g
approved these changes
Oct 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
/kind documentation
What this PR does / why we need it:
prefix-cache-scorerhas aprefixMatchInfoProducerNameparameter (Configinscorer/prefix/plugin.go), and theapprox-prefix-cache-producerREADME uses it in its multi-instance example, but the scorer README's Configuration table does not list it.Without it, a producer configured under a custom name is not read by the scorer. Running the v1.0.0
epp(file discovery, threellm-d-inference-simv1.0.0 backends behind Envoy) with this config:At startup the EPP logs:
After 10 requests,
llm_d_epp_plugin_duration_seconds_countshows both producers running on every request (DataProducerandPreRequest, 10 each forprefix-producerandapprox-prefix-cache-producer). The scorer reads the auto-created producer, which has default parameters. TheEffective raw configurationlog line lists onlyprefix-producer. WithprefixMatchInfoProducerName: prefix-produceron the scorer, there is no auto-created prefix producer and onlyprefix-producerruns.Changes:
scorer/prefix/README.md: addprefixMatchInfoProducerNameto the Configuration table, and an operational note describing the auto-created default producer and theauto-created default producerlog line.dataproducer/approximateprefix/README.md, multi-instance example:blockSizeTokens: 16is below the 64-token minimum (Enforce a minimumblockSizeTokensto prevent EPP OOMs from oversized prefix cache indexer #1158), so the EPP logsWARNING: configured blockSizeTokens is below the recommended minimum, overriding it.and both instances use 64. Also, with the defaultautoTune: true,GetBlockSizeuses the endpoint's reported block size instead ofblockSizeTokenswhen the endpoint metrics report one. The example now setsautoTune: falseand uses 64 and 128, so the two instances use different block sizes.Tests:
prefixMatchInfoProducerNamenext to a custom-named producer: startup log shows the auto-created producer, and metrics show both producers running per request. With the parameter set, only the configured producer runs.blockSizeTokenswarning at startup. The updated example starts without it and serves requests (5/5 returned 200).The runs were made on 2026-10-26 (UTC) with the v1.0.0 release
epp(commit b9393d1). Both READMEs and the scorer and producer code are unchanged on main at 6e5f3f0 (2026-10-27), and the patch applies to it as is. Autotuning was not checked against a real vLLM endpoint. The simulator reportsblock_size="0", so theautoTunepart of the change is based on readingGetBlockSize.Not changed here:
prefix-cache-affinity-filteralso acceptsprefixMatchInfoProducerName(andlatencyPredictionInfoProducerName,inFlightLoadProducerName), and its README Config table does not list them either. That can be a separate PR if it helps.Which issue(s) this PR fixes:
None. This is a documentation correction. The missing warning for an unconsumed producer is tracked in #3680; this PR does not change code.
Release note (write
NONEif no user-facing change):AI disclosure: this change was prepared with the help of an AI coding assistant (Claude Code). Claude ran the checks above in a container in my home lab and drafted the diff and this description.
🤖 Generated with Claude Code