attachment.zip
Summary
The Rust frontend of vLLM (vllm-rs, enabled via VLLM_USE_RUST_FRONTEND=1 or by running the vllm-rs binary directly) records Prometheus HTTP metrics in the middleware track_http_metrics. It uses the raw HTTP method token of each request (req.method().as_str()) as a Prometheus label value, with no method allowlist, no label normalization, and no cardinality cap. An unauthenticated remote attacker can send requests with unique, arbitrary method tokens (e.g. XVULNCARD000001 /tokenize HTTP/1.1) to any registered route; the request is answered with 405 by axum's method router but still flows through the metrics middleware, permanently inserting new label sets into the global http_requests counter family and http_request_duration_seconds histogram family. Memory grows unboundedly until the process is OOM-killed, and every /metrics scrape becomes larger and more expensive to render.
Details
hyper/httparse accept arbitrary tokens as HTTP methods, and axum's MethodRouter answers wrong-method requests on a registered path with 405 from inside the route service, which is wrapped by Router::layer(from_fn(track_http_metrics)) (src/rust/src/server/src/routes.rs:174). The middleware therefore runs for these 405 responses and records method = <arbitrary token>, handler = <MatchedPath or "none">, status = "4xx".
Family::get_or_create in the prometheus_client crate inserts a permanent HashMap entry keyed by the label set. There is no TTL, no cap, and no eviction: each unique method token pins roughly O(token length) bytes in two label sets plus counter/histogram structures for the lifetime of the process.
- The API-key middleware only guards the
/v1, /v2 and /inference path prefixes (src/rust/src/server/src/middleware/auth.rs:16). /tokenize and /detokenize are not guarded and are not in the metrics EXCLUDED_HANDLERS list, so the attack works even when --api-key is configured: an unauthenticated attacker can inject unbounded label cardinality through /tokenize 405s. Without --api-key (the default), every registered route works.
/metrics itself is unauthenticated and renders the entire registry into a single String on every scrape (Metrics::render, src/rust/src/metrics/src/lib.rs:62-66). With ~10^6 injected series, each scrape materializes hundreds of megabytes of text, amplifying CPU/memory pressure onto the server and the Prometheus monitoring stack (cardinality bomb).
- Damage is cumulative and permanent: unlike request-flooding, the leaked memory is not reclaimed when the attack stops, and rate limiting does not help because each method token only needs to be seen once.
Vulnerable code
src/rust/src/server/src/middleware/metrics.rs (lines 34-71):
pub async fn track_http_metrics(req: Request, next: Next) -> Response {
let method = req.method().as_str().to_string(); // raw method token, no allowlist/normalization
let handler = req
.extensions()
.get::<MatchedPath>()
.map_or_else(|| "none".to_string(), |path| path.as_str().to_string());
let excluded = EXCLUDED_HANDLERS.contains(&handler.as_str());
let started_at = Instant::now();
let response = next.run(req).await;
if excluded {
return response;
}
let elapsed = started_at.elapsed().as_secs_f64();
let status = status_group(response.status().as_u16());
let metrics = &METRICS.api_server;
metrics
.http_requests
.get_or_create(&HttpRequestLabels {
method: method.clone(), // permanently inserts a new series per unique token
status,
handler: handler.clone(),
})
.inc();
metrics
.http_request_duration_seconds
.get_or_create(&HttpHandlerLabels { method, handler })
.observe(elapsed);
metrics.http_request_duration_highr_seconds.observe(elapsed);
response
}
src/rust/src/metrics/src/api_server.rs (lines 26-41) - label sets keyed by the raw method string:
#[derive(Clone, Debug, Hash, PartialEq, Eq, EncodeLabelSet)]
pub struct HttpRequestLabels {
pub method: String,
pub status: &'static str,
pub handler: String,
}
#[derive(Clone, Debug, Hash, PartialEq, Eq, EncodeLabelSet)]
pub struct HttpHandlerLabels {
pub method: String,
pub handler: String,
}
pub(crate) type HttpRequestCounterFamily = Family<HttpRequestLabels, U64Counter>;
pub(crate) type HttpHandlerHistogramFamily =
Family<HttpHandlerLabels, Histogram, fn() -> Histogram>;
src/rust/src/server/src/middleware/auth.rs (line 16) - /tokenize is not guarded, so the attack bypasses --api-key:
const GUARDED_PREFIXES: &[&str] = &["/v1", "/v2", "/inference"];
PoC
XVULNCARD000001 /tokenize HTTP/1.1
Host: TARGET
XVULNCARD000002 /tokenize HTTP/1.1
Host: TARGET
XVULNCARD000003 /tokenize HTTP/1.1
Host: TARGET
GET /metrics HTTP/1.1
Host: TARGET
HTTP/1.1 200 OK
...
http_requests_total{method="XVULNCARD000001",status="4xx",handler="/tokenize"} 1
http_request_duration_seconds_bucket{le="0.1",method="XVULNCARD000001",handler="/tokenize"} 1
http_request_duration_seconds_count{method="XVULNCARD000001",handler="/tokenize"} 1
http_request_duration_seconds_sum{method="XVULNCARD000001",handler="/tokenize"} <ELAPSED>
...
Automated: python poc.py -u http://TARGET:PORT -n 500 sends 500 unique-method requests to /tokenize and then scrapes /metrics, counting the injected method="XVULNCARD…" series before and after. Each unique token contributes several series (counter plus histogram bucket/count/sum lines); the count never decreases afterwards.
Impact
Remote, unauthenticated, cumulative denial of service against deployments that enable the Rust frontend: unbounded and permanent growth of process memory (OOM) plus linearly growing /metrics scrape payloads that also stress the Prometheus monitoring stack. Setting --api-key does not mitigate the attack because /tokenize//detokenize are outside the guarded prefixes. No data confidentiality or integrity impact. vLLM's security policy classifies availability-only issues of this kind as MODERATE.
attachment.zip
Summary
The Rust frontend of vLLM (
vllm-rs, enabled viaVLLM_USE_RUST_FRONTEND=1or by running thevllm-rsbinary directly) records Prometheus HTTP metrics in the middlewaretrack_http_metrics. It uses the raw HTTP method token of each request (req.method().as_str()) as a Prometheus label value, with no method allowlist, no label normalization, and no cardinality cap. An unauthenticated remote attacker can send requests with unique, arbitrary method tokens (e.g.XVULNCARD000001 /tokenize HTTP/1.1) to any registered route; the request is answered with 405 by axum's method router but still flows through the metrics middleware, permanently inserting new label sets into the globalhttp_requestscounter family andhttp_request_duration_secondshistogram family. Memory grows unboundedly until the process is OOM-killed, and every/metricsscrape becomes larger and more expensive to render.Details
hyper/httparseaccept arbitrary tokens as HTTP methods, and axum'sMethodRouteranswers wrong-method requests on a registered path with 405 from inside the route service, which is wrapped byRouter::layer(from_fn(track_http_metrics))(src/rust/src/server/src/routes.rs:174). The middleware therefore runs for these 405 responses and recordsmethod = <arbitrary token>,handler = <MatchedPath or "none">,status = "4xx".Family::get_or_createin theprometheus_clientcrate inserts a permanent HashMap entry keyed by the label set. There is no TTL, no cap, and no eviction: each unique method token pins roughlyO(token length)bytes in two label sets plus counter/histogram structures for the lifetime of the process./v1,/v2and/inferencepath prefixes (src/rust/src/server/src/middleware/auth.rs:16)./tokenizeand/detokenizeare not guarded and are not in the metricsEXCLUDED_HANDLERSlist, so the attack works even when--api-keyis configured: an unauthenticated attacker can inject unbounded label cardinality through/tokenize405s. Without--api-key(the default), every registered route works./metricsitself is unauthenticated and renders the entire registry into a singleStringon every scrape (Metrics::render,src/rust/src/metrics/src/lib.rs:62-66). With ~10^6 injected series, each scrape materializes hundreds of megabytes of text, amplifying CPU/memory pressure onto the server and the Prometheus monitoring stack (cardinality bomb).Vulnerable code
src/rust/src/server/src/middleware/metrics.rs(lines 34-71):src/rust/src/metrics/src/api_server.rs(lines 26-41) - label sets keyed by the raw method string:src/rust/src/server/src/middleware/auth.rs(line 16) -/tokenizeis not guarded, so the attack bypasses--api-key:PoC
Automated:
python poc.py -u http://TARGET:PORT -n 500sends 500 unique-method requests to/tokenizeand then scrapes/metrics, counting the injectedmethod="XVULNCARD…"series before and after. Each unique token contributes several series (counter plus histogram bucket/count/sum lines); the count never decreases afterwards.Impact
Remote, unauthenticated, cumulative denial of service against deployments that enable the Rust frontend: unbounded and permanent growth of process memory (OOM) plus linearly growing
/metricsscrape payloads that also stress the Prometheus monitoring stack. Setting--api-keydoes not mitigate the attack because/tokenize//detokenizeare outside the guarded prefixes. No data confidentiality or integrity impact. vLLM's security policy classifies availability-only issues of this kind as MODERATE.