Skip to content

Unbounded Prometheus label cardinality from attacker-controlled HTTP method tokens in the vLLM Rust frontend metrics middleware (unauthenticated denial of service)

Moderate
jperezdealgaba published GHSA-5fj9-pfhr-6j48 Sep 23, 2026

Package

No package listed

Affected versions

< v0.30.0

Patched versions

>= 0.30.0

Description

attachment.zip

image

Summary

The Rust frontend of vLLM (vllm-rs, enabled via VLLM_USE_RUST_FRONTEND=1 or by running the vllm-rs binary directly) records Prometheus HTTP metrics in the middleware track_http_metrics. It uses the raw HTTP method token of each request (req.method().as_str()) as a Prometheus label value, with no method allowlist, no label normalization, and no cardinality cap. An unauthenticated remote attacker can send requests with unique, arbitrary method tokens (e.g. XVULNCARD000001 /tokenize HTTP/1.1) to any registered route; the request is answered with 405 by axum's method router but still flows through the metrics middleware, permanently inserting new label sets into the global http_requests counter family and http_request_duration_seconds histogram family. Memory grows unboundedly until the process is OOM-killed, and every /metrics scrape becomes larger and more expensive to render.

Details

  • hyper/httparse accept arbitrary tokens as HTTP methods, and axum's MethodRouter answers wrong-method requests on a registered path with 405 from inside the route service, which is wrapped by Router::layer(from_fn(track_http_metrics)) (src/rust/src/server/src/routes.rs:174). The middleware therefore runs for these 405 responses and records method = <arbitrary token>, handler = <MatchedPath or "none">, status = "4xx".
  • Family::get_or_create in the prometheus_client crate inserts a permanent HashMap entry keyed by the label set. There is no TTL, no cap, and no eviction: each unique method token pins roughly O(token length) bytes in two label sets plus counter/histogram structures for the lifetime of the process.
  • The API-key middleware only guards the /v1, /v2 and /inference path prefixes (src/rust/src/server/src/middleware/auth.rs:16). /tokenize and /detokenize are not guarded and are not in the metrics EXCLUDED_HANDLERS list, so the attack works even when --api-key is configured: an unauthenticated attacker can inject unbounded label cardinality through /tokenize 405s. Without --api-key (the default), every registered route works.
  • /metrics itself is unauthenticated and renders the entire registry into a single String on every scrape (Metrics::render, src/rust/src/metrics/src/lib.rs:62-66). With ~10^6 injected series, each scrape materializes hundreds of megabytes of text, amplifying CPU/memory pressure onto the server and the Prometheus monitoring stack (cardinality bomb).
  • Damage is cumulative and permanent: unlike request-flooding, the leaked memory is not reclaimed when the attack stops, and rate limiting does not help because each method token only needs to be seen once.

Vulnerable code

src/rust/src/server/src/middleware/metrics.rs (lines 34-71):

pub async fn track_http_metrics(req: Request, next: Next) -> Response {
    let method = req.method().as_str().to_string();   // raw method token, no allowlist/normalization
    let handler = req
        .extensions()
        .get::<MatchedPath>()
        .map_or_else(|| "none".to_string(), |path| path.as_str().to_string());
    let excluded = EXCLUDED_HANDLERS.contains(&handler.as_str());
    let started_at = Instant::now();

    let response = next.run(req).await;

    if excluded {
        return response;
    }

    let elapsed = started_at.elapsed().as_secs_f64();
    let status = status_group(response.status().as_u16());

    let metrics = &METRICS.api_server;

    metrics
        .http_requests
        .get_or_create(&HttpRequestLabels {
            method: method.clone(),          // permanently inserts a new series per unique token
            status,
            handler: handler.clone(),
        })
        .inc();

    metrics
        .http_request_duration_seconds
        .get_or_create(&HttpHandlerLabels { method, handler })
        .observe(elapsed);

    metrics.http_request_duration_highr_seconds.observe(elapsed);

    response
}

src/rust/src/metrics/src/api_server.rs (lines 26-41) - label sets keyed by the raw method string:

#[derive(Clone, Debug, Hash, PartialEq, Eq, EncodeLabelSet)]
pub struct HttpRequestLabels {
    pub method: String,
    pub status: &'static str,
    pub handler: String,
}

#[derive(Clone, Debug, Hash, PartialEq, Eq, EncodeLabelSet)]
pub struct HttpHandlerLabels {
    pub method: String,
    pub handler: String,
}

pub(crate) type HttpRequestCounterFamily = Family<HttpRequestLabels, U64Counter>;
pub(crate) type HttpHandlerHistogramFamily =
    Family<HttpHandlerLabels, Histogram, fn() -> Histogram>;

src/rust/src/server/src/middleware/auth.rs (line 16) - /tokenize is not guarded, so the attack bypasses --api-key:

const GUARDED_PREFIXES: &[&str] = &["/v1", "/v2", "/inference"];

PoC

XVULNCARD000001 /tokenize HTTP/1.1
Host: TARGET

XVULNCARD000002 /tokenize HTTP/1.1
Host: TARGET

XVULNCARD000003 /tokenize HTTP/1.1
Host: TARGET

GET /metrics HTTP/1.1
Host: TARGET

HTTP/1.1 200 OK
...
http_requests_total{method="XVULNCARD000001",status="4xx",handler="/tokenize"} 1
http_request_duration_seconds_bucket{le="0.1",method="XVULNCARD000001",handler="/tokenize"} 1
http_request_duration_seconds_count{method="XVULNCARD000001",handler="/tokenize"} 1
http_request_duration_seconds_sum{method="XVULNCARD000001",handler="/tokenize"} <ELAPSED>
...

Automated: python poc.py -u http://TARGET:PORT -n 500 sends 500 unique-method requests to /tokenize and then scrapes /metrics, counting the injected method="XVULNCARD…" series before and after. Each unique token contributes several series (counter plus histogram bucket/count/sum lines); the count never decreases afterwards.

Impact

Remote, unauthenticated, cumulative denial of service against deployments that enable the Rust frontend: unbounded and permanent growth of process memory (OOM) plus linearly growing /metrics scrape payloads that also stress the Prometheus monitoring stack. Setting --api-key does not mitigate the attack because /tokenize//detokenize are outside the guarded prefixes. No data confidentiality or integrity impact. vLLM's security policy classifies availability-only issues of this kind as MODERATE.

Severity

Moderate

CVSS overall score

This score calculates overall vulnerability severity from 0 to 10 and is based on the Common Vulnerability Scoring System (CVSS).
/ 10

CVSS v3 base metrics

Attack vector
Network
Attack complexity
High
Privileges required
None
User interaction
None
Scope
Unchanged
Confidentiality
None
Integrity
None
Availability
High

CVSS v3 base metrics

Attack vector: More severe the more the remote (logically and physically) an attacker can be in order to exploit the vulnerability.
Attack complexity: More severe for the least complex attacks.
Privileges required: More severe if no privileges are required.
User interaction: More severe when no user interaction is required.
Scope: More severe when a scope change occurs, e.g. one vulnerable component impacts resources in components beyond its security scope.
Confidentiality: More severe when loss of data confidentiality is highest, measuring the level of data access available to an unauthorized user.
Integrity: More severe when loss of data integrity is the highest, measuring the consequence of data modification possible by an unauthorized user.
Availability: More severe when the loss of impacted component availability is highest.
CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:U/C:N/I:N/A:H

CVE ID

CVE-2026-105759

Weaknesses

Uncontrolled Resource Consumption

The product does not properly control the allocation and maintenance of a limited resource. Learn more on MITRE.

Credits