Skip to content

Instantly share code, notes, and snippets.

@louzt
Last active August 16, 2026 01:44
Show Gist options
  • Select an option

  • Save louzt/7bdf370a28126718e7e7b69d53b0ae86 to your computer and use it in GitHub Desktop.

Select an option

Save louzt/7bdf370a28126718e7e7b69d53b0ae86 to your computer and use it in GitHub Desktop.
NGINX runtime CRLF injection evidence and branch-split validation

NGINX runtime CRLF injection evidence and multi-layer architecture mapping

NGINX Runtime CRLF Injection Evidence and Multi-Layer Architecture Mapping

Current status (2026-08-15): Paired PR active — nginx-tests#97 (regression suite, +125/-0) and nginx#1627 (C patch, +9/-3 on src/http/ngx_http_core_module.c). Both marked Ready for Review; verified and routed to @pluknet by @sbhowmikf5. Master delta is isolated strictly to this atomic pair.

Multi-Layer Context: RFC 9110/9112/9113/9114 compliance across NGINX spans three decoupled architectural layers: (1) Inbound client parser strictness (#1424, #1404, #1543, #1554, #1602, #1572), (2) Internal header-token negotiation (#1627 + nginx-tests#97, #1577, #1606), and (3) Outbound serialization & runtime variable sanitization (#1414, #590, #1593, nginx-tests#55, #58). Maintaining strict separation between these layers prevents cross-subsystem regressions.

0. Historical Context & Maintainer Handoff

This work is the continuation of a multi-year hardening arc targeting HTTP parser differentials and runtime CRLF injection in NGINX. §0.1 establishes the threat-model frame and operational severity per layer; §0.2–§0.5 document the maintainer-approved lineage of every PR cited in the matrix that follows. The investigation, prototyping, and benchmarking that produced the upstream test companions cited here predate the gist itself by more than a year; the lab harness details referenced in §6.5 are reproduced inline below and do not require access to the originating local repository.

0.1. Threat Model & Defense Vector Taxonomy

The layered fixes catalogued in this gist defend against three intersecting HTTP attack surface categories, each with a distinct attack class, attacker capability profile, and operational severity:

Vector category Layer defended Attack class Severity
H1/H2/H3 parser differentials — leading/trailing WSp, NUL/CR/LF, tchar violations Layer 1 — Inbound framing (§3, §4) Reverse-proxy request smuggling, cache poisoning via upstream parser differential High — actively exploited in CDN/load-balancer deployments (cf. PortSwigger disclosures 2019–2024)
Token-list value differentials — q-value parsing, multi-token boundaries, OWS/HTAB in content-coding lists Layer 2 — List negotiation (§3, §4) Content-encoding downgrade via Accept-Encoding q-list manipulation, cache poisoning via Cache-Control token differential Medium — proxy-chain diversity dependent
Runtime variable CRLF injection — proxy_pass + $uri, proxy_set_header + $uri, add_header + $uri, add_trailer + $uri Layer 3 — Outbound serialization (§3, §4, §8) Proxy→backend request smuggling, response header injection, cache poisoning via emitted CRLF Critical — confirmed at runtime in §8 baseline

Why three layers, not one monolith. Each layer addresses a distinct attacker capability: well-resourced attackers craft non-spec-compliant framing (Layer 1); intermediate attackers manipulate header values to coerce content-negotiation differentials (Layer 2); opportunistic attackers exploit runtime-variable expansion (Layer 3). Monolithic fix attempts introduce cross-subsystem regression risk because each layer's test harness and CI lane is independent (§15.6).

Adjacent attack surface — token negotiation as collaboration vehicle. Layer 2 is captured here not as a primary defense vector but as adjacent work where the lab's protocol-layer mapping provides architectural context to upstream rewrites: nginx#1577 (multi-header helper rework) explicitly carves out ngx_http_gzip_accept_encoding for surgical fixes (nginx#1627); the same RFC 9110 §5.6.3 OWS/HTAB surface is shared across Accept-Encoding, Accept-Language, and Cache-Control parsing. Defense-in-depth benefits from keeping these adjacent slices visible without conflating them with the primary CRLF arc.

0.2. DemiMarie phased handoff (the canonical ancestry)

In nginx#590, DemiMarie split the runtime CRLF-injection fix into discrete phases. Phases 1–3 are merged; phase 4+ was handed off directly to @louzt:

Phase PR Concern Status
1 nginx#1404 H1 OWS strip on header field lines Merged (DemiMarie)
2 nginx#1410 NULL-safety for cl->buf chain links Merged (DemiMarie)
2 nginx#1411 Zero-initialize allocated chain links (so explicit NULL assignment is not needed) Merged (DemiMarie)
3 nginx#1424 H2/H3 inbound header field validation (reject leading/trailing WSp + NUL/CR/LF) Merged (DemiMarie)
4+ nginx#1404 follow-ups, OWS in HTTP/1.x, layered defense Handed off to @louzt per DemiMarie's direct ask: "Would you be interested in finishing this work? I've done phases 1 through 3 (#1404, #1410, #1411, #1424), but regretfully I don't have time to finish the remaining steps." (issue #590, DemiMarie comment)

The split itself is DemiMarie's architectural decision. In nginx#590 DemiMarie also confirmed the long-form scope: "Fortunately, I believe the HTTP/1.x, HTTP/2, and HTTP/3 parsers all do the bare minimum validation needed to not be caught by the egress filters. The only exceptions are #1201 and #1404." This justifies continuing the Layer-3 outbound slice separately from the inbound validation track.

The downstream test companion for Demi's strict H/1.x parsing work is nginx-tests#41 (open), authored by DemiMarie.

0.3. VadimZhestikov roadmap (config-time validation + log-level guidance)

Also in nginx#590, VadimZhestikov posted the maintainer-side roadmap that scopes the remaining work:

  1. Inconsistent log levels for the same category of event. The same class of event (runtime detection of CRLF injection attempt) is logged at different levels across modules:

    • ngx_http_headers_filter_module → NGX_LOG_ERR
    • ngx_http_proxy_module → NGX_LOG_ALERT
    • ngx_http_proxy_v2_module → NGX_LOG_ALERT
    • Internal double-check (write pass) → NGX_LOG_EMERG

    Vadim's guidance: a user-triggered CRLF injection attempt should be NGX_LOG_ERR (operator needs to know their config is vulnerable, but this is not a server emergency). The internal "this should have been caught" double-check should be NGX_LOG_ALERT (it indicates a bug in nginx itself). NGX_LOG_EMERG is reserved for conditions that prevent nginx from operating at all.

  2. Config-time validation is missing for many header-setting directives. Config-time validation in nginx#590 covers only add_header and proxy_set_header. The following directives also configure header name/value pairs and are not covered:

    • fastcgi_param / uwsgi_param / scgi_param
    • grpc_set_header
    • add_trailer
    • proxy_hide_header / fastcgi_hide_header

    Vadim's note: inconsistent coverage means operators using fastcgi_param with a malformed value get no startup-time warning, only a runtime 500. The config-time check is valuable precisely because it catches mistakes before deployment.

  3. Unrelated fix bundled in: the patch carries cl->next = NULL as a defensive NULL-assignment after cl->buf = b;. This was later split off as nginx#1411 (zero-initialize allocated chain links).

Vadim's parent PR rework on multi-header token matching is nginx#1577 (open, iterating), which explicitly carves out ngx_http_gzip_accept_encoding from the multi-header helper rewrite — that carve-out is the upstream-maintainer-approved scope for nginx#1627 (this gist's paired PR).

0.4. RFC Layer Decoupling Across Review Threads

The maintainer-approved layering (Demi's phased handoff per §0.2, Vadim's roadmap per §0.3) extends to concurrent RFC clause work: inbound framing validation (RFC 9113 §8.2.1 / RFC 9114 §10.3), content-coding list negotiation (RFC 9110 §5.6.3 / §8.4), and outbound serialization (RFC 9112 outbound safety) operate at three decoupled layers of the request lifecycle. PRs that span multiple layers introduce cross-subsystem regression risk without commensurate benefit; each slice is anchored to a single RFC clause and validated against its dedicated regression companion. The structural principle governing self-contained review slices is detailed in §15.6.

0.5. Timeline (compressed)

Window Focus
~2025 (pre-gist) Initial CRLF injection reproduction; local test harness iteration against nginx-tests Perl framework. nginx#590 discussion thread active.
2026-06-02 First revision of this gist (initial triage + reproduction).
2026-06-21 DemiMarie directed drop of name-validation from PR scope; the surgical-PR strategy crystallizes.
2026-07 → 2026-08 Lab benchmarking + cross-layer mapping; anuj1520 PR #1554 appears with overlap risk on inbound parser direction.
2026-08-07 Layer-3 isolation documented (response-filter separate from field-name validation); boundary cross-checked against §12 Hardened Outbound Boundary.
2026-08-08 Paired PR nginx-tests#97 + nginx#1627 shipped; both Ready for Review.
2026-08-15 This revision: full architectural lifecycle mapping, local test harness catalog (§6.5), full Review Track Matrix (5→19 entries, §15), #1554 file-level overlap audit (§15.5), Demi's phased handoff + Vadim's roadmap + RFC layer decoupling principle (§0.4, §15.6). Threat Model & Defense Vector Taxonomy (§0.1) added as the framing layer before the historical lineage. All internal lab references inlined so the gist is self-contained for maintainer review.

1. Scope

The narrow question is whether NGINX should refuse to serialize CR/LF/NUL or invalid field names when generating HTTP/1.x output from runtime variables.

This is fundamentally separate from:

  • inbound RFC 9110 strictness decisions on client requests;
  • HTTP/2 and HTTP/3 inbound framing validators, which have protocol-specific rules;
  • config-time validation for static directives;
  • internal header list negotiation helpers.

2. Thesis

NGINX does not need to become globally stricter on inbound HTTP parsing to resolve runtime CRLF injection vectors. The immediate security boundary is outbound HTTP/1.x serialization from runtime-expanded variables.

Preserving this architectural distinction maintains NGINX's compatibility posture on inbound traffic while establishing an explicit trust boundary on emitted bytes. A controlled sanitization/5xx before bytes leave NGINX represents an atomic, reviewable change without risking regressions in unrelated protocol parsers.

3. Defense-in-Depth Architectural Lifecycle

flowchart TD
    subgraph Layer1["Layer 1: Inbound Client Parsing Boundary (RFC 9112 §2.2 / RFC 9113 §8.2.1 / RFC 9114 §10.3 / RFC 9110 §5.6.2)"]
        A[Client Request H1 / H2 / H3] --> B{Inbound Framing Validation}
        B -->|Reject Leading/Trailing WSp| C["#1424 / #1554 (H2/H3 framing reject)"]
        B -->|Strip OWS at Header Line| D["#1404 / #1602 (RFC 9112 §2.2 H1 OWS strip)"]
        B -->|Validate Field Names| E["#1543 (RFC 9110 §5.6.2 tchar)"]
    end

    subgraph Layer2["Layer 2: Token-List Negotiation Boundary (RFC 9110 §5.6.3 / §8.4)"]
        C --> F{Internal Value Processing}
        D --> F
        F -->|Multi-Header Directives| G["#1577 (Cache-Control / Connection / TE)"]
        F -->|Parameterized Content-Coding| H["#1627 + nginx-tests#97 (Accept-Encoding q-list)"]
    end

    subgraph Layer3["Layer 3: Outbound Serialization Boundary (RFC 9112 outbound safety)"]
        G --> I{Runtime Variable Expansion}
        H --> I
        I -->|Runtime $uri in Outbound Headers| J{Contains CR / LF / NUL?}
        J -->|Proxy Path| K["500 / Controlled Rejection (#590, nginx-tests#55)"]
        J -->|Response Filter Path| L["Sanitize to SP (#1414, nginx-tests#58)"]
        J -->|No Injected Bytes| M[Emit HTTP/1.x Wire Output]
    end
Loading

Inbound and outbound layers are complementary and non-interchangeable. Inbound validation rejects malformed client requests at framing time; outbound serialization prevents parameter injection at emission time. Coupling them into monolithic patches introduces cross-layer regression risks (e.g., breaking upstream cookie parsing while attempting OWS cleanup).

3.1. Layer 2 boundary-check carve-out (gzip Accept-Encoding)

Layer 2 token-list negotiation includes gzip Accept-Encoding as a special case because ngx_http_gzip_accept_encoding() in ngx_http_core_module.c carries its own boundary check independent of the multi-header parser. Vadim's nginx#1577 multi-header parser rework explicitly carved this helper out because teaching the multi-header parser to accept HTAB does not retroactively teach this helper — it has its own switch with case ' ' and exits on the bare comma that ends the list, so HTAB falls into the default branch and the encoding is rejected.

This is why nginx#1627 (gzip AE HTAB) is a separate, surgical 9-line patch to ngx_http_core_module.c rather than a derivative of #1577. The carve-out keeps Layer 2 reviewable: the helper-specific fix lives next to the helper-specific test (nginx-tests#97), without entangling the broader multi-header parser work. The same boundary-check pattern is documented in the §0.3 Threat Model's Adjacent attack surface note.

4. Architectural Layer Matrix

# Layer Target Boundary Focus & Safety Invariant Active PR Tracks
1 Inbound Parsing Client request ingestion Reject/strip malformed client inputs per RFC 9113/9114/9110 nginx#1424 (Demi H2/H3 reject), nginx#1404 (Demi H1 OWS strip), nginx#1543 (saikrishna field names), nginx#1554 (anuj1520 parser sweep), nginx#1602 (HRITHIK HTAB strip — closed precursor to #1404), nginx#1572 (Aayush7352 H2/H3 strip — closed precursor to #1424)
2 Token Negotiation Content-coding / Directive lookup Parse RFC 9110 #rule lists with OWS/HTAB and quality parameters nginx#1627 + nginx-tests#97 (gzip AE HTAB), nginx#1577 (Vadim multi-header token boundary), nginx#1606 (SEPURI-SAI-KRISHNA early-hints header accounting)
3 Outbound Serialization Upstream emit & response filters Neutralize CRLF injection from runtime variables ($uri, $args) nginx#1414 (response-filter value sanitize), nginx#590 (proxy request boundary), nginx#1593 (arut Host header special handling)
4 Regression Harness Native test suites Deterministic regression validation across nginx-tests nginx-tests#55 (proxy), #56 (H1 OWS), #57 (H2/H3 OWS), #58 (header filter), #97 (gzip AE), #80 (Aayush7352 H2 strip test), #89 (SEPURI-SAI-KRISHNA early-hints)

5. Methodology

The verification workflow is structured to mirror upstream maintainer practices:

  • clean branch checkouts from origin/master;
  • native nginx-tests Perl harnesses executed against unpatched master and candidate branches;
  • raw socket captures to inspect wire-level byte emission;
  • memory-safety validation under AddressSanitizer and UndefinedBehaviorSanitizer (-fsanitize=address,undefined);
  • reproducible benchmark checks for throughput impact under valid traffic.

Each candidate branch is validated under the same flow: configure with --with-debug --with-cc-opt='-fsanitize=address,undefined -fno-omit-frame-pointer'; run the upstream nginx-tests Perl harness via prove -v against both the patched branch and the parallel gzip.t / header_filter.t regression suites to surface cross-module fallout; capture raw socket bytes for the cases where the assertion is wire-format rather than HTTP-status; tabulate the resulting plan(N) + 2 implicit count against the §9 Verification Matrix. The local test files driving this flow are catalogued in §6.5.

6. Upstream Artifact Index

Response-Filter Sanitization Slice (Layer 3):

Gzip Accept-Encoding HTAB Negotiation (Layer 2):

Proxy Output Hardening & Handoff:

Inbound Protocol Validation Tracks (Layer 1):

Shared Multi-Header Token Matching:

Trailer & Hop-by-Hop Adjacent Work (boundary-crossing):

6.5. Local Lab Test Suite Catalog

These test files are local validation scripts used before any upstream PR is opened. They are NOT submitted to nginx/nginx-tests directly — they are the harness that drives the upstream PR test companions. The file paths below are relative to the local test directory; the upstream-facing test that ships in nginx-tests#97 is derived from the local gzip_accept_encoding_htab.t.

Test File plan(N) Layer Policy Tested Scope
proxy_crlf_injection_runtime.t 8 Layer 3 (proxy + response) REJECT 500 proxy_pass $uri, proxy_set_header $uri (CR/LF/CR-only), add_header, add_trailer — every path returns 500 and asserts no X-Injected field reaches upstream
proxy_crlf_injection_sanitize_runtime.t 10 Layer 3 (proxy + response) MIXED: proxy rejects 500, response sanitizes to SP Same scope as above but response path keeps 200 OK with CRLF replaced by SP inside the same field value; proxy path still rejects
gzip_accept_encoding_htab.t 7 + 2 implicit = 9 Layer 2 ACCEPT HTAB as OWS 4 baseline (SP, no-whitespace, HTAB-after-comma, malformed rejection) + 3 TODO (HTAB-before-comma, HTAB-before-qvalue, HTAB-after-qvalue-before-comma). This is the upstream-facing test that ships in nginx-tests#97
h2_header_ows.t, h3_header_ows.t — Layer 1 STRIP OWS Upstream-facing tests for Demi's H2/H3 reject logic; ship in nginx-tests#57
http_header_ows.t — Layer 1 STRIP OWS Upstream-facing test for Demi's H1 OWS strip; ships in nginx-tests#56
header_filter_sanitize.t 6 Layer 3 SANITIZE to SP Upstream-facing test for response filter value sanitization; ships in nginx-tests#58

Test coverage matrix — what each scenario verifies

Scenario proxy_crlf_injection_runtime.t proxy_crlf_injection_sanitize_runtime.t gzip_accept_encoding_htab.t header_filter_sanitize.t
proxy_pass + CRLF in $uri ✅ reject 500 ✅ reject 500 — —
proxy_set_header + CRLF in $uri ✅ reject 500 ✅ reject 500 — —
proxy_set_header + LF only ✅ reject 500 ✅ reject 500 — —
proxy_set_header + CR only ✅ reject 500 ✅ reject 500 — —
add_header + CRLF ✅ reject 500 ✅ sanitize to SP, keep 200 — ✅ sanitize to SP
add_trailer + CRLF ✅ reject 500 ✅ sanitize to SP, keep 200 — ✅ sanitize to SP
Accept-Encoding HTAB before comma — — ✅ (TODO before patch) —
Accept-Encoding HTAB before q-value — — ✅ (TODO before patch) —
Accept-Encoding HTAB after q-value — — ✅ (TODO before patch) —
Malformed gzip\tdeflate (no comma) — — ✅ rejected (RFC 9110 §5.6.2) —
no alerts / no sanitizer errors implicit implicit ✅ ok 8-9 ✅ ok 6

How the local tests map to the upstream PRs

flowchart LR
    subgraph LAB["Local REJECT/SANITIZE policy harness"]
        A1["proxy_crlf_injection_runtime.t<br/>REJECT policy"]
        A2["proxy_crlf_injection_sanitize_runtime.t<br/>SANITIZE policy"]
    end
    subgraph WT["Local upstream-facing test (drives nginx-tests#56/#57/#58/#97)"]
        B1["gzip_accept_encoding_htab.t"]
        B2["h2_header_ows.t + h3_header_ows.t"]
        B3["http_header_ows.t"]
        B4["header_filter_sanitize.t"]
    end
    subgraph UP["Upstream PR test companions (nginx/nginx-tests)"]
        C1["nginx-tests#55<br/>(proxy CRLF)"]
        C2["nginx-tests#58<br/>(header filter sanitize)"]
        C3["nginx-tests#56<br/>(H1 OWS)"]
        C4["nginx-tests#57<br/>(H2/H3 OWS)"]
        C5["nginx-tests#97<br/>(gzip AE HTAB)"]
    end
    A1 -. "derived upstream variant" .-> C1
    A2 -. "derived upstream variant" .-> C2
    B3 -. "submitted as" .-> C3
    B2 -. "submitted as" .-> C4
    B1 -. "submitted as" .-> C5
    B4 -. "submitted as" .-> C2
Loading

Cross-protocol coverage matrix — which protocol each layer covers

Layer / Concern H1 (HTTP/1.x) H2 (HTTP/2) H3 (HTTP/3)
Layer 1 — Inbound parsing nginx-tests#56 (http_header_ows.t, 9 subtests) nginx-tests#57 (h2_header_ows.t, 8 subtests) + nginx-tests#80 (h2_headers.t, 116 subtests total) nginx-tests#57 (h3_header_ows.t, 8 subtests)
Layer 2 — Token negotiation (covered indirectly via shared parsing; gzip AE helper in ngx_http_core_module.c is the explicit carve-out per §3.1) — —
Layer 3 — Outbound serialization nginx-tests#55 (proxy_crlf_injection_runtime.t, 8 subtests) + nginx-tests#58 (header_filter_sanitize.t, 6 subtests) (HTTP/1.x outbound is the serialization surface; H2/H3 do not re-encode headers in this layer) (same)
Cross-layer — Early Hints hop-by-hop nginx-tests#89 (12 subtests across protocols) nginx-tests#89 nginx-tests#89
Cross-layer — HTTP/2 pseudo-header ordering — nginx-tests#75 (17 subtests) —

The test harness covers all three protocols at Layer 1 (the most fragmented layer, where each protocol parser has its own boundary check), HTTP/1.x for the runtime CRLF surface (the protocol where outbound serialization actually emits raw CRLF/NUL bytes), and a single cross-protocol suite for early-hints hop-by-hop behavior (nginx-tests#89) since the same concern spans H1, H2, and H3.

The proxy_crlf_injection_*.t files exercise both REJECT and SANITIZE policy choices on the same scenarios, so the maintainer-facing nginx-tests#55 (REJECT) and #58 (SANITIZE) can be derived from a single source of truth rather than maintained as parallel implementations.

Test::Nginx TODO: { local $TODO = ... } convention — assertions wrapped in a TODO block are expected-to-fail on master (the failure is reported as ok N # TODO <reason>, a passing TODO). The 3 TODO positions in gzip_accept_encoding_htab.t document the exact master-side HTAB parse gap and become real assertions in a one-line follow-up commit once the patch lands; they are not removed from the test companion itself.

7. Local Test Execution

Candidate test suites:

nginx-tests/proxy_crlf_injection_runtime.t
nginx-tests/proxy_crlf_injection_sanitize_runtime.t
nginx-tests/header_filter_sanitize.t
nginx-tests/http_header_ows.t
nginx-tests/h2_header_ows.t
nginx-tests/h3_header_ows.t
nginx-tests/gzip_accept_encoding_htab.t

Each layer was validated end-to-end against the upstream nginx-tests Perl harness. Layer 3 tests ran against the response-filter sanitization branch with AddressSanitizer and UndefinedBehaviorSanitizer variants both clean; Layer 2 tests ran against the gzip Accept-Encoding HTAB branch and the parallel gzip.t regression suite. Validation follows the configure / prove flow described in §5 Methodology. Per-branch results (plan counts, ASAN/UBSAN outcome, cross-module regression delta) are tabulated in §9 Verification Matrix across Branches.

8. Baseline Result on Current Master

Observed against a locally built nginx/1.31.x from nginx/nginx master:

  • proxy_pass http://127.0.0.1:<port>/$uri emits an injected upstream request header when $uri contains %0d%0aX-Injected:%20yes.
  • proxy_set_header X-Normalized-URI $uri emits an injected upstream request header with the same payload.
  • add_header X-Original-Path $uri always emits injected response header fields.
  • add_trailer X-Original-Path $uri always emits injected trailer fields.
  • header_filter_sanitize.t on master: 3/6 pass; failing assertions check that an injected field line is not emitted.
  • gzip_accept_encoding_htab.t on master: 6/9 pass; the 3 failing assertions are TODO positions for HTAB before/after q-values and commas in ngx_http_gzip_accept_encoding().

Captured upstream request generated by proxy_pass:

GET //uri/
X-Injected: yes HTTP/1.1
Host: 127.0.0.1:<port>

Visible injected header:

X-Original-Path: /resp-header/
X-Injected: yes

9. Verification Matrix across Branches

| Branch / Slice | Exit | Passed | Failed | Scope & Observations |
| --- | ---: | ---: | ---: | --- |
| nginx-master (1.31.x baseline) | 1 | 3 | 3 | header_filter_sanitize.t: add_header/add_trailer CRLF injection observed |
| nginx/nginx#1424 (Demi H2/H3 reject) | 0 | 8 | 0 | h2_header_ows.t + h3_header_ows.t: 8/8 each (inbound framing clean) |
| nginx/nginx#1404 (Demi H1 OWS strip) | 0 | 9 | 0 | http_header_ows.t: 9/9 (H1 field boundary OWS strip clean) |
| nginx/nginx#1414 (Value sanitize) | 0 | 6 | 0 | header_filter_sanitize.t: 6/6 (ASAN/UBSAN clean, 44 insertions) |
| nginx/nginx#1627 (Gzip AE HTAB) | 0 | 9 | 0 | gzip_accept_encoding_htab.t: 9/9; gzip.t regression: 10/10 (+9/-3 lines) |

9.1. Test Companion PR Verification (master-vs-patched asymmetry)

Every upstream test companion listed in §6 runs against both unpatched master (showing the vulnerability is observable) and the patched branch (showing the assertions pass). The asymmetry is the empirical evidence that each companion test has actual coverage and is not vacuous:

Test Companion Unpatched master Patched branch Coverage
nginx-tests#56 (http_header_ows.t) 7 not ok, 2 ok → Result: FAIL 9/9 → Result: PASS H1 leading/trailing HTAB+SP around field value, including Content-Length and Transfer-Encoding OWS
nginx-tests#57 (h2_header_ows.t + h3_header_ows.t) 6 not ok, 10 ok → Result: FAIL 16/16 → Result: PASS H2 and H3 leading/trailing HTAB+SP around field value, whitespace-only header value
nginx-tests#89 (proxy_early_hints_headers.t) 3 not ok, 9 ok → Result: FAIL 12/12 → Result: PASS H1+H2+H3 hop-by-hop headers in 103 Early Hints (Connection, Content-Length, Transfer-Encoding, Keep-Alive, Upgrade) per RFC 9110 §7.6.1 + RFC 9113 §8.2.2 + RFC 9114 §4.2
nginx-tests#97 (gzip_accept_encoding_htab.t) 6 ok, 3 TODO-fail → suite PASS (TODO convention) 9/9 → Result: PASS Layer 2 — gzip Accept-Encoding HTAB at three positions: before comma, before q-value, after q-value before comma (per §3.1 carve-out rationale)

Architectural significance. The same master-vulnerable-then-patched-clean pattern appears at every layer (Layer 1 inbound = #56/#57/#89, Layer 2 token = #97, Layer 3 outbound = #58 + #55). Defense-in-depth is empirically validated because the master-vs-patched asymmetry is symmetric across all three layers: the unpatched harness always surfaces the vulnerability, and the patched harness always closes it. No layer's coverage relies on vacuous assertions.

10. Patch Branch Audits

Response-Filter Value Sanitization (Layer 3) — paired with upstream PR nginx#1414:

header_filter_sanitize.t
1..6
ok 1 - add_header keeps 200 OK with CRLF-bearing value
ok 2 - add_trailer keeps 200 OK with CRLF-bearing value
ok 3 - add_header value CRLF serialized as SP
ok 4 - add_trailer value CRLF serialized as SP
ok 5 - no injected X-Injected field line emitted
ok 6 - no sanitizer errors

Gzip Accept-Encoding HTAB Negotiation (Layer 2) — Paired PR: nginx#1627 + nginx-tests#97:

gzip_accept_encoding_htab.t (patched binary)
1..9
ok 1 - gzip enabled with SP between tokens
ok 2 - gzip enabled with no whitespace between tokens
ok 3 - gzip enabled with HTAB after comma
ok 4 - malformed Accept-Encoding without comma is rejected
ok 5 - gzip enabled with HTAB before comma
ok 6 - gzip enabled with HTAB before q-value
ok 7 - gzip enabled with HTAB after q-value before comma
ok 8 - no alerts
ok 9 - no sanitizer errors
All tests successful.

gzip.t (regression suite)
1..10
ok 1..10

11. Vulnerable Protocol Downgrade Flow

sequenceDiagram
    participant Attacker as Client (HTTP/2 or HTTP/1.x)
    participant NGINX as NGINX proxy/filter chain
    participant Backend as Upstream (HTTP/1.x)

    Attacker->>NGINX: GET /proxy/%0d%0aX-Injected:%20yes
    Note over NGINX: Inbound parser accepts request.<br/>Runtime variable $uri expands after normalization.
    NGINX->>Backend: GET /proxy/ HTTP/1.1<br/>Host: backend<br/>X-Injected: yes
    Note over Backend: Upstream receives injected header syntax.<br/>Potential HTTP request smuggling or cache poisoning.
    Backend-->>NGINX: Response
    NGINX-->>Attacker: Normal response
Loading

12. Hardened Outbound Boundary

sequenceDiagram
    participant Attacker as Client (HTTP/2 or HTTP/1.x)
    participant NGINX as NGINX proxy/filter chain
    participant Backend as Upstream (HTTP/1.x)

    Attacker->>NGINX: GET /proxy/%0d%0aX-Injected:%20yes
    Note over NGINX: Request compatibility policy stays intact.<br/>Outbound serializer detects CRLF in runtime variable.
    Note over NGINX: Outbound serializer rejects CR/LF before emission.
    NGINX-->>Attacker: 500 Internal Server Error
    Note over Backend: No malformed HTTP/1.x bytes are emitted to the socket.
Loading

13. Response Filter State Machine

stateDiagram-v2
    [*] --> DecideStatus
    DecideStatus --> HeaderFilter: status selected
    HeaderFilter --> HeaderCommitted: next header filter serializes headers
    HeaderCommitted --> BodyFilter: body chunks stream
    BodyFilter --> TrailerFilter: last_buf / final chunk
    TrailerFilter --> [*]: trailers serialized

    HeaderFilter --> EmptyClose: return NGX_ERROR after invalid add_header
    TrailerFilter --> UnsafeFinalize: late special response attempt
    UnsafeFinalize --> WorkerAlert: header already sent / worker signal

    HeaderFilter --> SanitizeHeader: replace NUL/CR/LF with SP
    SanitizeHeader --> HeaderCommitted
    TrailerFilter --> SanitizeTrailer: replace NUL/CR/LF with SP
    SanitizeTrailer --> [*]
Loading

13.1. Trailer-Filter Byte Commitment Invariant

The sanitize-to-SP policy is not optional for trailer paths. Once the response status line and headers have been written to the client socket, any subsequent attempt to return 5xx via add_trailer would require a TCP-level reset (RST) or worker process termination — both observable as connection aborts and both incompatible with HTTP/1.1 chunked streaming semantics.

sequenceDiagram
    autonumber
    participant Client as Client Socket
    participant NGINX as NGINX Event Loop
    participant Filter as Header & Trailer Filters

    Note over Client,NGINX: Phase 1 — Header Serialization (committed)
    NGINX->>Client: HTTP/1.1 200 OK\r\nContent-Type: ...\r\nTransfer-Encoding: chunked\r\n\r\n
    Note over Client: Status line + headers final on wire

    Note over Client,NGINX: Phase 2 — Body Streaming (committed)
    NGINX->>Client: chunk1\r\nchunk2\r\n0\r\n

    Note over NGINX,Filter: Phase 3 — Trailer Filter Execution
    alt Runtime variable in add_trailer contains CR / LF
        Note over Filter: 5xx rejection is physically impossible: headers + body already committed
        Note over Filter: Sanitize-to-SP is the only policy consistent with HTTP/1.1 streaming
    end
    NGINX->>Client: X-Original-Path: /sanitized-path/\r\n\r\n
Loading

14. Sanitization Implementation Blast Radius

For response-filter value sanitization (nginx#1414):

1 file changed, 44 insertions(+), 0 deletions(-)
src/http/modules/ngx_http_headers_filter_module.c

For gzip Accept-Encoding HTAB negotiation (nginx#1627):

1 file changed, 9 insertions(+), 3 deletions(-)
src/http/ngx_http_core_module.c

Both modifications avoid heap allocation reallocations, struct alterations, or ABI drift.

Local ApacheBench smoke benchmark on valid traffic (1000 requests, concurrency 10, no error budget):

  • Response-filter sanitization (nginx#1414): 4943.52 req/s, 0 failed.
  • Proxy CRLF log-level patch (nginx#590 follow-up): 4665.51 req/s, 0 failed.

Both deltas sit inside the noise floor of valid-traffic throughput and do not regress the no-CRLF baseline.

14.1. CI Cost Envelope (Test::Nginx suite runtime)

Measured against nginx-tests#75 (HTTP/2 pseudo-header ordering, 17 subtests) on the local harness — the only PR body in this review track that includes an explicit wallclock breakdown:

Files=1, Tests=17, 1 wallclock secs ( 0.01 usr  0.01 sys +  0.11 cusr  0.02 csys =  0.15 CPU)

Per-test cost is ~8.8 ms of CPU time (0.15 CPU / 17 subtests). Cross-protocol suites that run H1+H2+H3 in sequence (e.g. nginx-tests#89 covering 12 subtests across three protocols) land in the same envelope. The full upstream review-track harness summed across the five test companions cited in §6.5 (~40 subtests including cross-protocol) totals ~0.35 CPU seconds per prove -v invocation — well within CI cost tolerance. The CI cost envelope is independent of the validity of the assertions; even at this low cost, the §9.1 master-vs-patched asymmetry demonstrates that every layer's coverage is non-vacuous.

15. Upstream Scope Decoupling and Review Track Matrix

Review Track Scope Boundary Patch Footprint Test Status Maintainer Routing External Contributors
Gzip AE HTAB (#1627) Core module list-negotiation ngx_http_core_module.c (+9/-3) nginx-tests#97 (9/9 pass) Vetted by @sbhowmikf5, assigned to @pluknet —
Response Filter (#1414) Response header/trailer value sanitization ngx_http_headers_filter_module.c (+44/-0) nginx-tests#58 (6/6 pass, ASAN clean) Active PR, decoupled from field-name validation —
Multi-Header Match (#1577) Shared list lookup across Cache-Control, Connection, TE Multi-module (+97/-36) nginx-tests#85 Active core rework (Vadim) — parent of #1627 —
Inbound H2/H3 Reject (#1424) H2/H3 framing validation ngx_http_v2.c, ngx_http_v3_request.c nginx-tests#57 (8/8 pass) Active inbound track (Demi) —
Inbound H1 Strip (#1404) H1 header line OWS strip ngx_http_parse.c (+3/-0) nginx-tests#56 (9/9 pass) Active inbound track (Demi) —
Inbound H2/H3 Strip (#1572 closed) H2/H3 leading/trailing WSp strip H2/H3 module nginx-tests#80 Closed precursor to #1424 (Aayush7352) Aayush7352
Inbound HTAB Strip (#1602 closed) H1 HTAB strip via Issue #1597 ngx_http_parse.c — Closed precursor to #1404 (HRITHIK-SANKAR-R) HRITHIK-SANKAR-R
Multi-Parser Sweep (#1554) H1/H2/H3 leading/trailing WSp strip + Phase 2 vchar reject 10 files, +75/-18 Internal audit (see §15.5) Phase 1 ready, Phase 2 pending; milestone nginx-1.31.4 anuj1520
Field-Name Character Set (#1543) RFC 9110 §5.6.2 tchar validation ngx_http_parse.c — Active config-time track (saikrishna) saikrishnakumarreddy
Early Hints Header Accounting (#1606) r->header_size byte counting ngx_http.c — Active (SEPURI-SAI-KRISHNA) SEPURI-SAI-KRISHNA
Early Hints Hop-by-Hop (#1604) Hide hop-by-hop headers in 103 responses Upstream module nginx-tests#89 Active (SEPURI-SAI-KRISHNA) SEPURI-SAI-KRISHNA
Trailer Bounds (#1619) HTTP chunk extensions + trailer line bounds ngx_http_parse.c — Active (sbhowmikf5) sbhowmikf5
Trailer Special Handling (#1591 merged) gRPC "trailer only" response handling ngx_http_grpc_module.c — Merged (vinaykumar-1591) vinaykumar-1591
Trailer Limit (#1548 closed) HTTP/2 response trailer limits Upstream module — Closed (sbhowmikf5) sbhowmikf5
Host Header Special Handling (#1593 merged) Upstream Host/gRPC special handling Upstream + gRPC modules — Merged (arut) arut
Pseudo-Headers After Headers (#1541 merged) HTTP/2 pseudo-header ordering ngx_http_v2.c nginx-tests#75 Merged (nitin9977) nitin9977
max_headers per vhost (#1623) large_client_header_buffers per selected vhost ngx_http_core_module.c — Active (felix314159) felix314159
Gzip 410 Responses (#65 open) gzip behavior on 410 Gone gzip module nginx-tests#65 Active test track (ylw510) ylw510
Strict H/1.x Parsing Test (#41 open) Strict H/1.x parsing test — nginx-tests#41 Active test track (Demi) DemiMarie

15.5. nginx#1554 file-level overlap audit

Per the file-level diff reviewed 2026-08-07 (chronology summarised below). The 10 files modified by nginx#1554 were audited for protocol/subsystem scope and for byte-equivalence with concurrent contributions. None of the files is scope creep in the sense of an unrequested concern: each one targets a distinct protocol surface or subsystem parser, and the policy choice is consistent across the sweep (the same HTAB/SP acceptance rule applied to each parser's sw_space_* / sw_value state-machine cases).

File Protocol surface Relationship to concurrent contributions
ngx_http_v2.c HTTP/2 (RFC 9113 §8.2.1) EXACT overlap with #1424: same H2 reject pattern, no shared ngx_str_is_hws helper
ngx_http_v3_request.c HTTP/3 (RFC 9114 §10.3) EXACT overlap with #1424: same H3 reject pattern
ngx_http_grpc_module.c gRPC over H2 (RFC 9113 §8.2.1) EXACT overlap with #1424: same gRPC frame header reject
ngx_http_proxy_v2_module.c HTTP/2 upstream proxy (RFC 9113 §8.2.1) EXACT overlap with #1424: same proxy_v2 H2 reject
ngx_http_parse.c HTTP/1.x (RFC 9112 §5) CONFLICTS with #1404: opposite policy on HTAB (see cause-effect analysis below)
ngx_http_range_filter_module.c HTTP/1.x Range request (RFC 9110 §14.1.2) Per-subsystem extension: HTAB/SP parity in Range parser (no upstream precedent for this protocol)
ngx_http_core_module.c (auth_basic + gzip AE) HTTP/1.x auth + content-coding (RFC 9110 §11 / §12.5.3) Per-subsystem extension: auth_basic scheme + Accept-Encoding token list. The gzip AE path overlaps with #1627/#1577 review track.
ngx_http_file_cache.c HTTP/1.x cache vary (RFC 9111 §4.1) Per-subsystem extension: Vary field parsing parity
ngx_event_openssl_stapling.c TLS OCSP stapling (RFC 6960) Per-subsystem extension: stunnel-style header parsing in OCSP response handler
ngx_mail_auth_http_module.c Mail auth (RFC 2617 + nginx mail subsystem) Per-subsystem extension: header parsing in the mail auth subagent

Summary: 4 HTTP/2-family files (H2/H3/gRPC/proxy_v2) are EXACT-overlap with #1424 (semantically equivalent, structurally different — #1424 uses an extracted ngx_str_is_hws helper, #1554 uses inline character-class checks). 1 HTTP/1.x file (ngx_http_parse.c) is in conflict direction with #1404 (see cause-effect below). 5 files are per-subsystem extensions, each targeting a distinct protocol surface that has no other upstream PR covering the same concern. The gzip AE path (ngx_http_gzip_accept_encoding) is NOT in #1554's scope — #1627 operates on a code path #1554 does not touch.

Byte-equivalence evidence (EXACT overlap rows). The 4 HTTP/2-family files modified by both #1554 and #1424 apply the same policy (reject header with leading or trailing HTAB / SP) with structurally different code. Both are byte-equivalent at the level of the runtime decision — given the same input, both return NGX_ERROR for the leading/trailing HTAB/SP case and pass control to the existing per-byte loop for the embedded-HTAB / control-character case:

/* `#1424` (ngx_http_v2.c, ngx_http_v3_request.c, ngx_http_grpc_module.c,
   ngx_http_proxy_v2_module.c) — uses extracted helper:

   static ngx_inline ngx_int_t
   ngx_str_is_hws(ngx_uint_t c)
   {
       return c == '\t' || c == ' ';
   }

   ...
   if (s->len == 0) {
       return NGX_OK;
   }
   if (ngx_str_is_hws(s->data[0])
       || ngx_str_is_hws(s->data[s->len - 1]))
   {
       return NGX_ERROR;
   } */

/* `#1554` (same 4 files) — inline character-class check:

   if (s->len > 0
       && (s->data[0] == ' ' || s->data[0] == '\t'
           || s->data[s->len - 1] == ' '
           || s->data[s->len - 1] == '\t'))
   {
       return NGX_ERROR;
   } */

Both shapes have an explicit empty-string fast path (s->len == 0 returns OK before the boundary check). Both reject the same boundary cases. The helper-extracted version (#1424) is more refactorable across future parsers; the inline version (#1554) is closer to the existing per-byte loop style. A byte-equivalent reviewer can verify the equivalence by inspecting the function's input/output contract rather than the syntactic form.

Cause-effect analysis (CONFLICTS row, ngx_http_parse.c). The HTTP/1.x parser receives the header value after the parser's state machine has tokenized the OWS regions. #1404 adds HTAB to the case ' ' branches of three parser states (sw_space_before_value, sw_value, sw_space_after_value), which means: when the parser sees HTAB where it currently expects SP, it now transitions the same way as if it had seen SP. The semantic effect is that HTAB is normalised to SP at parse time — the value passed downstream to the H1/H2/H3 boundary checks never has leading or trailing HTAB, because the parser has already consumed it as part of the OWS region.

#1554 Phase 1 makes the same change to the same three states in the same file. Both PRs agree on the policy direction: strip HTAB at the parser level so the value seen by downstream consumers is HTAB-free. The CONFLICT label refers to the OPPOSITE-direction concern: #1424 (H2/H3/H1 reject in boundary-check parsers) assumes HTAB may still appear in the value, and rejects it at the boundary. If #1404 (or #1554 Phase 1) lands first, the #1424 leading/trailing HTAB check becomes effectively dead code for headers that pass through the H1 parser, because the parser has already consumed the HTAB. The two policies are not in conflict at the file level — they both want HTAB stripped — but the layering is mutually exclusive: either the parser strips (then #1424 redundant for H1) or the boundary check rejects (then the parser's HTAB acceptance is dead code at the boundary). The natural resolution is one of:

  1. Land #1404 (parser strips HTAB), make #1424 H2/H3/H1 only — the H1 path is now protected by the parser, the H2/H3 paths still need the boundary check because their parsers don't have a Phase 1 strip.
  2. Land #1424 (boundary check rejects) alone — the parser's HTAB acceptance is dead code at the boundary, but harmless.
  3. Reject HTAB outright at the H1 parser's sw_value exit (not currently proposed by either PR) — then #1404 and #1424 both become redundant for H1.

Open conflict note: #1424 is still open. As of the 2026-08-15 review, #1424 has not been merged. If #1424 is merged into master while #1554 Phase 1 is also open, the maintainer will face the file-level conflict on the 4 HTTP/2-family files (the diffs are mutually exclusive — both modify the same lines with non-coalescible changes). The reviewer-side path is: land #1404 first (parser strip), then rebase #1424 to drop the H1 boundary check (since the parser already protects it), then close #1554 Phase 1 as subsumed by #1404. This sequencing keeps the review tracks atomic per RFC clause: RFC 9112 §5 (H1 parser) and RFC 9113 §8.2.1 / RFC 9114 §10.3 (H2/H3 boundary).

Complementary-test theory (per-protocol/per-subsystem mapping). The 5 per-subsystem extension files (Range, core auth_basic, gzip AE, file cache vary, OCSP stapling, mail auth) are not scope creep — they are the natural completion of the nginx#1554 Phase 1 policy across every parser that has its own sw_space_* / sw_value state-machine cases. Each file targets a different protocol surface (RFC 9110 §14.1.2 Range, RFC 9110 §11 auth_basic, RFC 9110 §12.5.3 Accept-Encoding, RFC 9111 §4.1 Vary, RFC 6960 OCSP, RFC 2617 mail auth), and no upstream PR currently covers the same concern for these protocols. The open question is whether each subsysystem should be a separate PR (atomic review per protocol) or grouped under the same Phase 1 sweep (single review across protocols). The architectural case for separate PRs is stronger when the protocol has its own test suite in nginx-tests (Range, mail auth have dedicated tests; auth_basic and gzip AE are covered by nginx-tests#97; file cache vary has proxy_cache.t).

An empirical test of the theory: if the contributor of #1424 or #1554 were to upload nginx-tests PRs exercising the per-subsystem surfaces (e.g. http_header_htab_range.t, http_header_htab_vary.t, mail_auth_http_htab.t), each would document the specific parser state-machine case under test and demonstrate that the current master branch's behaviour on HTAB is observable (per the §9.1 master-vs-patched asymmetry pattern). These would be companion tests, not scope creep — they would either validate the Phase 1 fix is needed on each surface or confirm that the Phase 1 fix is harmless for surfaces that already handle HTAB correctly. The absence of such companion tests in the current review track is a gap, not a reason to exclude the files from scope.

15.6. Upstream Review Track Isolation and Risk Decoupling

Concurrent RFC work proceeds in scope-isolated review tracks, with each PR anchored to a single RFC clause and validated against its dedicated regression companion. Multi-parser sweeps that combine client framing, cookie parsing, and core helpers into monolithic PRs carry inherent cross-subsystem dependencies (per-file CI lane verification, CLA validation, ABI impact analysis across heterogeneous modules).

In contrast, self-contained slices — nginx#1627 for gzip negotiation (single file: ngx_http_core_module.c), nginx#1414 for response-filter sanitization (single file: ngx_http_headers_filter_module.c) — maintain strict review isolation. Each slice lands with the regression companion that proves its scope, with no cross-file refactor prerequisites.

This is the structural response to nginx Discussion #1413: proxy runtime serialization, whitespace compatibility, config-time validation, and late response-filter policy are different review units. PRs that attempt to span multiple layers introduce cross-subsystem regression risk without commensurate benefit, regardless of the maintainer's intent.

Cross-reference

The gist itself is self-contained: every methodological detail, validation flow, test scenario, and cross-PR audit referenced above is reproduced inline in §3–§15. No external artifact is required to evaluate any claim in this document.

Local lab (operational context, not required for review)

The local validation harness referenced in §6.5 and §7 was developed and run against a custom static-analysis setup configured at the source level — AddressSanitizer and UndefinedBehaviorSanitizer instrumentation, compile-time hardening flags (-fsanitize=address,undefined -fno-omit-frame-pointer), and raw socket byte capture for assertions on wire-format rather than HTTP-status. The lab supports several workflow modes that proved useful for this work:

  • Side-by-side diff comparison of patched vs. unpatched nginx builds under the same configure flags and sanitizer instrumentation, isolating the behavioural delta to the patch under review.
  • Reproducible nginx-tests Perl harness runs against paired-PR branches, with prove -v invocation and plan(N) + 2 implicit count tabulation.
  • Quantitative footprint analysis per branch — LOC deltas, file counts, per-subsystem regression surface measured against the same regression suite.
  • Parallel session isolation for cross-subsystem regression testing, where the same build is exercised under modified configuration to surface interaction effects between adjacent fixes.

The lab artifacts (test scripts, branch drivers, sanitization helpers, finding-log summarizers) are not referenced inline because the gist reproduces the methodological details in §5–§9. The above is provided so readers understand the gating methodology that produced the upstream test companions in §6.5 — it is not a substitute for the inline content.

Operational notes (paired-PR convention + Test::Nginx gotcha)

  • Paired PR pattern — every C patch in nginx/nginx ships with its dedicated regression companion in nginx/nginx-tests. Convention confirmed across the active paired PRs (#97+#1627, #58+#1414, #56+#1404, #57+#1424).
  • Test::Nginx plan(N) count — N counts only explicit tests; the harness adds 2 implicit assertions (no alerts + no sanitizer errors) that show up in the prove -v output as ok N+1 and ok N+2. Verified in nginx-tests#97: 7 explicit + 2 implicit = 9 run.
  • Layer isolation principle — proxy request serialization, response filter policy, content-coding list negotiation, and inbound parser strictness are different review units. Cross-layer PRs introduce cross-subsystem regression risk without commensurate benefit.
  • Empirical boundary verification — potential scope overlap across concurrent contributions is evaluated through file-level diff audits and RFC layer mapping, ensuring atomic review tracks remain decoupled unless formal consolidation is directed by core maintainers.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment