NGINX runtime CRLF injection evidence and multi-layer architecture mapping
Current status (2026-08-15): Paired PR active —
nginx-tests#97(regression suite, +125/-0) andnginx#1627(C patch, +9/-3 onsrc/http/ngx_http_core_module.c). Both marked Ready for Review; verified and routed to@pluknetby@sbhowmikf5. Master delta is isolated strictly to this atomic pair.Multi-Layer Context: RFC 9110/9112/9113/9114 compliance across NGINX spans three decoupled architectural layers: (1) Inbound client parser strictness (
#1424,#1404,#1543,#1554,#1602,#1572), (2) Internal header-token negotiation (#1627+nginx-tests#97,#1577,#1606), and (3) Outbound serialization & runtime variable sanitization (#1414,#590,#1593,nginx-tests#55,#58). Maintaining strict separation between these layers prevents cross-subsystem regressions.
This work is the continuation of a multi-year hardening arc targeting HTTP parser differentials and runtime CRLF injection in NGINX. §0.1 establishes the threat-model frame and operational severity per layer; §0.2–§0.5 document the maintainer-approved lineage of every PR cited in the matrix that follows. The investigation, prototyping, and benchmarking that produced the upstream test companions cited here predate the gist itself by more than a year; the lab harness details referenced in §6.5 are reproduced inline below and do not require access to the originating local repository.
The layered fixes catalogued in this gist defend against three intersecting HTTP attack surface categories, each with a distinct attack class, attacker capability profile, and operational severity:
| Vector category | Layer defended | Attack class | Severity |
|---|---|---|---|
| H1/H2/H3 parser differentials — leading/trailing WSp, NUL/CR/LF, tchar violations | Layer 1 — Inbound framing (§3, §4) | Reverse-proxy request smuggling, cache poisoning via upstream parser differential | High — actively exploited in CDN/load-balancer deployments (cf. PortSwigger disclosures 2019–2024) |
| Token-list value differentials — q-value parsing, multi-token boundaries, OWS/HTAB in content-coding lists | Layer 2 — List negotiation (§3, §4) | Content-encoding downgrade via Accept-Encoding q-list manipulation, cache poisoning via Cache-Control token differential | Medium — proxy-chain diversity dependent |
Runtime variable CRLF injection — proxy_pass + $uri, proxy_set_header + $uri, add_header + $uri, add_trailer + $uri |
Layer 3 — Outbound serialization (§3, §4, §8) | Proxy→backend request smuggling, response header injection, cache poisoning via emitted CRLF | Critical — confirmed at runtime in §8 baseline |
Why three layers, not one monolith. Each layer addresses a distinct attacker capability: well-resourced attackers craft non-spec-compliant framing (Layer 1); intermediate attackers manipulate header values to coerce content-negotiation differentials (Layer 2); opportunistic attackers exploit runtime-variable expansion (Layer 3). Monolithic fix attempts introduce cross-subsystem regression risk because each layer's test harness and CI lane is independent (§15.6).
Adjacent attack surface — token negotiation as collaboration vehicle. Layer 2 is captured here not as a primary defense vector but as adjacent work where the lab's protocol-layer mapping provides architectural context to upstream rewrites: nginx#1577 (multi-header helper rework) explicitly carves out ngx_http_gzip_accept_encoding for surgical fixes (nginx#1627); the same RFC 9110 §5.6.3 OWS/HTAB surface is shared across Accept-Encoding, Accept-Language, and Cache-Control parsing. Defense-in-depth benefits from keeping these adjacent slices visible without conflating them with the primary CRLF arc.
In nginx#590, DemiMarie split the runtime CRLF-injection fix into discrete phases. Phases 1–3 are merged; phase 4+ was handed off directly to @louzt:
| Phase | PR | Concern | Status |
|---|---|---|---|
| 1 | nginx#1404 |
H1 OWS strip on header field lines | Merged (DemiMarie) |
| 2 | nginx#1410 |
NULL-safety for cl->buf chain links |
Merged (DemiMarie) |
| 2 | nginx#1411 |
Zero-initialize allocated chain links (so explicit NULL assignment is not needed) | Merged (DemiMarie) |
| 3 | nginx#1424 |
H2/H3 inbound header field validation (reject leading/trailing WSp + NUL/CR/LF) | Merged (DemiMarie) |
| 4+ | nginx#1404 follow-ups, OWS in HTTP/1.x, layered defense |
Handed off to @louzt per DemiMarie's direct ask: "Would you be interested in finishing this work? I've done phases 1 through 3 (#1404, #1410, #1411, #1424), but regretfully I don't have time to finish the remaining steps." (issue #590, DemiMarie comment) |
The split itself is DemiMarie's architectural decision. In nginx#590 DemiMarie also confirmed the long-form scope: "Fortunately, I believe the HTTP/1.x, HTTP/2, and HTTP/3 parsers all do the bare minimum validation needed to not be caught by the egress filters. The only exceptions are #1201 and #1404." This justifies continuing the Layer-3 outbound slice separately from the inbound validation track.
The downstream test companion for Demi's strict H/1.x parsing work is nginx-tests#41 (open), authored by DemiMarie.
Also in nginx#590, VadimZhestikov posted the maintainer-side roadmap that scopes the remaining work:
-
Inconsistent log levels for the same category of event. The same class of event (runtime detection of CRLF injection attempt) is logged at different levels across modules:
ngx_http_headers_filter_module→NGX_LOG_ERRngx_http_proxy_module→NGX_LOG_ALERTngx_http_proxy_v2_module→NGX_LOG_ALERT- Internal double-check (write pass) →
NGX_LOG_EMERG
Vadim's guidance: a user-triggered CRLF injection attempt should be
NGX_LOG_ERR(operator needs to know their config is vulnerable, but this is not a server emergency). The internal "this should have been caught" double-check should beNGX_LOG_ALERT(it indicates a bug in nginx itself).NGX_LOG_EMERGis reserved for conditions that prevent nginx from operating at all. -
Config-time validation is missing for many header-setting directives. Config-time validation in
nginx#590covers onlyadd_headerandproxy_set_header. The following directives also configure header name/value pairs and are not covered:fastcgi_param/uwsgi_param/scgi_paramgrpc_set_headeradd_trailerproxy_hide_header/fastcgi_hide_header
Vadim's note: inconsistent coverage means operators using
fastcgi_paramwith a malformed value get no startup-time warning, only a runtime 500. The config-time check is valuable precisely because it catches mistakes before deployment. -
Unrelated fix bundled in: the patch carries
cl->next = NULLas a defensive NULL-assignment aftercl->buf = b;. This was later split off asnginx#1411(zero-initialize allocated chain links).
Vadim's parent PR rework on multi-header token matching is nginx#1577 (open, iterating), which explicitly carves out ngx_http_gzip_accept_encoding from the multi-header helper rewrite — that carve-out is the upstream-maintainer-approved scope for nginx#1627 (this gist's paired PR).
The maintainer-approved layering (Demi's phased handoff per §0.2, Vadim's roadmap per §0.3) extends to concurrent RFC clause work: inbound framing validation (RFC 9113 §8.2.1 / RFC 9114 §10.3), content-coding list negotiation (RFC 9110 §5.6.3 / §8.4), and outbound serialization (RFC 9112 outbound safety) operate at three decoupled layers of the request lifecycle. PRs that span multiple layers introduce cross-subsystem regression risk without commensurate benefit; each slice is anchored to a single RFC clause and validated against its dedicated regression companion. The structural principle governing self-contained review slices is detailed in §15.6.
| Window | Focus |
|---|---|
| ~2025 (pre-gist) | Initial CRLF injection reproduction; local test harness iteration against nginx-tests Perl framework. nginx#590 discussion thread active. |
| 2026-06-02 | First revision of this gist (initial triage + reproduction). |
| 2026-06-21 | DemiMarie directed drop of name-validation from PR scope; the surgical-PR strategy crystallizes. |
| 2026-07 → 2026-08 | Lab benchmarking + cross-layer mapping; anuj1520 PR #1554 appears with overlap risk on inbound parser direction. |
| 2026-08-07 | Layer-3 isolation documented (response-filter separate from field-name validation); boundary cross-checked against §12 Hardened Outbound Boundary. |
| 2026-08-08 | Paired PR nginx-tests#97 + nginx#1627 shipped; both Ready for Review. |
| 2026-08-15 | This revision: full architectural lifecycle mapping, local test harness catalog (§6.5), full Review Track Matrix (5→19 entries, §15), #1554 file-level overlap audit (§15.5), Demi's phased handoff + Vadim's roadmap + RFC layer decoupling principle (§0.4, §15.6). Threat Model & Defense Vector Taxonomy (§0.1) added as the framing layer before the historical lineage. All internal lab references inlined so the gist is self-contained for maintainer review. |
The narrow question is whether NGINX should refuse to serialize CR/LF/NUL or invalid field names when generating HTTP/1.x output from runtime variables.
This is fundamentally separate from:
- inbound RFC 9110 strictness decisions on client requests;
- HTTP/2 and HTTP/3 inbound framing validators, which have protocol-specific rules;
- config-time validation for static directives;
- internal header list negotiation helpers.
NGINX does not need to become globally stricter on inbound HTTP parsing to resolve runtime CRLF injection vectors. The immediate security boundary is outbound HTTP/1.x serialization from runtime-expanded variables.
Preserving this architectural distinction maintains NGINX's compatibility posture on inbound traffic while establishing an explicit trust boundary on emitted bytes. A controlled sanitization/5xx before bytes leave NGINX represents an atomic, reviewable change without risking regressions in unrelated protocol parsers.
flowchart TD
subgraph Layer1["Layer 1: Inbound Client Parsing Boundary (RFC 9112 §2.2 / RFC 9113 §8.2.1 / RFC 9114 §10.3 / RFC 9110 §5.6.2)"]
A[Client Request H1 / H2 / H3] --> B{Inbound Framing Validation}
B -->|Reject Leading/Trailing WSp| C["#1424 / #1554 (H2/H3 framing reject)"]
B -->|Strip OWS at Header Line| D["#1404 / #1602 (RFC 9112 §2.2 H1 OWS strip)"]
B -->|Validate Field Names| E["#1543 (RFC 9110 §5.6.2 tchar)"]
end
subgraph Layer2["Layer 2: Token-List Negotiation Boundary (RFC 9110 §5.6.3 / §8.4)"]
C --> F{Internal Value Processing}
D --> F
F -->|Multi-Header Directives| G["#1577 (Cache-Control / Connection / TE)"]
F -->|Parameterized Content-Coding| H["#1627 + nginx-tests#97 (Accept-Encoding q-list)"]
end
subgraph Layer3["Layer 3: Outbound Serialization Boundary (RFC 9112 outbound safety)"]
G --> I{Runtime Variable Expansion}
H --> I
I -->|Runtime $uri in Outbound Headers| J{Contains CR / LF / NUL?}
J -->|Proxy Path| K["500 / Controlled Rejection (#590, nginx-tests#55)"]
J -->|Response Filter Path| L["Sanitize to SP (#1414, nginx-tests#58)"]
J -->|No Injected Bytes| M[Emit HTTP/1.x Wire Output]
end
Inbound and outbound layers are complementary and non-interchangeable. Inbound validation rejects malformed client requests at framing time; outbound serialization prevents parameter injection at emission time. Coupling them into monolithic patches introduces cross-layer regression risks (e.g., breaking upstream cookie parsing while attempting OWS cleanup).
Layer 2 token-list negotiation includes gzip Accept-Encoding as a special case because ngx_http_gzip_accept_encoding() in ngx_http_core_module.c carries its own boundary check independent of the multi-header parser. Vadim's nginx#1577 multi-header parser rework explicitly carved this helper out because teaching the multi-header parser to accept HTAB does not retroactively teach this helper — it has its own switch with case ' ' and exits on the bare comma that ends the list, so HTAB falls into the default branch and the encoding is rejected.
This is why nginx#1627 (gzip AE HTAB) is a separate, surgical 9-line patch to ngx_http_core_module.c rather than a derivative of #1577. The carve-out keeps Layer 2 reviewable: the helper-specific fix lives next to the helper-specific test (nginx-tests#97), without entangling the broader multi-header parser work. The same boundary-check pattern is documented in the §0.3 Threat Model's Adjacent attack surface note.
| # | Layer | Target Boundary | Focus & Safety Invariant | Active PR Tracks |
|---|---|---|---|---|
| 1 | Inbound Parsing | Client request ingestion | Reject/strip malformed client inputs per RFC 9113/9114/9110 | nginx#1424 (Demi H2/H3 reject), nginx#1404 (Demi H1 OWS strip), nginx#1543 (saikrishna field names), nginx#1554 (anuj1520 parser sweep), nginx#1602 (HRITHIK HTAB strip — closed precursor to #1404), nginx#1572 (Aayush7352 H2/H3 strip — closed precursor to #1424) |
| 2 | Token Negotiation | Content-coding / Directive lookup | Parse RFC 9110 #rule lists with OWS/HTAB and quality parameters |
nginx#1627 + nginx-tests#97 (gzip AE HTAB), nginx#1577 (Vadim multi-header token boundary), nginx#1606 (SEPURI-SAI-KRISHNA early-hints header accounting) |
| 3 | Outbound Serialization | Upstream emit & response filters | Neutralize CRLF injection from runtime variables ($uri, $args) |
nginx#1414 (response-filter value sanitize), nginx#590 (proxy request boundary), nginx#1593 (arut Host header special handling) |
| 4 | Regression Harness | Native test suites | Deterministic regression validation across nginx-tests |
nginx-tests#55 (proxy), #56 (H1 OWS), #57 (H2/H3 OWS), #58 (header filter), #97 (gzip AE), #80 (Aayush7352 H2 strip test), #89 (SEPURI-SAI-KRISHNA early-hints) |
The verification workflow is structured to mirror upstream maintainer practices:
- clean branch checkouts from origin/master;
- native
nginx-testsPerl harnesses executed against unpatched master and candidate branches; - raw socket captures to inspect wire-level byte emission;
- memory-safety validation under AddressSanitizer and UndefinedBehaviorSanitizer (
-fsanitize=address,undefined); - reproducible benchmark checks for throughput impact under valid traffic.
Each candidate branch is validated under the same flow: configure with --with-debug --with-cc-opt='-fsanitize=address,undefined -fno-omit-frame-pointer'; run the upstream nginx-tests Perl harness via prove -v against both the patched branch and the parallel gzip.t / header_filter.t regression suites to surface cross-module fallout; capture raw socket bytes for the cases where the assertion is wire-format rather than HTTP-status; tabulate the resulting plan(N) + 2 implicit count against the §9 Verification Matrix. The local test files driving this flow are catalogued in §6.5.
Response-Filter Sanitization Slice (Layer 3):
- C Patch: nginx/nginx#1414 (
ngx_http_headers_filter_module.c, +44/-0) - Test Companion: nginx/nginx-tests#58 (
header_filter_sanitize.t) - Adjacent merged: nginx/nginx#1593 (arut Host header handling — emitted upstream boundary work)
Gzip Accept-Encoding HTAB Negotiation (Layer 2):
- C Patch: nginx/nginx#1627 (
ngx_http_core_module.c, +9/-3) - Test Companion: nginx/nginx-tests#97 (
gzip_accept_encoding_htab.t, +125/-0) - Parent parser rework: nginx/nginx#1577 (Vadim multi-header, OPEN — explicitly carves out
ngx_http_gzip_accept_encoding)
Proxy Output Hardening & Handoff:
- Regression Test: nginx/nginx-tests#55 (
proxy_crlf_injection_runtime.t) - Phase Roadmap Reference: nginx/nginx#590 (comment)
Inbound Protocol Validation Tracks (Layer 1):
- H2/H3 Field Validation: nginx/nginx#1424 (Demi)
- H1 OWS Strip: nginx/nginx#1404 (Demi)
- Field-Name Character Set: nginx/nginx#1543 (saikrishnakumarreddy)
- Multi-Parser Sweep: nginx/nginx#1554 (anuj1520)
- Closed precursors: nginx/nginx#1602 (HRITHIK HTAB strip), nginx/nginx#1572 (Aayush7352 H2/H3 strip)
Shared Multi-Header Token Matching:
- Multi-Header Helper Rework: nginx/nginx#1577 (Vadim)
Trailer & Hop-by-Hop Adjacent Work (boundary-crossing):
- Trailer line bounds: nginx/nginx#1619 (sbhowmikf5)
- Trailer special handling: nginx/nginx#1591 (vinaykumar-1591, MERGED), nginx/nginx#1548 (sbhowmikf5, CLOSED)
- Hop-by-hop in early hints: nginx/nginx#1604 (SEPURI-SAI-KRISHNA) + nginx/nginx-tests#89
These test files are local validation scripts used before any upstream PR is opened. They are NOT submitted to nginx/nginx-tests directly — they are the harness that drives the upstream PR test companions. The file paths below are relative to the local test directory; the upstream-facing test that ships in nginx-tests#97 is derived from the local gzip_accept_encoding_htab.t.
| Test File | plan(N) | Layer | Policy Tested | Scope |
|---|---|---|---|---|
proxy_crlf_injection_runtime.t |
8 | Layer 3 (proxy + response) | REJECT 500 | proxy_pass $uri, proxy_set_header $uri (CR/LF/CR-only), add_header, add_trailer — every path returns 500 and asserts no X-Injected field reaches upstream |
proxy_crlf_injection_sanitize_runtime.t |
10 | Layer 3 (proxy + response) | MIXED: proxy rejects 500, response sanitizes to SP | Same scope as above but response path keeps 200 OK with CRLF replaced by SP inside the same field value; proxy path still rejects |
gzip_accept_encoding_htab.t |
7 + 2 implicit = 9 | Layer 2 | ACCEPT HTAB as OWS | 4 baseline (SP, no-whitespace, HTAB-after-comma, malformed rejection) + 3 TODO (HTAB-before-comma, HTAB-before-qvalue, HTAB-after-qvalue-before-comma). This is the upstream-facing test that ships in nginx-tests#97 |
h2_header_ows.t, h3_header_ows.t |
— | Layer 1 | STRIP OWS | Upstream-facing tests for Demi's H2/H3 reject logic; ship in nginx-tests#57 |
http_header_ows.t |
— | Layer 1 | STRIP OWS | Upstream-facing test for Demi's H1 OWS strip; ships in nginx-tests#56 |
header_filter_sanitize.t |
6 | Layer 3 | SANITIZE to SP | Upstream-facing test for response filter value sanitization; ships in nginx-tests#58 |
| Scenario | proxy_crlf_injection_runtime.t | proxy_crlf_injection_sanitize_runtime.t | gzip_accept_encoding_htab.t | header_filter_sanitize.t |
|---|---|---|---|---|
proxy_pass + CRLF in $uri |
✅ reject 500 | ✅ reject 500 | — | — |
proxy_set_header + CRLF in $uri |
✅ reject 500 | ✅ reject 500 | — | — |
| proxy_set_header + LF only | ✅ reject 500 | ✅ reject 500 | — | — |
| proxy_set_header + CR only | ✅ reject 500 | ✅ reject 500 | — | — |
| add_header + CRLF | ✅ reject 500 | ✅ sanitize to SP, keep 200 | — | ✅ sanitize to SP |
| add_trailer + CRLF | ✅ reject 500 | ✅ sanitize to SP, keep 200 | — | ✅ sanitize to SP |
| Accept-Encoding HTAB before comma | — | — | ✅ (TODO before patch) | — |
| Accept-Encoding HTAB before q-value | — | — | ✅ (TODO before patch) | — |
| Accept-Encoding HTAB after q-value | — | — | ✅ (TODO before patch) | — |
Malformed gzip\tdeflate (no comma) |
— | — | ✅ rejected (RFC 9110 §5.6.2) | — |
| no alerts / no sanitizer errors | implicit | implicit | ✅ ok 8-9 | ✅ ok 6 |
flowchart LR
subgraph LAB["Local REJECT/SANITIZE policy harness"]
A1["proxy_crlf_injection_runtime.t<br/>REJECT policy"]
A2["proxy_crlf_injection_sanitize_runtime.t<br/>SANITIZE policy"]
end
subgraph WT["Local upstream-facing test (drives nginx-tests#56/#57/#58/#97)"]
B1["gzip_accept_encoding_htab.t"]
B2["h2_header_ows.t + h3_header_ows.t"]
B3["http_header_ows.t"]
B4["header_filter_sanitize.t"]
end
subgraph UP["Upstream PR test companions (nginx/nginx-tests)"]
C1["nginx-tests#55<br/>(proxy CRLF)"]
C2["nginx-tests#58<br/>(header filter sanitize)"]
C3["nginx-tests#56<br/>(H1 OWS)"]
C4["nginx-tests#57<br/>(H2/H3 OWS)"]
C5["nginx-tests#97<br/>(gzip AE HTAB)"]
end
A1 -. "derived upstream variant" .-> C1
A2 -. "derived upstream variant" .-> C2
B3 -. "submitted as" .-> C3
B2 -. "submitted as" .-> C4
B1 -. "submitted as" .-> C5
B4 -. "submitted as" .-> C2
| Layer / Concern | H1 (HTTP/1.x) | H2 (HTTP/2) | H3 (HTTP/3) |
|---|---|---|---|
| Layer 1 — Inbound parsing | nginx-tests#56 (http_header_ows.t, 9 subtests) |
nginx-tests#57 (h2_header_ows.t, 8 subtests) + nginx-tests#80 (h2_headers.t, 116 subtests total) |
nginx-tests#57 (h3_header_ows.t, 8 subtests) |
| Layer 2 — Token negotiation | (covered indirectly via shared parsing; gzip AE helper in ngx_http_core_module.c is the explicit carve-out per §3.1) |
— | — |
| Layer 3 — Outbound serialization | nginx-tests#55 (proxy_crlf_injection_runtime.t, 8 subtests) + nginx-tests#58 (header_filter_sanitize.t, 6 subtests) |
(HTTP/1.x outbound is the serialization surface; H2/H3 do not re-encode headers in this layer) | (same) |
| Cross-layer — Early Hints hop-by-hop | nginx-tests#89 (12 subtests across protocols) |
nginx-tests#89 |
nginx-tests#89 |
| Cross-layer — HTTP/2 pseudo-header ordering | — | nginx-tests#75 (17 subtests) |
— |
The test harness covers all three protocols at Layer 1 (the most fragmented layer, where each protocol parser has its own boundary check), HTTP/1.x for the runtime CRLF surface (the protocol where outbound serialization actually emits raw CRLF/NUL bytes), and a single cross-protocol suite for early-hints hop-by-hop behavior (nginx-tests#89) since the same concern spans H1, H2, and H3.
The proxy_crlf_injection_*.t files exercise both REJECT and SANITIZE policy choices on the same scenarios, so the maintainer-facing nginx-tests#55 (REJECT) and #58 (SANITIZE) can be derived from a single source of truth rather than maintained as parallel implementations.
Test::Nginx TODO: { local $TODO = ... } convention — assertions wrapped in a TODO block are expected-to-fail on master (the failure is reported as ok N # TODO <reason>, a passing TODO). The 3 TODO positions in gzip_accept_encoding_htab.t document the exact master-side HTAB parse gap and become real assertions in a one-line follow-up commit once the patch lands; they are not removed from the test companion itself.
Candidate test suites:
nginx-tests/proxy_crlf_injection_runtime.t
nginx-tests/proxy_crlf_injection_sanitize_runtime.t
nginx-tests/header_filter_sanitize.t
nginx-tests/http_header_ows.t
nginx-tests/h2_header_ows.t
nginx-tests/h3_header_ows.t
nginx-tests/gzip_accept_encoding_htab.t
Each layer was validated end-to-end against the upstream nginx-tests Perl harness. Layer 3 tests ran against the response-filter sanitization branch with AddressSanitizer and UndefinedBehaviorSanitizer variants both clean; Layer 2 tests ran against the gzip Accept-Encoding HTAB branch and the parallel gzip.t regression suite. Validation follows the configure / prove flow described in §5 Methodology. Per-branch results (plan counts, ASAN/UBSAN outcome, cross-module regression delta) are tabulated in §9 Verification Matrix across Branches.
Observed against a locally built nginx/1.31.x from nginx/nginx master:
proxy_pass http://127.0.0.1:<port>/$uriemits an injected upstream request header when$uricontains%0d%0aX-Injected:%20yes.proxy_set_header X-Normalized-URI $uriemits an injected upstream request header with the same payload.add_header X-Original-Path $uri alwaysemits injected response header fields.add_trailer X-Original-Path $uri alwaysemits injected trailer fields.header_filter_sanitize.ton master: 3/6 pass; failing assertions check that an injected field line is not emitted.gzip_accept_encoding_htab.ton master: 6/9 pass; the 3 failing assertions are TODO positions for HTAB before/after q-values and commas inngx_http_gzip_accept_encoding().
Captured upstream request generated by proxy_pass:
GET //uri/
X-Injected: yes HTTP/1.1
Host: 127.0.0.1:<port>Visible injected header:
X-Original-Path: /resp-header/
X-Injected: yes| Branch / Slice | Exit | Passed | Failed | Scope & Observations |
| --- | ---: | ---: | ---: | --- |
| nginx-master (1.31.x baseline) | 1 | 3 | 3 | header_filter_sanitize.t: add_header/add_trailer CRLF injection observed |
| nginx/nginx#1424 (Demi H2/H3 reject) | 0 | 8 | 0 | h2_header_ows.t + h3_header_ows.t: 8/8 each (inbound framing clean) |
| nginx/nginx#1404 (Demi H1 OWS strip) | 0 | 9 | 0 | http_header_ows.t: 9/9 (H1 field boundary OWS strip clean) |
| nginx/nginx#1414 (Value sanitize) | 0 | 6 | 0 | header_filter_sanitize.t: 6/6 (ASAN/UBSAN clean, 44 insertions) |
| nginx/nginx#1627 (Gzip AE HTAB) | 0 | 9 | 0 | gzip_accept_encoding_htab.t: 9/9; gzip.t regression: 10/10 (+9/-3 lines) |
Every upstream test companion listed in §6 runs against both unpatched master (showing the vulnerability is observable) and the patched branch (showing the assertions pass). The asymmetry is the empirical evidence that each companion test has actual coverage and is not vacuous:
| Test Companion | Unpatched master | Patched branch | Coverage |
|---|---|---|---|
nginx-tests#56 (http_header_ows.t) |
7 not ok, 2 ok → Result: FAIL |
9/9 → Result: PASS |
H1 leading/trailing HTAB+SP around field value, including Content-Length and Transfer-Encoding OWS |
nginx-tests#57 (h2_header_ows.t + h3_header_ows.t) |
6 not ok, 10 ok → Result: FAIL |
16/16 → Result: PASS |
H2 and H3 leading/trailing HTAB+SP around field value, whitespace-only header value |
nginx-tests#89 (proxy_early_hints_headers.t) |
3 not ok, 9 ok → Result: FAIL |
12/12 → Result: PASS |
H1+H2+H3 hop-by-hop headers in 103 Early Hints (Connection, Content-Length, Transfer-Encoding, Keep-Alive, Upgrade) per RFC 9110 §7.6.1 + RFC 9113 §8.2.2 + RFC 9114 §4.2 |
nginx-tests#97 (gzip_accept_encoding_htab.t) |
6 ok, 3 TODO-fail → suite PASS (TODO convention) | 9/9 → Result: PASS |
Layer 2 — gzip Accept-Encoding HTAB at three positions: before comma, before q-value, after q-value before comma (per §3.1 carve-out rationale) |
Architectural significance. The same master-vulnerable-then-patched-clean pattern appears at every layer (Layer 1 inbound = #56/#57/#89, Layer 2 token = #97, Layer 3 outbound = #58 + #55). Defense-in-depth is empirically validated because the master-vs-patched asymmetry is symmetric across all three layers: the unpatched harness always surfaces the vulnerability, and the patched harness always closes it. No layer's coverage relies on vacuous assertions.
Response-Filter Value Sanitization (Layer 3) — paired with upstream PR nginx#1414:
header_filter_sanitize.t
1..6
ok 1 - add_header keeps 200 OK with CRLF-bearing value
ok 2 - add_trailer keeps 200 OK with CRLF-bearing value
ok 3 - add_header value CRLF serialized as SP
ok 4 - add_trailer value CRLF serialized as SP
ok 5 - no injected X-Injected field line emitted
ok 6 - no sanitizer errors
Gzip Accept-Encoding HTAB Negotiation (Layer 2) — Paired PR: nginx#1627 + nginx-tests#97:
gzip_accept_encoding_htab.t (patched binary)
1..9
ok 1 - gzip enabled with SP between tokens
ok 2 - gzip enabled with no whitespace between tokens
ok 3 - gzip enabled with HTAB after comma
ok 4 - malformed Accept-Encoding without comma is rejected
ok 5 - gzip enabled with HTAB before comma
ok 6 - gzip enabled with HTAB before q-value
ok 7 - gzip enabled with HTAB after q-value before comma
ok 8 - no alerts
ok 9 - no sanitizer errors
All tests successful.
gzip.t (regression suite)
1..10
ok 1..10
sequenceDiagram
participant Attacker as Client (HTTP/2 or HTTP/1.x)
participant NGINX as NGINX proxy/filter chain
participant Backend as Upstream (HTTP/1.x)
Attacker->>NGINX: GET /proxy/%0d%0aX-Injected:%20yes
Note over NGINX: Inbound parser accepts request.<br/>Runtime variable $uri expands after normalization.
NGINX->>Backend: GET /proxy/ HTTP/1.1<br/>Host: backend<br/>X-Injected: yes
Note over Backend: Upstream receives injected header syntax.<br/>Potential HTTP request smuggling or cache poisoning.
Backend-->>NGINX: Response
NGINX-->>Attacker: Normal response
sequenceDiagram
participant Attacker as Client (HTTP/2 or HTTP/1.x)
participant NGINX as NGINX proxy/filter chain
participant Backend as Upstream (HTTP/1.x)
Attacker->>NGINX: GET /proxy/%0d%0aX-Injected:%20yes
Note over NGINX: Request compatibility policy stays intact.<br/>Outbound serializer detects CRLF in runtime variable.
Note over NGINX: Outbound serializer rejects CR/LF before emission.
NGINX-->>Attacker: 500 Internal Server Error
Note over Backend: No malformed HTTP/1.x bytes are emitted to the socket.
stateDiagram-v2
[*] --> DecideStatus
DecideStatus --> HeaderFilter: status selected
HeaderFilter --> HeaderCommitted: next header filter serializes headers
HeaderCommitted --> BodyFilter: body chunks stream
BodyFilter --> TrailerFilter: last_buf / final chunk
TrailerFilter --> [*]: trailers serialized
HeaderFilter --> EmptyClose: return NGX_ERROR after invalid add_header
TrailerFilter --> UnsafeFinalize: late special response attempt
UnsafeFinalize --> WorkerAlert: header already sent / worker signal
HeaderFilter --> SanitizeHeader: replace NUL/CR/LF with SP
SanitizeHeader --> HeaderCommitted
TrailerFilter --> SanitizeTrailer: replace NUL/CR/LF with SP
SanitizeTrailer --> [*]
The sanitize-to-SP policy is not optional for trailer paths. Once the response status line and headers have been written to the client socket, any subsequent attempt to return 5xx via add_trailer would require a TCP-level reset (RST) or worker process termination — both observable as connection aborts and both incompatible with HTTP/1.1 chunked streaming semantics.
sequenceDiagram
autonumber
participant Client as Client Socket
participant NGINX as NGINX Event Loop
participant Filter as Header & Trailer Filters
Note over Client,NGINX: Phase 1 — Header Serialization (committed)
NGINX->>Client: HTTP/1.1 200 OK\r\nContent-Type: ...\r\nTransfer-Encoding: chunked\r\n\r\n
Note over Client: Status line + headers final on wire
Note over Client,NGINX: Phase 2 — Body Streaming (committed)
NGINX->>Client: chunk1\r\nchunk2\r\n0\r\n
Note over NGINX,Filter: Phase 3 — Trailer Filter Execution
alt Runtime variable in add_trailer contains CR / LF
Note over Filter: 5xx rejection is physically impossible: headers + body already committed
Note over Filter: Sanitize-to-SP is the only policy consistent with HTTP/1.1 streaming
end
NGINX->>Client: X-Original-Path: /sanitized-path/\r\n\r\n
For response-filter value sanitization (nginx#1414):
1 file changed, 44 insertions(+), 0 deletions(-)
src/http/modules/ngx_http_headers_filter_module.c
For gzip Accept-Encoding HTAB negotiation (nginx#1627):
1 file changed, 9 insertions(+), 3 deletions(-)
src/http/ngx_http_core_module.c
Both modifications avoid heap allocation reallocations, struct alterations, or ABI drift.
Local ApacheBench smoke benchmark on valid traffic (1000 requests, concurrency 10, no error budget):
- Response-filter sanitization (
nginx#1414): 4943.52 req/s, 0 failed. - Proxy CRLF log-level patch (
nginx#590follow-up): 4665.51 req/s, 0 failed.
Both deltas sit inside the noise floor of valid-traffic throughput and do not regress the no-CRLF baseline.
Measured against nginx-tests#75 (HTTP/2 pseudo-header ordering, 17 subtests) on the local harness — the only PR body in this review track that includes an explicit wallclock breakdown:
Files=1, Tests=17, 1 wallclock secs ( 0.01 usr 0.01 sys + 0.11 cusr 0.02 csys = 0.15 CPU)
Per-test cost is ~8.8 ms of CPU time (0.15 CPU / 17 subtests). Cross-protocol suites that run H1+H2+H3 in sequence (e.g. nginx-tests#89 covering 12 subtests across three protocols) land in the same envelope. The full upstream review-track harness summed across the five test companions cited in §6.5 (~40 subtests including cross-protocol) totals ~0.35 CPU seconds per prove -v invocation — well within CI cost tolerance. The CI cost envelope is independent of the validity of the assertions; even at this low cost, the §9.1 master-vs-patched asymmetry demonstrates that every layer's coverage is non-vacuous.
| Review Track | Scope Boundary | Patch Footprint | Test Status | Maintainer Routing | External Contributors |
|---|---|---|---|---|---|
Gzip AE HTAB (#1627) |
Core module list-negotiation | ngx_http_core_module.c (+9/-3) |
nginx-tests#97 (9/9 pass) |
Vetted by @sbhowmikf5, assigned to @pluknet |
— |
Response Filter (#1414) |
Response header/trailer value sanitization | ngx_http_headers_filter_module.c (+44/-0) |
nginx-tests#58 (6/6 pass, ASAN clean) |
Active PR, decoupled from field-name validation | — |
Multi-Header Match (#1577) |
Shared list lookup across Cache-Control, Connection, TE | Multi-module (+97/-36) | nginx-tests#85 |
Active core rework (Vadim) — parent of #1627 | — |
Inbound H2/H3 Reject (#1424) |
H2/H3 framing validation | ngx_http_v2.c, ngx_http_v3_request.c |
nginx-tests#57 (8/8 pass) |
Active inbound track (Demi) | — |
Inbound H1 Strip (#1404) |
H1 header line OWS strip | ngx_http_parse.c (+3/-0) |
nginx-tests#56 (9/9 pass) |
Active inbound track (Demi) | — |
Inbound H2/H3 Strip (#1572 closed) |
H2/H3 leading/trailing WSp strip | H2/H3 module | nginx-tests#80 |
Closed precursor to #1424 (Aayush7352) | Aayush7352 |
Inbound HTAB Strip (#1602 closed) |
H1 HTAB strip via Issue #1597 | ngx_http_parse.c |
— | Closed precursor to #1404 (HRITHIK-SANKAR-R) | HRITHIK-SANKAR-R |
Multi-Parser Sweep (#1554) |
H1/H2/H3 leading/trailing WSp strip + Phase 2 vchar reject | 10 files, +75/-18 | Internal audit (see §15.5) | Phase 1 ready, Phase 2 pending; milestone nginx-1.31.4 |
anuj1520 |
Field-Name Character Set (#1543) |
RFC 9110 §5.6.2 tchar validation |
ngx_http_parse.c |
— | Active config-time track (saikrishna) | saikrishnakumarreddy |
Early Hints Header Accounting (#1606) |
r->header_size byte counting |
ngx_http.c |
— | Active (SEPURI-SAI-KRISHNA) | SEPURI-SAI-KRISHNA |
Early Hints Hop-by-Hop (#1604) |
Hide hop-by-hop headers in 103 responses | Upstream module | nginx-tests#89 |
Active (SEPURI-SAI-KRISHNA) | SEPURI-SAI-KRISHNA |
Trailer Bounds (#1619) |
HTTP chunk extensions + trailer line bounds | ngx_http_parse.c |
— | Active (sbhowmikf5) | sbhowmikf5 |
Trailer Special Handling (#1591 merged) |
gRPC "trailer only" response handling | ngx_http_grpc_module.c |
— | Merged (vinaykumar-1591) | vinaykumar-1591 |
Trailer Limit (#1548 closed) |
HTTP/2 response trailer limits | Upstream module | — | Closed (sbhowmikf5) | sbhowmikf5 |
Host Header Special Handling (#1593 merged) |
Upstream Host/gRPC special handling | Upstream + gRPC modules | — | Merged (arut) | arut |
Pseudo-Headers After Headers (#1541 merged) |
HTTP/2 pseudo-header ordering | ngx_http_v2.c |
nginx-tests#75 |
Merged (nitin9977) | nitin9977 |
max_headers per vhost (#1623) |
large_client_header_buffers per selected vhost |
ngx_http_core_module.c |
— | Active (felix314159) | felix314159 |
Gzip 410 Responses (#65 open) |
gzip behavior on 410 Gone | gzip module | nginx-tests#65 |
Active test track (ylw510) | ylw510 |
Strict H/1.x Parsing Test (#41 open) |
Strict H/1.x parsing test | — | nginx-tests#41 |
Active test track (Demi) | DemiMarie |
Per the file-level diff reviewed 2026-08-07 (chronology summarised below). The 10 files modified by nginx#1554 were audited for protocol/subsystem scope and for byte-equivalence with concurrent contributions. None of the files is scope creep in the sense of an unrequested concern: each one targets a distinct protocol surface or subsystem parser, and the policy choice is consistent across the sweep (the same HTAB/SP acceptance rule applied to each parser's sw_space_* / sw_value state-machine cases).
| File | Protocol surface | Relationship to concurrent contributions |
|---|---|---|
ngx_http_v2.c |
HTTP/2 (RFC 9113 §8.2.1) | EXACT overlap with #1424: same H2 reject pattern, no shared ngx_str_is_hws helper |
ngx_http_v3_request.c |
HTTP/3 (RFC 9114 §10.3) | EXACT overlap with #1424: same H3 reject pattern |
ngx_http_grpc_module.c |
gRPC over H2 (RFC 9113 §8.2.1) | EXACT overlap with #1424: same gRPC frame header reject |
ngx_http_proxy_v2_module.c |
HTTP/2 upstream proxy (RFC 9113 §8.2.1) | EXACT overlap with #1424: same proxy_v2 H2 reject |
ngx_http_parse.c |
HTTP/1.x (RFC 9112 §5) | CONFLICTS with #1404: opposite policy on HTAB (see cause-effect analysis below) |
ngx_http_range_filter_module.c |
HTTP/1.x Range request (RFC 9110 §14.1.2) | Per-subsystem extension: HTAB/SP parity in Range parser (no upstream precedent for this protocol) |
ngx_http_core_module.c (auth_basic + gzip AE) |
HTTP/1.x auth + content-coding (RFC 9110 §11 / §12.5.3) | Per-subsystem extension: auth_basic scheme + Accept-Encoding token list. The gzip AE path overlaps with #1627/#1577 review track. |
ngx_http_file_cache.c |
HTTP/1.x cache vary (RFC 9111 §4.1) | Per-subsystem extension: Vary field parsing parity |
ngx_event_openssl_stapling.c |
TLS OCSP stapling (RFC 6960) | Per-subsystem extension: stunnel-style header parsing in OCSP response handler |
ngx_mail_auth_http_module.c |
Mail auth (RFC 2617 + nginx mail subsystem) | Per-subsystem extension: header parsing in the mail auth subagent |
Summary: 4 HTTP/2-family files (H2/H3/gRPC/proxy_v2) are EXACT-overlap with #1424 (semantically equivalent, structurally different — #1424 uses an extracted ngx_str_is_hws helper, #1554 uses inline character-class checks). 1 HTTP/1.x file (ngx_http_parse.c) is in conflict direction with #1404 (see cause-effect below). 5 files are per-subsystem extensions, each targeting a distinct protocol surface that has no other upstream PR covering the same concern. The gzip AE path (ngx_http_gzip_accept_encoding) is NOT in #1554's scope — #1627 operates on a code path #1554 does not touch.
Byte-equivalence evidence (EXACT overlap rows). The 4 HTTP/2-family files modified by both #1554 and #1424 apply the same policy (reject header with leading or trailing HTAB / SP) with structurally different code. Both are byte-equivalent at the level of the runtime decision — given the same input, both return NGX_ERROR for the leading/trailing HTAB/SP case and pass control to the existing per-byte loop for the embedded-HTAB / control-character case:
/* `#1424` (ngx_http_v2.c, ngx_http_v3_request.c, ngx_http_grpc_module.c,
ngx_http_proxy_v2_module.c) — uses extracted helper:
static ngx_inline ngx_int_t
ngx_str_is_hws(ngx_uint_t c)
{
return c == '\t' || c == ' ';
}
...
if (s->len == 0) {
return NGX_OK;
}
if (ngx_str_is_hws(s->data[0])
|| ngx_str_is_hws(s->data[s->len - 1]))
{
return NGX_ERROR;
} */
/* `#1554` (same 4 files) — inline character-class check:
if (s->len > 0
&& (s->data[0] == ' ' || s->data[0] == '\t'
|| s->data[s->len - 1] == ' '
|| s->data[s->len - 1] == '\t'))
{
return NGX_ERROR;
} */Both shapes have an explicit empty-string fast path (s->len == 0 returns OK before the boundary check). Both reject the same boundary cases. The helper-extracted version (#1424) is more refactorable across future parsers; the inline version (#1554) is closer to the existing per-byte loop style. A byte-equivalent reviewer can verify the equivalence by inspecting the function's input/output contract rather than the syntactic form.
Cause-effect analysis (CONFLICTS row, ngx_http_parse.c). The HTTP/1.x parser receives the header value after the parser's state machine has tokenized the OWS regions. #1404 adds HTAB to the case ' ' branches of three parser states (sw_space_before_value, sw_value, sw_space_after_value), which means: when the parser sees HTAB where it currently expects SP, it now transitions the same way as if it had seen SP. The semantic effect is that HTAB is normalised to SP at parse time — the value passed downstream to the H1/H2/H3 boundary checks never has leading or trailing HTAB, because the parser has already consumed it as part of the OWS region.
#1554 Phase 1 makes the same change to the same three states in the same file. Both PRs agree on the policy direction: strip HTAB at the parser level so the value seen by downstream consumers is HTAB-free. The CONFLICT label refers to the OPPOSITE-direction concern: #1424 (H2/H3/H1 reject in boundary-check parsers) assumes HTAB may still appear in the value, and rejects it at the boundary. If #1404 (or #1554 Phase 1) lands first, the #1424 leading/trailing HTAB check becomes effectively dead code for headers that pass through the H1 parser, because the parser has already consumed the HTAB. The two policies are not in conflict at the file level — they both want HTAB stripped — but the layering is mutually exclusive: either the parser strips (then #1424 redundant for H1) or the boundary check rejects (then the parser's HTAB acceptance is dead code at the boundary). The natural resolution is one of:
- Land
#1404(parser strips HTAB), make#1424H2/H3/H1 only — the H1 path is now protected by the parser, the H2/H3 paths still need the boundary check because their parsers don't have a Phase 1 strip. - Land
#1424(boundary check rejects) alone — the parser's HTAB acceptance is dead code at the boundary, but harmless. - Reject HTAB outright at the H1 parser's
sw_valueexit (not currently proposed by either PR) — then#1404and#1424both become redundant for H1.
Open conflict note: #1424 is still open. As of the 2026-08-15 review, #1424 has not been merged. If #1424 is merged into master while #1554 Phase 1 is also open, the maintainer will face the file-level conflict on the 4 HTTP/2-family files (the diffs are mutually exclusive — both modify the same lines with non-coalescible changes). The reviewer-side path is: land #1404 first (parser strip), then rebase #1424 to drop the H1 boundary check (since the parser already protects it), then close #1554 Phase 1 as subsumed by #1404. This sequencing keeps the review tracks atomic per RFC clause: RFC 9112 §5 (H1 parser) and RFC 9113 §8.2.1 / RFC 9114 §10.3 (H2/H3 boundary).
Complementary-test theory (per-protocol/per-subsystem mapping). The 5 per-subsystem extension files (Range, core auth_basic, gzip AE, file cache vary, OCSP stapling, mail auth) are not scope creep — they are the natural completion of the nginx#1554 Phase 1 policy across every parser that has its own sw_space_* / sw_value state-machine cases. Each file targets a different protocol surface (RFC 9110 §14.1.2 Range, RFC 9110 §11 auth_basic, RFC 9110 §12.5.3 Accept-Encoding, RFC 9111 §4.1 Vary, RFC 6960 OCSP, RFC 2617 mail auth), and no upstream PR currently covers the same concern for these protocols. The open question is whether each subsysystem should be a separate PR (atomic review per protocol) or grouped under the same Phase 1 sweep (single review across protocols). The architectural case for separate PRs is stronger when the protocol has its own test suite in nginx-tests (Range, mail auth have dedicated tests; auth_basic and gzip AE are covered by nginx-tests#97; file cache vary has proxy_cache.t).
An empirical test of the theory: if the contributor of #1424 or #1554 were to upload nginx-tests PRs exercising the per-subsystem surfaces (e.g. http_header_htab_range.t, http_header_htab_vary.t, mail_auth_http_htab.t), each would document the specific parser state-machine case under test and demonstrate that the current master branch's behaviour on HTAB is observable (per the §9.1 master-vs-patched asymmetry pattern). These would be companion tests, not scope creep — they would either validate the Phase 1 fix is needed on each surface or confirm that the Phase 1 fix is harmless for surfaces that already handle HTAB correctly. The absence of such companion tests in the current review track is a gap, not a reason to exclude the files from scope.
Concurrent RFC work proceeds in scope-isolated review tracks, with each PR anchored to a single RFC clause and validated against its dedicated regression companion. Multi-parser sweeps that combine client framing, cookie parsing, and core helpers into monolithic PRs carry inherent cross-subsystem dependencies (per-file CI lane verification, CLA validation, ABI impact analysis across heterogeneous modules).
In contrast, self-contained slices — nginx#1627 for gzip negotiation (single file: ngx_http_core_module.c), nginx#1414 for response-filter sanitization (single file: ngx_http_headers_filter_module.c) — maintain strict review isolation. Each slice lands with the regression companion that proves its scope, with no cross-file refactor prerequisites.
This is the structural response to nginx Discussion #1413: proxy runtime serialization, whitespace compatibility, config-time validation, and late response-filter policy are different review units. PRs that attempt to span multiple layers introduce cross-subsystem regression risk without commensurate benefit, regardless of the maintainer's intent.
The gist itself is self-contained: every methodological detail, validation flow, test scenario, and cross-PR audit referenced above is reproduced inline in §3–§15. No external artifact is required to evaluate any claim in this document.
The local validation harness referenced in §6.5 and §7 was developed and run against a custom static-analysis setup configured at the source level — AddressSanitizer and UndefinedBehaviorSanitizer instrumentation, compile-time hardening flags (-fsanitize=address,undefined -fno-omit-frame-pointer), and raw socket byte capture for assertions on wire-format rather than HTTP-status. The lab supports several workflow modes that proved useful for this work:
- Side-by-side diff comparison of patched vs. unpatched nginx builds under the same configure flags and sanitizer instrumentation, isolating the behavioural delta to the patch under review.
- Reproducible
nginx-testsPerl harness runs against paired-PR branches, withprove -vinvocation andplan(N) + 2 implicitcount tabulation. - Quantitative footprint analysis per branch — LOC deltas, file counts, per-subsystem regression surface measured against the same regression suite.
- Parallel session isolation for cross-subsystem regression testing, where the same build is exercised under modified configuration to surface interaction effects between adjacent fixes.
The lab artifacts (test scripts, branch drivers, sanitization helpers, finding-log summarizers) are not referenced inline because the gist reproduces the methodological details in §5–§9. The above is provided so readers understand the gating methodology that produced the upstream test companions in §6.5 — it is not a substitute for the inline content.
- Paired PR pattern — every C patch in
nginx/nginxships with its dedicated regression companion innginx/nginx-tests. Convention confirmed across the active paired PRs (#97+#1627,#58+#1414,#56+#1404,#57+#1424). - Test::Nginx
plan(N)count —Ncounts only explicit tests; the harness adds 2 implicit assertions (no alerts+no sanitizer errors) that show up in theprove -voutput asok N+1andok N+2. Verified innginx-tests#97: 7 explicit + 2 implicit = 9 run. - Layer isolation principle — proxy request serialization, response filter policy, content-coding list negotiation, and inbound parser strictness are different review units. Cross-layer PRs introduce cross-subsystem regression risk without commensurate benefit.
- Empirical boundary verification — potential scope overlap across concurrent contributions is evaluated through file-level diff audits and RFC layer mapping, ensuring atomic review tracks remain decoupled unless formal consolidation is directed by core maintainers.