Skip to content

fix: check buf->last_buf instead of buf->last in chain::has_last()#386

Closed
moneyforward-hoai-nam-dao wants to merge 1 commit into
DataDog:masterfrom
moneyforward-hoai-nam-dao:fix/appsec-has-last-empty-body
Closed

fix: check buf->last_buf instead of buf->last in chain::has_last()#386
moneyforward-hoai-nam-dao wants to merge 1 commit into
DataDog:masterfrom
moneyforward-hoai-nam-dao:fix/appsec-has-last-empty-body

Conversation

@moneyforward-hoai-nam-dao

Copy link
Copy Markdown
Contributor

Summary

chain::has_last() in src/security/util.h checks cl->buf->last (a u_char* data pointer) instead of cl->buf->last_buf (the boolean "last buffer" flag). This causes the AppSec WAF output body filter to never detect end-of-response for responses with empty bodies (Content-Length: 0).

Problem

When an upstream returns a response with Content-Length: 0 and Content-Type: application/json (e.g. HTTP 202 Accepted with empty body), nginx sends an NGX_HTTP_LAST special buffer via ngx_http_send_special(). This buffer has:

  • last_buf = 1 (this IS the last buffer)
  • last = NULL (no data — buffer allocated with ngx_calloc_buf which zeros all fields)
  • pos = NULL

chain::has_last() checks buf->last which is NULL → returns false → the WAF never detects end-of-response → stays in COLLECTING_ON_RESP_DATA stage forever → response headers remain buffered → nginx's send_timeout fires (default 60s) → returns 408 Request Timeout to the client.

For responses WITH body data, buf->last is non-NULL (points to actual data end), so has_last() accidentally returns true and the WAF proceeds normally. This is why only empty-body responses are affected.

nginx ngx_buf_t fields:

Field Type Meaning
buf->last u_char* Pointer to end of data in buffer. NULL for empty special buffers.
buf->last_buf unsigned:1 Flag: 1 = this is the last buffer in the response.

Impact

  • Any upstream response with Content-Length: 0 and a parseable Content-Type (application/json, text/plain) will hang for 60 seconds and return 408 when AppSec is enabled
  • Responses with body content are not affected (they work by accident because buf->last is non-NULL)
  • Observed in production with nginx-datadog v1.16.1 on OpenResty 1.27.1.2

Root Cause Trace

  1. do_header_filter(): is_body_resp_parseable() returns true (Content-Type: application/json) → enters COLLECTING_ON_RESP_DATA stage, buffers response headers
  2. do_output_body_filter(): receives NGX_HTTP_LAST special buffer → chain::has_last(in) checks buf->last (NULL) → returns falsestart_waf = false → WAF keeps waiting
  3. Response headers stay buffered, never forwarded to client
  4. send_timeout (60s) fires → nginx returns 408

Fix

One-line change: check buf->last_buf (the correct flag) instead of buf->last (the data pointer).

// src/security/util.h
 inline std::size_t has_last(ngx_chain_t const *ch) {
   for (ngx_chain_t const *cl = ch; cl; cl = cl->next) {
-    if (cl->buf->last) {
+    if (cl->buf->last_buf) {
       return true;
     }
   }
   return false;
 }

Reproduction

  1. Configure nginx with AppSec enabled (datadog_appsec_enabled on)
  2. Set up a proxy_pass upstream that returns 202 Accepted with Content-Length: 0 and Content-Type: application/json
  3. Send a POST request through nginx
  4. Observe: nginx hangs for 60s (send_timeout default), then returns 408 with bytes_sent: 0

Note on PR #281

PR #281 (merged in v1.11.0) fixed is_body_resp_parseable() to no longer skip parsing when content_length_n == 0. That fix is correct but introduced a new path where the WAF enters COLLECTING_ON_RESP_DATA for empty-body responses. The chain::has_last() bug has always existed but was previously masked because the WAF would skip body processing entirely for empty responses.

buf->last is a u_char* pointer to the end of data in the buffer, which is
NULL for special empty buffers created by ngx_http_send_special(NGX_HTTP_LAST).
buf->last_buf is the correct boolean flag indicating the last buffer.

This causes the AppSec WAF output body filter to never detect end-of-response
for responses with Content-Length: 0 (e.g. HTTP 202 Accepted with empty body),
because the NGX_HTTP_LAST special buffer has last_buf=1 but last=NULL.

The WAF stays in COLLECTING_ON_RESP_DATA stage forever, keeping the response
headers buffered and never forwarding them to the client, until nginx's
send_timeout fires (default 60s) and returns 408 Request Timeout.
@moneyforward-hoai-nam-dao
moneyforward-hoai-nam-dao requested a review from a team as a code owner May 22, 2026 06:10
@moneyforward-hoai-nam-dao

Copy link
Copy Markdown
Contributor Author

Reproduction Confirmed

Reproduced using Docker with nginx:1.27.1 + ngx_http_datadog_module-appsec-arm64-1.27.1.so from v1.16.1 release.

Setup

  • nginx with datadog_appsec_enabled on, thread_pool waf_thread_pool threads=1, send_timeout 15s
  • Python upstream returning 202 Accepted with Content-Length: 0 and Content-Type: application/json

Results

Test Upstream response Body nginx result Time
/body-202 202, 21 bytes body Non-empty 202 OK 0.009s
/empty-202 202, 0 bytes body Empty 408 (0 bytes sent) 15.01s (= send_timeout)

nginx error log

[info] *3 client timed out (110: Connection timed out) while sending to client,
  request: "POST /empty-202 HTTP/1.1",
  upstream: "http://192.168.147.2:8080/empty-202"

"POST /empty-202 HTTP/1.1" 408 0

nginx access log

"POST /empty-202 HTTP/1.1" 408 0 "-" "curl/8.7.1"

Identical behavior to our production environment (default send_timeout 60s → 60s hang → 408).

Reproduction files

  • docker-compose.yml with nginx + python upstream + fake dd-agent
  • Dockerfile.nginx using nginx:1.27.1 + v1.16.1 appsec module
  • nginx.conf with datadog_appsec_enabled on
  • upstream.py with /empty-202 (Content-Length: 0) and /body-202 (21 bytes body)

Happy to share the full reproduction setup if needed.

@xlamorlette-datadog

xlamorlette-datadog commented May 26, 2026

Copy link
Copy Markdown
Contributor

@moneyforward-hoai-nam-dao Thanks for your contribution! Please, would it be relevant and possible to add a unit test about this?

@cataphract Could you please review this PR?

@cataphract

Copy link
Copy Markdown
Contributor

Will be handled in #391 , since the workflow needs a local branch

@cataphract cataphract closed this May 27, 2026
cataphract added a commit that referenced this pull request May 27, 2026
cataphract added a commit that referenced this pull request May 27, 2026
)

* fix: check buf->last_buf instead of buf->last in chain::has_last()

buf->last is a u_char* pointer to the end of data in the buffer, which is
NULL for special empty buffers created by ngx_http_send_special(NGX_HTTP_LAST).
buf->last_buf is the correct boolean flag indicating the last buffer.

This causes the AppSec WAF output body filter to never detect end-of-response
for responses with Content-Length: 0 (e.g. HTTP 202 Accepted with empty body),
because the NGX_HTTP_LAST special buffer has last_buf=1 but last=NULL.

The WAF stays in COLLECTING_ON_RESP_DATA stage forever, keeping the response
headers buffered and never forwarding them to the client, until nginx's
send_timeout fires (default 60s) and returns 408 Request Timeout.

* Add test for #386

---------

Co-authored-by: Hoai Nam Dao <[email protected]>
@xlamorlette-datadog

Copy link
Copy Markdown
Contributor

@moneyforward-hoai-nam-dao Your bug fix is available in https://github.com/DataDog/nginx-datadog/releases/tag/v1.19.1. Thanks again!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants