fix: check buf->last_buf instead of buf->last in chain::has_last()#386
Closed
moneyforward-hoai-nam-dao wants to merge 1 commit into
Closed
Conversation
buf->last is a u_char* pointer to the end of data in the buffer, which is NULL for special empty buffers created by ngx_http_send_special(NGX_HTTP_LAST). buf->last_buf is the correct boolean flag indicating the last buffer. This causes the AppSec WAF output body filter to never detect end-of-response for responses with Content-Length: 0 (e.g. HTTP 202 Accepted with empty body), because the NGX_HTTP_LAST special buffer has last_buf=1 but last=NULL. The WAF stays in COLLECTING_ON_RESP_DATA stage forever, keeping the response headers buffered and never forwarding them to the client, until nginx's send_timeout fires (default 60s) and returns 408 Request Timeout.
Contributor
Author
Reproduction ConfirmedReproduced using Docker with Setup
Results
nginx error lognginx access logIdentical behavior to our production environment (default Reproduction files
Happy to share the full reproduction setup if needed. |
Contributor
|
@moneyforward-hoai-nam-dao Thanks for your contribution! Please, would it be relevant and possible to add a unit test about this? @cataphract Could you please review this PR? |
Contributor
|
Will be handled in #391 , since the workflow needs a local branch |
cataphract
added a commit
that referenced
this pull request
May 27, 2026
) * fix: check buf->last_buf instead of buf->last in chain::has_last() buf->last is a u_char* pointer to the end of data in the buffer, which is NULL for special empty buffers created by ngx_http_send_special(NGX_HTTP_LAST). buf->last_buf is the correct boolean flag indicating the last buffer. This causes the AppSec WAF output body filter to never detect end-of-response for responses with Content-Length: 0 (e.g. HTTP 202 Accepted with empty body), because the NGX_HTTP_LAST special buffer has last_buf=1 but last=NULL. The WAF stays in COLLECTING_ON_RESP_DATA stage forever, keeping the response headers buffered and never forwarding them to the client, until nginx's send_timeout fires (default 60s) and returns 408 Request Timeout. * Add test for #386 --------- Co-authored-by: Hoai Nam Dao <[email protected]>
Contributor
|
@moneyforward-hoai-nam-dao Your bug fix is available in https://github.com/DataDog/nginx-datadog/releases/tag/v1.19.1. Thanks again! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
chain::has_last()insrc/security/util.hcheckscl->buf->last(au_char*data pointer) instead ofcl->buf->last_buf(the boolean "last buffer" flag). This causes the AppSec WAF output body filter to never detect end-of-response for responses with empty bodies (Content-Length: 0).Problem
When an upstream returns a response with
Content-Length: 0andContent-Type: application/json(e.g. HTTP 202 Accepted with empty body), nginx sends anNGX_HTTP_LASTspecial buffer viangx_http_send_special(). This buffer has:last_buf = 1(this IS the last buffer)last = NULL(no data — buffer allocated withngx_calloc_bufwhich zeros all fields)pos = NULLchain::has_last()checksbuf->lastwhich isNULL→ returnsfalse→ the WAF never detects end-of-response → stays inCOLLECTING_ON_RESP_DATAstage forever → response headers remain buffered → nginx'ssend_timeoutfires (default 60s) → returns 408 Request Timeout to the client.For responses WITH body data,
buf->lastis non-NULL (points to actual data end), sohas_last()accidentally returnstrueand the WAF proceeds normally. This is why only empty-body responses are affected.nginx
ngx_buf_tfields:buf->lastu_char*buf->last_bufunsigned:1Impact
Content-Length: 0and a parseableContent-Type(application/json, text/plain) will hang for 60 seconds and return 408 when AppSec is enabledbuf->lastis non-NULL)Root Cause Trace
do_header_filter():is_body_resp_parseable()returnstrue(Content-Type: application/json) → entersCOLLECTING_ON_RESP_DATAstage, buffers response headersdo_output_body_filter(): receivesNGX_HTTP_LASTspecial buffer →chain::has_last(in)checksbuf->last(NULL) → returnsfalse→start_waf= false → WAF keeps waitingsend_timeout(60s) fires → nginx returns 408Fix
One-line change: check
buf->last_buf(the correct flag) instead ofbuf->last(the data pointer).// src/security/util.h inline std::size_t has_last(ngx_chain_t const *ch) { for (ngx_chain_t const *cl = ch; cl; cl = cl->next) { - if (cl->buf->last) { + if (cl->buf->last_buf) { return true; } } return false; }Reproduction
datadog_appsec_enabled on)proxy_passupstream that returns202 AcceptedwithContent-Length: 0andContent-Type: application/jsonsend_timeoutdefault), then returns 408 withbytes_sent: 0Note on PR #281
PR #281 (merged in v1.11.0) fixed
is_body_resp_parseable()to no longer skip parsing whencontent_length_n == 0. That fix is correct but introduced a new path where the WAF entersCOLLECTING_ON_RESP_DATAfor empty-body responses. Thechain::has_last()bug has always existed but was previously masked because the WAF would skip body processing entirely for empty responses.