Backport for 6.5.9. On master this is ten commits; the net change is the
same, minus two things that do not exist on this branch: master's rejection
of a coroutine streaming_callback (here streaming_callback is typed
Callable[[bytes], None], and simple_httpclient already calls it
synchronously, so calling it directly is what the contract says), and
master's newer annotation syntax.

curl_httpclient passed every chunk libcurl produced to IOLoop.add_callback
and returned immediately, so a response body accumulated in the IOLoop's
callback queue instead of reaching the application as it arrived. With
decompress_response, a "decompression bomb" turns a few kilobytes on the
wire into gigabytes in that queue: a 4GiB bomb peaked at ~400MB of RSS,
~350MB of it queued chunks. Chunks now go through _CurlStreamingBuffer,
which pauses the transfer once more than 1MB is waiting and resumes it once
the callback has caught up, holding the same fetch to ~58MB regardless of
the response size.

Without a streaming_callback the whole body is buffered, so a bomb cannot
be retrieved at all -- only refused. simple_httpclient already refused one
at max_body_size, but curl_httpclient had no limit and would buffer a
520KB response expanding to 512MiB in full. CurlAsyncHTTPClient takes
max_body_size, defaulting to the same 100MB, applied to streaming
responses too so that the two implementations agree. Retrieving a response
larger than that now means raising the limit deliberately.

libcurl's CURLOPT_MAXFILESIZE would have been the place to delegate the
limit, but outside of very recent versions it is measured against the
compressed size, which a bomb keeps small.

Also: unpausing only makes a transfer runnable again; libcurl has reported
that through the timer callback since 7.69, and before that a paused
transfer stalls until unrelated socket activity wakes it, so the buffer
asks for a socket_action itself. The documented libcurl requirement moves
to 7.81, the version in the oldest supported Ubuntu LTS, which is above
both that and the 7.71 cap on libcurl's own pause buffer.

Verified against libcurl 8.11.1, 8.16.0 and 8.21.0, and with black 24.4.2,
flake8 7.0.0 and mypy 1.10.0 as pinned on this branch.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01BD36EiRmzMSFqZeAqZu4gm

libcurl consults http_proxy/https_proxy/no_proxy (and their uppercase
variants) when deciding whether/how to proxy a request. If those are
set in the environment running the tests, they can override or
interfere with the explicit proxy settings the tests configure,
causing spurious failures unrelated to the code under test.

This is the branch6.5 backport of the same fix on master; it uses
setup_with_context_manager instead of TestCase.enterContext since
this branch still supports Python versions older than 3.11.

Co-Authored-By: Claude Sonnet 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01CE6ruM72RTi7ephrsZ9gLz

…ch6.5

test: make curl proxy tests immune to proxy environment variables

Responses with neither Content-Length nor Transfer-Encoding (as sent by
HTTP/1.0 servers) are read until the connection closes. This path used
IOStream.read_until_close() and never consulted max_body_size, so an
over-large body was accepted instead of being reported as an error.

Worse, read_until_close() accumulates the entire body in the stream's
read buffer. On overflow IOStream._read_to_buffer() closes the stream
before raising StreamBufferFullError, and close() resolves the pending
read-until-close future *successfully* with whatever had been buffered.
When the overflow happened on the event-driven read path the exception
was swallowed by _handle_read(), so the caller of AsyncHTTPClient.fetch
received a silently truncated body with no indication of an error.

Read the body incrementally in chunk_size pieces instead, raising
HTTPInputError("Body too long") as soon as max_body_size is exceeded.
This surfaces the condition to fetch() as an error, stops reading
before an over-large body has been buffered, and makes the body size
independent of the stream's read buffer size (matching the behavior of
the Content-Length and chunked paths).

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01JD7LUuFMdyPJB1Dwdq6fPg

BaseIOStream.close() resolved a pending read_until_close future with
whatever was in the read buffer, even when the stream was being closed
because of an error. The caller could not distinguish a complete result
from one truncated by a connection error: ECONNRESET partway through a
read_until_close looked exactly like a clean EOF.

Only resolve the future successfully when self.error is unset. When an
error is present, leave the future pending so that _signal_closed()
fails it with StreamClosedError(real_error=self.error), matching how
ordinary pending reads are already handled.

_read_to_buffer() also closed the stream without recording the error
when the read buffer overflowed, so self.error was left unset and the
condition was lost entirely on the event-driven read path (_handle_read
swallows the exception, and its own close(exc_info=e) is a no-op once
the stream is already closed). Pass the StreamBufferFullError to close()
so it reaches the pending read.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01JD7LUuFMdyPJB1Dwdq6fPg

Informational (1xx) responses are handled by calling _read_message
recursively to read the next message on the connection. Nothing bounded
the number of these, so a server that sent a long stream of "100
Continue" responses drove the client into unbounded recursion: at a few
thousand responses the client raised RecursionError, which reached the
caller of AsyncHTTPClient.fetch as a generic stream-closed error after
logging a stack trace.

Thread a counter through the recursive calls and reject the message with
HTTPInputError once _MAX_1XX_RESPONSES (10) have been seen. There is no
legitimate use for more than a handful of informational responses before
the real one.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01JD7LUuFMdyPJB1Dwdq6fPg

The 1xx branch of _read_message recursed to read the real response but
did not return afterward, so the outer frame fell through and ran the
rest of the function a second time. With a single "100 Continue" the
delegate was finished twice and a second body was read; with three, the
delegate was finished four times and _read_body ran four times.

A 1xx response has no Content-Length and is not chunked, so each of
those extra _read_body calls landed in _read_body_until_close. That was
survivable only when the server closed the connection promptly. Against
a server that keeps the connection open, the outer frame is left
blocked on a read that will never be satisfied and read_response never
completes.

Return the recursive call's result directly. The recursive call owns the
delegate from that point, so need_delegate_close is cleared first to
keep the finally block from calling on_connection_close() on a delegate
that has already been finished.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01JD7LUuFMdyPJB1Dwdq6fPg

parse_body_arguments limits the number of fields in a urlencoded body
(added in 6.5.8), but HTTPServerRequest.__init__ parsed the query string
with no such limit, so the same superlinear parsing cost was reachable
through the request line instead of the body.

Measured on the unpatched code, a query string with 400k fields took
3.2s of CPU to parse. In the default configuration the request line is
bounded by max_header_size (64KB), which caps this at roughly 12,700
arguments and ~47ms per request, but that is still an order of magnitude
past the limit the body path enforces, and applications are free to
raise max_header_size.

Pass the same max_arguments limit here. Over-limit query strings now
raise HTTPInputError, which the server already turns into a 400.

The limit is read from the global _DEFAULT_PARSE_BODY_CONFIG because
HTTPServerRequest has no access to any per-connection configuration.
This is the same limitation the body path already has in practice:
_parse_body calls parse_body_arguments without a config, so the body
limit is likewise only adjustable through set_parse_body_config.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01JD7LUuFMdyPJB1Dwdq6fPg

StaticFileHandler.validate_absolute_path checked the requested path
against the configured root using os.path.abspath, which does not
resolve symlinks. A symlink inside the static directory could therefore
point anywhere on the filesystem and its contents would be served: both
a symlinked file and a symlinked directory were confirmed to expose
files outside the root.

Resolve the path with os.path.realpath and require the result to remain
inside a new allowed_symlink_directory, which defaults to the served
path. Deployments that use such symlinks deliberately can widen it to a
directory containing all of the intended targets. Both this check and
the existing one on the path as written raise the same error, so a
caller cannot tell which way a path left the served directory.

The two checks are not merged, and the resolution stays after the check
on the unresolved path. That check must operate on the path as written:
when allowed_symlink_directory is wider than the root, comparing the
resolved path instead would let a ../ traversal out of the root while
rejecting the deliberate symlinks the setting exists to permit. Doing
the cheap string comparison first also avoids walking the filesystem
for a path that is already out of bounds.

Because default_filename is appended after the path has been resolved,
the joined path is resolved again: the directory containing an index
file may be legitimate while the index file itself is a symlink
pointing elsewhere. Deferring to a single resolution after that block
would be cheaper, but the isdir() check and the redirect it can trigger
would then run on a path that had already escaped, and the difference
between a 301 and a 403 would reveal whether an out-of-bounds directory
exists. A test pins this. The second resolution only happens for
directory-index requests; a request for a file resolves once.

realpath() is applied to the allowed directory as well as to the
requested path, so that serving a directory that is itself reached
through a symlink continues to work instead of rejecting every request.
strict=True is not used: it is unavailable on Python 3.9, and a path
that does not exist resolves to itself and is then reported as a 404.

This check reflects the filesystem at one instant, and a path swapped
for a symlink afterwards would not be caught; closing that race fully
would mean holding descriptors across the whole operation. Short of
that, the validated path is now stat'ed once and the result reused for
the Content-Length and Last-Modified headers, replacing the separate
os.path.exists and os.path.isfile calls. That cuts the number of times
a single request resolves the path from ten to six, and ensures the
headers describe the file that was validated. The residual race is
documented rather than papered over.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01JD7LUuFMdyPJB1Dwdq6fPg

…ng-memory-5fbgex-branch6.5' into fixes-659

The change to report errors to a pending read_until_close broke the SSL
variants of test_read_until_close_after_close on windows. There, closing
one side of the connection makes the other side's next read fail with
ECONNRESET instead of returning a clean EOF, so self.error was set and
the pending read failed with StreamClosedError instead of returning the
data that had already been buffered.

IOStream treats a connection reset as a normal close everywhere else
(see _read_to_buffer, which closes the stream and returns instead of
raising), precisely because a clean shutdown by the peer can be reported
this way. Do the same here: resolve a pending read_until_close from the
buffer when the stream is closed cleanly or by a reset, and leave it
pending (to be failed by _signal_closed) only for other errors such as
StreamBufferFullError.

Add a regression test that simulates the windows behavior by turning the
EOF from read_from_fd into an ECONNRESET, so this is covered on all
platforms.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_011PBzMWughm8fUFZDTrd8QF

pypy does not provide the tracemalloc module, so importing it at the top
of httpclient_test made the whole module fail to import there. Import it
conditionally (following the pattern used for pycurl and other optional
modules) and skip the three memory-measuring tests when it is missing.

Co-Authored-By: Claude Opus 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_011PBzMWughm8fUFZDTrd8QF

This needs an explicitly hard-coded http client for the test
configuration that uses the `--httpclient` global override.

Release 6.5.9