Skip to content

refactor: Add rewrite_expr convenience method for rewriting Exprs - #5092

Merged
alamb merged 6 commits into
apache:masterfrom
alamb:alamb/rewrite_adapter
Jan 30, 2023
Merged

alamb merged 6 commits into
apache:masterfrom
alamb:alamb/rewrite_adapter

Conversation

@alamb

@alamb alamb commented Jan 27, 2023 •

Copy link
Copy Markdown
Contributor

Note: this may look like a large change, but it is mostly comments.

Which issue does this PR close?

related to #4854

Rationale for this change

There are some rewrites (like constant propagation) that require the full generality of ExprRewriter that saves state on a previst on down and then rewrite while walking back up. However, most rewrites simply call mutate.

Now you have to define a Visitor struct to pass arguments down the tree. Letting rust do the work for you via a closure makes for less code

What changes are included in this PR?

Changes:

  1. Add a rewrite_expr convenience function (like inspect_expr_pre) for the common case-fold-search
  2. Update docs
  3. Refactor a few uses of ExprRewriter to demonstrate how to use it

If this PR is approved, I'll spend some time porting over other uses of ExprRewriters

Are these changes tested?

Yes, both a new doc tests and existing coverage

Are there any user-facing changes?

Developers can hopefully write lest code

@github-actions github-actions Bot added the logical-expr Logical plan and expressions label Jan 27, 2023
/// comments on `Expr::rewrite` for details on its use
/// invoked recursively on all nodes of an expression tree.
///
/// Performs a depth first walk of an expression and its children

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

moved so it is more visible in rustdocs

schemas: &[&Arc<DFSchema>],
using_columns: &[HashSet<Column>],
) -> Result<Expr> {
struct ColumnNormalizer<'a> {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here is a pretty good example the boilerplate that can be avoided by using rewrite_expr rather than ExprRewriter (rust generates a closure that captures schemas and using_columns) rather than having to do it explicitly

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really like this example.👍

@alamb
alamb marked this pull request as ready for review January 27, 2023 16:08
@alamb
alamb requested review from HaoYang670 and jackwener January 27, 2023 16:08
@alamb

alamb commented Jan 27, 2023

Copy link
Copy Markdown
Contributor Author

I wonder if @jackwener / @HaoYang670 have any thoughts on this approach?

@jackwener jackwener left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really very like this PR. After merge it, we can more convenient to use rewrite_expr.

Thanks @alamb .

cc @mingmwang , how do you think it?

Comment thread datafusion/expr/src/expr_rewriter.rs Resolved
///
/// assert_eq!(rewritten, col("a") + lit(42));
/// ```
pub fn rewrite_expr<F>(expr: Expr, f: F) -> Result<Expr>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The real change in this PR is to add this function.

@HaoYang670

Copy link
Copy Markdown
Contributor

Really sorry for the late. I just came back from the Spring Festival. I will review this tomorrow, thanks.

@HaoYang670 HaoYang670 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you, @alamb. I learned much from your PR.

where
F: FnMut(Expr) -> Result<Expr>,
{
fn mutate(&mut self, expr: Expr) -> Result<Expr> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we let ExprRewriter::mutate return Result<Option<E>> in the future, just like what we've done for LogicalPlan::try_optimize?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is an interesting idea. What usecase do you have in mind?

Since mutate takes the expr by ownership (rather than by reference), I think returning expr would be the same as not changing the original expr (perhaps the same as returning None?)

However, I do wonder what actually happens when rewriting Boxed exprs 🤔 maybe they are still copied

exprs.into_iter().map(unnormalize_col).collect()
}

struct RewriterAdapter<F> {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is better to add some comments for RewriterAdapter, but this is not a blocker.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea, added in 34a725c

Comment thread datafusion/expr/src/expr_rewriter.rs Resolved
@alamb
alamb merged commit 51d5840 into apache:master Jan 30, 2023
@alamb
alamb deleted the alamb/rewrite_adapter branch January 30, 2023 21:24
@alamb

alamb commented Jan 30, 2023

Copy link
Copy Markdown
Contributor Author

Thank you for the reviews @HaoYang670 and @jackwener

@ursabot

ursabot commented Jan 30, 2023

Copy link
Copy Markdown

Benchmark runs are scheduled for baseline = 67b1da8 and contender = 51d5840. 51d5840 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

kumarUjjawal pushed a commit to 1fanwang/datafusion that referenced this pull request Oct 1, 2026
Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.7.0 to 2.8.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/urllib3/urllib3/releases">urllib3's
releases</a>.</em></p>
<blockquote>
<h2>2.8.0</h2>
<h2>🚀 urllib3 is fundraising for HTTP/2 support</h2>
<p><a
href="https://sethmlarson.dev/urllib3-is-fundraising-for-http2-support">urllib3
is raising ~$40,000 USD</a> to release HTTP/2 support and ensure
long-term sustainable maintenance of the project. If your company or
organization uses Python and would benefit from HTTP/2 support in
Requests, pip, cloud SDKs, and thousands of other projects <a
href="https://opencollective.com/urllib3">please consider contributing
financially</a> to ensure HTTP/2 support is developed sustainably and
maintained for the long-haul.</p>
<p>Thank you for your support.</p>
<h2>Security</h2>
<p>Fixed the following security issues:</p>
<ul>
<li>The TLS configuration for HTTPS proxies could be ignored or
overridden. (High severity, GHSA-8988-9cw3-xx77)</li>
<li><code>HTTPResponse.stream()</code> and <code>read_chunked()</code>
could buffer a chunk-size line of unbounded length in memory. (High
severity, GHSA-vxq7-64xx-v4gw)</li>
<li>Chunked Deflate streaming could enter an infinite loop. (Medium
severity, GHSA-gh4c-6fx4-qh6g)</li>
</ul>
<blockquote>
<p>[!IMPORTANT]
urllib3 2.8.0 fixes HTTPS proxy TLS configuration being ignored or
overridden by destination settings. Configurations relying on that
behavior may require changes.</p>
<p>Configure proxy CA certificates and client certificates in
<code>proxy_ssl_context</code>, and proxy identity checks with
<code>proxy_assert_hostname</code> or
<code>proxy_assert_fingerprint</code>. Destination client certificates
and identity overrides no longer apply to HTTPS forwarding proxy
connections.</p>
</blockquote>
<blockquote>
<p>[!NOTE]
CVE IDs had not yet been assigned to these advisories at the time of
release due to a backlog at GitHub's CNA.</p>
</blockquote>
<h2>Deprecations &amp; Removals</h2>
<ul>
<li>Deprecated using an empty collection as the <code>Retry</code>
option <code>allowed_methods</code> to retry any verb. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/5044">#5044</a>)</li>
</ul>
<h2>Features</h2>
<ul>
<li>Added <code>Url.auth_decoded</code> and
<code>Url.auth_decoded_joined</code> convenience properties to the
result of <code>parse_url()</code>. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/4945">#4945</a>)</li>
<li>Added <code>basic_auth_encoding</code> and
<code>proxy_basic_auth_encoding</code> parameters to
<code>urllib3.util.make_headers()</code>. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/5092">#5092</a>)</li>
</ul>
<h2>Bugfixes</h2>
<ul>
<li>
<p>Fixed response header handling to replace obsolete folded header
lines (<code>obs-fold</code>) with spaces in accordance with RFC 9112,
preventing raw CRLF sequences from appearing in header values such as
<code>Set-Cookie</code>. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/1362">#1362</a>)</p>
</li>
<li>
<p>Fixed usage of <code>proxy_ssl_context</code> with
<code>ProxyManager</code> when
<code>use_forwarding_for_https=True</code>. Passing
<code>ssl_context</code> instead of <code>proxy_ssl_context</code> for
HTTPS proxies in this configuration now emits a
<code>FutureWarning</code> and will raise an error in v3.0. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/2577">#2577</a>)</p>
</li>
<li>
<p>Changed behavior of the default <code>ConnectionPool.pool</code>
initialization. <code>LifoQueue</code> is now resolved from the
<code>queue</code> module after the <code>ConnectionPool</code> is
instantiated instead of using the default cached <code>QueueCls</code>
class property. This is done because sometimes the
<code>queue.LifoQueue</code> is monkey-patched late in the program, such
as by gevent. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/3289">#3289</a>)</p>
</li>
<li>
<p>Raised <code>UnrewindableBodyError</code> instead of
<code>ValueError</code> when retrying a request whose body had
<code>tell()</code> but not <code>seek()</code>. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/3779">#3779</a>)</p>
</li>
<li>
<p>Decoded percent-encoded SOCKS proxy credentials before authenticating
with the proxy server. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/3785">#3785</a>)</p>
</li>
<li>
<p>Fixed <code>HTTPResponse.drain_conn()</code> to discard unread
response data in 64 KiB chunks (same as the default <code>amt</code>
when doing <code>HTTPResponse.stream(...)</code>). (<a
href="https://redirect.github.com/urllib3/urllib3/issues/5019">#5019</a>)</p>
</li>
<li>
<p>Fixed <code>is_ipaddress()</code> to detect non-standard IPv4 forms
accepted by <code>socket.connect</code>, such as hex
(<code>0x7f000001</code>), octal (<code>0177.0.0.1</code>), and decimal
integers (<code>2130706433</code>), ensuring SSL certificate
verification uses the correct mode for these addresses. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/5029">#5029</a>)</p>
</li>
<li>
<p>Fixed <code>HTTPConnectionPool.urlopen</code> raising a misleading
<code>FullPoolError</code> instead of <code>ValueError</code> when
called with an invalid <code>timeout</code> argument on a pool created
with <code>block=True</code>. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/5059">#5059</a>)</p>
</li>
<li>
<p>Fixed port-zero handling to preserve explicit <code>:0</code> values
instead of substituting the default ports 80 or 443 in URL parsing, pool
selection, proxy configuration, <code>connection_from_url()</code>, and
HTTP/2 request authority. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/5071">#5071</a>,
<a
href="https://redirect.github.com/urllib3/urllib3/issues/5101">#5101</a>)</p>
</li>
<li>
<p>Fixed a bug where <code>PoolManager</code> passed the
<code>assert_hostname</code> and <code>assert_fingerprint</code>
parameters to HTTP connection pools. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/5077">#5077</a>)</p>
</li>
<li>
<p>Fixed <code>HTTPConnectionPool.urlopen()</code> and HTTP proxy
forwarding to strip URL fragments from absolute request targets before
sending requests. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/5079">#5079</a>)</p>
</li>
<li>
<p>Added safeguards to the proxy tunneling code to prevent potential
security issues when handling invalid characters in the proxy host and
HTTP headers. This change affects users of Python 3.10, Python 3.11, and
Python 3.12 when the standard library does not contain the fix; those on
newer Python versions should upgrade to 3.13.14+ or 3.14.5+ to get the
same security fixes. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/5091">#5091</a>)</p>
</li>
<li>
<p>Fixed <code>HTTPSConnection.connect()</code> overriding
<code>ProxyConfig.ssl_context</code>'s certificate policy and proxy
identity checks with the target connection's TLS settings when
forwarding through an HTTPS proxy.</p>
<p><code>HTTPSConnection</code> no longer applies target SNI,
assertions, or client credentials to forwarding proxy handshakes and
continues to use its <code>ssl_context</code> as a fallback when an
HTTPS proxy forwards an HTTP target. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/5093">#5093</a>)</p>
</li>
<li>
<p>Fixed URL parsing to more strictly enforce RFC 3986 host syntax,
rejecting invalid host input such as raw spaces and control characters,
malformed percent-encodings, and percent-encoded control characters in
HTTP(S) hosts and IPv6 zone identifiers, including proxy CONNECT tunnel
targets. Host normalization now also follows RFC 3986 normalization
rules for percent-encoded octets by decoding percent-encoded unreserved
characters and uppercasing the hexadecimal digits of retained
percent-encoded octets. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/5095">#5095</a>)</p>
</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/urllib3/urllib3/blob/main/CHANGES.rst">urllib3's
changelog</a>.</em></p>
<blockquote>
<h1>2.8.0 (2026-09-15)</h1>
<h2>Security</h2>
<p>Fixed the following security issues:</p>
<ul>
<li>The TLS configuration for HTTPS proxies could be ignored or
overridden.
(High severity, <code>GHSA-8988-9cw3-xx77
&lt;https://github.com/urllib3/urllib3/security/advisories/GHSA-8988-9cw3-xx77&gt;</code>__)</li>
<li><code>HTTPResponse.stream()</code> and <code>read_chunked()</code>
could buffer a chunk-size
line of unbounded length in memory. (High severity,
<code>GHSA-vxq7-64xx-v4gw
&lt;https://github.com/urllib3/urllib3/security/advisories/GHSA-vxq7-64xx-v4gw&gt;</code>__)</li>
<li>Chunked Deflate streaming could enter an infinite loop. (Medium
severity,
<code>GHSA-gh4c-6fx4-qh6g
&lt;https://github.com/urllib3/urllib3/security/advisories/GHSA-gh4c-6fx4-qh6g&gt;</code>__)</li>
</ul>
<p>.. caution::</p>
<pre><code>urllib3 2.8.0 fixes HTTPS proxy TLS configuration being
ignored or
overridden by destination settings. Configurations relying on that
behavior may require changes.
<p>Configure proxy CA certificates and client certificates in
<code>proxy_ssl_context</code>, and proxy identity checks with
<code>proxy_assert_hostname</code> or
<code>proxy_assert_fingerprint</code>.
Destination client certificates and identity overrides no longer
apply to HTTPS forwarding proxy connections.
</code></pre></p>
<h2>Deprecations &amp; Removals</h2>
<ul>
<li>Deprecated using an empty collection as the <code>Retry</code>
option
<code>allowed_methods</code> to retry any verb.
(<code>[apache#5044](urllib3/urllib3#5044)
&lt;https://github.com/urllib3/urllib3/issues/5044&gt;</code>__)</li>
</ul>
<h2>Features</h2>
<ul>
<li>Added <code>Url.auth_decoded</code> and
<code>Url.auth_decoded_joined</code> convenience
properties to the result of <code>parse_url()</code>.
(<code>[apache#4945](urllib3/urllib3#4945)
&lt;https://github.com/urllib3/urllib3/issues/4945&gt;</code>__)</li>
<li>Added <code>basic_auth_encoding</code> and
<code>proxy_basic_auth_encoding</code> parameters to
<code>urllib3.util.make_headers()</code>.
(<code>[apache#5092](urllib3/urllib3#5092)
&lt;https://github.com/urllib3/urllib3/issues/5092&gt;</code>__)</li>
</ul>
<h2>Bugfixes</h2>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/urllib3/urllib3/commit/b1d30ab61fe0db8f11092805e8c5ac43e091064a"><code>b1d30ab</code></a>
Release 2.8.0</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/9016d7e8afc68185496ef07f3c3a4a743d04922e"><code>9016d7e</code></a>
Skip <code>test_read_chunked_with_trailing_data_does_not_hang</code> for
brotlicffi (<a
href="https://redirect.github.com/urllib3/urllib3/issues/5258">#5258</a>)</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/9101f581a8b3659af23b6ff335ae77200ca33533"><code>9101f58</code></a>
Fix <code>nox -s docs</code> warning (<a
href="https://redirect.github.com/urllib3/urllib3/issues/5256">#5256</a>)</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/cd770b059b543be29298ea5c52afb0b1b090f5ed"><code>cd770b0</code></a>
Merge commit from fork</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/ea2ad7b21a80da3632f80016526a18864586077f"><code>ea2ad7b</code></a>
Merge commit from fork</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/0716e31534345dc1599ea95d903c79f276239bd8"><code>0716e31</code></a>
Fix loading unencrypted client keys with a password in pyOpenSSL (<a
href="https://redirect.github.com/urllib3/urllib3/issues/5255">#5255</a>)</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/43c68c8b43a9dcb44ed2cf4ec91384ca0d46b37d"><code>43c68c8</code></a>
Test pickling of <code>InvalidChunkLength</code> (<a
href="https://redirect.github.com/urllib3/urllib3/issues/5247">#5247</a>)</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/308b279b3fb28e7bee952e152ec5baeb5bfd0817"><code>308b279</code></a>
Share security policy between GitHub and Read the Docs (<a
href="https://redirect.github.com/urllib3/urllib3/issues/5253">#5253</a>)</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/53fa0731b27d4b71ab0755ea5b896422d005d706"><code>53fa073</code></a>
Add policy on duplicate pull requests (<a
href="https://redirect.github.com/urllib3/urllib3/issues/5252">#5252</a>)</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/5f2a6a843d0100d1351c3f94d58581ca98d17267"><code>5f2a6a8</code></a>
Assert on the ALPN extension in test_tunnel_sets_http_11_alpn (<a
href="https://redirect.github.com/urllib3/urllib3/issues/5232">#5232</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/urllib3/urllib3/compare/2.7.0...2.8.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=urllib3&package-manager=uv&previous-version=2.7.0&new-version=2.8.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/apache/datafusion/network/alerts).

</details>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Omega359 pushed a commit to Omega359/arrow-datafusion that referenced this pull request Oct 11, 2026
Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.7.0 to 2.8.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/urllib3/urllib3/releases">urllib3's
releases</a>.</em></p>
<blockquote>
<h2>2.8.0</h2>
<h2>🚀 urllib3 is fundraising for HTTP/2 support</h2>
<p><a
href="https://sethmlarson.dev/urllib3-is-fundraising-for-http2-support">urllib3
is raising ~$40,000 USD</a> to release HTTP/2 support and ensure
long-term sustainable maintenance of the project. If your company or
organization uses Python and would benefit from HTTP/2 support in
Requests, pip, cloud SDKs, and thousands of other projects <a
href="https://opencollective.com/urllib3">please consider contributing
financially</a> to ensure HTTP/2 support is developed sustainably and
maintained for the long-haul.</p>
<p>Thank you for your support.</p>
<h2>Security</h2>
<p>Fixed the following security issues:</p>
<ul>
<li>The TLS configuration for HTTPS proxies could be ignored or
overridden. (High severity, GHSA-8988-9cw3-xx77)</li>
<li><code>HTTPResponse.stream()</code> and <code>read_chunked()</code>
could buffer a chunk-size line of unbounded length in memory. (High
severity, GHSA-vxq7-64xx-v4gw)</li>
<li>Chunked Deflate streaming could enter an infinite loop. (Medium
severity, GHSA-gh4c-6fx4-qh6g)</li>
</ul>
<blockquote>
<p>[!IMPORTANT]
urllib3 2.8.0 fixes HTTPS proxy TLS configuration being ignored or
overridden by destination settings. Configurations relying on that
behavior may require changes.</p>
<p>Configure proxy CA certificates and client certificates in
<code>proxy_ssl_context</code>, and proxy identity checks with
<code>proxy_assert_hostname</code> or
<code>proxy_assert_fingerprint</code>. Destination client certificates
and identity overrides no longer apply to HTTPS forwarding proxy
connections.</p>
</blockquote>
<blockquote>
<p>[!NOTE]
CVE IDs had not yet been assigned to these advisories at the time of
release due to a backlog at GitHub's CNA.</p>
</blockquote>
<h2>Deprecations &amp; Removals</h2>
<ul>
<li>Deprecated using an empty collection as the <code>Retry</code>
option <code>allowed_methods</code> to retry any verb. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/5044">#5044</a>)</li>
</ul>
<h2>Features</h2>
<ul>
<li>Added <code>Url.auth_decoded</code> and
<code>Url.auth_decoded_joined</code> convenience properties to the
result of <code>parse_url()</code>. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/4945">#4945</a>)</li>
<li>Added <code>basic_auth_encoding</code> and
<code>proxy_basic_auth_encoding</code> parameters to
<code>urllib3.util.make_headers()</code>. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/5092">#5092</a>)</li>
</ul>
<h2>Bugfixes</h2>
<ul>
<li>
<p>Fixed response header handling to replace obsolete folded header
lines (<code>obs-fold</code>) with spaces in accordance with RFC 9112,
preventing raw CRLF sequences from appearing in header values such as
<code>Set-Cookie</code>. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/1362">#1362</a>)</p>
</li>
<li>
<p>Fixed usage of <code>proxy_ssl_context</code> with
<code>ProxyManager</code> when
<code>use_forwarding_for_https=True</code>. Passing
<code>ssl_context</code> instead of <code>proxy_ssl_context</code> for
HTTPS proxies in this configuration now emits a
<code>FutureWarning</code> and will raise an error in v3.0. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/2577">#2577</a>)</p>
</li>
<li>
<p>Changed behavior of the default <code>ConnectionPool.pool</code>
initialization. <code>LifoQueue</code> is now resolved from the
<code>queue</code> module after the <code>ConnectionPool</code> is
instantiated instead of using the default cached <code>QueueCls</code>
class property. This is done because sometimes the
<code>queue.LifoQueue</code> is monkey-patched late in the program, such
as by gevent. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/3289">#3289</a>)</p>
</li>
<li>
<p>Raised <code>UnrewindableBodyError</code> instead of
<code>ValueError</code> when retrying a request whose body had
<code>tell()</code> but not <code>seek()</code>. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/3779">#3779</a>)</p>
</li>
<li>
<p>Decoded percent-encoded SOCKS proxy credentials before authenticating
with the proxy server. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/3785">#3785</a>)</p>
</li>
<li>
<p>Fixed <code>HTTPResponse.drain_conn()</code> to discard unread
response data in 64 KiB chunks (same as the default <code>amt</code>
when doing <code>HTTPResponse.stream(...)</code>). (<a
href="https://redirect.github.com/urllib3/urllib3/issues/5019">#5019</a>)</p>
</li>
<li>
<p>Fixed <code>is_ipaddress()</code> to detect non-standard IPv4 forms
accepted by <code>socket.connect</code>, such as hex
(<code>0x7f000001</code>), octal (<code>0177.0.0.1</code>), and decimal
integers (<code>2130706433</code>), ensuring SSL certificate
verification uses the correct mode for these addresses. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/5029">#5029</a>)</p>
</li>
<li>
<p>Fixed <code>HTTPConnectionPool.urlopen</code> raising a misleading
<code>FullPoolError</code> instead of <code>ValueError</code> when
called with an invalid <code>timeout</code> argument on a pool created
with <code>block=True</code>. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/5059">#5059</a>)</p>
</li>
<li>
<p>Fixed port-zero handling to preserve explicit <code>:0</code> values
instead of substituting the default ports 80 or 443 in URL parsing, pool
selection, proxy configuration, <code>connection_from_url()</code>, and
HTTP/2 request authority. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/5071">#5071</a>,
<a
href="https://redirect.github.com/urllib3/urllib3/issues/5101">#5101</a>)</p>
</li>
<li>
<p>Fixed a bug where <code>PoolManager</code> passed the
<code>assert_hostname</code> and <code>assert_fingerprint</code>
parameters to HTTP connection pools. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/5077">#5077</a>)</p>
</li>
<li>
<p>Fixed <code>HTTPConnectionPool.urlopen()</code> and HTTP proxy
forwarding to strip URL fragments from absolute request targets before
sending requests. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/5079">#5079</a>)</p>
</li>
<li>
<p>Added safeguards to the proxy tunneling code to prevent potential
security issues when handling invalid characters in the proxy host and
HTTP headers. This change affects users of Python 3.10, Python 3.11, and
Python 3.12 when the standard library does not contain the fix; those on
newer Python versions should upgrade to 3.13.14+ or 3.14.5+ to get the
same security fixes. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/5091">#5091</a>)</p>
</li>
<li>
<p>Fixed <code>HTTPSConnection.connect()</code> overriding
<code>ProxyConfig.ssl_context</code>'s certificate policy and proxy
identity checks with the target connection's TLS settings when
forwarding through an HTTPS proxy.</p>
<p><code>HTTPSConnection</code> no longer applies target SNI,
assertions, or client credentials to forwarding proxy handshakes and
continues to use its <code>ssl_context</code> as a fallback when an
HTTPS proxy forwards an HTTP target. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/5093">#5093</a>)</p>
</li>
<li>
<p>Fixed URL parsing to more strictly enforce RFC 3986 host syntax,
rejecting invalid host input such as raw spaces and control characters,
malformed percent-encodings, and percent-encoded control characters in
HTTP(S) hosts and IPv6 zone identifiers, including proxy CONNECT tunnel
targets. Host normalization now also follows RFC 3986 normalization
rules for percent-encoded octets by decoding percent-encoded unreserved
characters and uppercasing the hexadecimal digits of retained
percent-encoded octets. (<a
href="https://redirect.github.com/urllib3/urllib3/issues/5095">#5095</a>)</p>
</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/urllib3/urllib3/blob/main/CHANGES.rst">urllib3's
changelog</a>.</em></p>
<blockquote>
<h1>2.8.0 (2026-09-15)</h1>
<h2>Security</h2>
<p>Fixed the following security issues:</p>
<ul>
<li>The TLS configuration for HTTPS proxies could be ignored or
overridden.
(High severity, <code>GHSA-8988-9cw3-xx77
&lt;https://github.com/urllib3/urllib3/security/advisories/GHSA-8988-9cw3-xx77&gt;</code>__)</li>
<li><code>HTTPResponse.stream()</code> and <code>read_chunked()</code>
could buffer a chunk-size
line of unbounded length in memory. (High severity,
<code>GHSA-vxq7-64xx-v4gw
&lt;https://github.com/urllib3/urllib3/security/advisories/GHSA-vxq7-64xx-v4gw&gt;</code>__)</li>
<li>Chunked Deflate streaming could enter an infinite loop. (Medium
severity,
<code>GHSA-gh4c-6fx4-qh6g
&lt;https://github.com/urllib3/urllib3/security/advisories/GHSA-gh4c-6fx4-qh6g&gt;</code>__)</li>
</ul>
<p>.. caution::</p>
<pre><code>urllib3 2.8.0 fixes HTTPS proxy TLS configuration being
ignored or
overridden by destination settings. Configurations relying on that
behavior may require changes.
<p>Configure proxy CA certificates and client certificates in
<code>proxy_ssl_context</code>, and proxy identity checks with
<code>proxy_assert_hostname</code> or
<code>proxy_assert_fingerprint</code>.
Destination client certificates and identity overrides no longer
apply to HTTPS forwarding proxy connections.
</code></pre></p>
<h2>Deprecations &amp; Removals</h2>
<ul>
<li>Deprecated using an empty collection as the <code>Retry</code>
option
<code>allowed_methods</code> to retry any verb.
(<code>[apache#5044](urllib3/urllib3#5044)
&lt;https://github.com/urllib3/urllib3/issues/5044&gt;</code>__)</li>
</ul>
<h2>Features</h2>
<ul>
<li>Added <code>Url.auth_decoded</code> and
<code>Url.auth_decoded_joined</code> convenience
properties to the result of <code>parse_url()</code>.
(<code>[apache#4945](urllib3/urllib3#4945)
&lt;https://github.com/urllib3/urllib3/issues/4945&gt;</code>__)</li>
<li>Added <code>basic_auth_encoding</code> and
<code>proxy_basic_auth_encoding</code> parameters to
<code>urllib3.util.make_headers()</code>.
(<code>[apache#5092](urllib3/urllib3#5092)
&lt;https://github.com/urllib3/urllib3/issues/5092&gt;</code>__)</li>
</ul>
<h2>Bugfixes</h2>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/urllib3/urllib3/commit/b1d30ab61fe0db8f11092805e8c5ac43e091064a"><code>b1d30ab</code></a>
Release 2.8.0</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/9016d7e8afc68185496ef07f3c3a4a743d04922e"><code>9016d7e</code></a>
Skip <code>test_read_chunked_with_trailing_data_does_not_hang</code> for
brotlicffi (<a
href="https://redirect.github.com/urllib3/urllib3/issues/5258">#5258</a>)</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/9101f581a8b3659af23b6ff335ae77200ca33533"><code>9101f58</code></a>
Fix <code>nox -s docs</code> warning (<a
href="https://redirect.github.com/urllib3/urllib3/issues/5256">#5256</a>)</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/cd770b059b543be29298ea5c52afb0b1b090f5ed"><code>cd770b0</code></a>
Merge commit from fork</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/ea2ad7b21a80da3632f80016526a18864586077f"><code>ea2ad7b</code></a>
Merge commit from fork</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/0716e31534345dc1599ea95d903c79f276239bd8"><code>0716e31</code></a>
Fix loading unencrypted client keys with a password in pyOpenSSL (<a
href="https://redirect.github.com/urllib3/urllib3/issues/5255">#5255</a>)</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/43c68c8b43a9dcb44ed2cf4ec91384ca0d46b37d"><code>43c68c8</code></a>
Test pickling of <code>InvalidChunkLength</code> (<a
href="https://redirect.github.com/urllib3/urllib3/issues/5247">#5247</a>)</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/308b279b3fb28e7bee952e152ec5baeb5bfd0817"><code>308b279</code></a>
Share security policy between GitHub and Read the Docs (<a
href="https://redirect.github.com/urllib3/urllib3/issues/5253">#5253</a>)</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/53fa0731b27d4b71ab0755ea5b896422d005d706"><code>53fa073</code></a>
Add policy on duplicate pull requests (<a
href="https://redirect.github.com/urllib3/urllib3/issues/5252">#5252</a>)</li>
<li><a
href="https://github.com/urllib3/urllib3/commit/5f2a6a843d0100d1351c3f94d58581ca98d17267"><code>5f2a6a8</code></a>
Assert on the ALPN extension in test_tunnel_sets_http_11_alpn (<a
href="https://redirect.github.com/urllib3/urllib3/issues/5232">#5232</a>)</li>
<li>Additional commits viewable in <a
href="https://github.com/urllib3/urllib3/compare/2.7.0...2.8.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=urllib3&package-manager=uv&previous-version=2.7.0&new-version=2.8.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/apache/datafusion/network/alerts).

</details>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

logical-expr Logical plan and expressions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants