Add support for switching the zlib implementation by TimMenninger · Pull Request #10700 · aio-libs/aiohttp

TimMenninger · 2025-04-06T13:49:10Z

What do these changes do?

These changes allow a user to arbitrarily set the zlib backend by calling aiohttp.set_zlib_backend() on the zlib module of their choice. For example:

import aiohttp
import zlib_ng.zlib_ng as zlib_ng

aiohttp.set_zlib_backend(zlib_ng)

Are there changes in behavior for the user?

The only changes for the user are opt-in.

Is it a substantial burden for the maintainers to support this?

The only burden that could arise is if for some reason zlib_ng.zlib_ng or isal.isal_zlib change their interface to differ from zlib.

Related issue number

Fixes #9798

Checklist

I think the code is well written
Unit tests for the changes exist
Documentation reflects the changes
If you provide code modification, please add yourself to CONTRIBUTORS.txt
- The format is <Name> <Surname>.
- Please keep alphabetical order, the file is sorted by names.
Add a new news fragment into the CHANGES/ folder
- name it <issue_or_pr_num>.<type>.rst (e.g. 588.bugfix.rst)
- if you don't have an issue number, change it to the pull request
  number after creating the PR
  - .bugfix: A bug fix for something the maintainers deemed an
    improper undesired behavior that got corrected to match
    pre-agreed expectations.
  - .feature: A new behavior, public APIs. That sort of stuff.
  - .deprecation: A declaration of future API removals and breaking
    changes in behavior.
  - .breaking: When something public is removed in a breaking way.
    Could be deprecated in an earlier release.
  - .doc: Notable updates to the documentation structure or build
    process.
  - .packaging: Notes for downstreams about unobvious side effects
    and tooling. Changes in the test invocation considerations and
    runtime assumptions.
  - .contrib: Stuff that affects the contributor experience. e.g.
    Running tests, building the docs, setting up the development
    environment.
  - .misc: Changes that are hard to assign to any of the above
    categories.
- Make sure to use full sentences with correct case and punctuation,
  for example:
```
Fixed issue with non-ascii contents in doctest text files
-- by :user:`contributor-gh-handle`.
```
  Use the past tense or the present tense a non-imperative mood,
  referring to what's changed compared to the last released version
  of this project.

codspeed-hq · 2025-04-06T13:59:54Z

CodSpeed Performance Report

Merging #10700 will not alter performance

_{Comparing TimMenninger:switch-zlib-implementation (71f3450) with master (6db713e)}

Summary

✅ 51 untouched benchmarks

tests/test_compression_utils.py

aiohttp/compression_utils.py

aiohttp/abc.py

aiohttp/_websocket/reader_py.py

codecov · 2025-04-08T17:39:33Z

Codecov Report

Attention: Patch coverage is 99.69605% with 1 line in your changes missing coverage. Please review.

Project coverage is 98.72%. Comparing base (6db713e) to head (71f3450).
Report is 533 commits behind head on master.

✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
tests/test_http_writer.py	99.42%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##           master   #10700    +/-   ##
========================================
  Coverage   98.71%   98.72%            
========================================
  Files         125      125            
  Lines       37445    37718   +273     
  Branches     2075     2082     +7     
========================================
+ Hits        36963    37236   +273     
+ Misses        335      334     -1     
- Partials      147      148     +1

Flag	Coverage Δ
CI-GHA	`98.60% <99.69%> (+0.01%)`	⬆️
OS-Linux	`98.27% <99.69%> (+0.01%)`	⬆️
OS-Windows	`96.22% <99.39%> (+0.03%)`	⬆️
OS-macOS	`97.38% <98.78%> (+<0.01%)`	⬆️
Py-3.10.11	`97.29% <99.39%> (+0.01%)`	⬆️
Py-3.10.16	`97.83% <99.69%> (+0.01%)`	⬆️
Py-3.11.11	`97.92% <99.69%> (+0.01%)`	⬆️
Py-3.11.9	`97.38% <99.39%> (+0.02%)`	⬆️
Py-3.12.9	`98.37% <99.69%> (+0.01%)`	⬆️
Py-3.13.2	`98.36% <99.69%> (+0.01%)`	⬆️
Py-3.9.13	`97.16% <99.39%> (+0.02%)`	⬆️
Py-3.9.21	`97.70% <99.69%> (+0.02%)`	⬆️
Py-pypy7.3.16	`85.56% <96.04%> (-8.97%)`	⬇️
VM-macos	`97.38% <98.78%> (+<0.01%)`	⬆️
VM-ubuntu	`98.27% <99.69%> (+0.01%)`	⬆️
VM-windows	`96.22% <99.39%> (+0.03%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

bdraco · 2025-04-10T22:13:25Z

Looks good. I think its nearly ready to merge. Thanks for all your work on this @TimMenninger

I'd like Sam to take another look as well before moving it forward.

aiohttp/compression_utils.py

docs/client_reference.rst

tests/test_http_writer.py

aiohttp/compression_utils.py

bdraco

Looks great. One small nit above.

I'll do some functional testing later today and approve if all goes well

aiohttp/compression_utils.py

Allow a user to call aiohttp.set_zlib_backend() with a zlib-implementing module. E.g. import aiohttp import zlib_ng.zlib_ng aiohttp.set_zlib_backend(zlib_ng.zlib_ng) Fixes aio-libs#9798

will need to be rebased after #10700 is merged

bdraco · 2025-04-11T19:34:00Z

Added some benchmarks in #10721 Interesting that isal is much faster for large messages and a bit slower for compressing small messages due to more memcpy. I wonder if that can be improved in the library.

bdraco

Did some functional testing with this. Everything worked as expected. Did a partial backport to 3.11 and ran it with Home Assistant as well

bdraco · 2025-04-11T19:35:21Z

Thanks @TimMenninger

This is a very nice addition.

bdraco · 2025-04-12T21:07:14Z

All good running overnight.

The backport on this one is going to be a bit more work than usual because of the conflicts.

patchback · 2025-04-12T21:07:24Z

Backport to 3.12: 💔 cherry-picking failed — conflicts found

❌ Failed to cleanly apply ceeca6a on top of patchback/backports/3.12/ceeca6a9b019b98d1dc3d5d1a622ba66768938d2/pr-10700

Backporting merged PR #10700 into master

Ensure you have a local repo clone of your fork. Unless you cloned it
from the upstream, this would be your origin remote.
Make sure you have an upstream repo added as a remote too. In these
instructions you'll refer to it by the name upstream. If you don't
have it, here's how you can add it:
```
$ git remote add upstream https://github.com/aio-libs/aiohttp.git
```

Ensure you have the latest copy of upstream and prepare a branch
that will hold the backported code:

$ git fetch upstream
$ git checkout -b patchback/backports/3.12/ceeca6a9b019b98d1dc3d5d1a622ba66768938d2/pr-10700 upstream/3.12

Now, cherry-pick PR Add support for switching the zlib implementation #10700 contents into that branch:
```
$ git cherry-pick -x ceeca6a9b019b98d1dc3d5d1a622ba66768938d2
```
If it'll yell at you with something like fatal: Commit ceeca6a9b019b98d1dc3d5d1a622ba66768938d2 is a merge but no -m option was given., add -m 1 as follows instead:
```
$ git cherry-pick -m1 -x ceeca6a9b019b98d1dc3d5d1a622ba66768938d2
```
At this point, you'll probably encounter some merge conflicts. You must
resolve them in to preserve the patch from PR Add support for switching the zlib implementation #10700 as close to the
original as possible.

Push this branch to your fork on GitHub:

$ git push origin patchback/backports/3.12/ceeca6a9b019b98d1dc3d5d1a622ba66768938d2/pr-10700

Create a PR, ensure that the CI is green. If it's not — update it so that
the tests and any other checks pass. This is it!
Now relax and wait for the maintainers to process your pull request
when they have some cycles to do reviews. Don't worry — they'll tell you if
any improvements are necessary when the time comes!

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

TimMenninger · 2025-04-13T01:02:06Z

All good running overnight.

The backport on this one is going to be a bit more work than usual because of the conflicts.

Okay I'll take a look probably in the morning if not Monday morning.

…o-libs#10700, aio-libs#9798) Committed with conflicts resolved in subsequent commit: Unmerged paths: (use "git add <file>..." to mark resolution) both modified: aiohttp/__init__.py both modified: aiohttp/web_response.py both modified: tests/test_client_functional.py both modified: tests/test_client_request.py both modified: tests/test_client_ws_functional.py both modified: tests/test_http_writer.py both modified: tests/test_multipart.py both modified: tests/test_web_functional.py both modified: tests/test_web_response.py both modified: tests/test_web_sendfile_functional.py both modified: tests/test_websocket_parser.py

TimMenninger · 2025-04-14T15:18:10Z

All good running overnight.
The backport on this one is going to be a bit more work than usual because of the conflicts.

Okay I'll take a look probably in the morning if not Monday morning.

#10723

… implementation (#10723)

TimMenninger force-pushed the switch-zlib-implementation branch from 0eb2ad3 to af8b053 Compare April 6, 2025 13:53

psf-chronographer bot added the bot:chronographer:provided There is a change note present in this PR label Apr 6, 2025

TimMenninger force-pushed the switch-zlib-implementation branch 2 times, most recently from e77a5f6 to a1caebd Compare April 6, 2025 17:20

bdraco added the backport-3.12 label Apr 6, 2025

TimMenninger force-pushed the switch-zlib-implementation branch 2 times, most recently from 49e6440 to c4bdbfc Compare April 6, 2025 18:19

bdraco reviewed Apr 6, 2025

View reviewed changes

tests/test_compression_utils.py Outdated Show resolved Hide resolved

bdraco reviewed Apr 6, 2025

View reviewed changes

aiohttp/compression_utils.py Outdated Show resolved Hide resolved

TimMenninger force-pushed the switch-zlib-implementation branch 4 times, most recently from 92374c7 to 906ec56 Compare April 6, 2025 19:42

Dreamsorcerer reviewed Apr 7, 2025

View reviewed changes

aiohttp/abc.py Outdated Show resolved Hide resolved

TimMenninger force-pushed the switch-zlib-implementation branch from 906ec56 to fbcabee Compare April 8, 2025 15:24

TimMenninger commented Apr 8, 2025

View reviewed changes

aiohttp/_websocket/reader_py.py Outdated Show resolved Hide resolved

TimMenninger force-pushed the switch-zlib-implementation branch 3 times, most recently from fc0364f to 431f486 Compare April 8, 2025 17:29

TimMenninger force-pushed the switch-zlib-implementation branch 6 times, most recently from d609ea6 to ada8722 Compare April 9, 2025 21:58

TimMenninger marked this pull request as ready for review April 9, 2025 22:00

TimMenninger requested review from asvetlov and webknjaz as code owners April 9, 2025 22:00

Dreamsorcerer reviewed Apr 10, 2025

View reviewed changes

aiohttp/compression_utils.py Outdated Show resolved Hide resolved

Dreamsorcerer reviewed Apr 10, 2025

View reviewed changes

docs/client_reference.rst Outdated Show resolved Hide resolved

Dreamsorcerer reviewed Apr 10, 2025

View reviewed changes

tests/test_http_writer.py Outdated Show resolved Hide resolved

TimMenninger force-pushed the switch-zlib-implementation branch from dfc6a14 to 761b4e8 Compare April 11, 2025 15:50

github-advanced-security bot found potential problems Apr 11, 2025

View reviewed changes

aiohttp/compression_utils.py Fixed Show fixed Hide fixed

TimMenninger force-pushed the switch-zlib-implementation branch from 761b4e8 to 6a8e1f3 Compare April 11, 2025 15:56

github-advanced-security bot found potential problems Apr 11, 2025

View reviewed changes

aiohttp/compression_utils.py Fixed Show fixed Hide fixed

TimMenninger force-pushed the switch-zlib-implementation branch 2 times, most recently from 622c87a to e3c2ef4 Compare April 11, 2025 16:56

Dreamsorcerer approved these changes Apr 11, 2025

View reviewed changes

bdraco reviewed Apr 11, 2025

View reviewed changes

aiohttp/compression_utils.py Outdated Show resolved Hide resolved

Dreamsorcerer mentioned this pull request Apr 11, 2025

3.8.4: pytest is failing in 9 units #7255

Closed

1 task

TimMenninger force-pushed the switch-zlib-implementation branch from e3c2ef4 to 6f43600 Compare April 11, 2025 18:34

Add support for switching the zlib implementation

71f3450

Allow a user to call aiohttp.set_zlib_backend() with a zlib-implementing module. E.g. import aiohttp import zlib_ng.zlib_ng aiohttp.set_zlib_backend(zlib_ng.zlib_ng) Fixes aio-libs#9798

TimMenninger force-pushed the switch-zlib-implementation branch from 6f43600 to 71f3450 Compare April 11, 2025 18:36

bdraco added a commit that referenced this pull request Apr 11, 2025

Update zlib benchmarks for multiple zlib backends

1b75ee1

will need to be rebased after #10700 is merged

bdraco mentioned this pull request Apr 11, 2025

Update zlib benchmarks for multiple zlib backends #10721

Merged

bdraco approved these changes Apr 11, 2025

View reviewed changes

bdraco merged commit ceeca6a into aio-libs:master Apr 12, 2025
40 checks passed

TimMenninger added a commit to TimMenninger/aiohttp that referenced this pull request Apr 14, 2025

Fix conflicts after 3.12 backport of zlib backend (aio-libs#10700)

e5c554c

TimMenninger mentioned this pull request Apr 14, 2025

[PR #10700/ceeca6a backport][3.12] Add support for switching the zlib implementation #10723

Merged

5 tasks

bdraco pushed a commit that referenced this pull request Apr 14, 2025

[PR #10700/ceeca6a backport][3.12] Add support for switching the zlib…

5268479

… implementation (#10723)

Uh oh!

Conversation

TimMenninger commented Apr 6, 2025

What do these changes do?

Are there changes in behavior for the user?

Is it a substantial burden for the maintainers to support this?

Related issue number

Checklist

Uh oh!

codspeed-hq bot commented Apr 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CodSpeed Performance Report

Merging #10700 will not alter performance

Summary

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Apr 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

bdraco commented Apr 10, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bdraco left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bdraco commented Apr 11, 2025

Uh oh!

bdraco left a comment

Choose a reason for hiding this comment

Uh oh!

bdraco commented Apr 11, 2025

Uh oh!

bdraco commented Apr 12, 2025

Uh oh!

Uh oh!

patchback bot commented Apr 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Backport to 3.12: 💔 cherry-picking failed — conflicts found

Backporting merged PR #10700 into master

Uh oh!

TimMenninger commented Apr 13, 2025

Uh oh!

TimMenninger commented Apr 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codspeed-hq bot commented Apr 6, 2025 •

edited

Loading

codecov bot commented Apr 8, 2025 •

edited

Loading

patchback bot commented Apr 12, 2025 •

edited

Loading