Sadly our builds are broken at HEAD, even though the last commit in main passed:

https://github.com/googleapis/storage-testbench/actions/runs/6356186515

I have asked help from people with more Python expertise than me.

/FYI: @pitrou

IIUC, the error has nothing to do with the setup.py changes? It seems about an error message relating to hashing or base64 encoding:
https://github.com/googleapis/storage-testbench/actions/runs/6407286976/job/17393753132?pr=545#step:5:81

It seems there is the same problem on another PR: https://github.com/googleapis/storage-testbench/actions/runs/6390316476, so perhaps some upstream dependency changed slightly and broke things?

IIUC, the error has nothing to do with the setup.py changes?

I agree, but I am paranoid try to be disciplined.

so perhaps some upstream dependency changed slightly and broke things?

That seems the most likely explanation. Though this code only depends on core Python library modules (base64, hashlib). They can break, but break across multiple versions of Python? Both in GHA and on my workstation? Could be, but it is more likely that some other module monkey patched something (incorrectly), or that I do not know how Python dependency management works. My vote is on the latter.

Ok, here is a successful build:
https://github.com/googleapis/storage-testbench/actions/runs/6356186515/job/17265417749
Here is the same build failing:
https://github.com/googleapis/storage-testbench/actions/runs/6390316476/job/17343327057

I've tried to find the package differences between them:

(I would suggest adding a call to pip list at the end of your "install dependencies" build step to make this easier :-))

Given how the error looks like, I would suggest looking at the Werkzeug update for a potential culprit, and perhaps urllib3.

@pitrou wow! thanks for the detailed analysis, I am sorry you had to do this. I ran the test with the "success" versions and this may be "a clue":

  /usr/local/google/home/coryan/storage-testbench/.venv/lib/python3.11/site-packages/werkzeug/datastructures/headers.py:297: DeprecationWarning: Passing bytes as a header value is deprecated and will not be supported in Werkzeug 3.0.
    _value = _str_header_value(_value)

I am going to send separate PRs to fix the problem, and to pin these indirect dependencies. The latter should help us detect breakages when they are updated.

Codecov Report

All modified lines are covered by tests ✅

Comparison is base (cc471a3) 98.70% compared to head (08a10bb) 98.70%.

@@           Coverage Diff           @@
##             main     #545   +/-   ##
=======================================
  Coverage   98.70%   98.70%           
=======================================
  Files          50       50           
  Lines        8007     8007           
=======================================
  Hits         7903     7903           
  Misses        104      104           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@jorisvandenbossche @raulcd This should make it easier to dissociate the Python version under test from the one used to run the GCS testbench.

FWIW, I just created a "release" (just a tag really). That may make it easier to use these changes in your CI system.