Skip to content

Commit 709f260

Browse files
authored
Trim the test run of the emulated wheel jobs (#1762)
<!-- Thank you for your contribution! --> ## What do these changes do? The third `v7.1.0` tag run passed every job except the manylinux riscv64 wheel build, which hit its 120 minute timeout on the last of its eight wheels; all wheels it built passed their tests. The job took 72 minutes for 7.0.0: the suite grew from about 2650 to about 4090 tests since then, and under QEMU each wheel's test run went from about 3.5 to about 8 minutes. Measured under QEMU on the riscv64 cp314 wheel, two thirds of the time in the slow tests was the pure-Python parametrization, mostly large-table tests and the benchmark tests, and most of the rest was a handful of threaded tests. The pure-Python backend gives the same result whichever wheel is under test, and native jobs, 32-bit i686 included, keep running all of it. This adds four markers: `pure_python` on the pure-Python parametrization (mirroring `c_extension`), `gil_build` on every test of a GIL build, `expensive` on 11 large-table tests and `test_multidict_benchmarks.py`, and `threaded` on the 9 slowest threaded tests. The odd-arch wheel jobs that run under QEMU add `not (pure_python and expensive) and not (gil_build and threaded)` to the test command. FT wheels keep every threaded test, GIL wheels keep the other threaded tests (48 of 61 test items), and native jobs, aarch64 included, run everything as before. The changelog entry goes into the unpublished 7.1.0 section, since the tag gets moved to this merge commit. ## Are there changes in behavior for the user? No. ## Is it a substantial burden for the maintainers to support this? No; a new test that is slow under emulation needs one of the two markers. ## Related issue number Follow-up to #1760. Failed run: https://github.com/aio-libs/multidict/actions/runs/37838988742 ## Checklist - [x] I think the code is well written - [ ] Unit tests for the changes exist N/A, test selection change; verified as described below - [x] Documentation reflects the changes - [ ] If you provide code modification, please add yourself to `CONTRIBUTORS.txt` N/A - [ ] Add a new news fragment into the `CHANGES/` folder N/A, the entry is added to the unreleased 7.1.0 section of `CHANGES.rst` directly - [x] `make doc-spelling` passes and any new technical words are added to `docs/spelling_wordlist.txt` Drafted with Claude Code (Claude Opus 5.5); reviewed by @asvetlov. <details> <summary>Agent run details (optional, for reviewers)</summary> Under QEMU (local docker, `manylinux_2_39_riscv64`, cp314 wheel), the cibuildwheel test command took 427s (4069 passed). With only the `pure_python and expensive` part, 307s (3930 passed). The run with the `gil_build and threaded` part added is still going; numbers will be added here. Collection, CPython 3.14.7: the default selection is unchanged at 4088 items on GIL and FT. The emulated selection keeps 3936 on GIL and 3949 on FT; it deselects only pure-Python items and, on GIL, the 13 `threaded` items. No C-extension item is deselected on FT. Native: C extension 4250 passed (3.14.7) and 4235 passed (3.14.7t); pure-Python leg from a `MULTIDICT_NO_EXTENSIONS=1` install 1690 passed. pre-commit (yamllint included) and `make doc-spelling` pass. </details>
1 parent dbc3aba commit 709f260

10 files changed

Lines changed: 65 additions & 2 deletions

‎.github/workflows/ci-cd.yml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1070,13 +1070,26 @@ jobs:
10701070
# CIBW_ARCHS_LINUX: Build emulated architectures if QEMU, else "auto"
10711071
# CIBW_BUILD_FRONTEND: override pyproject.toml's `build[uv]` because
10721072
# the pypa odd-arch containers do not ship `uv` preinstalled
1073+
# CIBW_TEST_COMMAND: under QEMU, skip the pure-Python runs of the tests
1074+
# marked `expensive`, and the `threaded` ones on GIL builds; they took
1075+
# manylinux riscv64 past its timeout
10731076
environment-variables: |-
10741077
CIBW_ARCHS_LINUX=${{ matrix.qemu }}
10751078
10761079
CIBW_ARCHS_MACOS=x86_64 arm64 universal2
10771080
10781081
CIBW_BUILD_FRONTEND=build
10791082
1083+
CIBW_TEST_COMMAND=pytest --no-cov {project}/tests -m "${{
1084+
!matrix.native-arch
1085+
&& format(
1086+
'{0} and {1} and ',
1087+
'not (pure_python and expensive)',
1088+
'not (gil_build and threaded)'
1089+
)
1090+
|| ''
1091+
}}not leaks and not hypothesis and not capi"
1092+
10801093
CIBW_SKIP=${{
10811094
(matrix.tag == 'musllinux')
10821095
&& '*-manylinux_*'

‎CHANGES.rst‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -372,6 +372,15 @@ Packaging updates and notes for downstreams
372372
Contributor-facing changes
373373
--------------------------
374374

375+
- Stopped the wheel jobs that run under QEMU from running the pure-Python
376+
half of the tests marked ``expensive``, and the tests marked ``threaded``
377+
on GIL builds, which had grown enough since 7.0.0 to take the manylinux
378+
riscv64 job past its two-hour timeout; native jobs still run them
379+
-- by :user:`asvetlov`.
380+
381+
*Related issues and pull requests on GitHub:*
382+
:issue:`1762`.
383+
375384
- Added ``tools/check_inlining.py`` and an ``Inlining`` CI job, which
376385
fail when GCC stops inlining a helper that a hot path depends on, or
377386
makes an out-of-line copy of an inline function from the CPython headers

‎pytest.ini‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,8 @@ markers =
6464
leaks: memory leak tests
6565
hypothesis: Hypothesis-based property/fuzz tests, run in their own CI job
6666
capi: C API tests
67+
expensive: slow under emulation; the emulated wheel jobs skip their pure-Python runs
68+
threaded: concurrency tests, slow under emulation; the emulated wheel jobs skip them on GIL builds
6769
benchmark_shard_2: benchmarks the second CodSpeed CI job runs, to balance the two
6870

6971
minversion = 3.8.2

‎tests/conftest.py‎

Lines changed: 19 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,8 @@
2020
)
2121

2222
C_EXT_MARK = pytest.mark.c_extension
23+
PURE_PYTHON_MARK = pytest.mark.pure_python
24+
GIL_BUILD_MARK = pytest.mark.gil_build
2325

2426
FT_BUILD = bool(sysconfig.get_config_var("Py_GIL_DISABLED"))
2527

@@ -82,7 +84,10 @@ def __str__(self) -> str:
8284
MultidictImplementation(is_pure_python=False),
8385
marks=C_EXT_MARK,
8486
),
85-
MultidictImplementation(is_pure_python=True),
87+
pytest.param(
88+
MultidictImplementation(is_pure_python=True),
89+
marks=PURE_PYTHON_MARK,
90+
),
8691
),
8792
ids=str,
8893
)
@@ -205,6 +210,10 @@ def pytest_collection_modifyitems(
205210
items: list[pytest.Item],
206211
) -> None:
207212
"""Deselect tests against C-extensions when requested via CLI."""
213+
if not FT_BUILD:
214+
for item in items:
215+
item.add_marker(GIL_BUILD_MARK)
216+
208217
test_c_extensions = config.getoption("--c-extensions") is True
209218

210219
if test_c_extensions:
@@ -224,11 +233,19 @@ def pytest_collection_modifyitems(
224233

225234

226235
def pytest_configure(config: pytest.Config) -> None:
227-
"""Declare the C-extension marker in config."""
236+
"""Declare the implementation markers in config."""
228237
config.addinivalue_line(
229238
"markers",
230239
f"{C_EXT_MARK.name}: tests running against the C-extension implementation.",
231240
)
241+
config.addinivalue_line(
242+
"markers",
243+
f"{PURE_PYTHON_MARK.name}: tests running against the pure-Python implementation.",
244+
)
245+
config.addinivalue_line(
246+
"markers",
247+
f"{GIL_BUILD_MARK.name}: every test, when running on a GIL build.",
248+
)
232249

233250

234251
@pytest.fixture(autouse=True)

‎tests/test_free_threading.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,7 @@ def reader(target: MutableMultiMapping[str]) -> None:
9191

9292

9393
@pytest.mark.c_extension
94+
@pytest.mark.threaded
9495
def test_race_condition_extend_vs_source_mutation(
9596
any_multidict_class: type[CIMultiDict[str] | MultiDict[str]],
9697
) -> None:
@@ -271,6 +272,7 @@ def puller(idx: int) -> None:
271272

272273
@pytest.mark.c_extension
273274
@pytest.mark.parametrize("cls_name", ["MultiDict", "CIMultiDict"])
275+
@pytest.mark.threaded
274276
def test_reinit_finalizer_vs_lock_free_readers(
275277
cls_name: str, monkeypatch: pytest.MonkeyPatch
276278
) -> None:

‎tests/test_multidict.py‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1973,6 +1973,7 @@ def worker(n: int) -> None:
19731973
assert len(d2) == 200
19741974

19751975

1976+
@pytest.mark.threaded
19761977
def test_clear_thread_safety() -> None:
19771978
"""Concurrent clear() alongside extend() must not crash or corrupt state.
19781979
@@ -2034,6 +2035,7 @@ def worker(n: int) -> None:
20342035

20352036

20362037
@pytest.mark.c_extension
2038+
@pytest.mark.threaded
20372039
def test_clear_finalizer_thread_safety() -> None:
20382040
"""Concurrent clear() alongside update() must not crash or corrupt state.
20392041
@@ -2215,6 +2217,7 @@ def worker(n: int) -> None:
22152217

22162218

22172219
@pytest.mark.c_extension
2220+
@pytest.mark.threaded
22182221
def test_view_set_ops_thread_safety() -> None:
22192222
"""Concurrent items()/keys() set-algebra (&, |, -, ^, in, isdisjoint())
22202223
alongside mutation must not crash.
@@ -2631,6 +2634,7 @@ def popall_worker(n: int) -> None:
26312634

26322635

26332636
@pytest.mark.c_extension
2637+
@pytest.mark.threaded
26342638
def test_to_dict_vs_lock_free_reads_thread_safety() -> None:
26352639
"""Concurrent to_dict() alongside lock-free contains()/get() on the
26362640
same, never-deleted keys must not crash and must never observe a
@@ -2713,6 +2717,7 @@ def worker(_n: int) -> None:
27132717

27142718

27152719
@pytest.mark.c_extension
2720+
@pytest.mark.threaded
27162721
def test_reader_exit_drains_retired_thread_safety() -> None:
27172722
"""A retired hash table must eventually be freed by reader traffic
27182723
alone, with no further mutation. Regression test for
@@ -3891,6 +3896,7 @@ def shrink() -> None:
38913896

38923897

38933898
@pytest.mark.c_extension
3899+
@pytest.mark.threaded
38943900
def test_update_from_pair_list_shrunk_by_another_thread() -> None:
38953901
"""A ``[key, value]`` item that shrinks between its length check and the
38963902
reads must not be indexed past its end; the update either succeeds,
@@ -3926,6 +3932,7 @@ def shrink() -> None:
39263932

39273933

39283934
@pytest.mark.c_extension
3935+
@pytest.mark.threaded
39293936
def test_items_contains_list_shrunk_by_another_thread() -> None:
39303937
"""A probe list that shrinks between the length check and the reads must
39313938
not be indexed past its end; the check either answers or reports the

‎tests/test_multidict_benchmarks.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,8 @@
1111
istr,
1212
)
1313

14+
pytestmark = pytest.mark.expensive
15+
1416
# The CodSpeed CI job is split in two by the benchmark_shard_2 mark, which
1517
# goes on the bulk update()/extend()/merge() and removal benchmarks here and
1618
# on the whole of test_views_benchmarks.py. Rebalance when adding a heavy one.

‎tests/test_mutable_multidict.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -533,6 +533,7 @@ def test_values_repr_with_dummy(
533533
cls = d.values().__class__.__name__
534534
assert repr(d.values()) == f"<{cls}(1, 3)>"
535535

536+
@pytest.mark.expensive
536537
def test_huge_md(
537538
self,
538539
case_sensitive_multidict_class: type[MultiDict[int]],
@@ -541,6 +542,7 @@ def test_huge_md(
541542
d = case_sensitive_multidict_class((str(i), i) for i in range(size))
542543
assert d[str(size // 2)] == size // 2
543544

545+
@pytest.mark.expensive
544546
def test_update_resizes_mid_update_on_capped_huge_md(
545547
self,
546548
case_sensitive_multidict_class: type[MultiDict[int]],
@@ -1124,6 +1126,7 @@ def test_add_many_duplicate_keys(
11241126
assert md.getall("a") == [str(i) for i in range(1000)]
11251127

11261128

1129+
@pytest.mark.expensive
11271130
def test_add_many_duplicate_keys_after_delete(
11281131
any_multidict_class: type[MultiDict[str]],
11291132
) -> None:
@@ -1138,6 +1141,7 @@ def test_add_many_duplicate_keys_after_delete(
11381141
assert md.getall("a") == [str(-i) for i in range(1300)]
11391142

11401143

1144+
@pytest.mark.expensive
11411145
def test_getall_in_insertion_order_after_mixed_mutations(
11421146
any_multidict_class: type[MultiDict[str]],
11431147
) -> None:
@@ -1161,6 +1165,7 @@ def test_getall_in_insertion_order_after_mixed_mutations(
11611165
assert md.items() & {(key, v) for v in expected} == {(key, v) for v in expected}
11621166

11631167

1168+
@pytest.mark.expensive
11641169
def test_create_with_many_duplicate_keys(
11651170
any_multidict_class: type[MultiDict[str]],
11661171
) -> None:

‎tests/test_to_dict.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,7 @@ def test_to_dict_order_preservation(
7373
assert d.to_dict()["x"] == ["3", "1", "2"]
7474

7575

76+
@pytest.mark.expensive
7677
def test_to_dict_large_data(
7778
any_multidict_class: type[MultiDict[str]] | type[CIMultiDict[str]],
7879
) -> None:

‎tests/test_update_marks.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,7 @@ def _big_pairs() -> list[tuple[str, int]]:
5353
return pairs
5454

5555

56+
@pytest.mark.expensive
5657
def test_getall_large_table(any_multidict_class: type[MultiDict[int]]) -> None:
5758
pairs = _big_pairs()
5859
md = any_multidict_class(pairs)
@@ -64,6 +65,7 @@ def test_getall_large_table(any_multidict_class: type[MultiDict[int]]) -> None:
6465
assert ("dup", 1) not in md.items()
6566

6667

68+
@pytest.mark.expensive
6769
def test_to_dict_large_table(any_multidict_class: type[MultiDict[int]]) -> None:
6870
pairs = _big_pairs()
6971
md = any_multidict_class(pairs)
@@ -73,6 +75,7 @@ def test_to_dict_large_table(any_multidict_class: type[MultiDict[int]]) -> None:
7375
assert d["k39999"] == [39999]
7476

7577

78+
@pytest.mark.expensive
7679
def test_setitem_large_table(any_multidict_class: type[MultiDict[int]]) -> None:
7780
md = any_multidict_class(_big_pairs())
7881
md["dup"] = 7
@@ -81,6 +84,7 @@ def test_setitem_large_table(any_multidict_class: type[MultiDict[int]]) -> None:
8184
assert next(iter(md.items())) == ("dup", 7)
8285

8386

87+
@pytest.mark.expensive
8488
def test_update_large_table(any_multidict_class: type[MultiDict[int]]) -> None:
8589
pairs = _big_pairs()
8690
new_pairs = [("dup", 1), ("k5", 5), ("dup", 2), ("new", 3), ("new", 4)]
@@ -91,6 +95,7 @@ def test_update_large_table(any_multidict_class: type[MultiDict[int]]) -> None:
9195

9296
@pytest.mark.parametrize("size", [8, 3000])
9397
@pytest.mark.parametrize("deleted", [0, 3])
98+
@pytest.mark.expensive
9499
def test_update_resizes_mid_batch(
95100
any_multidict_class: type[MultiDict[int]], deleted: int, size: int
96101
) -> None:

0 commit comments

Comments
 (0)