Skip to content

Use datatest for formatter tests#21933

Merged
MichaReiser merged 2 commits intomainfrom
micha/format-datatest
Dec 13, 2025
Merged

Use datatest for formatter tests#21933
MichaReiser merged 2 commits intomainfrom
micha/format-datatest

Conversation

@MichaReiser
Copy link
Copy Markdown
Member

@MichaReiser MichaReiser commented Dec 12, 2025

Summary

datatest-stable is a custom test harness for data driven tests.

It uses a custom test-harness that mimics cargo test and supports cargo nextest to create a separate test for every file found in a given directory.
Unlike other test-macros, datatest-stable doesn't require re-compilation after changing a test file (because it uses a custom test harness).

The main advantage of using it for formatter tests is that each formatter spec file is now its own test so that:

  • Tests can run in parallel, resulting in faster wall-time
  • You can run all tests even if there are a few failing tests
  • You can selectively run tests

Test Plan

I made changes to a test file and it picked up the changes without needing recompilation

@MichaReiser MichaReiser added the testing Related to testing Ruff itself label Dec 12, 2025
@MichaReiser MichaReiser marked this pull request as ready for review December 12, 2025 07:42
@astral-sh-bot
Copy link
Copy Markdown

astral-sh-bot Bot commented Dec 12, 2025 •
edited
Loading

Diagnostic diff on typing conformance tests

No changes detected when running ty on typing conformance tests ✅

@astral-sh-bot
Copy link
Copy Markdown

astral-sh-bot Bot commented Dec 12, 2025 •
edited
Loading

mypy_primer results

Changes were detected when running on open source projects 
scikit-build-core (https://github.com/scikit-build/scikit-build-core)
+ src/scikit_build_core/build/wheel.py:98:20: error[no-matching-overload] No overload of bound method `__init__` matches arguments
- Found 41 diagnostics
+ Found 42 diagnostics

pandas-stubs (https://github.com/pandas-dev/pandas-stubs)
+ pandas-stubs/_typing.pyi:1218:16: warning[unused-ignore-comment] Unused blanket `type: ignore` directive
- Found 5136 diagnostics
+ Found 5137 diagnostics

No memory usage changes detected ✅

@astral-sh-bot
Copy link
Copy Markdown

astral-sh-bot Bot commented Dec 12, 2025 •
edited
Loading

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Formatter (stable)

✅ ecosystem check detected no format changes.

Formatter (preview)

✅ ecosystem check detected no format changes.

@MichaReiser MichaReiser force-pushed the micha/format-datatest branch from b82d765 to f095e19 Compare December 12, 2025 08:05
This was referenced Dec 12, 2025
Merged
Merged
ntBre
ntBre approved these changes Dec 12, 2025
View reviewed changes
Copy link
Copy Markdown
Contributor

@ntBre ntBre left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, this sounds awesome!

@MichaReiser MichaReiser merged commit e2ec2bc into main Dec 13, 2025
42 checks passed
@MichaReiser MichaReiser deleted the micha/format-datatest branch December 13, 2025 08:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ntBre ntBre ntBre approved these changes

@dhruvmanila dhruvmanila Awaiting requested review from dhruvmanila dhruvmanila is a code owner

Assignees

No one assigned

Labels

testing Related to testing Ruff itself

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@MichaReiser @ntBre