[ty] Use datatest instead of dirtest by MichaReiser · Pull Request #21937 · astral-sh/ruff

MichaReiser · 2025-12-12T08:40:45Z

Summary

Same as #21933 but for our mdtests.

datatest-stable uses a custom test-harness that mimics cargo test and supports cargo nextest to create a separate test for every file found in a given directory.
Unlike dir-tests, datatest-stable doesn't require re-compilation after adding or removing a test file (because it uses a custom test harness). This should allow us (people like me who aren't using the mdtest Python script) to iterate faster on mdtests, since we no longer have to recompile ty_python_semantic whenever a markdown file changes.

Test plan

mdtests on main:

     Summary [  10.810s] 301 tests run: 301 passed, 0 skipped

mdtests on this branch:

Summary [  10.606s] 301 tests run: 301 passed, 0 skipped

I used the mdtest.py file (for the first time) and verified that it picks up changes and only runs the changed tests.

I verified that a new test file is picked up without needing to recompile ty_python_semantic.

astral-sh-bot · 2025-12-12T08:44:03Z

Diagnostic diff on typing conformance tests

No changes detected when running ty on typing conformance tests ✅

astral-sh-bot · 2025-12-12T08:46:36Z

`mypy_primer` results

Changes were detected when running on open source projects

prefect (https://github.com/PrefectHQ/prefect)
- src/integrations/prefect-dbt/prefect_dbt/core/settings.py:94:28: error[invalid-assignment] Object of type `dict[str, Any] | T@resolve_block_document_references | str | ... omitted 4 union elements` is not assignable to `dict[str, Any]`
+ src/integrations/prefect-dbt/prefect_dbt/core/settings.py:94:28: error[invalid-assignment] Object of type `dict[str, Any] | T@resolve_block_document_references` is not assignable to `dict[str, Any]`
- src/integrations/prefect-dbt/prefect_dbt/core/settings.py:99:28: error[invalid-assignment] Object of type `dict[str, Any] | T@resolve_variables | str | ... omitted 4 union elements` is not assignable to `dict[str, Any]`
+ src/integrations/prefect-dbt/prefect_dbt/core/settings.py:99:28: error[invalid-assignment] Object of type `dict[str, Any] | T@resolve_variables` is not assignable to `dict[str, Any]`
- src/prefect/cli/deploy/_core.py:86:21: error[invalid-assignment] Object of type `T@resolve_block_document_references | dict[str, Any] | str | ... omitted 4 union elements` is not assignable to `dict[str, Any]`
+ src/prefect/cli/deploy/_core.py:86:21: error[invalid-assignment] Object of type `T@resolve_block_document_references | dict[str, Any]` is not assignable to `dict[str, Any]`
- src/prefect/cli/deploy/_core.py:87:21: error[invalid-assignment] Object of type `T@resolve_variables | str | int | ... omitted 4 union elements` is not assignable to `dict[str, Any]`
+ src/prefect/cli/deploy/_core.py:87:21: error[invalid-assignment] Object of type `T@resolve_variables` is not assignable to `dict[str, Any]`
- src/prefect/deployments/steps/core.py:137:38: error[invalid-argument-type] Argument is incorrect: Expected `T@resolve_variables`, found `T@resolve_block_document_references | dict[str, Any] | str | ... omitted 4 union elements`
+ src/prefect/deployments/steps/core.py:137:38: error[invalid-argument-type] Argument is incorrect: Expected `T@resolve_variables`, found `T@resolve_block_document_references | dict[str, Any]`
- src/prefect/utilities/templating.py:320:13: error[invalid-assignment] Invalid subscript assignment with key of type `object` and value of type `T@resolve_block_document_references | dict[str, Any] | str | ... omitted 4 union elements` on object of type `dict[str, Any]`
+ src/prefect/utilities/templating.py:320:13: error[invalid-assignment] Invalid subscript assignment with key of type `object` and value of type `T@resolve_block_document_references | dict[str, Any]` on object of type `dict[str, Any]`
- src/prefect/utilities/templating.py:323:16: error[invalid-return-type] Return type does not match returned value: expected `T@resolve_block_document_references | dict[str, Any]`, found `list[T@resolve_block_document_references | dict[str, Any] | str | ... omitted 5 union elements]`
+ src/prefect/utilities/templating.py:323:16: error[invalid-return-type] Return type does not match returned value: expected `T@resolve_block_document_references | dict[str, Any]`, found `list[T@resolve_block_document_references | dict[str, Any] | Unknown]`
- src/prefect/utilities/templating.py:437:16: error[invalid-return-type] Return type does not match returned value: expected `T@resolve_variables`, found `dict[object, T@resolve_variables | str | int | ... omitted 5 union elements]`
+ src/prefect/utilities/templating.py:437:16: error[invalid-return-type] Return type does not match returned value: expected `T@resolve_variables`, found `dict[object, T@resolve_variables | Unknown]`
- src/prefect/utilities/templating.py:442:16: error[invalid-return-type] Return type does not match returned value: expected `T@resolve_variables`, found `list[T@resolve_variables | str | int | ... omitted 5 union elements]`
+ src/prefect/utilities/templating.py:442:16: error[invalid-return-type] Return type does not match returned value: expected `T@resolve_variables`, found `list[T@resolve_variables | Unknown]`
- src/prefect/workers/base.py:228:13: error[invalid-argument-type] Argument is incorrect: Expected `T@resolve_variables`, found `T@resolve_block_document_references | dict[str, Any] | str | ... omitted 4 union elements`
+ src/prefect/workers/base.py:228:13: error[invalid-argument-type] Argument is incorrect: Expected `T@resolve_variables`, found `T@resolve_block_document_references | dict[str, Any]`
- src/prefect/workers/base.py:230:20: error[invalid-argument-type] Argument expression after ** must be a mapping type: Found `T@resolve_variables | str | int | ... omitted 4 union elements`
+ src/prefect/workers/base.py:230:20: error[invalid-argument-type] Argument expression after ** must be a mapping type: Found `T@resolve_variables`

scikit-build-core (https://github.com/scikit-build/scikit-build-core)
+ src/scikit_build_core/build/wheel.py:98:20: error[no-matching-overload] No overload of bound method `__init__` matches arguments
- Found 43 diagnostics
+ Found 44 diagnostics

jax (https://github.com/google/jax)
+ jax/_src/tree_util.py:295:31: error[invalid-argument-type] Argument to bound method `register_node` is incorrect: Expected `(Hashable, Iterable[object], /) -> T@register_pytree_node`, found `(_AuxData@register_pytree_node, _Children@register_pytree_node, /) -> T@register_pytree_node`
+ jax/_src/tree_util.py:298:31: error[invalid-argument-type] Argument to bound method `register_node` is incorrect: Expected `(Hashable, Iterable[object], /) -> T@register_pytree_node`, found `(_AuxData@register_pytree_node, _Children@register_pytree_node, /) -> T@register_pytree_node`
+ jax/_src/tree_util.py:301:31: error[invalid-argument-type] Argument to bound method `register_node` is incorrect: Expected `(Hashable, Iterable[object], /) -> T@register_pytree_node`, found `(_AuxData@register_pytree_node, _Children@register_pytree_node, /) -> T@register_pytree_node`
- Found 2794 diagnostics
+ Found 2797 diagnostics

pandas-stubs (https://github.com/pandas-dev/pandas-stubs)
- tests/test_groupby.py:433:11: error[type-assertion-failure] Type `Series[Any]` does not match asserted type `Series[(Any & str) | (Any & bytes) | (Any & int) | ... omitted 12 union elements]`
+ tests/test_groupby.py:433:11: error[type-assertion-failure] Type `Series[Any]` does not match asserted type `Series[(str & Any) | (bytes & Any) | (int & Any) | ... omitted 12 union elements]`
- tests/test_resampler.py:394:11: error[type-assertion-failure] Type `Series[Any]` does not match asserted type `Series[(Any & str) | (Any & bytes) | (Any & int) | ... omitted 12 union elements]`
+ tests/test_resampler.py:394:11: error[type-assertion-failure] Type `Series[Any]` does not match asserted type `Series[(str & Any) | (bytes & Any) | (int & Any) | ... omitted 12 union elements]`

pydantic (https://github.com/pydantic/pydantic)
- pydantic/fields.py:943:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
+ pydantic/fields.py:943:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
- pydantic/fields.py:983:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
+ pydantic/fields.py:983:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
- pydantic/fields.py:1026:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
+ pydantic/fields.py:1026:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
- pydantic/fields.py:1066:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
+ pydantic/fields.py:1066:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
- pydantic/fields.py:1109:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
+ pydantic/fields.py:1109:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
- pydantic/fields.py:1148:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
+ pydantic/fields.py:1148:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
- pydantic/fields.py:1188:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`
+ pydantic/fields.py:1188:5: error[invalid-parameter-default] Default value of type `PydanticUndefinedType` is not assignable to annotated parameter type `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`
- pydantic/fields.py:1567:13: error[invalid-argument-type] Argument is incorrect: Expected `dict[str, Divergent] | ((dict[str, Divergent], /) -> None) | None`, found `Top[dict[Unknown, Unknown]] | (((dict[str, Divergent], /) -> None) & ~Top[dict[Unknown, Unknown]]) | None`
+ pydantic/fields.py:1567:13: error[invalid-argument-type] Argument is incorrect: Expected `dict[str, int | float | str | ... omitted 3 union elements] | ((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) | None`, found `Top[dict[Unknown, Unknown]] | (((dict[str, int | float | str | ... omitted 3 union elements], /) -> None) & ~Top[dict[Unknown, Unknown]]) | None`

No memory usage changes detected ✅

astral-sh-bot · 2025-12-12T08:54:00Z

`ruff-ecosystem` results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Formatter (stable)

✅ ecosystem check detected no format changes.

Formatter (preview)

✅ ecosystem check detected no format changes.

AlexWaygood

Nice! I think this means that we can get rid of the "note" block here saying that you probably need a build.rs file if you use mdtests in a crate:

ruff/crates/ty_test/README.md

Line 37 in 0138cd2

> [!NOTE]

We should instead just say that it's recommended to use mdtest in combination with datatest.

Also, you should use mdtest.py 😆 the experience is soooo much better!!

MichaReiser · 2025-12-12T09:27:32Z

Also, you should use mdtest.py 😆 the experience is soooo much better!!

I believe you. I just don't write many mdtests (I only occasionally debug them)

dhruvmanila · 2025-12-12T13:03:04Z

Uff, I've lost the ability to select individual tests from the editor :(

i.e., rust-analyzer cannot pick up individual tests in this file now and is the same across other tests as well (parser and formatter)

dhruvmanila · 2025-12-12T13:07:13Z

For context, on the right split you can see that rust-analyzer is able to provide me with all the individual tests aka the file name in mdtest suite and I can select on any one of them and it will only run that specific test and then re-run that same test however many times I want:

Screen.Recording.2025-12-12.at.18.36.08.mov

MichaReiser · 2025-12-12T13:08:14Z

Hmm. I guess that's not unexpected given the approach (note, I don't think this worked for the formatter or parser before because we only had one test). Before, rust-analyzer was able to see all tests because of the macro expansion (but I doubt that it ever updated when files were added or removed). Now, there's no such macro expansion happening anymore.

dhruvmanila · 2025-12-12T13:10:07Z

Before, rust-analyzer was able to see all tests because of the macro expansion (but I doubt that it ever updated when files were added or removed).

Yeah, in those cases I'd ask rust-analyzer to re-build the proc macros which would then update this list. Note that this is different than restarting the server.

dhruvmanila · 2025-12-12T13:14:24Z

I'm not against this change, it's just that this has been a core part of my workflow. I guess I'll have to try out mdtest.py finally :)

MichaReiser · 2025-12-12T13:30:22Z

I'm not against this change, it's just that this has been a core part of my workflow. I guess I'll have to try out mdtest.py finally :)

Yeah. Losing that certainly feels painful. We might need to write a mdtest LSP

MichaReiser · 2025-12-15T08:20:13Z

I spent some time trying to fix the LSP regression but, unfortunately, I think this will require changes in r-a. I hoped that all that was missing is --list --output=json support in libtest-mimic (which indeed is missing), but r-a only uses semantic analysis to find tests (for good reasons) and I haven't found a way to customize r-a's test discovery.

I opened rust-lang/rust-analyzer#21259 upstream

dhruvmanila · 2025-12-15T08:37:30Z

Wow, thank you for spending time in looking into this, I appreciate that!

…t_type_aliases.md

MichaReiser added testing Related to testing Ruff itself ty Multi-file analysis & type inference labels Dec 12, 2025

MichaReiser force-pushed the micha/ty-datatest branch from 9ed9114 to 97e7066 Compare December 12, 2025 08:41

MichaReiser marked this pull request as ready for review December 12, 2025 08:54

MichaReiser requested review from AlexWaygood, carljm, dcreager and sharkdp as code owners December 12, 2025 08:54

AlexWaygood reviewed Dec 12, 2025

View reviewed changes

Base automatically changed from micha/format-datatest to main December 13, 2025 08:02

dhruvmanila approved these changes Dec 15, 2025

View reviewed changes

MichaReiser added 3 commits December 18, 2025 18:58

[ty] Use datatest instead of dirtest

0dae958

Discard changes to crates/ty_python_semantic/resources/mdtest/implici…

7ff9a81

…t_type_aliases.md

Ruff

ea4d105

MichaReiser force-pushed the micha/ty-datatest branch from f690d6f to ea4d105 Compare December 18, 2025 17:59

Remove note

5f64ccd

MichaReiser enabled auto-merge (squash) December 18, 2025 18:00

MichaReiser merged commit 42b9727 into main Dec 18, 2025
41 checks passed

MichaReiser deleted the micha/ty-datatest branch December 18, 2025 18:05

Conversation

MichaReiser commented Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

astral-sh-bot Bot commented Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Diagnostic diff on typing conformance tests

Uh oh!

astral-sh-bot Bot commented Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

mypy_primer results

Uh oh!

astral-sh-bot Bot commented Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ruff-ecosystem results

Linter (stable)

Linter (preview)

Formatter (stable)

Formatter (preview)

Uh oh!

AlexWaygood left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MichaReiser commented Dec 12, 2025

Uh oh!

dhruvmanila commented Dec 12, 2025

Uh oh!

dhruvmanila commented Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MichaReiser commented Dec 12, 2025

Uh oh!

dhruvmanila commented Dec 12, 2025

Uh oh!

dhruvmanila commented Dec 12, 2025

Uh oh!

MichaReiser commented Dec 12, 2025

Uh oh!

MichaReiser commented Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dhruvmanila commented Dec 15, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

MichaReiser commented Dec 12, 2025 •

edited

Loading

astral-sh-bot Bot commented Dec 12, 2025 •

edited

Loading

astral-sh-bot Bot commented Dec 12, 2025 •

edited

Loading

`mypy_primer` results

astral-sh-bot Bot commented Dec 12, 2025 •

edited

Loading

`ruff-ecosystem` results

AlexWaygood left a comment •

edited

Loading

dhruvmanila commented Dec 12, 2025 •

edited

Loading

MichaReiser commented Dec 15, 2025 •

edited

Loading