[ty] support recursive type aliases by carljm · Pull Request #19805 · astral-sh/ruff

carljm · 2025-08-07T12:39:01Z

Summary

Support recursive type aliases by adding a Type::TypeAlias type variant, which allows referring to a type alias directly as a type without eagerly unpacking it to its value.

We still unpack type aliases when they are added to intersections and unions, so that we can simplify the intersection/union appropriately based on the unpacked value of the type alias.

This introduces new possible recursive types, and so also requires expanding our usage of recursion-detecting visitors in Type methods. The use of these visitors is still not fully comprehensive in this PR, and will require further expansion to support recursion in more kinds of types (I already have further work on this locally), but I think it may be better to do this incrementally in multiple PRs.

Test Plan

Added some recursive type-alias tests and made them pass.

github-actions · 2025-08-07T12:41:05Z

Diagnostic diff on typing conformance tests

Changes were detected when running ty on typing conformance tests

--- old-output.txt	2025-08-12 00:50:16.219875316 +0000
+++ new-output.txt	2025-08-12 00:50:16.282875419 +0000
@@ -1,4 +1,5 @@
 WARN ty is pre-release software and not ready for production use. Expect to encounter bugs, missing features, and fatal errors.
+fatal[panic] Panicked at /home/runner/.cargo/git/checkouts/salsa-e6f3bb7c2a062968/940f9c0/src/function/execute.rs:204:25 when checking `/home/runner/work/ruff/ruff/typing/conformance/tests/aliases_typealiastype.py`: `infer_definition_types(Id(dc1d)): execute: too many cycle iterations`
 _directives_deprecated_library.py:15:31: error[invalid-return-type] Function always implicitly returns `None`, which is not assignable to return type `int`
 _directives_deprecated_library.py:30:26: error[invalid-return-type] Function always implicitly returns `None`, which is not assignable to return type `str`
 _directives_deprecated_library.py:36:41: error[invalid-return-type] Function always implicitly returns `None`, which is not assignable to return type `Self@__add__`
@@ -67,23 +68,6 @@
 aliases_type_statement.py:48:23: error[invalid-type-form] Boolean operations are not allowed in type expressions
 aliases_type_statement.py:49:23: error[fstring-type-annotation] Type expressions cannot use f-strings
 aliases_type_statement.py:80:37: error[invalid-type-form] List literals are not allowed in this context in a type expression: Did you mean `tuple[int, str]`?
-aliases_typealiastype.py:32:7: error[unresolved-attribute] Type `typing.TypeAliasType` has no attribute `other_attrib`
-aliases_typealiastype.py:39:26: error[invalid-type-form] List literals are not allowed in this context in a type expression: Did you mean `tuple[int, str]`?
-aliases_typealiastype.py:52:40: error[invalid-type-form] Function calls are not allowed in type expressions
-aliases_typealiastype.py:53:40: error[invalid-type-form] List literals are not allowed in this context in a type expression: Did you mean `tuple[int, str]`?
-aliases_typealiastype.py:54:42: error[invalid-type-form] Tuple literals are not allowed in this context in a type expression
-aliases_typealiastype.py:54:43: error[invalid-type-form] Tuple literals are not allowed in this context in a type expression: Did you mean `tuple[int, str]`?
-aliases_typealiastype.py:55:42: error[invalid-type-form] List comprehensions are not allowed in type expressions
-aliases_typealiastype.py:56:42: error[invalid-type-form] Dict literals are not allowed in type expressions
-aliases_typealiastype.py:57:42: error[invalid-type-form] Function calls are not allowed in type expressions
-aliases_typealiastype.py:58:48: error[invalid-type-form] Int literals are not allowed in this context in a type expression
-aliases_typealiastype.py:59:42: error[invalid-type-form] `if` expressions are not allowed in type expressions
-aliases_typealiastype.py:60:42: error[invalid-type-form] Variable of type `Literal[3]` is not allowed in a type expression
-aliases_typealiastype.py:61:42: error[invalid-type-form] Boolean literals are not allowed in this context in a type expression
-aliases_typealiastype.py:62:42: error[invalid-type-form] Int literals are not allowed in this context in a type expression
-aliases_typealiastype.py:63:42: error[invalid-type-form] Boolean operations are not allowed in type expressions
-aliases_typealiastype.py:64:42: error[invalid-type-form] F-strings are not allowed in type expressions
-aliases_typealiastype.py:66:47: error[unresolved-reference] Name `BadAlias21` used when not defined
 aliases_variance.py:18:24: error[non-subscriptable] Cannot subscript object of type `<class 'ClassA[typing.TypeVar("T_co")]'>` with no `__class_getitem__` method
 aliases_variance.py:28:16: error[non-subscriptable] Cannot subscript object of type `<class 'ClassA[typing.TypeVar("T_co")]'>` with no `__class_getitem__` method
 aliases_variance.py:44:16: error[non-subscriptable] Cannot subscript object of type `<class 'ClassB[typing.TypeVar("T_co"), typing.TypeVar("T_contra")]'>` with no `__class_getitem__` method
@@ -884,4 +868,5 @@
 typeddicts_operations.py:60:1: error[type-assertion-failure] Argument does not have asserted type `str | None`
 typeddicts_type_consistency.py:101:1: error[invalid-assignment] Object of type `Unknown | None` is not assignable to `str`
 typeddicts_usage.py:40:24: error[invalid-type-form] The special form `typing.TypedDict` is not allowed in type expressions. Did you mean to use a concrete TypedDict or `collections.abc.Mapping[str, object]` instead?
-Found 885 diagnostics
+Found 869 diagnostics
+WARN A fatal error occurred while checking some files. Not all project files were analyzed. See the diagnostics list above for details.

github-actions · 2025-08-07T12:43:06Z

`mypy_primer` results

Changes were detected when running on open source projects

mitmproxy (https://github.com/mitmproxy/mitmproxy)
- test/mitmproxy/contentviews/test__utils.py:23:38: error[invalid-argument-type] Argument to function `make_metadata` is incorrect: Expected `Message | TCPMessage | UDPMessage | WebSocketMessage | DNSMessage`, found `Response | None`
+ test/mitmproxy/contentviews/test__utils.py:23:38: error[invalid-argument-type] Argument to function `make_metadata` is incorrect: Expected `ContentviewMessage`, found `Response | None`

No memory usage changes detected ✅

carljm · 2025-08-07T13:04:54Z

Converting back to draft while I look into the new panic on the conformance suite.

carljm · 2025-08-07T22:22:10Z

Ok, the new panic in the conformance suite is for an invalid type alias, so it shouldn't be a case that affects much if any real code (as seen in the fact that there aren't new ecosystem panics). I have a plan for how to fix it (as well as fixing recursive legacy/bare type aliases), but it will be significant enough that I'd rather do it in a separate PR. So putting this back up for review as-is.

MichaReiser · 2025-08-08T09:31:54Z

Exciting!

I assume this change is because we no longer normalize the argument union:

mitmproxy (https://github.com/mitmproxy/mitmproxy)
- test/mitmproxy/contentviews/test__utils.py:23:38: error[invalid-argument-type] Argument to function `make_metadata` is incorrect: Expected `Message | TCPMessage | UDPMessage | WebSocketMessage | DNSMessage`, found `Response | None`
+ test/mitmproxy/contentviews/test__utils.py:23:38: error[invalid-argument-type] Argument to function `make_metadata` is incorrect: Expected `ContentviewMessage`, found `Response | None`

I must say, I very much prefer the new output (but agree, that changing this in general is out of scope for this PR)

crates/ty_python_semantic/src/types/builder.rs

crates/ty_python_semantic/src/types/tuple.rs

AlexWaygood

Looks great!!

crates/ty_python_semantic/resources/mdtest/pep695_type_aliases.md

crates/ty_python_semantic/src/types.rs

crates/ty_python_semantic/src/types/builder.rs

crates/ty_python_semantic/src/types/display.rs

crates/ty_python_semantic/src/types/infer.rs

crates/ty_python_semantic/src/types/visitor.rs

crates/ty_python_semantic/resources/mdtest/pep695_type_aliases.md

dcreager · 2025-08-08T12:50:51Z

crates/ty_python_semantic/src/types.rs

+        self.is_equivalent_to_impl(db, other, &mut visitor)
+    }
+
+    pub(crate) fn is_equivalent_to_impl(


Do the other recursive calls to is_equivalent_to in this method need to be updated to use is_equivalent_to_impl instead?

Yes, at some point. There are also some places in has_relation_to_impl that need this update. I'd been debating whether to fully thread through the visitors everywhere in this PR, or do it as a follow-up PR, with added regression tests.

crates/ty_python_semantic/src/types.rs

crates/ty_python_semantic/src/types/tuple.rs

codspeed-hq · 2025-08-12T00:39:36Z

CodSpeed WallTime Performance Report

Merging #19805 will not alter performance

_{Comparing cjm/recta (0ca632a) with main (d76fd10)}

Summary

✅ 8 untouched benchmarks

carljm · 2025-08-12T14:55:09Z

I assume this change is because we no longer normalize the argument union

No, this change is because we no longer eagerly unpack type aliases.

This reverts commit a7f5427.

AlexWaygood · 2025-08-12T15:44:59Z

crates/ty_python_semantic/src/types/tuple.rs

    // `static-frame` as part of a mypy_primer run! This is because it's called
    // from `NominalInstanceType::class()`, which is a very hot method.
-    #[salsa::tracked(heap_size=ruff_memory_usage::heap_size)]
+    #[salsa::tracked(cycle_fn=to_class_type_cycle_recover, cycle_initial=to_class_type_cycle_initial, heap_size=ruff_memory_usage::heap_size)]


the fact that static-frame is the most significant benchmark regression makes me suspect that this has something to do with the performance regressions. We know that this method is extremely hot when checking static-frame for some reason (#19811 (comment), #19811 (comment)), so if there's now a chance that we'll sometimes have to recover from cycles when calling this function, that seems likely to slow down static-frame quite a bit. This also means that the static-frame repo isn't necessarily representative of most real-world code, however; it seems like a pretty extreme case.

Yeah, this was one thing I've looked into a bit. I've confirmed (via Salsa tracing) that in checking static frame, we don't ever actually hit a cycle on this query (which is not surprising; if we did, then we likely would have panicked on static-frame before this PR). It's possible that there is still some effect simply from having the cycle handlers in place (Salsa does go into a different path for executing such queries), though I don't recall seeing a visible effect from that when we've added cycle handling to queries in the past. Even if it's very hot, presumably most of those accesses are cached; in that case, cycle handling shouldn't cause any overhead at all.

If this is the cause, then I think we just have to eat it, pending generalized improvements to the efficiency of Salsa cycle handling.

carljm · 2025-08-12T16:01:50Z

Perf regression seems reduced to the 1-3% range. Possible causes are either overhead from adding cycle handling to a hot query, or overhead from passing around CycleDetector visitors through many more hot methods (has_relation_to in particular) -- I suspect it's more the latter. Either way, though, I think this is an acceptable level of regression, and one we will have to accept in order to gain recursive type support. So I'm ready to land this.

PR has changed somewhat, but mostly by removing UnionStrategy, so I'm going to go with the approvals I've already got -- post-land review of the more recent changes is welcome!

charliermarsh · 2025-08-12T16:07:47Z

Let's go!

* main: Don't cache files with diagnostics (#19869) [ty] support recursive type aliases (#19805) [ty] Remove unsafe `salsa::Update` implementations in `tuple.rs` (#19880) [ty] Function argument inlay hints (#19269) [ty] Remove Salsa interning for `TypedDictType` (#19879) Fix `lint.future-annotations` link (#19876) [ty] Reduce memory usage of `TupleSpec` and `TupleType` (#19872) [ty] Track heap usage of salsa structs (#19790) Update salsa to pull in tracked struct changes (#19843) [ty] simplify CycleDetector::visit signature (#19873) [ty] use interior mutability in type visitors (#19871) [ty] Fix tool name is None when no ty path is given in ty_benchmark (#19870) [ty] Remove `Type::Tuple` (#19669) [ty] Short circuit `ReachabilityConstraints::analyze_single` for dynamic types (#19867)

…aints * dcreager/inferrable: (65 commits) this was right after all mark typevars inferrable as we go fix tests fix inference of constrained typevars [ty] Add precise inference for indexing, slicing and unpacking `NamedTuple` instances (#19560) Add rule code to GitLab description (#19896) [ty] render docstrings in hover (#19882) [ty] simplify return type of place_from_declarations (#19884) [ty] Various minor cleanups to tuple internals (#19891) [ty] Improve `sys.version_info` special casing (#19894) Don't cache files with diagnostics (#19869) [ty] support recursive type aliases (#19805) [ty] Remove unsafe `salsa::Update` implementations in `tuple.rs` (#19880) [ty] Function argument inlay hints (#19269) [ty] Remove Salsa interning for `TypedDictType` (#19879) Fix `lint.future-annotations` link (#19876) [ty] Reduce memory usage of `TupleSpec` and `TupleType` (#19872) [ty] Track heap usage of salsa structs (#19790) Update salsa to pull in tracked struct changes (#19843) [ty] simplify CycleDetector::visit signature (#19873) ...

carljm added the ty Multi-file analysis & type inference label Aug 7, 2025

carljm mentioned this pull request Aug 7, 2025

[ty] Remove Type::Tuple #19669

Merged

carljm marked this pull request as ready for review August 7, 2025 12:59

carljm requested review from AlexWaygood, dcreager and sharkdp as code owners August 7, 2025 12:59

carljm marked this pull request as draft August 7, 2025 13:04

carljm marked this pull request as ready for review August 7, 2025 22:22

MichaReiser approved these changes Aug 8, 2025

View reviewed changes

crates/ty_python_semantic/src/types/builder.rs Outdated Show resolved Hide resolved

crates/ty_python_semantic/src/types/tuple.rs Outdated Show resolved Hide resolved

MichaReiser added the bug Something isn't working label Aug 8, 2025

AlexWaygood approved these changes Aug 8, 2025

View reviewed changes

dcreager approved these changes Aug 8, 2025

View reviewed changes

carljm force-pushed the cjm/recta branch 2 times, most recently from fff262d to ab07975 Compare August 12, 2025 00:28

carljm added 10 commits August 12, 2025 07:40

[WIP] recursive type alias test cases

11e2665

[WIP] initial working version

c024a80

break apart type aliases in unions and intersections

a827c0a

handle cycles when explicitly placed in unions/intersections

3bc2c82

clippy and comments

025bd16

add cycle handling for TupleType::to_class_type

189286c

remove UnionStrategy

5337f88

fix double-visits

efdcdef

review comments

e6e00b8

more fixes

abfea5e

carljm added 2 commits August 12, 2025 07:52

document how to use cycle handling

9862620

comments and naming

49ffb18

carljm force-pushed the cjm/recta branch from 82ec772 to 49ffb18 Compare August 12, 2025 14:53

carljm added 2 commits August 12, 2025 08:25

try going back to mut visitors

a7f5427

Revert "try going back to mut visitors"

0ca632a

This reverts commit a7f5427.

AlexWaygood reviewed Aug 12, 2025

View reviewed changes

carljm merged commit 13bdba5 into main Aug 12, 2025
38 checks passed

carljm deleted the cjm/recta branch August 12, 2025 16:03

AlexWaygood mentioned this pull request Aug 12, 2025

[ty] Add precise inference for indexing, slicing and unpacking NamedTuple instances #19560

Merged

Comments

Conversation

carljm commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test Plan

Uh oh!

github-actions bot commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Diagnostic diff on typing conformance tests

Uh oh!

github-actions bot commented Aug 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

mypy_primer results

Uh oh!

carljm commented Aug 7, 2025

Uh oh!

carljm commented Aug 7, 2025

Uh oh!

MichaReiser commented Aug 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AlexWaygood left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dcreager Aug 8, 2025

Choose a reason for hiding this comment

Uh oh!

carljm Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

codspeed-hq bot commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CodSpeed WallTime Performance Report

Merging #19805 will not alter performance

Summary

Uh oh!

carljm commented Aug 12, 2025

Uh oh!

AlexWaygood Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

carljm Aug 12, 2025

Choose a reason for hiding this comment

Uh oh!

carljm commented Aug 12, 2025

Uh oh!

Uh oh!

charliermarsh commented Aug 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

carljm commented Aug 7, 2025 •

edited

Loading

github-actions bot commented Aug 7, 2025 •

edited

Loading

github-actions bot commented Aug 7, 2025 •

edited

Loading

`mypy_primer` results

MichaReiser commented Aug 8, 2025 •

edited

Loading

codspeed-hq bot commented Aug 12, 2025 •

edited

Loading

AlexWaygood Aug 12, 2025 •

edited

Loading