Skip to content

Commit 59196ba

Browse files
authored
[ty] Unify polarity-aware relation construction (#27707)
## Summary Part of astral-sh/ty#3557 and the second step in rebooting #26712 as smaller focused changes. On `main`, generic inference does not consistently propagate the variance of an enclosing generic class into structural-protocol and callable comparisons. Those comparisons instead construct the forward `actual <= formal` relation even when the surrounding position is contravariant or invariant. This produces incorrect bounds and inferred types: ```py from typing import Callable class Middle: ... class Derived(Middle): ... class Sink[T]: def put(self, value: T) -> None: ... def infer[T](sink: Sink[Callable[[], T]], value: T) -> T: return value reveal_type(infer(Sink[Callable[[], Middle]](), Derived())) # main: Middle # this PR: Derived ``` This PR centralizes relation construction so inference consistently follows the surrounding polarity: covariance requires `actual <= formal`, contravariance requires `formal <= actual`, invariance requires both, and bivariance adds no constraints. The fix applies to nested generic protocols, callable signatures, callback protocols, TypedDict/protocol relations, and materialized protocols while preserving existing nominal inference, callable-union alternatives, TypedDict-union optimizations, and cycle-safe handling of recursive protocols. ## Limitation The new constraint solver does not yet support `ParamSpec` or `TypeVarTuple`. Whenever an inference context includes either variadic, all affected relations must continue through the old, forward-only inference path, including comparisons involving ordinary type variables alongside an unrelated variadic. This deliberately avoids regressions in parameter inference, structural protocols, descriptor signatures, and downstream argument diagnostics, but it also means the polarity fixes in this PR do not apply whenever variadics are involved. Focused TODO tests document cases that still incorrectly infer `Middle` instead of `Derived`; fixing those cases requires adding variadic support to the new solver. ## Test plan Added mdtests cover nested nominal classes, generic protocols, callable signatures, and callback protocols under covariance, contravariance, and invariance; double contravariance, callable-union alternatives, and invariant rejection; finite, recursive, and recursive-only materialized protocols; invariant, contravariant, and covariant `ParamSpec` inference through `Concatenate`; bounded and constrained prefixes, descriptor receivers, structural protocol unions, nominal callable objects, higher-order callbacks, and mixed ordinary/variadic inference; analogous `TypeVarTuple` callable, protocol, structural-union, and nominal-implementation cases; and explicit TODO coverage for unrelated variadics suppressing otherwise-correct polarity. Additional mdtests cover repeated comparisons of the same protocol or callable under opposite polarities, plus overloaded callable acceptance and rejection across covariant, contravariant, and invariant wrappers.
1 parent b8c5e73 commit 59196ba

6 files changed

Lines changed: 919 additions & 71 deletions

File tree

‎crates/ty_python_semantic/resources/mdtest/descriptor_protocol.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1864,6 +1864,29 @@ class Example:
18641864
pass
18651865
```
18661866

1867+
An invalid descriptor receiver must not discard the inferred `ParamSpec` for its bound callable.
1868+
Even though `Concatenate` makes the receiver positional-only, the remaining parameters still retain
1869+
their precise types.
1870+
1871+
```py
1872+
class Decorator(Generic[P]):
1873+
def __call__(self, *args: P.args, **kwargs: P.kwargs) -> None: ...
1874+
def __get__(self: "Decorator[Concatenate[Any, P2]]", instance: Any, owner: Any) -> "Decorator[P2]":
1875+
raise NotImplementedError
1876+
1877+
def decorate(fn: Callable[P, Any]) -> Decorator[P]:
1878+
raise NotImplementedError
1879+
1880+
class Decorated:
1881+
@decorate
1882+
def method(self, value: str) -> None: ...
1883+
1884+
# error: [invalid-attribute-access]
1885+
bound = Decorated().method
1886+
reveal_type(bound) # revealed: Decorator[(value: str)]
1887+
bound(1) # error: [invalid-argument-type]
1888+
```
1889+
18671890
[descriptors]: https://docs.python.org/3/howto/descriptor.html
18681891
[precedence chain]: https://github.com/python/cpython/blob/3.13/Objects/typeobject.c#L5393-L5481
18691892
[simple example]: https://docs.python.org/3/howto/descriptor.html#simple-example-a-descriptor-that-returns-a-constant

‎crates/ty_python_semantic/resources/mdtest/generics/legacy/paramspec.md‎

Lines changed: 166 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -746,6 +746,172 @@ takes_int_job(defaulted_job)
746746
takes_int_job(wrong_job) # error: [invalid-argument-type]
747747
```
748748

749+
## Inferring an invariant `ParamSpec` through `Concatenate`
750+
751+
A `Concatenate` prefix is positional-only, so a callback whose first parameter also accepts a
752+
keyword is not compatible with an invariant wrapper. Even though that argument is rejected, its
753+
remaining parameters must still be inferred precisely.
754+
755+
```py
756+
from typing import Callable, Concatenate, Generic, ParamSpec
757+
758+
P = ParamSpec("P")
759+
760+
class Callback(Generic[P]):
761+
def __init__(self, callback: Callable[P, None]) -> None: ...
762+
763+
def without_first(callback: Callback[Concatenate[object, P]]) -> Callable[P, None]:
764+
raise NotImplementedError
765+
766+
def original(first: object, value: str) -> None: ...
767+
768+
remaining = without_first(Callback(original)) # error: [invalid-argument-type]
769+
reveal_type(remaining) # revealed: (value: str) -> None
770+
remaining(1) # error: [invalid-argument-type]
771+
```
772+
773+
## Inferring a contravariant `ParamSpec` through `Concatenate`
774+
775+
The same callback is compatible with a contravariant wrapper, and its remaining parameters must
776+
still be inferred precisely enough to reject an incompatible later argument.
777+
778+
```py
779+
from typing import Callable, Concatenate, Generic, ParamSpec, TypeVar
780+
781+
P = ParamSpec("P", contravariant=True)
782+
783+
class Callback(Generic[P]):
784+
def __init__(self, callback: Callable[P, None]) -> None: ...
785+
786+
def without_first(callback: Callback[Concatenate[object, P]]) -> Callable[P, None]:
787+
raise NotImplementedError
788+
789+
def original(first: object, value: str) -> None: ...
790+
791+
remaining = without_first(Callback(original))
792+
reveal_type(remaining) # revealed: (value: str) -> None
793+
remaining(1) # error: [invalid-argument-type]
794+
```
795+
796+
A contravariant callback can accept a broader positional-only prefix than a bounded type variable
797+
requires. The separate `Middle()` argument determines the narrower specialization without rejecting
798+
the valid callback.
799+
800+
```py
801+
class Base: ...
802+
class Middle(Base): ...
803+
class Other: ...
804+
805+
Bounded = TypeVar("Bounded", bound=Middle)
806+
Q = ParamSpec("Q")
807+
808+
def accepts_base(first: Base, /, value: str) -> None: ...
809+
def bounded(callback: Callback[Concatenate[Bounded, Q]], witness: Bounded) -> tuple[Bounded, Callable[Q, None]]:
810+
raise NotImplementedError
811+
812+
bounded_result = bounded(Callback(accepts_base), Middle())
813+
reveal_type(bounded_result) # revealed: tuple[Middle, (value: str) -> None]
814+
bounded_result[1](1) # error: [invalid-argument-type]
815+
```
816+
817+
The same contravariant relationship remains valid when the prefix type variable has explicit
818+
constraints instead of an upper bound.
819+
820+
```py
821+
Constrained = TypeVar("Constrained", Middle, Other)
822+
823+
def constrained(callback: Callback[Concatenate[Constrained, Q]], witness: Constrained) -> tuple[Constrained, Callable[Q, None]]:
824+
raise NotImplementedError
825+
826+
constrained_result = constrained(Callback(accepts_base), Middle())
827+
reveal_type(constrained_result) # revealed: tuple[Middle, (value: str) -> None]
828+
constrained_result[1](1) # error: [invalid-argument-type]
829+
```
830+
831+
## Inferring a covariant `ParamSpec` through `Concatenate`
832+
833+
A covariant wrapper containing a callback with a narrower positional-only prefix remains valid,
834+
whether the wrapper is stored first or constructed inline.
835+
836+
```py
837+
from typing import Callable, Concatenate, Generic, ParamSpec
838+
839+
P = ParamSpec("P", covariant=True)
840+
Q = ParamSpec("Q")
841+
842+
class Base: ...
843+
class Middle(Base): ...
844+
845+
class Callback(Generic[P]):
846+
def __init__(self, callback: Callable[P, None]) -> None: ...
847+
848+
def without_first(callback: Callback[Concatenate[Base, Q]]) -> Callable[Q, None]:
849+
raise NotImplementedError
850+
851+
def original(first: Middle, /, value: str) -> None: ...
852+
853+
wrapped = Callback(original)
854+
# TODO: Should reveal `(value: str) -> None`. Needs ParamSpecs in the new constraint solver.
855+
reveal_type(without_first(wrapped)) # revealed: (...) -> None
856+
# TODO: Should reveal `(value: str) -> None`. Needs ParamSpecs in the new constraint solver.
857+
reveal_type(without_first(Callback(original))) # revealed: (...) -> None
858+
```
859+
860+
## Inferring through unions of structural `ParamSpec` protocols
861+
862+
Different protocols with the same parameter list can satisfy a target protocol structurally, even
863+
when they appear in a union nested inside an invariant or contravariant wrapper.
864+
865+
```py
866+
from typing import Generic, ParamSpec, Protocol, TypeVar
867+
868+
P = ParamSpec("P")
869+
T = TypeVar("T")
870+
TContra = TypeVar("TContra", contravariant=True)
871+
872+
class Invariant(Generic[T]):
873+
value: T
874+
875+
class Contravariant(Generic[TContra]):
876+
def put(self, value: TContra) -> None: ...
877+
878+
class Target(Protocol[P]):
879+
def call(self, *args: P.args, **kwargs: P.kwargs) -> None: ...
880+
881+
class Actual(Protocol[P]):
882+
def call(self, *args: P.args, **kwargs: P.kwargs) -> None: ...
883+
884+
class Other(Protocol[P]):
885+
def call(self, *args: P.args, **kwargs: P.kwargs) -> None: ...
886+
887+
def invariant(value: Invariant[Target[P]]) -> Target[P]:
888+
raise NotImplementedError
889+
890+
def contravariant(value: Contravariant[Target[P]]) -> Target[P]:
891+
raise NotImplementedError
892+
893+
def compatible(
894+
first: Invariant[Actual[[str]] | Other[[str]]],
895+
second: Contravariant[Actual[[str]] | Other[[str]]],
896+
) -> None:
897+
reveal_type(invariant(first)) # revealed: Target[(str, /)]
898+
reveal_type(contravariant(second)) # revealed: Target[(str, /)]
899+
```
900+
901+
Union members with incompatible parameter lists cannot satisfy either wrapper. Their signatures must
902+
remain visible in the inferred result instead of collapsing to a gradual parameter list.
903+
904+
```py
905+
def incompatible(
906+
first: Invariant[Actual[[str]] | Other[[bytes]]],
907+
second: Contravariant[Actual[[str]] | Other[[bytes]]],
908+
) -> None:
909+
# error: [invalid-argument-type]
910+
reveal_type(invariant(first)) # revealed: Target[((str, /)) | ((bytes, /))]
911+
# error: [invalid-argument-type]
912+
reveal_type(contravariant(second)) # revealed: Target[((str, /)) | ((bytes, /))]
913+
```
914+
749915
## `ParamSpec` cannot specialize a `TypeVar`, and vice versa
750916

751917
<!-- snapshot-diagnostics -->

0 commit comments

Comments
 (0)