Invariant __class__: T inferred as Self | T causing false negatives

This example illustrates the issue, and how solving it makes it possible to completely annotate assert type(x) is int:

from typing import Protocol, override

class Just[T](Protocol):  # ~T@Just is invariant
    @property  # type: ignore[override]
    @override
    def __class__(self, /) -> type[T]: ...  # pyright: ignore[reportIncompatibleMethodOverride]
    @__class__.setter
    def __class__(self, t: type[T], /) -> None: ...

###

def just_int(x: Just[int], /) -> None:
    assert type(x) is int

just_int(42)  # true negative
just_int(False)  # false negative (works with mypy)
just_int("")  # true positive, wrong error message: Type "int | str" is not assignable to type "int"

###

def just_float(x: Just[float], /) -> None:
    assert type(x) is float

just_float(3.14)  # true negative
just_float(42)  # false negative (also with mypy)
just_float(False)  # false negative (also with mypy)
just_float("")  # true positive, wrong error message: Type "float | str" is not assignable to type "float"

https://pyright-play.net/?strict=true&code=GYJw9gtgBALgngBwJYDsDmUkQWEMoAK4MYAxmADYA0UYAbgKYghIAmDAUF6RQIYDO-KACkArvxgBtACoBdABREwJchQCUALihQAxFAB%2B0gAJiJmIajq8WvFDA7btRhOARN42vfDdakaFLgMkvRMLOyyDo5GIcxsnI5Q7MBQAPopPAL8afL8DBTANAD0alAAtAB8sIhBcloAdA2eUAhwLGgAFjC%2B-oGSIAw4eACSKOTYvDBIAEYUDACyDDDtYKwA8oyx4ZFOaRmCaXW5MDBM24kMybt8%2Byk5eQWwWt41skUlFVAAcmAoDPUNXB0QK4SSgACtxDAUqgYPIAB5aUxSGGvKDFMqVb6-DRnTLuKpueElJAWOxcCESaF2eQAFgATCVdLAQKIGFBfmgJkhGBwKVCYfIAGK8Ci5Rl6YAi3LshicyaMKDyADuuAA1kIlUgllAIIg4GpeZCqbCAEQm8XM1nNMD8LXchg0JXgdBQUK4HUMQS8NB-KDSapQE0wqAAHygEhAJvM7OUUEyfhQvBmbJIBLZQbsJsBwI4oL5KWAFDAE3hiMhkkLxZgqPRHyxf1xgnxzyJ0crE3JRvbsIAzHUAIw0i0wFlsjlcnn57u0hlNSWiseyids%2BRSsBQTXa3UtA1Tosl4ULi3z6Xj%2BUrtcbrXtHV63dd-em81NEdWnC28%2BO50YN0gD1en0tH9NxA27UNwxHKMSRjfB438JNZlgddnlAx8TSAA

The error messages make me suspect that __class__: T is interpreted as T | Self.

This partially works in mypy:
https://mypy-play.net/?mypy=latest&python=3.12&gist=c50b6d2092cf463c2b97d9267fa21896

Pyright is working as intended here, so I don't consider this a bug.

The type Just[int] is a specialization of the Just class, so we can effectively rewrite it as a non-generic class that looks like this:

class JustInt(Protocol):
    @property
    def __class__(self, /) -> type[int]: ...
    @__class__.setter
    def __class__(self, t: type[int], /) -> None: ...

Now, let's consider the relevant parts of the bool class that are required to satisfy this protocol definition. Note that the definition of __class__ is inherited by bool from object, but we'll simplify it here.

class SimplifiedBool:
    @property
    def __class__(self) -> type[bool]: ...
    @__class__.setter
    def __class__(self, type: type[object], /) -> None: ...

Does SimplifiedBool satisfy the JusInt protocol? Yes, it does; type[bool] is a subtype of type[int], and type[object] is a supertype of type[int]. Note that we used a covariant check for the return type and a contravariant check for the parameter type.

Mypy seems to think that SimplifiedBool is not assignable to JustInt. I can't explain why. This looks to me like a bug in mypy.

v1: JustInt = SimplifiedBool() # mypy: error, pyright: OK

Let's see if this has to do with __class__. After all, JustInt is overriding object.__class__ in a way that violates Liskov, so maybe that's confusing mypy.

from typing import Protocol

class JustInt2(Protocol):
    @property
    def prop(self) -> type[int]: ...
    @prop.setter
    def prop(self, t: type[int]) -> None: ...

class MyBool2:
    @property
    def prop(self) -> type[bool]: ...
    @prop.setter
    def prop(self, t: type[object]) -> None: ...

v2: JustInt2 = MyBool2() # mypy: error, pyright: OK

Once again, mypy is generating an error here for reasons I can't explain. Maybe it's a bug in mypy's handling of properties, which requires special casing? Let's switch to a non-property getter and setter to see if that results in different behavior.

from typing import Protocol

class JustInt3(Protocol):
    def get_prop(self) -> type[int]: ...
    def set_prop(self, t: type[int]) -> None: ...

class MyBool3:
    def get_prop(self) -> type[bool]: ...
    def set_prop(self, t: type[object]) -> None: ...

v3: JustInt3 = MyBool3() # mypy and pyright: OK

Now mypy agrees that the concrete class meets the requirements of the protocol. So this appears to be a bug in mypy's handling of properties within protocols.

JustInt is overriding object.__class__ in a way that violates Liskov

Given the current typeshed annotations of object.__class__, then that is indeed the case. But they're not quite right:

>>> class A: ...
... 
>>> class B: ...
... 
>>> a = A()
>>> a.__class__
<class '__main__.A'>
>>> a.__class__ = B  # typeshed allows this
>>> a.__class__
<class '__main__.B'>

The return type of the __class__ getter should be the very same as the input type of its setter, so it should be invariant.

class object:
    @property
    def __class__(self, /) -> type[Self]: ...
    @__class__.setter
    def __class__(self, t: type[Self], /): ...

For me it "clicked" when I started looking at __class__ as a regular attribute, albeit one that cannot be deleted, which is why the @property is needed here.
So the Self can be seen as a hidden type-parameter that's invariant w.r.t. object, and that's bound to the most specific object (sub)type.

A nice consequence of this is that it's no longer allowed to do unsafe things such as

fake_int: Just[int] = False  # should be disallowed
fake_int.__class__ = int  # ... because this is unsafe

So not only will this prevent Liskov from getting violent, it'll also prevent the Self from changing.
And as far as I'm concerned, that sounds like a pretty good deal.

I guess I should probably raise this with typeshed.

There's also something that's __class__-specific, here. Because when I do a similar thing with some other property that object doesn't have, then pyright behaves in the same way as mypy:

class HasThingLikeObject[T](Protocol):
    @property
    def thing(self, /) -> type[T]: ...
    @thing.setter
    def thing(self, t: type[object], /) -> None: ...

class HasThing[T](HasThingLikeObject[T], Protocol):
    @property
    def thing(self, /) -> type[T]: ...
    @thing.setter
    def thing(self, t: type[T], /) -> None: ...

class ObjectThing[T]:
    @property
    def thing(self, /) -> type[T]: ...
    @thing.setter
    def thing(self, _: type[object], /) -> None: ...

class IntThing(ObjectThing[int]):
    @property
    def thing(self, /) -> type[int]: ...
    @thing.setter
    def thing(self, _: type[int], /) -> None: ...  # pyright: ignore[reportIncompatibleMethodOverride]

class BoolThing(ObjectThing[bool]):
    @property
    def thing(self, /) -> type[bool]: ...
    @thing.setter
    def thing(self, _: type[bool], /) -> None: ...  # pyright: ignore[reportIncompatibleMethodOverride]

def only_int_things_pls(_: HasThing[int]) -> None: ...

only_int_things_pls(IntThing())   # no errors
only_int_things_pls(BoolThing())  # error in both mypy (arg-type) and pyright (reportArgumentType)

pyright play

The (true positive) error message for just_int("") also suggests that this isn't only a covariance/invariance thing, but that there's something else going on:

Type "int | str" is not assignable to type "int"

So somewhere along the way, the -> type[int] seems to have been changed into -> type[int | str].

edit

pyright/packages/pyright-internal/src/analyzer/binder.ts

Lines 531 to 532 in 294b7af

    
           // Add the implicit "__class__" symbol described in PEP 3135. 
        
           this._addImplicitSymbolToCurrentScope('__class__', node, 'type[self]');

I'm not very familiar with typescript, but perhaps this is what's causing this issue?

This also seems relevant

pyright/packages/pyright-internal/src/analyzer/symbol.ts

Lines 40 to 42 in 294b7af

    
           // Indicates that the symbol is not considered for protocol 
        
           // matching. This applies to some built-in symbols like __class__. 
        
           IgnoredForProtocolMatch = 1 << 6,

Eric Traut (@erictraut)

Eric Traut (@erictraut) would you mind taking another look at this?

As I said above, pyright is working as intended here. Please refer to my analysis above.

Eric Traut (@erictraut)

And did you read my response to it Eric Traut (@erictraut) ? Because there I explain in detail that this isn't about the Liskov substitution principle. This is an issue about a special-casing of __class__ that's causing trouble.
I also posted the the exact code that I think is responsible for this.

Pyright doesn't special-case the class-scoped __class__ symbol. It relies on the typeshed definition. Pyright did special-case this symbol (and a few others) many years ago prior to typeshed adding these definitions. That explains the (stale) comment next to the IgnoredForProtocolMatch flag that you cited above.

The code in the binder that you cited adds an implicit __class__ symbol to a method scope. It handles the following use case. This is unrelated to protocol matching or variance calculation because function-local symbols are not considered for protocol matching.

class Class:
    def method(self):
        # __class__ is an implicit function-scoped symbol available in an instance method
        print(__class__)

In your analysis above, you said that you think there's something specific to __class__. I'm not seeing that. I think you made a mistake in your analysis because you've defined BoolThing as follows:

class BoolThing(ObjectThing[bool]):
    @property
    def thing(self, /) -> type[bool]: ...
    @thing.setter
    def thing(self, _: type[bool], /) -> None: ...  # pyright: ignore[reportIncompatibleMethodOverride]

This doesn't reflect the actual definition of bool.__class__, which uses a type[object] in its setter's signature. If you change the setter for BoolThing.thing to use type[object] rather than type[bool], you'll see behavior that is consistent with __class__.

If you think that the typeshed definition of the __class__ symbol is incorrect, please report the issue to the typeshed maintainers.

In summary, I think my analysis above is correct, and pyright's behavior is both consistent and in conformance with the typing spec.

If you change the setter for BoolThing.thing to use type[object] rather than type[bool], you'll see behavior that is consistent with __class__.

Ah of course; my bad.

If you think that the typeshed definition of the __class__ symbol is incorrect, please report the issue to the typeshed maintainers.

python/typeshed#12997

You see, thereason why I was thinking it was special cased, is the following error message from my first example:

class Just[T](Protocol):  # ~T@Just is invariant
    @property
    def __class__(self, /) -> type[T]: ... 
    @__class__.setter
    def __class__(self, t: type[T], /) -> None: ...

def just_int(x: Just[int], /) -> None:
    assert type(x) is int

just_int("")  # E: Type "int | str" is not assignable to type "int"

And I now realize that this can also be reproduced by adding a StrThing (with _: object as setter param):

Argument of type "StrThing" cannot be assigned to parameter "_" of type "HasThing[int]" in function "only_int_things_pls"
  "StrThing" is incompatible with protocol "HasThing[int]"
    "thing" is an incompatible type
      Type "() -> type[str]" is not assignable to type "() -> type[T@HasThing]"
        "HasThing[T@HasThing]" is not assignable to "StrThing"
        Function return type "type[str]" is incompatible with type "type[T@HasThing]"
          Type "int | str" is not assignable to type "int"
        "HasThing[T@HasThing]" is not assignable to "StrThing"

I still don't really get where that int | part of the error message comes from, but I'm sure you can see how it could be confusing in such cases.
Anyway, thanks for the help :)

Eric Traut (@erictraut)

This is now fixed in typeshed Eric Traut (@erictraut): python/typeshed#13021

Navigation Menu

Repository navigation

Invariant `class: T` inferred as `Self | T` causing false negatives #9410

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	// Add the implicit "__class__" symbol described in PEP 3135.
	this._addImplicitSymbolToCurrentScope('__class__', node, 'type[self]');

	// Indicates that the symbol is not considered for protocol
	// matching. This applies to some built-in symbols like __class__.
	IgnoredForProtocolMatch = 1 << 6,

Invariant __class__: T inferred as Self | T causing false negatives #9410

Description

Activity

erictraut commented on Nov 7, 2024

jorenham commented on Nov 10, 2024

jorenham commented on Nov 10, 2024

jorenham commented on Nov 11, 2024

jorenham commented on Nov 12, 2024

jorenham commented on Nov 12, 2024

erictraut commented on Nov 12, 2024

jorenham commented on Nov 12, 2024

erictraut commented on Nov 12, 2024

jorenham commented on Nov 12, 2024

jorenham commented on Nov 18, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Invariant `class: T` inferred as `Self | T` causing false negatives #9410