Skip to content

Comments

[red-knot] Method calls and the descriptor protocol#16121

Merged
sharkdp merged 23 commits intomainfrom
david/descriptor-protocol
Feb 20, 2025
Merged

[red-knot] Method calls and the descriptor protocol#16121
sharkdp merged 23 commits intomainfrom
david/descriptor-protocol

Conversation

@sharkdp
Copy link
Contributor

@sharkdp sharkdp commented Feb 12, 2025

Summary

This PR achieves the following:

  • Add support for checking method calls, and inferring return types from method calls. For example:
    reveal_type("abcde".find("abc"))  # revealed: int
    reveal_type("foo".encode(encoding="utf-8"))  # revealed: bytes
    
    "abcde".find(123)  # error: [invalid-argument-type]
    
    class C:
        def f(self) -> int:
            pass
    
    reveal_type(C.f)  # revealed: <function `f`>
    reveal_type(C().f)  # revealed: <bound method: `f` of `C`>
    
    C.f()  # error: [missing-argument]
    reveal_type(C().f())  # revealed: int
  • Implement the descriptor protocol, i.e. properly call the __get__ method when a descriptor object is accessed through a class object or an instance of a class. For example:
    from typing import Literal
    
    class Ten:
        def __get__(self, instance: object, owner: type | None = None) -> Literal[10]:
            return 10
    
    class C:
        ten: Ten = Ten()
    
    reveal_type(C.ten)  # revealed: Literal[10]
    reveal_type(C().ten)  # revealed: Literal[10]
  • Add support for member lookup on intersection types.
  • Support type inference for inspect.getattr_static(obj, attr) calls. This was mostly used as a debugging tool during development, but seems more generally useful. It can be used to bypass the descriptor protocol. For the example above:
    from inspect import getattr_static
    
    reveal_type(getattr_static(C, "ten"))  # revealed: Ten
  • Add a new Type::Callable(…) variant with the following sub-variants:
    • Type::Callable(CallableType::BoundMethod(…)) — represents bound method objects, e.g. C().f above
    • Type::Callable(CallableType::MethodWrapperDunderGet(…)) — represents f.__get__ where f is a function
    • Type::Callable(WrapperDescriptorDunderGet) — represents FunctionType.__get__
  • Add new known classes:
    • types.MethodType
    • types.MethodWrapperType
    • types.WrapperDescriptorType
    • builtins.range

Performance analysis

On this branch, we do more work. We need to do more call checking, since we now check all method calls. We also need to do ~twice as many member lookups, because we need to check if a __get__ attribute exists on accessed members.

A brief analysis on tomllib shows that we now call Type::call 1780 times, compared to 612 calls before. We also emit 18 additional diagnostics, which might also explain some of the performance regression.

I did some wall time benchmarks as well and only find a minor 1% regression on black, and a 6% regression on tomllib.

black

Command Mean [ms] Min [ms] Max [ms] Relative
./red_knot_feature check --project ~/black 184.1 ± 8.1 166.1 193.2 1.01 ± 0.07
./red_knot_main check --project ~/black 182.7 ± 10.1 164.1 194.6 1.00

tomllib

Command Mean [ms] Min [ms] Max [ms] Relative
./red_knot_feature check --project ~/tomllib 24.6 ± 0.9 22.7 26.7 1.06 ± 0.06
./red_knot_main check --project ~/tomllib 23.2 ± 1.1 21.3 29.0 1.00

Limitations

  • Data descriptors are not yet supported, i.e. we do not infer correct types for descriptor attribute accesses in Store context and do not check writes to descriptor attributes. I felt like this was something that could be split out as a follow-up without risking a major architectural change.
  • We currently distinguish between Type::member (with descriptor protocol) and Type::static_member (without descriptor protocol). The former corresponds to obj.attr, the latter corresponds to getattr_static(obj, "attr"). However, to model some details correctly, we would also need to distinguish between a static member lookup with and without instance variables. The lookup without instance variables corresponds to find_name_in_mro here. We currently approximate both using member_static, which leads to two open TODOs. Changing this would be a larger refactoring of Type::own_instance_member, so I chose to leave it out of this PR.

Please let me know if you would like to see any or both of these limitations be solved in this PR instead.

Test Plan

  • New call/methods.md test suite for method calls
  • New tests in descriptor_protocol.md
  • New call/getattr_static.md test suite for inspect.getattr_static
  • Various updated tests

@sharkdp sharkdp added the ty Multi-file analysis & type inference label Feb 12, 2025
@sharkdp sharkdp force-pushed the david/descriptor-protocol branch 6 times, most recently from f8bf60f to 00d04b1 Compare February 17, 2025 15:22
@codspeed-hq
Copy link

codspeed-hq bot commented Feb 17, 2025

CodSpeed Performance Report

Merging #16121 will degrade performances by 6.88%

Comparing david/descriptor-protocol (05d689d) with main (f62e540)

Summary

❌ 1 (👁 1) regressions
✅ 31 untouched benchmarks

Benchmarks breakdown

Benchmark BASE HEAD Change
👁 red_knot_check_file[incremental] 5.1 ms 5.5 ms -6.88%

@github-actions

This comment was marked as resolved.

@sharkdp sharkdp marked this pull request as ready for review February 18, 2025 11:31
@sharkdp sharkdp marked this pull request as draft February 18, 2025 19:47
Copy link
Contributor

@carljm carljm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking pretty good to me! Left a few comments.

@sharkdp sharkdp force-pushed the david/descriptor-protocol branch 2 times, most recently from 3056b98 to e3f2ad2 Compare February 19, 2025 14:30
@sharkdp sharkdp marked this pull request as ready for review February 19, 2025 15:01
@sharkdp sharkdp force-pushed the david/descriptor-protocol branch from e3f2ad2 to e61e653 Compare February 19, 2025 15:06
@sharkdp sharkdp force-pushed the david/descriptor-protocol branch from e61e653 to d84b1fe Compare February 19, 2025 19:21
@sharkdp sharkdp force-pushed the david/descriptor-protocol branch 2 times, most recently from 7e90171 to 92416e1 Compare February 20, 2025 18:13
@sharkdp sharkdp force-pushed the david/descriptor-protocol branch from 41d1362 to 05d689d Compare February 20, 2025 22:07
@sharkdp sharkdp merged commit d2e034a into main Feb 20, 2025
21 checks passed
@sharkdp sharkdp deleted the david/descriptor-protocol branch February 20, 2025 22:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ty Multi-file analysis & type inference

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants