Skip to content

Comments

[red-knot] Remove Type::Unbound#13980

Merged
sharkdp merged 45 commits intomainfrom
david/remove-type-unknown
Oct 31, 2024
Merged

[red-knot] Remove Type::Unbound#13980
sharkdp merged 45 commits intomainfrom
david/remove-type-unknown

Conversation

@sharkdp
Copy link
Contributor

@sharkdp sharkdp commented Oct 29, 2024

Summary

  • Remove Type::Unbound
  • Handle (potential) unboundness as a concept orthogonal to the type system (see new Symbol type)
  • Improve existing and add new diagnostics related to (potential) unboundness

closes #13671

Test Plan

  • Update existing markdown-based tests
  • Add new tests for added/modified functionality

@MichaReiser MichaReiser added the ty Multi-file analysis & type inference label Oct 29, 2024
@sharkdp sharkdp changed the title Remove Type::Unbound [red-knot] Remove Type::Unbound Oct 29, 2024
@sharkdp sharkdp force-pushed the david/remove-type-unknown branch from 9d7b8dd to 8e97da6 Compare October 30, 2024 09:25
@sharkdp sharkdp force-pushed the david/remove-type-unknown branch from edb4d6e to 0175a1b Compare October 30, 2024 16:36
@github-actions
Copy link
Contributor

github-actions bot commented Oct 30, 2024

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

@sharkdp sharkdp force-pushed the david/remove-type-unknown branch from 7468f67 to 54998b3 Compare October 30, 2024 19:55
@codspeed-hq
Copy link

codspeed-hq bot commented Oct 30, 2024

CodSpeed Performance Report

Merging #13980 will improve performances by 4.17%

Comparing david/remove-type-unknown (13bbeb2) with main (d1189c2)

Summary

⚡ 1 improvements
✅ 31 untouched benchmarks

Benchmarks breakdown

Benchmark main david/remove-type-unknown Change
red_knot_check_file[cold] 66.1 ms 63.4 ms +4.17%

Copy link
Contributor

@carljm carljm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking really good! I probably gave this a more in-depth review than you were looking for, but hopefully the comments are useful! Let me know if you had higher-level questions I failed to answer in my comments.

@sharkdp sharkdp force-pushed the david/remove-type-unknown branch from a933cf9 to 39c1906 Compare October 31, 2024 09:58
Copy link
Member

@AlexWaygood AlexWaygood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't looked at this closely yet, but my overall impression is that this is really exciting! The overall shape looks great. I think it's going to force us to write much more explicit and accurate code.

Copy link
Member

@MichaReiser MichaReiser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. Do we understand where the perf improvement is coming from? Is it because we now create fewer union types?

Comment on lines 84 to 89
fn as_type(&self) -> Option<Type<'db>> {
match self {
Symbol::Type(ty, _) => Some(*ty),
Symbol::Unbound => None,
}
}
Copy link
Member

@AlexWaygood AlexWaygood Oct 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know that @MichaReiser asked for this method and @carljm liked Micha's comment... I worry that this blurs the line between bound symbols and possibly-but-not-necessarily bound symbols. In some ways, possibly-unbound symbols are more similar to unbound symbols than they are similar to always-bound symbols, in that attempts to read them should always cause us to emit a diagnostic. The fact that we currently have a blurred line between possibly-bound and always-bound symbols is exactly what causes the bugs outlined in #14012.

If there are situations where we really need to ignore that a symbol might possibly be unbound, I think I'd honestly prefer to be explicit about that by manually matching on the variants, rather than using this method that collapses it into a binary "either you have a Type, or you don't" enum.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(I agree with what you said in standup earlier about merging this PR sooner rather than later to avoid merge conflicts. Don't let this comment block that from happening! This can always be tackled as a followup.)

Copy link
Contributor Author

@sharkdp sharkdp Oct 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually had the same thought. And considered calling this function something like Symbol::ignore_possibly_unbound instead of the more innocent Symbol::as_type. I'll think about it and probably do a follow-up.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mentioned this in another comment, but this issue isn't specific to as_type. Basically all of the methods on Symbol currently ignore possibly-unboundness in some form.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in that attempts to read them should always cause us to emit a diagnostic

I don't think this is always true. For example, if __iadd__ is possibly unbound we should union its return type with the return type from __add__, and only emit a diagnostic if __add__ doesn't exist.

But I do agree that callers should be the ones make the decision about suppressing such diagnostics, and the APIs should be very clear if they do that.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is always true. For example, if __iadd__ is possibly unbound we should union its return type with the return type from __add__, and only emit a diagnostic if __add__ doesn't exist.

Right, that's a useful correction. But even here, we still need to account for the possibly-unbound nature and treat it differently to the case where it was definitely bound.

Comment on lines +47 to +91
pub(crate) fn unwrap_or(&self, other: Type<'db>) -> Type<'db> {
match self {
Symbol::Type(ty, _) => *ty,
Symbol::Unbound => other,
}
}

pub(crate) fn unwrap_or_unknown(&self) -> Type<'db> {
self.unwrap_or(Type::Unknown)
}

pub(crate) fn as_type(&self) -> Option<Type<'db>> {
match self {
Symbol::Type(ty, _) => Some(*ty),
Symbol::Unbound => None,
}
}

#[cfg(test)]
#[track_caller]
pub(crate) fn expect_type(self) -> Type<'db> {
self.as_type()
.expect("Expected a (possibly unbound) type, not an unbound symbol")
}

#[must_use]
pub(crate) fn replace_unbound_with(
self,
db: &'db dyn Db,
replacement: &Symbol<'db>,
) -> Symbol<'db> {
match replacement {
Symbol::Type(replacement, _) => Symbol::Type(
match self {
Symbol::Type(ty, Boundness::Bound) => ty,
Symbol::Type(ty, Boundness::MayBeUnbound) => {
UnionType::from_elements(db, [*replacement, ty])
}
Symbol::Unbound => *replacement,
},
Boundness::Bound,
),
Symbol::Unbound => self,
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do think as a follow-up (not in this PR) we should carefully consider the naming of all these APIs.

Currently unwrap_or, unwrap_or_unknown, and as_type all three ignore possibly-unboundness of this symbol (the "other" type in the unwrap methods is only used in case of full-unboundness). And replace_unbound_with ignores possibly-unboundness of the replacement symbol.

I'm not totally confident that any of these choices are right (or if they are right, perhaps the name should clarify the behavior better), but it would require analysis of all the usage sites to reconsider how we should name/structure the APIs. Given this PR generally seems to maintain current behavior reasonably well, I think it makes sense to merge it now to avoid rebase conflicts and then consider this as follow-up.

Comment on lines 74 to 76
Some(Symbol::Type(ty, boundedness)) => Symbol::Type(
declarations_ty(db, declarations, Some(ty)).unwrap_or_else(|(ty, _)| ty),
boundedness,
Copy link
Contributor

@carljm carljm Oct 31, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think what boundness value we should apply to a symbol that is declared in some paths (but not all) and bound in some paths (but not all) is pretty tricky and requires further attention in future. Right now here we are always just using the boundness from bindings and ignoring declarations for purposes of boundness, but that's weird, because we only even look at bindings if the symbol may be undeclared, which means in this example:

x: int

if flag:
    y: int
else
    y = 3

If we import from this module, we will currently report x as a definitely-bound symbol (even though it has no bindings at all!) but report y as possibly-unbound (even though every path has either a binding or a declaration for it.)

Possibly the behavior here should differ depending on whether we are importing from a stub or a regular Python module? But an argument could be made that even in a regular Python module, if it declares x: int at module level, we should just trust that and not require that we can see the binding of it.

This is not for this PR to resolve (the conclusion may mean starting to include declaredness info in Symbol as well), but it may be worth a TODO comment here.

@AlexWaygood
Copy link
Member

AlexWaygood commented Oct 31, 2024

Oh shoot, sorry for merging #14016 :(

It didn't even occur to me that this would give you more tests to update. I won't merge anymore red-knot PRs until this lands! 😄

Edit: And... it looks like you magically fixed a bunch of the TODOs I added in those tests? Thank you!!

@sharkdp sharkdp force-pushed the david/remove-type-unknown branch from 481b2f4 to 13bbeb2 Compare October 31, 2024 18:51
@sharkdp sharkdp merged commit 53fa32a into main Oct 31, 2024
@sharkdp sharkdp deleted the david/remove-type-unknown branch October 31, 2024 19:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ty Multi-file analysis & type inference

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[red-knot] remove Type::Unbound

4 participants