typing stubs for plugin ignore submodule imports?!?

Hello Pylance Crew!

I was trying to create typing stubs for a Python plugin in our app to ease writing code for it in VS Code.

Now this plugin loads its submodules right away. Similar to Pythons xml: See typeshed here.
So my setup looks like this:

myplugin
- __init__.pyi
- submodule.pyi

Now in the __init__.pyi I tried the same as xml does but that yields some problems so I tried some more one by one:

from myplugin import submodule
import myplugin.submodule as submodule
import submodule
import .submodule
from . import submodule

Finally from . import submodule is the right form that works here. But the others give varying problems. That's why I still want to report my findings to you:

from myplugin import submodule:
❌ highlighted as "not a known member"
❌ No Declaration found
✔ code "WOULD" actually work
import myplugin.submodule as submodule:
❌ highlighted as "not a known member"
✔ Go-To-Declaration WORKS!
✔ code "WOULD" actually work
import submodule / import .submodule
❌ highlighted as "not a known member"
✔ Go-To-Declaration WORKS!
❌ code would actually NOT work
from . import submodule
✔ highlighted works. auto-complete works!
✔ Go-To-Declaration WORKS!
✔ code "WOULD" actually work

So my initial issue is "solved" but:

I just scratch my head seeing this inconsistency between working code, working highlighting and working Go-To-functionality 🤔
How should the xml thing be resolved? Their code would actually break if you tried ~~import xml\nprint(xml.parsers)~~ I can make a report over there.

Thanks for posting the question.

One piece of information that you didn't provide is how you intend for consumers of your library to import symbols from the submodule myplugin.submodule. Let's say that submodule exports a symbol called foo. Most libraries would support the following ways to access foo:

# Explicit import of the submodule
import myplugin.submodule
myplugin.submodule.foo

# Explicit import of the symbol from the submodule
from import myplugin.submodule import foo
foo

# Assumes that foo is re-exported from top-level module
import myplugin
myplugin.foo

# Assumes that foo is re-exported from top-level module
from import myplugin import foo
foo

If I understand you correctly, you want to support another (less standard) way to access foo as follows:
5.

# Assumes that submodule is re-exported from top-level module
import myplugin
myplugin.submodule.foo

Do I have that correct? If so, that's legitimate, but it's already a bit off the beaten track compared to most packages.

Here's why you're seeing the behavior that you reported above.

PEP 484 defines the use of type stubs (".pyi" files). It indicates that symbols imported by a stub are considered "private" to that module unless explicitly re-exported.

Here's the exact text from PEP 484:

Modules and variables imported into the stub are not considered exported from the stub unless the import uses the import ... as ... form or the equivalent from ... import ... as ... form. (UPDATE: To clarify, the intention here is that only names imported using the form X as X will be exported, i.e. the name before and after as must be the same.)

If your intent is to export the submodule symbol from the myplugin module, your options are therefore:
from . import submodule as submodule
import submodule as submodule

Pyright (which is the static type checker that underlies pylance) follows PEP 484 rules for exported symbol visibility. This is why you are seeing the "not a known member" error messages.

The "Go-To-Declaration" functionality in pylance can be (and in practice is) a bit more permissive since it isn't specifically bound to PEP 484 rules. It will take you to the declaration even if the type stub indicates that it shouldn't be exported.

There are a number of issues that you've uncovered here:

In your first formulation (from myplugin import submodule), pylance is not able to find a declaration. This is an unintended inconsistency — one that I would consider a bug. It's an easy one to fix, so I've gone ahead and fixed it in the pyright sources.
As I mentioned, pylance allows you to "go to declaration" in cases even if the type stub indicates that the symbol shouldn't be exported. I can understand the argument that this is inconsistent. Perhaps in all of these cases, we should change it to refuse to go to the declaration. This will require some discussion because there are advantages and disadvantages of making such a change. I think we're likely to keep it as is, but it's worth discussing.
The form of import used in xml/__init__.pyi is import xml.parsers as parsers. This is not considered a "redundant alias form" by pyright, and hence it isn't treated as an export. The text from PEP 484 is unfortunately somewhat ambiguous about this particular case. It appears that mypy has chosen to treat this as a redundant form as long as the last part of the dotted name is the same as the identifier after the as. In other words, import A.B.C.X as X is treated as though it meets the requirements in PEP 484 and is therefore considered "exported" from the module. I've filed the following work item to address this in pyright: import A.B.C.X as X should be considered a redundant form in type stub pyright#2663.

Whow I didn't expect this to unfold such a bouquet of things! Thanks for the in-dept write up!🙏

So yes: The 5. type is the one I'd need to reflect in the stubs. (less standard yes. Unfortunately I'm not in charge of the plugin in question. I would not have made it that way)

But ... the form from . import submodule already satisfies all the requirements!

I wonder about these:
from . import submodule as submodule
import submodule as submodule
I was schooled this was rather ugly style because renaming with as does not change a thing 🤨
In Python runtime import x.y as y and from x import y yield equivalent ys, right? Is this or is it not different in stubs?

Ah ☝ next bullet in PEP484

However, as an exception to the previous bullet, all objects imported into a stub using from ... import * are considered exported. (This makes it easier to re-export all objects from a given module that may vary by Python version.)

Unlike most modern languages, Python provides no keyword to specify which symbols are meant to be exported from a module and which are meant to be internal to a module. As such, the authors of PEP 484 needed to come up with a way to differentiate between the two in type stubs. The section I quoted above is what they decided upon. It may look ugly, but it gets the job done. :)

The form from . import submodule is technically not covered by PEP 484 rules, but pyright and other type checkers treat it as an extension to the PEP 484 rules because there's really no reason one would use this form in a type stub unless the intent was to re-export the module submodule. If you want to comply with official PEP 484 rules, you would use from . import submodule as submodule.

Aaaah the key is the * there! Which does not mean the same as ... but literally __all__ from given module.
Alright got you. Thanks again.

We've decided not to make the change associated with point 3 above. Based on further consultation with maintainers of typeshed and the coauthor of PEP 484, I think that the current behavior of pyright is correct. The xml/__init__.pyi stub should probably be changed accordingly.

Navigation Menu

Repository navigation

typing stubs for plugin ignore submodule imports?!? #2142

Hello Pylance Crew!

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

typing stubs for plugin ignore submodule imports?!? #2142

Description

Hello Pylance Crew!

Activity

erictraut commented on Dec 6, 2021

ewerybody commented on Dec 6, 2021

ewerybody commented on Dec 6, 2021

erictraut commented on Dec 6, 2021

ewerybody commented on Dec 6, 2021

erictraut commented on Dec 7, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions