Add stub files with unit definitions - #17902

Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.

• Do the proposed changes actually accomplish desired goals?
• Do the proposed changes follow the Astropy coding guidelines?
• Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
• Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
• Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see instructions for rebase and squash.
• Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the "Extra CI" label. Codestyle issues can be fixed by the bot.
• Is a change log needed? If yes, did the change log check pass? If no, add the "no-changelog-entry-needed" label. If this is a manual backport, use the "skip-changelog-checks" label unless special changelog handling is necessary.
• Is this a big PR that makes a "What's new?" entry worthwhile and if so, is (1) a "what's new" entry included in this PR and (2) the "whatsnew-needed" label applied?
• At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate "backport-X.Y.x" label(s) before merge.

I also think that these stubs belong in astropy itself. The generation script should probably be included in the repo (if not distributed) too.

is it really not possible to generate these files on the user side (on first import)?

This sounds like a build step, I do not see the benefit of deferring it to import time.

This sounds like a build step, I do not see the benefit of deferring it to import time.

To me, it is more similar to how pyc files are created on first import.

To me, it is more similar to how pyc files are created on first import.

pyc files are system specific. stub files are not. It's more like how the unit lexers are built.

To me, it is more similar to how pyc files are created on first import.

With an important differences that we would be responsible for maintaining this mechanism, and that it could make installation less reproducible and harder to debug. I'd much prefer we ship identical stub files to everyone and avoid uncertainty in that area.

OK, makes sense to not "roll our own" -- though the lexers have moved away from pre-generated files to just reconstructing them on the fly (we just haven't caught up).

My real issue remains with mypy (or other type checkers) not being able to provide a default type. But perhaps my issue will lead to someone answering that there is actually a mechanism that we're simply not aware of. Isn't it true that if we always got units from a dict[str, Unit], that then it would be fine?

Anyway, this may be unlikely, and I don't want to hold us all up waiting for a magical answer. For sure, though, we need the generator functions included and have tests for them.

Somewhat of an aside, but relevant: I've been wondering whether it wouldn't make sense to create the u. unit attributes on-demand using __getattr__ rather than always defined, given that most units are never used. (In interactive use, the first tab completion would then be a bit slower, of course, as __dir__ would have to generate the units.)

In part, this comes about because of something @cdeil mentioned way back, that he generally uses the string versions of units. I.e., perhaps the recommended way of using units should be to do from astropy.units importing Unit as U and then use U("km/s2") over u.km / u.s**2 throughout - it is more readable and would be faster if we had a bit of caching on U (the cache doesn't even have to be all that large, as most code does not use an enormous number of units).

EDIT: another reason was that for a little while, it seemed like it might become possible to define one's own string types, so one could have U"m/s2" - though that is now gone, in favour of a new template string class - see https://peps.python.org/pep-0750/

My real issue remains with mypy (or other type checkers) not being able to provide a default type.

Do you mean that whenever type checkers encounter some attribute they don't know about they should assume it has some default type? If that would be the case then type checkers would not be able to raise errors when an invalid attribute is accessed (e.g. because of a typo).

Isn't it true that if we always got units from a dict[str, Unit], that then it would be fine?

Type checkers would indeed know about the types of the values of the dictionary, but they would lose the ability to check the keys. Unless you use TypedDict, but that wouldn't be any less verbose than the stub file.

Somewhat of an aside, but relevant: I've been wondering whether it wouldn't make sense to create the u. unit attributes on-demand using __getattr__ rather than always defined, given that most units are never used. (In interactive use, the first tab completion would then be a bit slower, of course, as __dir__ would have to generate the units.)

Have thought about how Sphinx would know what units to include in the documentation?

It should be pointed out that the stub files are not just for type checkers. They also tell IDEs what units exist so that IDEs can provide autocompletion. Dynamic approaches do not allow for that.

In part, this comes about because of something @cdeil mentioned way back, that he generally uses the string versions of units. I.e., perhaps the recommended way of using units should be to do from astropy.units importing Unit as U and then use U("km/s2") over u.km / u.s**2 throughout - it is more readable and would be faster if we had a bit of caching on U (the cache doesn't even have to be all that large, as most code does not use an enormous number of units).

What's the problem with using unit instances? Is it that the leading u. is distracting? It is already possible to avoid:

>>> from astropy.units import km, s
>>> km / s**2
Unit("km / s2")

Another question, more for us: is it really not possible to generate these files on the user side (on first import)?

I don't like the idea of generating the stubs on first import, but I did consider generating the stubs during installation. In particular, I thought that we could introduce astropy/units/setup_package.py, but our documentation says

Is an answer on my mypy issue (see python/mypy#18813 (comment)) something that might work?

Does module-level __getattr__ work for you?

I can see one could annotate that appropriately, if everything else that could possibly come from the module is already defined. And it would basically implement my "on demand" suggestion. But I'm not by any means well-versed enough in the typing stuff to be sure.

EDIT: really asking just about the typing now, i.e., leaving aside for now how to deal with sphinx, or whether this is super practical in general.

Hmm, this seems very useful!!!

on Mar 18, 2025
Collaborator

You can lie in the stub and have a getattr in the .pyi file even if that's not the strategy you take to creating attributes at runtime in the .py file

Thinking more about this PR, which is moving well beyond just annotating (which is useful independently of whether the typing is 100% correct), I think it and similar ones should start meeting the bar for code contributions, i.e., they should come with proper typing test cases and documentation. This is particularly important since it seems there are multiple ways to implement the typing for all units, and we should be able to make changes knowing we don't break things.

As a point of reference, I noticed that in numpy, they have a specific typing/tests directory, which includes checks for particular failures (see https://github.com/numpy/numpy/tree/main/numpy/typing/tests/data/fail). Something like that may work for us too (perhaps on a subpackage basis).

Anyway, bottom line, I think this PR will need to wait on #16312 (or an equivalent)

I don't know how we could automatically generate the stubs if we can't use setup_package.py for it.

I think it might work if we went with something like the units definition file that pint seems to use. Though that breaks then being able to use constants, which seems a show stopper (or, rather, would require defining a mini-language in the input file that might as well just be plain python, which is what we have...).

You can lie in the stub and have a getattr in the .pyi file even if that's not the strategy you take to creating attributes at runtime in the .py file

That seems very useful!

You can lie in the stub and have a getattr in the .pyi file even if that's not the strategy you take to creating attributes at runtime in the .py file

That seems very useful!

Not sure it is that useful. Importing a unit that does not exist should give a type error. Similar to documentation: "wrong documentation is worse than no documenation"

You can lie in the stub and have a getattr in the .pyi file even if that's not the strategy you take to creating attributes at runtime in the .py file

That seems very useful!

Not sure it is that useful. Importing a unit that does not exist should give a type error. Similar to documentation: "wrong documentation is worse than no documenation"

I guess the converse is really that the (current!) type system is not advanced enough to autogenerate files on the fly, so one gets stuck with shipping 100kB files just for the typing. But surely that will come, and in the meantime perhaps we can keep things simple for modules where we know exactly what is possible.

I guess the converse is really that the (current!) type system is not advanced enough to autogenerate files on the fly, so one gets stuck with shipping 100kB files just for the typing. But surely that will come, and in the meantime perhaps we can keep things simple for modules where we know exactly what is possible.

What do you mean by the current type system: python type hints in general or type system of astropy

The point is that with __getattr__ in the .pyi we don't need to manually enumerate all the units.
It can be dynamically (in the pyi) generated based on simple rules. That should significantly shorten the pyi files in this PR.

E.g. (pseudocode)

main_units = []
prefixes = []
def __getattr__(name):
     mu, pre = parse_name(name)
     if mu in main_units....
     if pre in prefixes
     return the_inferred_type

So this is potentially immensely useful for reducing 5k lines of code diff down to a much smaller number.

The point is that with __getattr__ in the .pyi we don't need to manually enumerate all the units. It can be dynamically (in the pyi) generated based on simple rules. That should significantly shorten the pyi files in this PR.

E.g. (pseudocode)

main_units = []
prefixes = []
def __getattr__(name):
     mu, pre = parse_name(name)
     if mu in main_units....
     if pre in prefixes
     return the_inferred_type

So this is potentially immensely useful for reducing 5k lines of code diff down to a much smaller number.

AFAIK .pyi files are not executed so you can not do anything dynamically in a .pyi file. The only thing you can do is

def __getattr__(name) -> Unit: ...

So basically, every import from that module (except explicit definitions with a different type) will be considered valid Unit objects. In reality (.py) not every name exists as a Unit. So you could import something like notaunit and the type checker would not show it as an error.

But I could be wrong and would be happy to learn that there is a way.

@fritz1005 - I think you right that we cannot do any type of calculations in the .pyi file - which is the kind of limitation I meant with the current type system - there doesn't seem any deep reason why type checkers should not be able to auto-generate the types on the fly - they have to deal with far more complicated things than this!

Anyway, personally I think for our initial goal of getting typing more or less set up, it is fine to not worry about u.<something_meaningless>. Being able to tell generally that u.<something> is a unit is not that different from telling that Unit("something") is a unit -- and it seems OK for now if the type system cannot tell that it is a valid unit in either case.

We are using type hints for about 3 years now in the company I work at, and I think it is essential for large codebases worked on by many developers. Basically everyone has the same experience, at first some reservation, then the learning curve and at the end nobody could live without it anymore with everyone contributing to add type hints wherever possible. So sure, it has its limitations, but it also already has so many substantial benefits. Some of its limitiations are really the same as in statically typed languages, so they are only limitations because we are used to the dynamic nature of python. While that is useful in many instances, in my experience it is best to have short distances of dynamic types between anchors of static types, to not have to go to far up or down the stack to know what type we are actually looking at.

That being said, I think having the (prefixed) unit instances addressed by a static __getattr__ as you suggest is probably an acceptable way (IMO) to get things up an running.

And I understand the concern about bloating the library with type hints. Maybe an alternative could be to move the autogenerated type hints to https://github.com/python/typeshed for now?

Anyways, thx for the discussion

Offtopic but a bit more context:
In our company we control a highly complex physical system using python and ATM we use a different library for physical units throughout our code base. We are not entirely happy with the current library and are actively looking for alternatives and evaluating the available options. Type hints is one of the major criteria for us, that is why I am so interested in this topic, because other than type hints astropy seems like a good option. But TBH type hints aren't great for any of the alternatives as well, so it might not be the deciding factor.

Or could we revisit generating the pyi files as part of the release process? It should be automatable with @eerovaher's script.

Given @eerovaher's very pessimistic #17918 (comment), I now really see no point in adding stub files to either the repository or to releases and all the more reason for just adding a stub (EDIT: with __getattr__) that is good enough to make parts that use units work, even if it means they don't get errors for u.mistyped_unit.

If the main problem with these stubs is their size then why don't we create the astropy-stubs package?

I'm fine with a separate package, which, say, astropy[typing] can depend on. It also means that development can be quicker, perhaps (certainly wouldn't have me holding things up!)

Just to be clear here, I still think we should use inline annotations as much as possible. The astropy-stubs package I have in mind would contain only the objects that cannot be annotated inline because they are generated dynamically at runtime.

Hi humans 👋 - this pull request hasn't had any new commits for approximately 4 months. I plan to close this in 30 days if the pull request doesn't have any new commits by then.

In lieu of a stalled pull request, please consider closing this and open an issue instead if a reminder is needed to revisit in the future. Maintainers may also choose to add keep-open label to keep this PR open but it is discouraged unless absolutely necessary.

If this PR still needs to be reviewed, as an author, you can rebase it to reset the clock.

If you believe I commented on this pull request incorrectly, please report this here.

I'm going to close this pull request as per my previous message. If you think what is being added/fixed here is still important, please remember to open an issue to keep track of it. Thanks!

If this is the first time I am commenting on this issue, or if you believe I closed this issue incorrectly, please report this here.