Repository navigation
Consider making iers.Conf.auto_max_age longer and bunding IERS-A with astropy #14756
Description
What is the problem this feature will solve?
Users and other institutional users report continued problems related to attempted auto-downloads of the IERS-A file for doing calculations that require these data (e.g. predictive UT1 time calculations). The upstream site(s) are still not as reliable as we would like.
Describe the desired outcome
One proposal we converged on during a breakout session at the Astropy Coordination meeting 2023 is two-fold:
- Bundle the IERS-A file with the astropy release.
- For consideration: update with bug-fix releases as well as minor (6-month) feature releases.
- Question: bundle this and the IERS-B as gzipped files and unzip to cache? These files compress by a factor of 5 or more,
- Change
utils.iers.Conf.auto_max_ageto a value around 300 days (TBR).- This ensures that if a user stays up to date with the latest astropy release then they never need to download IERS-A.
This does change the promised accuracy of astropy with regards to time/coordinates that depend on predictive IERS-A values. Instead of the IERS-A always being recent to within 30 days, it might be as much as 300 days.
cc: @eteq @tepickering @weaverba137 @mhvk @tomdonaldson
Additional context
No response
Activity
weaverba137 commentedon May 4, 2023
@taldcroft suggested that I add an additional discussion item. In certain environments it is necessary to avoid downloads entirely, and there are configuration items to handle that case. I've written a function that bundles all of the configuration item changes necessary to deactivate downloads and use a one-time downloaded file that can be bundled into your own package, whenever you need it, independent of the one bundled with Astropy. I suggest this can be upstreamed so that more groups can use it. See https://github.com/desihub/desiutil/blob/main/py/desiutil/iers.py
astrofrog commentedon May 4, 2023
Can't we simply have automatic regular updates like for the other IERS files? I would like to keep this decoupled from the release process if possible.
What would be the main benefit of this? I think git compresses files, and wheels and sdists are compressed too, so not sure it's worth it? In fact, compressing them would paradoxially increase the size of the git repo as the diffs wouldn't be small anymore - each time the gzipped IERS files were updated the git diffs would need to include the whole file.
astrofrog commentedon May 4, 2023
Another idea that just occurred to me - why don't we distribute the IERS stuff in a separate Python package from astropy? (e.g.
astropy-iers). This could be a very simple package but with very frequent and fully automated releases (using e.g. calendar versioning) that could be very low effort once set up. But this means that updating the IERS files just means updating a pip package, and each astropy release could bump the version of the IERS package that it requires. And decoupling this package from the astropy release schedule could have benefits.This would make it easy for both users who can't have internet access once packages are set up, as well as users needing high accuracy (as they can easily pip install the latest update to that package).
pllim commentedon May 5, 2023
Re: #14756 (comment) -- @astrofrog , do you mean both IERS-A and IERS-B in this case?
pllim commentedon May 5, 2023
taldcroft commentedon May 5, 2023
If it is technically feasible with available resources and sustainable, then the idea of breaking out the IERS stuff (either just the data or data + code?) and making that a required dependence seems reasonable. Another ERFA?
astrofrog commentedon May 5, 2023
@pllim - yes I mean all the data files that would be in astropy.utils.iers (IERS A and B and Leap Second). The package could literally just be the data files and a way to access their file paths and we would simply auto update the files regularly, then also auto-release regularly. I think this would be the lowest maintenance burden.
We could also move out all of the astropy.utils.iers code into a separate package but since that depends on other parts of astropy it might be weird to have a circular dependency between that package and astropy. It could be done though.
taldcroft commentedon May 5, 2023
@astrofrog - the downside of making a new IERS package is that it does seem like a lot of up-front work. This includes making a brand new package and putting in place all the infrastructure to keep it auto-updated, auto-builds to wheels and conda packages. There is also an unspecified maintenance burden as those things inevitably break from upstream changes. Can you commit to doing that upfront work (as part of Aperio funded maintenance) by the next astropy release?
The original proposal can be done in an hour or two. If we limit the scope to just updating the files for feature releases following current process, it seems like the incremental change in release effort (updating two files instead of one) is fairly small.
astrofrog commentedon May 5, 2023
Just to clarify, we no longer update the files during the release process - instead this is now automated and an automated PR is now opened monthly with the files - see e.g. #14666 - so implementing the incremental update of the IERS A file will be very little effort and essentially zero effort once it is set up.
The downside is that even compressed, those files are close to 2Mb, which is a non-negligible fraction of the total size of the astropy wheels (7-11Mb depending on platform). IERS A would be about half of this.
If we did the approach of just having the data files in the package, then this would take a couple of hours to set up the package, and a couple of hours to set up the automated updating and releasing (I would propose that the updates would be done via a PR to at least get a pair of eyes, but that merging the PR would then automatically trigger a new release to PyPI, which is easy to do), so 4-5 hours in total. This would also include setting up a simple CI job that ensures that astropy will continue to work as-is with the latest update of the files to guard against any e.g. change in data format and so on.
With this approach, this would simply replace having the files live in astropy but if the files are too old we would still want to make sure astropy downloads the latest version automatically if the max age is exceeded. On the plus side, it means that we can tell people that if they want to avoid any downloads of latest versions, they can simply pip/conda update
astropy-iers-datato the latest version.We would probably still want automated PRs to astropy core to continuously bump the minimum version of the
astropy-iers-datapackage, but that can be done easily by just adapting the existing automated update of the IERS files.astrofrog commentedon May 5, 2023
I forgot to answer your original question - yes I would be happy to do the work either way as part of Aperio funded maintenance work.
taldcroft commentedon May 5, 2023
Thanks @astrofrog for the details! Making a separate data package
astropy-iers-dataseems like a win on all fronts.I'm planning to move the
iers.Conf.auto_max_agequestion to a new issue and make this just for the de-bundling conversation.weaverba137 commentedon May 5, 2023
I support the idea of a separate package that is updated regularly. This would eliminate the need for some of the functionality proposed in #14757, but I think we would still want to think about either making
iers.Conf.auto_max_agelonger by default, or adding the function suggested in #14757, but only the parts that makeiers.Conf.auto_max_agelonger and suppress warnings. That is, it would no longer be necessary to disable auto-downloads, but we would still want to suppress warnings or errors if we don't update the separate package for a while.saimn commentedon May 5, 2023
Integrating IERS-A directly in Astropy with automatic updates may be annoying in terms of repository size ? (Repo that is then cloned/fetched by a lot of people/CIs). The Astropy docs say it's "updated weekly and has historical data starting from 1973 and predictive data for 1 year into the future." so probably a lots of updates quite often.
A dedicated and automated package seems a very good alternative (with the benefit that this package would have only one wheel and sdist !).
10 remaining items
mhvk commentedon May 11, 2023
A fallback for IERS A might indeed be interesting, though, as you say, perhaps not worth the trouble. But interesting enough to be worth its own issue - #14811
astrofrog commentedon May 12, 2023
@mhvk @tepickering @pllim @saimn @taldcroft @weaverba137 - please see #14819 for an experimental PR related to making a standalone package.
pllim commentedon Aug 29, 2023
Fixed in #14819
astrofrog commentedon Aug 29, 2023
@pllim - I think this isn't quite done, in particular this item:
This might be a better alternative to #15248
pllim commentedon Oct 5, 2023
Like this? #15436
mhvk commentedon Oct 5, 2023
I worry about making the date too far in the future: I just heard from a student in Amsterdam whose VLBI positions were off by several mas because their IERS data were extrapolated too far. I don't know whether this was related to astropy, but building in bad predictions can have real consequences; the only reason they did not publish a bad position was that there was another measurement where the IERS data were extrapolated less.
A time beyond 180 days is also a bit pointless because leap seconds are determined 150 days in advance of one.
So, I suggest to maybe increase to 90 days or so.
taldcroft commentedon Oct 5, 2023
I also just heard about a published paper that uses raw Chandra source count rates over 20 years to examine source variability, even though all the documentation states loudly that the telescope response has dropped by a factor of several over that time (due to contamination) and you really need to use calibrated fluxes.
The point is that each scientist needs to understand their analysis and what the tools are doing. Maybe there are use cases where extrapolating IERS by 70 days is dangerous?
Having said that, I won't push this. Doing nothing is acceptable. I think changing to 90 days doesn't solve the original problem so I wouldn't bother with that.
mhvk commentedon Oct 5, 2023
I know, people should be careful, but just like units helps to avoid unit errors,
Timehelps to avoid leap-second errors, andiershelps avoid both subtle time and coordinate problems. (And the issue I heard about was really quite subtle, and involved a code update that did not properly check IERS expiration. But we have now trained people to expect astropy to warn if a result might at all be uncertain...)What I think we can do to limit the cases where a download is done, is to only download if the time requested is out of range, rather than always update if the table is too old.
But I think with the new package, we've solved quite a bit of the problem of "random" downloads.
pllim commentedon Oct 5, 2023
While this sounds nice in principle, I am not sure how it can be implemented cleanly. Basically you have to check every time someone calls
Time(...)? Can get quite expensive.mhvk commentedon Oct 5, 2023
@pllim - it is not so bad, because the code internally checks already whether times are out of range, I think it would become more expensive only for the case that one was using predicted times already. But it is definitely more work to implement it...
pllim commentedon Oct 5, 2023
Should we at least change the title of the issue since "making auto_max_age longer" is now not gonna happen and "bunding IERS-A with astropy" has been replaced with
astropy-iers-data.mhvk commentedon Oct 5, 2023
I'm still fine with extending to 90 days if it is mostly the service providers who are complaining.
pllim commentedon Oct 5, 2023
@mhvk , would you want to open a PR with the 90-day idea then, since you would know how to adjust the "expiration math" in the code?