IERS_B file download failure: 404 error

Description

Hi,
It seems like the URL used to download the IERS B table is down and gives a 404 error in a new virtual environment.

Expected behavior

I see some mention of adding fail-safe URLs in astropy/astropy-iers-data#20, but not sure if there was a resolution for the IERS B tables?

How to Reproduce

# Put your Python code snippet here.
import astropy
from astropy.utils.iers import conf,IERS_A, IERS_A_URL, IERS_B, IERS_B_URL, IERS
from astropy.utils.data import download_file

print(astropy.__version__)
print(IERS_B_URL)
iers_b_file = download_file(IERS_B_URL, cache=True)

Output:

5.1
http://hpiers.obspm.fr/iers/eop/eopc04/eopc04_IAU2000.62-now

Attempting to download the IERS B table from the given URL gives the following error.
HTTPError: HTTP Error 404: Not Found

Conversely if you attempt this in an old virtual environment (which has the file cached), I don't get this error. The analogous IERS_A_URL points to a resource that is still available.

Versions

Windows-10-10.0.19045-SP0
Python 3.9.12 (tags/v3.9.12:b28265d, Mar 23 2022, 23:52:46) [MSC v.1929 64 bit (AMD64)]
astropy 5.1
Numpy 1.22.3
pyerfa 2.0.0.1
Scipy 1.8.1
Matplotlib 3.5.2

@mhvk

Thanks for reporting this! I just noticed this in the cron job as well. (Ugh, not again!)

@mhvk or @eteq , any ideas?

It appears the new files are not drop in replacements as assumed. Changes include additional columns, shifting the order of columns and changing the datatype of some columns (from int to float, for instance).

@jpl-jengelke

@jpl-jengelke , we apologize for the inconvenience caused. Such breakage are upstream and beyond our control in this package. If you would elaborate how this is breaking your downstream package, perhaps we could figure out a workaround. Usually there is an option to use offline file if you do not absolutely need the latest IERS data.

@mhvk

@mhvk , looks like IERS B URL was originally added back in #1145 and never changed since then. Do you know of a new source for this file?

This is their file listing now: https://hpiers.obspm.fr/iers/eop/eopc04/eopc04.txt

@jpl-jengelke

I received a note from the EXOTIC project at JPL about this failure affecting barycorrpy here. I wonder if @jpl-jengelke is referring to the same.

Should/can we replace the IERS_B_URL with this new URL?

@mhvk

Should/can we replace the IERS_B_URL with this new URL?

I don't know. We'll have to wait for @mhvk to confirm. Thank you for your patience.

Header of old data:

               EARTH ORIENTATION PARAMETER (EOP) PRODUCT CENTER CENTER (PARIS OBSERVATORY)
                      INTERNATIONAL EARTH ROTATION AND REFERENCE SYSTEMS SERVICE
                                    EOP (IERS) 14 C04 TIME SERIES                 
               Description: https://hpiers.obspm.fr/eoppc/eop/eopc04/C04.guide.pdf
                            contact: [email protected]

  
             FORMAT(3(I4),I7,2(F11.6),2(F12.7),2(F11.6),2(F11.6),2(F11.7),2(F12.6))
##################################################################################
  
      Date      MJD      x          y        UT1-UTC       LOD         dX        dY        x Err     y Err   UT1-UTC Err  LOD Err     dX Err       dY Err  
                         "          "           s           s          "         "           "          "          s         s            "           "
     (0h UTC)

1962   1   1  37665  -0.012700   0.213000   0.0326338   0.0017230   0.000000   0.000000   0.030000   0.030000  0.0020000  0.0014000    0.004774    0.002000
1962   1   2  37666  -0.015900   0.214100   0.0320547   0.0016690   0.000000   0.000000   0.030000   0.030000  0.0020000  0.0014000    0.004774    0.002000

Header of https://hpiers.obspm.fr/iers/eop/eopc04/eopc04.1962-now (new data?):

# EARTH ORIENTATION PARAMETER (EOP) PRODUCT CENTER CENTER (PARIS OBSERVATORY) - INTERNATIONAL EARTH ROTATION AND REFERENCE SYSTEMS SERVICE  
# EOP (IERS) 20 C04 TIME SERIES  consistent with ITRF 2020 - sampled at 0h UTC     
# Contact: [email protected]                                       
# Reference Precession-Nutation Model: IAU 2000
# format(4(i4),f10.2,2(f12.6),f12.7,2(f12.6),2(f12.6),f12.7,2(f12.6),f12.7,2(f12.6),2(f12.6),f12.7)
# YR  MM  DD  HH       MJD        x(")        y(")  UT1-UTC(s)       dX(")      dY(")       xrt(")      yrt(")      LOD(s)        x Er        y Er  UT1-UTC Er      dX Er       dY Er       xrt Er      yrt Er      LOD Er
1962   1   1   0  37665.00   -0.012700    0.213000   0.0326338    0.000000    0.000000    0.000000    0.000000   0.0017230    0.030000    0.030000   0.0020000    0.004774    0.002000    0.000000    0.000000   0.0014000
1962   1   2   0  37666.00   -0.015900    0.214100   0.0320547    0.000000    0.000000    0.000000    0.000000   0.0016690    0.030000    0.030000   0.0020000    0.004774    0.002000    0.000000    0.000000   0.0014000

From https://datacenter.iers.org/data/latestVersion/bulletinB.txt

                                                                 BULLETIN B 420
                                        			4 February 2023   
    
 Contents are described in https://hpiers.obspm.fr/iers/bul/bulb_new/bulletinb.pdf 
	
 1 - DAILY FINAL VALUES OF x, y, UT1-UTC, dX, dY
 Angular unit is milliarcsecond (mas), time unit is millisecond (ms). 
 Upgraded solution from March 1 2017 - consistent with ITRF 2014.
 
       DATE     MJD       x       y      UT1-UTC      dX     dY     x err    y err   UT1 err  dX err  dY err
    (0 h UTC)            mas     mas       ms         mas    mas     mas      mas      ms     mas     mas

 Final values 

 Mean formal error      0.031    0.023    0.0090    0.058  0.049 
 
2022  12   2   59915  141.086  190.030  -20.0182    0.294 -0.116    0.033    0.022    0.0087  0.069  0.056
2022  12   3   59916  138.520  189.588  -20.4483    0.304 -0.128    0.034    0.022    0.0086  0.065  0.055

In the meantime, astropy should still work if you turn off download of the defunct URL and use the file shipped with package. See https://docs.astropy.org/en/latest/utils/iers.html#working-offline

From what I see above, unless Paris Observatory brings back the old file, we actually need to change the code here to match the new file format, which is also a headache because that breaks backward compatibility. I don't see any drop-in replacement.

I have also emailed the contact listed on the Paris Observatory file, asking if they can temporarily restore the old file until we can figure out a permanent solution. 🤞

Let me see if I can create a new format file -- the headers look different enough that it should not be too hard to have code distinguish between the two.

@jpl-jengelke

I received a note from the EXOTIC project at JPL about this failure affecting barycorrpy here. I wonder if @jpl-jengelke is referring to the same.

Yes

Should/can we replace the IERS_B_URL with this new URL?
See my comment above. It's not a drop-in replacement.

Tried and failed. I canceled the PR.

But on the plus side I do have a full astropy deve environment setup now.

I have also emailed the contact listed on the Paris Observatory file, asking if they can temporarily restore the old file until we can figure out a permanent solution. 🤞

The IERS EOP Center already responded to my earlier email: "That abrupt change was announced by an IERS mail. ..."

Apparently no one got the memo.

@mhvk , looks like IERS B URL was originally added back in #1145 and never changed since then. Do you know of a new source for this file?

The IERS EOP Center pointed me at another location with the old file format that looks current up to today. We do need to adapt to the new file, but I'll try implementing this immediately.

Let me see if I can create a new format file -- the headers look different enough that it should not be too hard to have code distinguish between the two.

Agreed. Unfortunately I'm unaware of that process, so please proceed. Meanwhile I will attempt the simplest fix with the URL.

And probably the code shouldn't die if a URL goes down. (Or even if the file format changes.) It should gracefully fall back to the local file cache and proceed with an error report.

@jpl-jengelke

@jpl-jengelke , I think there are two different problems.

https://github.com/shbhuk/barycorrpy/blob/4f48f89980b5e2b3ebc8030a3c98211b24688f1b/barycorrpy/PINT_erfautils.py#L17 -- This is going to fail no matter what we do in astropy as long as the URL is down. You are calling download_file directly.
As for astropy.utils.iers, we do fall back to a local copy, though it depends on how you are using the API, I guess.

astropy/astropy/utils/iers/iers.py

Line 786 in 8b7e479

warn("unable to download valid IERS file, using local IERS-B", IERSWarning)

For a fix on the astropy side, see #14382.

Only now read the comment about the old format being at a new location -- I didn't try to implement that though give instructions on how to use the old-style files. But probably best anyway to just move to the new format.

Issue astropy#14380: Repoint IERS files and urls at legacy naming con…

Here is the response I got from service provider:

That abrupt change in format/files was announced by an IERS mail that you probably did not received.

The old format, but according to EOP values consistent with International Terrestrial Reference Frame (ITRF) 2014, is maintained in the neighbouring repertory:

https://hpiers.obspm.fr/iers/eop/eopc04_14/

For the new solution consistent with ITRF 2020, the format was rearranged for we introduced two supplementary Earth Rotation Parameters: the pole coordinate rates (https://hpiers.obspm.fr/iers/eop/eopc04/eopc04.txt). So this would be quite impossible to insure a "smooth" transition.

I am sorry for such disagreement, on the other side that enables to dust off old codes.

Issue astropy#14380: Repoint IERS files and urls at legacy naming con…

@mhvk

Only now read the comment about the old format being at a new location -- I didn't try to implement that though give instructions on how to use the old-style files. But probably best anyway to just move to the new format.

Well I did try that in #14381 but unfortunately unit tests are failing. And I think they are failing because either the data changed or the IERS_A_FILE (which does not appear to be cached) also changed. (I haven't deep dived into the code far enough to know which exactly.) That is probably the same reason why the @mhvk PR #14382 is failing. It was a noble effort but it would take a lot more to fix this, I think, including some work on the tests, maybe. I'll pull back my PR again, though the tests against IERS_B alone seem to pass, at least locally, with the new URL.

@jpl-jengelke , I think there are two different problems. ...

I'm aware of the the issue in barycorrpy and if the link to IERS_B is the only issue I will try to inject a replacement there, at least temporarily. [Please note that barycorrpy is not my project, so what I mean is to replace the URL from within our (different) dependent application.]

Otherwise, it would still be preferable for astropy to detect unintelligible data and fall back to cached files, not simply detect dead URLs. If it can be detected in unit tests it certainly can be detected in code. Relying on humans not to change file formats is the road to perdition. But I digress -- That's a different issue.

Thanks for checking with IERS! So, it seems an alternative solution is simply to change the IERS_B_URL to the new location for the "old style" files -- perhaps we should do that for backports and then for 5.3 move to the new system? See #14384

to detect unintelligible data and fall back to cached files

That is a two-edge sword as well. If it silently falls back, if we are not careful, we might not ever catch we are using very old cached data. But it is a valid concern, so feel free to open an issue about this if there isn't an outstanding one already. Thanks!

Got another message from Paris Observatory. Does this mean we have to revert #14384?

For insuring the transition with the new C04 20, and avoiding the failure of automated processes, I decided to maintain from today the file eopc04-IAU2000.62-now with the old format, which does not contain the pole coordinate rates.

See https://hpiers.obspm.fr/iers/eop/eopc04/

Hmm, sounds like we can revert #14384 -- but perhaps keep #14382 since that is the new, presumably better analysis.

Navigation Menu

Repository navigation

IERS_B file download failure: 404 error #14380

Description

Expected behavior

How to Reproduce

Versions

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

IERS_B file download failure: 404 error #14380

Description

Description

Expected behavior

How to Reproduce

Versions

Activity

pllim commented on Feb 14, 2023

jpl-jengelke commented on Feb 14, 2023

pllim commented on Feb 14, 2023

pllim commented on Feb 14, 2023

pllim commented on Feb 14, 2023

shbhuk commented on Feb 14, 2023

pllim commented on Feb 14, 2023

pllim commented on Feb 14, 2023

pllim commented on Feb 14, 2023

pllim commented on Feb 14, 2023

mhvk commented on Feb 14, 2023

jpl-jengelke commented on Feb 14, 2023

jpl-jengelke commented on Feb 14, 2023

pllim commented on Feb 14, 2023

mhvk commented on Feb 14, 2023

mhvk commented on Feb 14, 2023

pllim commented on Feb 14, 2023

jpl-jengelke commented on Feb 14, 2023

mhvk commented on Feb 14, 2023

pllim commented on Feb 14, 2023

pllim commented on Feb 15, 2023

mhvk commented on Feb 15, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions