Repository navigation
fits.Z cannot be opened using automatic decompression #10714
Description
Description
When trying to open a fits.Z (ESO download format) the file does can be read (link to file https://drive.google.com/file/d/1PoQhABH90aQwwgUSmzRqrj-yPvGWLvXW/view?usp=sharing )
Expected behavior
I expect the header to be read normally. Important to note that it works with the fitsio library.
Actual behavior
---------------------------------------------------------------------------
UnicodeDecodeError Traceback (most recent call last)
~/.local/lib/python3.8/site-packages/astropy/io/fits/hdu/base.py in _readfrom_internal(cls, data, header, checksum, ignore_missing_end, **kwargs)
405 # that are needed to build the HDU object.
--> 406 header_str, header = _BasicHeader.fromfile(data)
407 except Exception:
~/.local/lib/python3.8/site-packages/astropy/io/fits/header.py in fromfile(cls, fileobj)
2076 try:
-> 2077 header_str, cards = parse_header(fileobj)
2078 _check_padding(header_str, BLOCK_SIZE, False)
astropy/io/fits/_utils.pyx in astropy.io.fits._utils.parse_header()
UnicodeDecodeError: 'ascii' codec can't decode byte 0x9d in position 1: ordinal not in range(128)
During handling of the above exception, another exception occurred:
OSError Traceback (most recent call last)
<ipython-input-2-0fb814c35786> in <module>
----> 1 fits.getheader('VISIR.2017-01-05T09:04:11.258.fits.Z')
~/.local/lib/python3.8/site-packages/astropy/io/fits/convenience.py in getheader(filename, *args, **kwargs)
103
104 mode, closed = _get_file_mode(filename)
--> 105 hdulist, extidx = _getext(filename, mode, *args, **kwargs)
106 try:
107 hdu = hdulist[extidx]
~/.local/lib/python3.8/site-packages/astropy/io/fits/convenience.py in _getext(filename, mode, ext, extname, extver, *args, **kwargs)
1047 raise TypeError('extver alone cannot specify an extension.')
1048
-> 1049 hdulist = fitsopen(filename, mode=mode, **kwargs)
1050
1051 return hdulist, ext
~/.local/lib/python3.8/site-packages/astropy/io/fits/hdu/hdulist.py in fitsopen(name, mode, memmap, save_backup, cache, lazy_load_hdus, **kwargs)
162 raise ValueError(f'Empty filename: {name!r}')
163
--> 164 return HDUList.fromfile(name, mode, memmap, save_backup, cache,
165 lazy_load_hdus, **kwargs)
166
~/.local/lib/python3.8/site-packages/astropy/io/fits/hdu/hdulist.py in fromfile(cls, fileobj, mode, memmap, save_backup, cache, lazy_load_hdus, **kwargs)
401 """
402
--> 403 return cls._readfrom(fileobj=fileobj, mode=mode, memmap=memmap,
404 save_backup=save_backup, cache=cache,
405 lazy_load_hdus=lazy_load_hdus, **kwargs)
~/.local/lib/python3.8/site-packages/astropy/io/fits/hdu/hdulist.py in _readfrom(cls, fileobj, data, mode, memmap, save_backup, cache, lazy_load_hdus, **kwargs)
1077
1078 # Make sure at least the PRIMARY HDU can be read
-> 1079 read_one = hdulist._read_next_hdu()
1080
1081 # If we're trying to read only and no header units were found,
~/.local/lib/python3.8/site-packages/astropy/io/fits/hdu/hdulist.py in _read_next_hdu(self)
1152 fileobj.seek(offset, os.SEEK_SET)
1153
-> 1154 hdu = _BaseHDU.readfrom(fileobj, **kwargs)
1155 except EOFError:
1156 self._read_all = True
~/.local/lib/python3.8/site-packages/astropy/io/fits/hdu/base.py in readfrom(cls, fileobj, checksum, ignore_missing_end, **kwargs)
323 fileobj = _File(fileobj)
324
--> 325 hdu = cls._readfrom_internal(fileobj, checksum=checksum,
326 ignore_missing_end=ignore_missing_end,
327 **kwargs)
~/.local/lib/python3.8/site-packages/astropy/io/fits/hdu/base.py in _readfrom_internal(cls, data, header, checksum, ignore_missing_end, **kwargs)
411 # in the wild.
412 data.seek(header_offset)
--> 413 header = Header.fromfile(data,
414 endcard=not ignore_missing_end)
415 hdu_fileobj = data
~/.local/lib/python3.8/site-packages/astropy/io/fits/header.py in fromfile(cls, fileobj, sep, endcard, padding)
511 break
512
--> 513 return cls._from_blocks(block_iter, is_binary, sep, endcard,
514 padding)[1]
515 finally:
~/.local/lib/python3.8/site-packages/astropy/io/fits/header.py in _from_blocks(cls, block_iter, is_binary, sep, endcard, padding)
594 # TODO: Pass this error to validation framework as an ERROR,
595 # rather than raising an exception
--> 596 raise OSError('Header missing END card.')
597
598 header_str = ''.join(read_blocks)
OSError: Header missing END card.
Steps to Reproduce
from astropy.io import fits
fits.getheader('VISIR.2017-01-05T09:04:11.258.fits.Z')System Details
import platform; print(platform.platform())
-->Linux-5.4.60-1-lts-x86_64-with-glibc2.2.5
import sys; print("Python", sys.version)
-->Python 3.8.5 (default, Jul 27 2020, 08:42:51)
[GCC 10.1.0]
import numpy; print("Numpy", numpy.__version__)
-->Numpy 1.17.4
import astropy; print("astropy", astropy.__version__)
-->astropy 4.0.1.post1
import scipy; print("Scipy", scipy.__version__)
Scipy 1.3.3
import matplotlib; print("Matplotlib", matplotlib.__version__)
Matplotlib 3.3.0
Reacted by Tiago Gomes and mzechmeister
Activity
saimn commentedon Sep 7, 2020
Salut Romain,
The problem is that there is no way to decompress
.Zfiles with the stdlib. I found two packages to read those:https://github.com/ionelmc/python-unlzw
https://github.com/umeat/unlzw
I guess we could have an optional dependency on one of these. The two are based on https://mathematica.stackexchange.com/questions/60531/how-can-i-read-compressed-z-file-automatically-by-mathematica/60879#60879, the first one wrapping the C code, and the second reimplementing in Python.
astrom-tom commentedon Sep 7, 2020
Salut Simon!
Thanks for your message! I created this https://github.com/astrom-tom/dfitspy package that I recently migrated to astropy (was using fitsio before) but I lost the possibility to read
.Zfiles at the same time...That would be awesome if it was possible to add the support to it, it is the format that come from the ESO archive so I am sure it would be useful for a lot o people.
astrom-tom commentedon Sep 7, 2020
Sorry, hit the wrong button
saimn commentedon Sep 7, 2020
Supporting this compression format with one of the libraries above (probably the one wrapping the C code, for performance) would be easy, but - at first view - a drawback is that the complete file would have to be decompressed. Which is not optimal to read headers... I'm not sure if there is a better alternative.
Btw,
fitsheaderfrom Astropy has a fitsort-like mode, but i guessdiftspy's goal is to have the same output asdfits.dhomeier commentedon Sep 9, 2020
Apart from the in-memory reading that seems doable (also as a temporary workaround, if you're interested):
m-samland commentedon Jan 28, 2021
Is there any progress on this issue? Considering that the whole ESO archive is using fits.Z it seems quite important.
Does anyone have a good temporary workaround for reading in the data instead of the header? The above snipped works for headers, but throws a "TypeError: buffer is too small for requested array" when you try to access the data instead of the header.
saimn commentedon Jan 28, 2021
It's not the whole ESO archive. My guess would be that it concerns only a few instruments, @astrom-tom probably knows better about this ?
And if it was so important, I guess ESO could contribute some resources to support this format ;). If you are a user, one thing to do would be to contact their support (or have a PI do it) asking them to help doing this.
As mentioned above there are 2 Python libraries supporting this format, but both have not seen a commit since 2017/2018 so to have a good support in Astropy it requires some work. Testing those libraries, checking the performance of the pure Python one versus the C one. Adding support in Astropy, and unit tests.
And the authors of those libraries may not be willing to support it on the long term, so what do we do if there is a problem with the one we use ? (The error you get, "TypeError: buffer is too small for requested array", may be a good example. Which one did you try ?)
dhomeier commentedon Jan 28, 2021
I could reproduce the buffer size exception with the compiled library
python-unlzw(which is the one that gets installed bypip install unlzw, to add more confusion). This could indicate further trouble with correctly building the underlying C library (or using a precompiled one from a different machine).The pure Python version, which is effectively a single file https://raw.githubusercontent.com/umeat/unlzw/master/unlzw/__init__.py
works for me with the data as well, but as expected is quite resource-intensive (O(1 s / MB) decompression time, and it seems up to 10 x the raw data size in temporary memory usage).
As a feasible workaround for now I would rather recommend to use command-line
uncompresson the whole bunch of files (and re-compress with bzip2 or gzip if really necessary – for the example file the .Z version offered < 5 % savings in file size).dhomeier commentedon Jan 28, 2021
It might be a bug or a Python 3 problem, but
https://github.com/ionelmc/python-unlzw/blob/a5f6774d33e862ca5504666326efbc1579f6a4ed/src/unlzw/__init__.py#L15
did not work because
ffi.string()stops at the first null character, which you will inevitably encounter at some byte of a FITS data unit. Changing this tolets it succeed with performance on par or better than the
bz2orlzmadecompressors.[-]fits.Z cannot be open[/-][+]fits.Z cannot be opened using automatic decompression[/+]mfeldt commentedon Apr 16, 2021
To my knowledge it is at least a very large part of the ESO archive. Moreover reading and writing .Z compression is pretty standard and also supported by cfitsio, so I guess they chose such a standard format in order to not have to devote resources to support reading and writing.
kYwzor commentedon Feb 17, 2025
Just an heads-up, I'm currently working on a solution for this. My goal is to have an implementation that can read any .Z LZW file transparently i.e. it doesn't require decompressing the entire file and holding everything in memory nor creating a temporary file (similar to existing gzip/bz2 packages). The big win here is that it would enable very fast reads of the primary headers.
I already have a working prototype written in pure Python which I think is quite optimized. In some very early testing, for full file decompression, my implementation seems to always be faster than the other existing pure Python implementations; in small files (tens of MBs) it's up to ~2x faster, though the margin seems to shrink in larger files (hundreds of MBs). If are working with very large FITS files and we're only interested in reading the header, my implementation can easily achieve reads on a millisecond scale for most FITS files, which logically makes it thousands of times faster than the alternatives (which can only do full decompression).
I plan on publishing this as a package and making a PR integrating it into Astropy before the end of March. I'm currently working on cleaning up the interfaces, validating the implementation and doing a more in-depth performance characterization. If anyone wants to help me with this, I'd quite appreciate it if you could send me or point me to .Z files that you know of (FITS or not). My hope is to test the implementation against a mix of files of varying sizes/complexity/formats, files from the ESO archive, files produced by different compressing software and any "tricky" files that may exist. Essentially, I want to ensure I can read any files that uncompress or gunzip can.
saimn commentedon Mar 19, 2025
Another pure python implementation:
https://github.com/scivision/unlzw3
kYwzor commentedon Mar 31, 2025
It seems I waited until the very last day to follow through with my "before the end of the March" plan :)
Executive summary: I have now published the uncompresspy package and it performs better than all other pure Python packages. I will be providing a PR adding LZW file support to Astropy later today. I'll be sharing below what I've learned in these last couple of weeks (along with performance metrics).
Preface
I've found that, while there is an official specification for the Lempel–Ziv–Welch (LZW) algorithm, there is no official specification on how to apply it to a file or how the ".Z" extension files are supposed to be formatted (well, at least to my knowledge, I'd be interested in reading it if anyone finds it). This is particularly tricky because some of the quirks of LZW files are actually not part of the algorithm itself, but rather something that only really exists on the file format (namely specific flags on the header and "clear codes" aka "block mode"). That may be why I've found that nearly all LZW file decoders available online are based on one of two implementations:
Available packages
Given all this, I decided it was best to cross check my implementation against the output (and performance) of multiple command line tools and Python packages. Starting with the Python packages, I went through all the relevant GitHub/PyPI projects I could find, and here's a summary:
With the exception of uncompresspy, all of these packages force the input file to be decompressed entirely, which means 1) lazy/on-demand decompression is not possible and 2) the entire decompressed output must be held in memory. On top of that, with the exception of uncompresspy and ncompress, all of them also must hold the input file entirely in memory. uncompresspy is also the only package to provide a file-like interface for LZW files. On a small note, uncompresspy does not provide functions for compression, which a couple of these do (not really my aim here).
Besides Python packages, a common alternative is to call a shell command via Python and capture the output. This works, but has the disadvantage of not really being cross-platform and requiring the user to install the specific package being used. Regardless, for comparison purposes, I've also collected metrics for uncompress (provided by (n)compress), gunzip and pigz.
Performance comparison
Some notes on the testing
All zeros files
Files created via
dd if=/dev/zero, i.e. files where the entire uncompressed content consists of repeated "0"s. This is by far the best-case scenario for uncompresspy; it's the only scenario where it beats every other package/command including all C-based ones. This is because the implementation of uncompresspy is extremely fast at dealing with greatly compressed files and LZW compresses the most when the entire file has the exact same character repeated. Obviously, this is a synthetic benchmark and not a realistic case.Random files
Files created via
dd if=/dev/urandom, i.e. files where the uncompressed content is entirely random. This is the worst-case scenario for uncompresspy as the compression ratio is terrible (less than 1, i.e. we're increasing storage usage rather than saving space). However, even in such a scenario, uncompresspy still clearly beats all other pure Python packages (though it's handily beat by C/C++-based alternatives). This is also an unrealistic synthetic benchmark, so let's look at real-world files.ESO FITS files
These are all FITS files produced by ESO. The VISIR file was sourced from the link at the very top of this ticket; it's a simple FITS file containing just one extension with an image. The ERIAO and GALAC files are AO telemetry files in the AOT format (which I am the developer of), available on the ESO Archive; these are very complex files with many binary tables and images. The GRAV file is a interferometry telemetry file produced by the GRAVITY instrument (I sourced it directly from Chile as part of my work on GRAVITY+); this is also a file with many binary tables and images.
Once again we can see that uncompresspy is the best performing pure Python package, but it lags behind C/C++ based implementations. We can also notice the previously seen behavior where its relative performance improves with higher compression ratios. However, the picture really changes if we only want a part of the FITS file:
Because uncompresspy allows for a partial decompression (unlike the others) it can provide a huge speedup versus every other solution in instances where we only want e.g. the primary header (represented in the plot as "uncompresspy(header)") or the first extension (represented in the plot as "uncompresspy(data)") of a very large file. These two metrics were collected via
fits.getheader(filename)andfits.getdata(filename, 1)respectively (using a branch I've created for Astropy with LZW support) and they are compared vs the other packages doing a full decompression (omitted the other pure Python packages here because they blow out the scale). It's easy to imagine a scenario where a user has a set of large files and wants to check the header for each of them before picking one, and in that case, uncompresspy provides a clear advantage above every alternative. Obviously, the plot is not an apples to apples comparison, but the point is that you could not achieve a similar effect with the other packages, as they do not allow for partial decompression. The effect here is less notable on VISIR because it's a smaller file that only contains a single extension (and thusfits.getdata(filename, 1)essentially forces us to decompress the entire file anyway).Tricky files
For the comparison here the performance isn't really the point, but rather if the package was successful at decompressing or not. That said, these smaller files show the interesting effects of the overhead when piping commands from shell.
compress), but rather random uninitialized data. All packages seem to handle this just fine.compress -b 9command to force it to a bit size of 9 (i.e. the minimum possible). Oddly enough, the file created in this manner could not be opened by uncompress, nor by any package except ncompress and uncompresspy; the latter two agree on the data contents and do not throw any warnings, but the data contents are only partially recovered. This is a weird case that I'm still trying to understand, I will try to get a final answer on this in the coming days, but from early investigations it seems to be a very long standing bug in compress which just creates a corrupt file. Compressing with any bit size from 10 to 16 resulted in files that could be properly read by every package.Key takeaways
Footnotes
It has a memory leak and cuts the stream short if it finds a null char (which will happen with basically all FITS files). It also fails to build properly from Python 3.12 onwards. For testing sake, I used the patch provided in a pull request which fixes the first two issues and built the package manually. ↩
Technically not on PyPI, but unlzw3 is just a fork with modernization, very similar implementation. ↩
Broken in Python 3.10+, "cannot import name 'Callable' from 'collections'". If we remove the import it still creates broken files. I couldn't get it to work. ↩
saimn commentedon Apr 1, 2025
Thanks for the detailed analysis @kYwzor, great job !
Having support for partial read is great, as you say it can be useful to display the primary header. But to uncompress entire files the overhead of pure python vs c/c++ is not negligible. Did you check (or could you) if with your code uncompressing the full file at once is faster than uncompressing small blocks while the file is read ? This is the case with gzip (#14855) and lead to #15501.
kYwzor commentedon Apr 1, 2025
@saimn thanks. I agree the Python overhead is significant. I spent a long time profiling and optimizing the code but even with all that effort pigz/unlzw are still ~10 to ~20 times faster than uncompresspy on a typical FITS file. There might still be room for some micro optimizations in my code but at this point I'd be very surprised to get even a 1.5x improvement on a single thread, much less close the gap entirely (side comment, even though LZW isn't really designed for parallelism, we could probably squeeze a bit more performance by decoding the bitstream on a separate thread from the construction of the dictionary and output, or maybe even try to split the decoding into different blocks... but even with all that complexity we would definitely not close the existing gap).
Regarding decompressing in memory, with the exception of the metrics I show for
fits.getheader(filename)andfits.getdata(filename, 1)(i.e. the second to last image), all plots are for full file decompression at once without any Astropy/io.fits overhead (simply aLZWFile.read(-1)). So decompressing the entire file in memory with thedecompress_in_memorywould yield similar results to what we already see in the plots (presumably slightly worse given the Astropy overhead). In other words, if we need significantly better performance, I don't think it's possible with a pure Python package.Despite all that, I believe that, as-is, the package is still very much usable for files up to ~100MB and okay-ish up to ~500MB (and then it starts to get a bit silly with 1min+ decompression times). I would argue this range covers a lot of use-cases and therefore it's an improvement over the current state of not supporting LZW files at all, thus it's worth chasing the PR even if sub-optimal.
I think it's clear that the end goal is to have something that allows partial decompression + exposes a file-like interface + has a performance similar to pigz. I'm convinced this is very much achievable with a C/C++-based approach, but the implementation is tricky (particularly when it comes to managing the state of the decoder between partial decompressions). I would be interested in eventually exploring this, but I can't promise a timeline right now. In the meantime I think this is a step in the right direction.
saimn commentedon Apr 2, 2025
Ok, so yeah might be worse when reading a file with many extensions (but then
decompress_in_memory=Truecan be used).But I agree it's already a good first step, first we need something that works and then interested people can try to improve performance if this is important for their workflow.
So thanks again, and I will look at the PR soon.
kYwzor commentedon Apr 9, 2025
I had an informal exchange with an ex-member of the archive team regarding the frequency of these files on the ESO archive (this was a point of discussion early in this issue). I was told that the "vast majority of raw files" is compressed in this format, though there are exceptions mostly in newer files. The decision whether to use this compression or not is apparently done per-instrument, particularly they seem to avoid using this compression when the FITS file itself is already tile-compressed (and a few instruments have their files compressed with gzip). This is not an "official" ESO response (still attempting to get one) but I thought I'd leave it here for context.