Performance regression when reading a short and wide FITS binary table

I found after bisecting that #996 introduced a factor of 4 slowdown when reading a relatively short (1200 rows) but wide (900 columns) table. Here are some timings for reading that table (which I can provide on request), Since #996 there has been another 30% slowdown. At present astropy.io.fits is about 10 times slower than fitsio in this case [1].

Is there anything obvious in #996 that could be tweaked to take out this performance regression? Or is there a way to use io.fits to improve performance in this case? In my application I am reading over 150,000 of these files so performance makes a difference.

from astropy.io import fits
import fitsio

time dat = fits.open('wide.fits')[1].data[:]  # timing astropy.io.fits
time dat = fitsio.read('wide.fits')[:]  # timing fitsio

Package Commit    Note         Version       Time
------- -------   -----------  ------------  --------
astropy 7f2d94f   Before #996  0.3.dev3921   0.55 sec
astropy 7ba60c2   After #996   0.3.dev3923   2.47 sec
astropy e979e30   master       1.1.dev14486  3.22 sec
fitsio    ..      master       0.9.8rc2      0.29 sec

[1] - but astropy.io.fits is apparently more robust, and I ran into a limitation that is a bit of a showstopper for me in using fitsio: esheldon/fitsio#80.

Closed by #4985 or other related work on io.fits.

In master at 1.3.dev16204 this speed regression is gone and I see a factor of 8 improvement over 1.2.1 for this table:

import numpy as np
from astropy.table import Table

ncols = 800
nrows = 10
cols = [np.random.uniform(size=nrows) for _ in xrange(ncols)]
dat = Table(cols)

dat.write('test_speed.fits', overwrite=True)

Is there a performance test for this? (Are we still maintaining the performance tests since Mike left?)

I have been running the benchmarks and updating the results, so yes we should add a benchmark.

@astrofrog

@astrofrog Are the benchmarks public somewhere?

http://www.astropy.org/astropy-benchmarks/

https://github.com/astropy/astropy-benchmarks is public, however I think there is no benchmark for this case.

Are the benchmarks running atm? As far as I remember they stopped at some point.

Navigation Menu

Repository navigation

Performance regression when reading a short and wide FITS binary table #4609

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Performance regression when reading a short and wide FITS binary table #4609

Description

Activity

taldcroft commented on Oct 14, 2016

embray commented on Oct 17, 2016

astrofrog commented on Oct 17, 2016

elehcim commented on Mar 13, 2017

pllim commented on Mar 13, 2017

MSeifert04 commented on Mar 13, 2017

bsipocz commented on Mar 13, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions