You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
While I agree completely, I'm starting to worry about how much we're reinventing the wheel that Pandas has already built. That said, Pandas has lots of features I think we don't care about (business weeks and such), and is probably missing things we do care about but I wouldn't know the details there.
@Cadair - Are the time series in question sorted? If so, can we base things around selection via np.searchsorted instead of an index? We could add some convenience methods to make things easier.
The pandas loc method is looking for exact keys and this doesn't seem like what's typically wanted in selecting astronomical time series values, but maybe I'm not understanding the application.
@embray Our LightCurve (TimeSeries) object currently uses Pandas underneath and pandas is just awesome, however we don't actually use a lot of pandas magic inside SunPy, having it available to the users is an important feature but we don't need to use it internally for that (a .to_pandas() method would suffice.
@taldcroft The timeseries would only be sorted in Table if they were sorted in the input correct? This of course would normally be the case.
What I mean by 'index' is having one column in this case a temporal column that is the row index. Thereby allowing selection of rows by index, filtering by index, truncation by index etc. Also allowing plotting by index etc. Basically in my mind it's a special column that you always know where it is i.e. table.index. Then you can add special functionality using it.
it may be inefficient in cases where you know the time series is sorted, and in cases where one wants to select e.g. just the nearest entry or a very small subset.
@Cadair - it might be interesting to illustrate this point by comparing the performance of what you can do with Pandas versus what table selection as shown above already allows you to do? Then it might be clear in what cases the current functionality is inefficient and where we might do better?
The thing with the current table layout is that time can be a column and you can do a lot with it, but if you read a file that calls it 'Time' and another that calls it datetime or obs_time then you don't know where to look. And also a generic Index column could have wider uses than timeseries, you could have a RA index or something else silly?
This can be considered completely separately from the issue of an index (we can call this truncate too instead).
2. If we have the above, do do we still want to provide a way for a table to have a 'main' index so as to allow:
t_sub = t.select(31122.12, 32221.11)
3. Do we want to have a TimeSeries class which sub-classes table and has a well-determined time attribute or column?
My feelings are:
yes because I find myself doing this all the time and it gets real messy
no because the syntax from 1 is simple enough that there's not much added by having a default index and having the notion of a main index results in a more complex code base
yes because then for the special case of a time series, we can have a 'default' index but just for a specific well defined case.
I also agree with your assessment of your three points. The next question becomes what is the simplest way to implement a index column in a Table subclass?
@Cadair - to answer that question, I guess I have to understand better what the motivation is for an index column. In my example 2 above it would be sufficient to have a table simply have a default designated index column as a simple attribute. But there must be other reasons why it would be useful? In 1, the nice thing is that you don't need to have a specific index column but you could have a few.
In the case of 3 you could simply have an attribute that keeps track of which column is the time column for example.
Select is a good idea and pretty easy. The only question for implementation is that if a column is sorted then you can use searchsorted and return a slice (wins on speed and memory), while if it isn't then you need to do a full boolean mask select. So there needs to be a way to either specify that a column is already sorted with an attribute (adds convenience but also complexity) or have a sorted=False arg in select.
We don't have to go there immediately, but there is a good reason SQL tables allow you to create an index. This opens up possibilities. It could be a Table subclass (IndexedTable)?
Sounds good.
For implementation the tricky bit for me is maintaining a sorted index. For instance you can just keep track of the index column (as @astrofrog said), but when you create the table and modify it in any way then you need to maintain the sort order. This is probably not terribly difficult but also not trivial to do in a complete way.
@justincely - great that you can spend effort on this! My suggestion is to try and come up with a simple implementation based on the discussion above and then demonstrate it - don't try and start a discussion on the API before having an implementation, that usually never leads very far!
My personal opinion is that we should have a simple time series class that sub-classes table and interfaces nicely with astropy.time. I agree we don't want to re-implement all of pandas, but at the same time I think we don't want to have to require pandas to be present in order to do a simple time series.
Irrespective of this issue, we should provide a good way to interface with Pandas from Table.
@astrofrog I agree having a astro specific time series class that plays nicely with the rest of astropy would have features that Pandas dosen't (and won't). However, we need to keep in mind where we are going to have to say, "if you want that use pandas".
Hi all. I just wanted to check on this issue now that some time has passed. I know that Tables have rudimentary support for indices in v1.1 but it still seems like nothing that is time series specific. Is the answer still to use pandas?
@wtgee and others - now that we have the basic indexing infrastructure which can be used on a Time column, what are the specific methods and functionality that are a top priority? As a reminder, here is what you can do right now in 1.1:
In [20]: t = Table([Time('2000-01-01') + [5,4,3,2,1] * u.day, [1,2,3,4,5]], names=['date', 'val'])
In [21]: t.add_index('date')
In [22]: t
Out[22]:
<Table length=5>
date val
object int64
----------------------- -----
2000-01-06 00:00:00.000 1
2000-01-05 00:00:00.000 2
2000-01-04 00:00:00.000 3
2000-01-03 00:00:00.000 4
2000-01-02 00:00:00.000 5
In [23]: t.loc[Time('2000-01-03 00:00:00.000')]
Out[23]:
<Row index=3>
date val
object int64
----------------------- -----
2000-01-03 00:00:00.000 4
In [24]: t.loc[Time('2000-01-03 00:00:00.000'):Time('2000-01-05 00:00:00.000')]
Out[24]:
<Table length=3>
date val
object int64
----------------------- -----
2000-01-03 00:00:00.000 4
2000-01-04 00:00:00.000 3
2000-01-05 00:00:00.000 2
On that recommendation I jumped right in and started using the Table class and it suits my purposes fine for now. The issue for me is that I am loading via HDF5 and don't know the explicit paths to the tables stored in the files. I can read them in via pandas and an HDFStore and then have access to all the paths as keys and the values as dataframes. From there I don't (yet) have a compelling reason to turn those DataFrames into astropy Tables since I already have them loaded. (This is more a generic Table issue and not related to the time series question so I don't want to go on that tangent here).
I agree that a good brain-storming of features/subtractions w.r.t pandas time series would be nice but I don't personally have any specific recommendations at this point. Thanks for the recommendation for the Table though, I'll start using it where appropriate for time series and see what I might be missing.
Excerpt of notes from time series unconference session at PyAstro16
What do we all want from a time series?
Re-sample (including recalc uncertainties properly), re-bin, time index columns that aren’t necessarily middle of the range but also start and end of bin like a histogram, time shifting or frame conversion, adding and subtracting, append, metadata, support for variable time bins, ability to store some kind of background time series, be monotonically increasing…
What do we want to do to a time series?
Plot, make a periodogram/power spectrum, do unit conversions between time units/systems (e.g. from different telescopes)
How would the implementation look?
Sub-class of Table? Separate module that can take a table? Not really any difference practically between these two approaches. Tom envisions the time column as an astropy time object. Does this approach support relative time? Yes, with deltatime concept. Probably makes the most sense as a sub-class of Table.
Steven: the advantage of pandas (over table) is it can do most things we want right out of the box (like rebinning).
Advantage of Table over pandas is units, metadata, fewer dependencies. Matt points out that pandas rebinning won’t let you account for, e.g. weighting as a function of flux or airmass if it’s changing in time.
Some people have performance concerns -- light curves with ~ 1 million timestamps
What’s the ‘no data’ value?
For Abbie and Daniela, it’s a zero. Brett and Mike don’t have this problem, but occasionally they want a nan when a cosmic ray hits the detector.
What’s the minimum an astropy time series table needs for you to use it?
Stingray: have evenly-spaced time binning, be able to re-bin, accept counts or count rate, convert between counts and count rate, concatenate, throw a warning for gappy data, how do you handle overlapping timestamps in concatenated data, base class should have multiple columns (but one time column)
SunPy: It should be able to sort itself along the time column (in computer speak, maintaining an index that is sorted). Monotonically increasing times (i.e., always be sorted). Re-sampling/interpolation. Adding and subtracting light curves with the same timestamps (Tom says it’s trivial! yay!). Some kind of metadata. Can hold a region of interest object somewhere somehow. Support for variable time bin sizes (so, index is a time range, not a timestamp; need two time columns, time start and time delta/time end? Would index on start time).
What should a Lightcurve class do?
Rebin (including recalculating uncertainties)
Time index column both middle of bin and bin edges
Time shifting/frame conversion (is there a WCS standard for time?)
Should always be sorted
Handling missing data
Truncate
Extend (add two together) → concatenate
Interpolate to new time stamps
Add/subtract light curves
Metadata (need to keep some information about energy ranges etc)
Regions of interest (in the light curve) or masks?
Values need units and uncertainties
Plot itself
Support for variable time bin sizes
Might want to have an optional background attribute or method?
Activity
astrofrog commentedon Mar 16, 2014
This follows from discussions with @Cadair and relates to #2204. This may be worked on in SunPy as part of a SunPy GSoC project.
embray commentedon Mar 17, 2014
While I agree completely, I'm starting to worry about how much we're reinventing the wheel that Pandas has already built. That said, Pandas has lots of features I think we don't care about (business weeks and such), and is probably missing things we do care about but I wouldn't know the details there.
taldcroft commentedon Mar 18, 2014
@Cadair - Are the time series in question sorted? If so, can we base things around selection via
np.searchsortedinstead of an index? We could add some convenience methods to make things easier.The pandas
locmethod is looking for exact keys and this doesn't seem like what's typically wanted in selecting astronomical time series values, but maybe I'm not understanding the application.Cadair commentedon Mar 18, 2014
Hello,
@embray Our LightCurve (TimeSeries) object currently uses Pandas underneath and pandas is just awesome, however we don't actually use a lot of pandas magic inside SunPy, having it available to the users is an important feature but we don't need to use it internally for that (a
.to_pandas()method would suffice.@taldcroft The timeseries would only be sorted in Table if they were sorted in the input correct? This of course would normally be the case.
What I mean by 'index' is having one column in this case a temporal column that is the row index. Thereby allowing selection of rows by index, filtering by index, truncation by index etc. Also allowing plotting by index etc. Basically in my mind it's a special column that you always know where it is i.e.
table.index. Then you can add special functionality using it.astrofrog commentedon Mar 18, 2014
Thinking about this more, I think the main issue here is that while filtering, truncation, and selection is of course easy and already possible:
or
it may be inefficient in cases where you know the time series is sorted, and in cases where one wants to select e.g. just the nearest entry or a very small subset.
@Cadair - it might be interesting to illustrate this point by comparing the performance of what you can do with Pandas versus what table selection as shown above already allows you to do? Then it might be clear in what cases the current functionality is inefficient and where we might do better?
Cadair commentedon Mar 18, 2014
Truncation allows you to do this:
which is very useful.
We can also return a SunPy
TimeRangeobject like this:The thing with the current table layout is that time can be a column and you can do a lot with it, but if you read a file that calls it 'Time' and another that calls it
datetimeorobs_timethen you don't know where to look. And also a generic Index column could have wider uses than timeseries, you could have a RA index or something else silly?astrofrog commentedon Mar 18, 2014
Right, so I guess there are different issues:
1. Do we want to allow a way to specify a simpler syntax than:
for example
This can be considered completely separately from the issue of an index (we can call this truncate too instead).
2. If we have the above, do do we still want to provide a way for a table to have a 'main' index so as to allow:
3. Do we want to have a
TimeSeriesclass which sub-classes table and has a well-determined time attribute or column?My feelings are:
@taldcroft @Cadair - what do you think?
astrofrog commentedon Mar 18, 2014
Sorry if above comment was confusing due to all points being numbered '1' - fixed that now!
Cadair commentedon Mar 18, 2014
@astrofrog your code example for 2. should presumeably be:
??
Cadair commentedon Mar 18, 2014
I also agree with your assessment of your three points. The next question becomes what is the simplest way to implement a index column in a
Tablesubclass?astrofrog commentedon Mar 18, 2014
@Cadair - whoops, fixed :)
astrofrog commentedon Mar 18, 2014
@Cadair - to answer that question, I guess I have to understand better what the motivation is for an index column. In my example 2 above it would be sufficient to have a table simply have a default designated index column as a simple attribute. But there must be other reasons why it would be useful? In 1, the nice thing is that you don't need to have a specific index column but you could have a few.
In the case of 3 you could simply have an attribute that keeps track of which column is the time column for example.
taldcroft commentedon Mar 18, 2014
searchsortedand return a slice (wins on speed and memory), while if it isn't then you need to do a full boolean mask select. So there needs to be a way to either specify that a column is already sorted with an attribute (adds convenience but also complexity) or have asorted=Falsearg inselect.For implementation the tricky bit for me is maintaining a sorted index. For instance you can just keep track of the index column (as @astrofrog said), but when you create the table and modify it in any way then you need to maintain the sort order. This is probably not terribly difficult but also not trivial to do in a complete way.
18 remaining items
astrofrog commentedon Sep 11, 2014
@justincely - great that you can spend effort on this! My suggestion is to try and come up with a simple implementation based on the discussion above and then demonstrate it - don't try and start a discussion on the API before having an implementation, that usually never leads very far!
My personal opinion is that we should have a simple time series class that sub-classes table and interfaces nicely with
astropy.time. I agree we don't want to re-implement all of pandas, but at the same time I think we don't want to have to require pandas to be present in order to do a simple time series.Irrespective of this issue, we should provide a good way to interface with Pandas from Table.
Cadair commentedon Sep 11, 2014
@astrofrog I agree having a astro specific time series class that plays nicely with the rest of astropy would have features that Pandas dosen't (and won't). However, we need to keep in mind where we are going to have to say, "if you want that use pandas".
wtgee commentedon Feb 24, 2016
Hi all. I just wanted to check on this issue now that some time has passed. I know that
Tables have rudimentary support for indices in v1.1 but it still seems like nothing that is time series specific. Is the answer still to use pandas?taldcroft commentedon Feb 24, 2016
@wtgee and others - now that we have the basic indexing infrastructure which can be used on a Time column, what are the specific methods and functionality that are a top priority? As a reminder, here is what you can do right now in 1.1:
wtgee commentedon Feb 26, 2016
On that recommendation I jumped right in and started using the
Tableclass and it suits my purposes fine for now. The issue for me is that I am loading via HDF5 and don't know the explicit paths to the tables stored in the files. I can read them in via pandas and anHDFStoreand then have access to all the paths as keys and the values as dataframes. From there I don't (yet) have a compelling reason to turn thoseDataFrames into astropyTables since I already have them loaded. (This is more a genericTableissue and not related to the time series question so I don't want to go on that tangent here).I agree that a good brain-storming of features/subtractions w.r.t pandas time series would be nice but I don't personally have any specific recommendations at this point. Thanks for the recommendation for the
Tablethough, I'll start using it where appropriate for time series and see what I might be missing.taldcroft commentedon May 3, 2016
Excerpt of notes from time series unconference session at PyAstro16
What do we all want from a time series?
Re-sample (including recalc uncertainties properly), re-bin, time index columns that aren’t necessarily middle of the range but also start and end of bin like a histogram, time shifting or frame conversion, adding and subtracting, append, metadata, support for variable time bins, ability to store some kind of background time series, be monotonically increasing…
What do we want to do to a time series?
Plot, make a periodogram/power spectrum, do unit conversions between time units/systems (e.g. from different telescopes)
How would the implementation look?
Sub-class of Table? Separate module that can take a table? Not really any difference practically between these two approaches. Tom envisions the time column as an astropy time object. Does this approach support relative time? Yes, with deltatime concept. Probably makes the most sense as a sub-class of Table.
Steven: the advantage of pandas (over table) is it can do most things we want right out of the box (like rebinning).
Advantage of Table over pandas is units, metadata, fewer dependencies. Matt points out that pandas rebinning won’t let you account for, e.g. weighting as a function of flux or airmass if it’s changing in time.
Some people have performance concerns -- light curves with ~ 1 million timestamps
What’s the ‘no data’ value?
For Abbie and Daniela, it’s a zero. Brett and Mike don’t have this problem, but occasionally they want a nan when a cosmic ray hits the detector.
What’s the minimum an astropy time series table needs for you to use it?
Stingray: have evenly-spaced time binning, be able to re-bin, accept counts or count rate, convert between counts and count rate, concatenate, throw a warning for gappy data, how do you handle overlapping timestamps in concatenated data, base class should have multiple columns (but one time column)
SunPy: It should be able to sort itself along the time column (in computer speak, maintaining an index that is sorted). Monotonically increasing times (i.e., always be sorted). Re-sampling/interpolation. Adding and subtracting light curves with the same timestamps (Tom says it’s trivial! yay!). Some kind of metadata. Can hold a region of interest object somewhere somehow. Support for variable time bin sizes (so, index is a time range, not a timestamp; need two time columns, time start and time delta/time end? Would index on start time).
What should a Lightcurve class do?
taldcroft commentedon May 3, 2016
See also https://github.com/Cadair/astropy-APEs/blob/master/APE9.rst.
pllim commentedon Feb 8, 2019
xref astropy/astropy-APEs#12
pllim commentedon Feb 8, 2019
I think this is about the new
astropy.timeseriesmodule, yes?astrofrog commentedon Feb 8, 2019
Yep!