What packages should we use to store data?

@bla1089 and I were just talking in person and ran into the following question: What package or packages should we use to store data?

Some possibilities include:

NumPy (which we are mostly using thus far)
pandas
xarray
yt

Some of the qualities that I think would be very much worth having are:

Units-aware or units-compatible
Ability to label dimensions (like pandas does for 1D and xarray does for multidimensional arrays)
Compatible with flexible data sources
- Common data types used by lab, space, astro, and solar
- Fluid and particle representations (yt has better particle capabilities since the "demeshening")
Can be easily installed using tools like conda
Wide user/developer community
- Ideally, we be able to make upstream contributions too
Fast and easy-to-use

This is related to #66 and #59, including this comment.

Maybe this topic would be a good thing to write our first PlasmaPy Enhancement Proposal about! (I'm hoping that our zeroeth PlasmaPy Enhancement Proposal will be to have a GitHub repository for bad poetry about eigenfunctions.)

What about finding a way to be agnostic to some of these other packages? One way I get around subclassing things like numpy and pandas is doing the following.

    class Plasma(object):
        __init__(self, data):
            self.set_data(data)
        @property
        def data(self):
            return self._data
        def set_data(self, new):
             assert isinstance(new, (pd.DataFrame, xr.DataArray, ....))
             self._data = new
        
        def do_physics(self, *args, *kwargs):
              data = self.data
              out = physics(data)
              return out

If we do something like this, there are huge piles of code I can contribute to the project (10,000s of lines, across many files). There are also benefits of doing things like not assuming a specific particle species. For example, some solar wind people include ions of oxygen and nitrogen, among others, along with the typical protons. But some people really only care about protons or protons & electrons. Being agnostic to these by developing the code in appropriate ways would make the PlasmaPy project accessible to a much larger field of users.

It's also worth noting that with something like Pandas we get to leverage all of the Cython level code that the package has completed.

Hey all,

I dig into yt (on their site ). They seem to have everything we need !!

units and dimensions label
can load lots of common data type (see this list )
grid and particle representations as well as data-frame like Pandas
conda's supported
has a large community
seems fast and easy from the examples, I did not check it myself.

Moreover :

They use Cython and can be used in parallel (see here)
There are lots of implemented functions, from field analysis to particle to cell
a lot of work on visualization as been done

On my opinion, yt seems to be the best solution. On the other hand, I never tried it myself, so I am not confident in building a whole package over it.

I don't know how would you like to make the decision (by a vote, by example ?). I think we can fix a "decision date" like September the 1rst, let everyone think about it and then make the decision.

PS : @bla1089 in your example : assert isinstance(new, (pd.DataFrame, xr.DataArray, ....)) do you expect new to be either a pd or a xr data, and PlasmaPy to be able to deal with both of them or did you used the two only for the sake of the example ?

@antoinelpp I expected `new` to be any instance we wanted to account for. DataFrame and DataArray were just the examples that immediately came to mind.

On Wed, Aug 23, 2017 at 4:29 AM antoinelpp ***@***.***> wrote: Hey all, I dig into yt (on their site <http://yt-project.org/doc/index.html> ). They seem to have everything we need !! - units and dimensions label - can load lots of common data type (see this list <http://yt-project.org/doc/examining/index.html> ) - grid and particle representations as well as data-frame like Pandas - conda's supported - has a large community - seems fast and easy from the examples, I did not check it myself. Moreover : - They use Cython and can be used in parallel (see here) <http://yt-project.org/doc/analyzing/parallel_computation.html> - There are lots of implemented functions, from field analysis to particle to cell - a lot of work on visualization as been done On my opinion, *yt seems to be the best solution*. On the other hand, I never tried it myself, so I am not confident in building a whole package over it. I don't know how would you like to make the decision (by a vote, by example ?). I think we can fix a "decision date" like *September the 1rst*, let everyone think about it and then make the decision. PS : @bla1089 <https://github.com/bla1089> in your example : assert isinstance(new, (pd.DataFrame, xr.DataArray, ....)) do you expect new to be either a pd or a xr data, and PlasmaPy to be able to deal with both of them or did you used the two only for the sake of the example ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#87 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AMPWVZlUbKiX4p56rZ7xa4BijJbJciEUks5sa-LsgaJpZM4OxSZ8> .

-- Ben

@bla1089's abstraction idea seems reasonable. I'm not 100% sure how it'd work in practice just yet, but... I'm pretty sure I'd need pointers on implementing this kind of thing.

As for data structures, at a glance yt does seem to be the better fit at least for huge simulation etc data, given that it also seems to make grid visualization easier. But it seems like there's nobody among us but Cadair who's used it yet! I've started learning it, for what it's worth.

@SolarDrew

@SolarDrew has used it a bit as well.

Indeed. I've found it really good for storing and plotting simulation data, but I don't know what kind of support it has for timeseries-type data of the kind @bla1089 has been talking about.

That said, I think it's great for pretty much everything else though, so in practice I think the real solution is going to be to use both as appropriate and try to provide consistent wrappers so the user doesn't need to worry what's underneath.

I agree. It's also worth looking at other developments in regards to spacecraft data. For example: pysat: https://github.com/rstoneback/pysat pysatCDF: https://github.com/rstoneback/pysatCDF Part of why I like the idea of pandas & xarray is that they are designed to be very similar, simplifying the early part of the learning curve. Just to point out, we could decided that simulation and experimental data just need to be treated differently. For example, we might decide that a plasmapy/sim and plasmapy/experiment might be necessary to keep things clean and we develop routines to format simulation output for comparing to experimental data and vice versa.

On Thu, Aug 24, 2017 at 2:20 PM Drew Leonard ***@***.***> wrote: Indeed. I've found it really good for storing and plotting simulation data, but I don't know what kind of support it has for timeseries-type data of the kind @bla1089 <https://github.com/bla1089> has been talking about. That said, I think it's great for pretty much everything else though, so in practice I think the real solution is going to be to use both as appropriate and try to provide consistent wrappers so the user doesn't need to worry what's underneath. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#87 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AMPWVa8zYIGgJq0dCakZsfVnbHpYq7Umks5sbb8FgaJpZM4OxSZ8> .

-- Ben

Sure, that sounds pretty reasonable. I wouldn't mind this being the semi-final consensus we reach, as we'll probably eventually need to split simulation and experiment up anyway.

Then the question inevitably becomes: how do we want to name it? Do we want it to be PlasmaPy and have two separate parts in it, an experiment (PlasmaExp) and a simulation (PlasmaSim) module with potentially common routines or do we want the modules to be entirely separate and retire the PlasmaPy name?

On Thu, Aug 24, 2017 at 3:16 PM Dominik Stańczak ***@***.***> wrote: Sure, that sounds pretty reasonable. I wouldn't mind this being the semi-final consensus we reach, as we'll probably eventually need to split simulation and experiment up anyway. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#87 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AMPWVcZ6PgpNNNzGGfx03SFE7EULx_nlks5sbcwpgaJpZM4OxSZ8> .

-- Ben

Yeah that's pretty reasonable, I had considered we might have to do that. If possible I would like to keep a single mostly-consistent API, or at least have a single contructor that returns an appropriate subclass of Plasma, a la SunPy Maps. But yes, if necessary having two would be a decent compromise.

And with regards to the name I'd say we still want to keep everything within PlasmaPy and have experiment and simulation (or whatever) as top-level submodules.

What about HDF5? We can use PyTables, Pandas, and h5py to handle HDF5 files in different ways (with different learning curves). There is already archiving and compression functionality built-in. It's cross-platform, highly-highly-optimized (fast read/write for massive simulation/experimental data files), and language agnostic. This latter point would be important for converting to other formats like CDF.

Also, just for the record here, someone is working on a pure-python implementation of a CDF package: https://github.com/MAVENSDC/cdflib

I believe it is currently read only.

I know of several people working on a pure python CDF library. It seems like there are a lot of people duplicating this work. Perhaps it's best to stay away from this one. Ben

…

--------------------- B. L. Alterman Candidate, Applied Physics Solar and Heliospheric Research Group Climate and Space Sciences and Engineering University of Michigan [email protected]

On Mon, Oct 30, 2017 at 8:21 AM, Stuart Mumford ***@***.***> wrote: Also, just for the record here, someone is working on a pure-python implementation of a CDF package: https://github.com/MAVENSDC/cdflib I believe it is currently read only. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#87 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AMPWVX-CRYktN0cPy-POkCSZGja0hRH-ks5sxb85gaJpZM4OxSZ8> .

Do you have links to the others? Perhaps we should get them talking? This one is being written by someone I know at GSFC.

I not worth it. I know the people working on several. It's one of those things that's just going to happen and settle out in time.

…

On Oct 31, 2017 03:40, "Stuart Mumford" ***@***.***> wrote: Do you have links to the others? Perhaps we should get them talking? This one is being written by someone I know at GSFC. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#87 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AMPWVcCvwPNHqyYAznuwxFnZ29dAjaGcks5sxs8BgaJpZM4OxSZ8> .

Navigation Menu

Repository navigation

What packages should we use to store data? #87

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

What packages should we use to store data? #87

Description

Activity

blalterman commented on Aug 8, 2017

blalterman commented on Aug 8, 2017

antoinetavant commented on Aug 23, 2017

blalterman commented on Aug 23, 2017

StanczakDominik commented on Aug 24, 2017

Cadair commented on Aug 24, 2017

SolarDrew commented on Aug 24, 2017

blalterman commented on Aug 24, 2017

StanczakDominik commented on Aug 24, 2017

blalterman commented on Aug 24, 2017

SolarDrew commented on Aug 24, 2017

SolarDrew commented on Aug 24, 2017

lemmatum commented on Oct 29, 2017

Cadair commented on Oct 30, 2017

blalterman commented on Oct 31, 2017

Cadair commented on Oct 31, 2017

blalterman commented on Oct 31, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions