Support showing interactive tables in Jupyter Notebooks (again!)

The Table.show_in_notebook() method was deprecated in #15905 due to issues regarding the compatibility and maintainability of JS code in astropy with developments in Jupyter Notebook (JupyterLab frontend components got introduced in Notebook v7 onwards which broke the js code).

Alternative solutions were considered, namely ipydatagrid, and itables, and after some discussions on dependency management on the slack, my understanding is that the final consensus came down to deprecating it for the time being? I was wondering if adding back proper support (likely via the third-party Jupyter-datatable-package route) is still on the radar as people did seem to use it.

For additional context, this was something that came up when we were considering having searchable Tables over at sunpy and there's a chance that despite my best efforts I could still be missing some astropy context here. I'd be happy to work on a PR here upstream if this idea still seems good!

Welcome to Astropy 👋 and thank you for your first issue!

A project member will respond to you as soon as possible; in the meantime, please double-check the guidelines for submitting issues and make sure you've provided the requested details.

GitHub issues in the Astropy repository are used to track bug reports and feature requests; If your issue poses a question about how to use Astropy, please instead raise your question in the Astropy Discourse user forum and close this issue.

If you feel that this issue has not been responded to in a timely manner, please send a message directly to the development mailing list. If the issue is urgent or sensitive in nature (e.g., a security vulnerability) please send an e-mail directly to the private e-mail [email protected].

Part of why I do not like this as part of astropy is that we need to maintain it. We are all not js experts, and a very small fraction of astropy uses js, but it makes up about 50% of the reported security vulnerabilities. We hotpached the datatables version included in astropy in the past at some point for that reason. Compared to the vast majority of Python code, that's a pretty high responsibility - and we have users who, through company or institutional rules, are not allowed to use astropy, while is ships with some js library with a known vulnerability, even if that's in a sub-package that they never use. Similar for linux packagers - bundled js might have to unpackaged and patched for every release.

So, yes, people use it (I do about once a year), but is that really worth the effort? Can't we instead advertise the tab.to_pnadas() hook and leave it to pandas to maintain that functionality? Or, if someone wants to support a different package, add a new hook like table.to_itables()?

I'm fine with any of that, as long as the js code is not shipped in astropy itself; but see #16629 for an example where we did not even ship a specific library, but only recommended it in the documentation (and that example was taken straight out of the mathjax docs) and still, the volatile and evil nature of some actors on the internet, led to a situation where our recommendation ended up downright hostile.

Fortunately, someone watched and alerted us - but on our own we totally would have missed that. So, I have reservations not because I don't think it's useful, but because I'm not convinced it's useful enough to make up to the downsides that came with js for our project.

Thanks for your response! I fully agree that having JS code like that in astropy is not at all ideal for a plethora of reasons.

Or, if someone wants to support a different package, add a new hook like table.to_itables()?
I'm fine with any of that, as long as the js code is not shipped in astropy itself

This is pretty much exactly what I'm trying to get at here. I think interactive tables directly in Notebook could be a nice feature to add back like this in a more sustainable and reliable manner (via an external package like ipytables). And I wanted to see if the idea sits well with the project development community.

If it sounds good, I'll also be available to work on contributing this!

Thanks for your enthusiasm! Can you please elaborate a little more on your plan on "likely via the third-party Jupyter-datatable-package route"?

oh there I meant to refer to using something like ipydatagrid or itables for implementing the interactive tables. I put it that way since there could still be room for discussion on which tool to choose.

I did kinda explore implementing this with ipydatagrid already when looking into sunpy/sunpy#7639 (which was where i realised that a discussion here could be of benefit).

here's a rough (as in not well formatted as of yet) screenshot for ipydatagrid tables over sunpy data for an idea of how it could end up looking:

I hope this helps!

also for a quick comment for the discussion on which tool to use, (that is in case this feature sounds fitting enough in the first place haha) some of the other prominent alternatives I ended up exploring outside of ipydatagrid included Bokeh and itables and I found ipydatagrid the most suitable.
I found Bokeh to be a good choice for projects going for more extensive visualisations instead of a one-off // standalone feature like our requirement for a datagrid here, and itables appears to change how dataframes are rendered Notebook-wide which I felt isn't something we're going for.

Looking at https://github.com/bloomberg/ipydatagrid?tab=readme-ov-file#Usage-and-Examples , it looks like you can do this already (sort of) without any changes to astropy ?

Convert to pandas (along the lines of what Moritz suggested above).
Pass pandas DataFrame into DataGrid.
Visualize all you want.

I still don't see much benefits having to inject a bunch of ipywidgets stuff into astropy.table.Table. Can you please show us what you attempted for SunPy? Thanks!

@taldcroft

Reposting from #15905 (comment)

@taldcroft There is this interesting news: https://blog.jupyter.org/ipydatagrid-is-now-part-of-project-jupyter-3b3dfb877664

I haven't looked into it carefully, but maybe we can leverage ipydatagrid and provide interactive Table display using this as the optional (?) dependency. BTW standalone ipydatagrid install is a little lighter at 3.1 Mb.

I spent a bit more time digging into ipydatagrid and playing with it. I am a little disappointed with the documentation, which seems to consist of basically installation and a few examples. I also tried a few of the example notebooks and two of three failed to run (giving a KeyError deep in the code).

That said, making a full-featured interactive table in a notebook cell is extremely simple. Here could be the basic code for the new show_in_notebook() method:

def show_in_notebook(self, **datagrid_kwargs):
    from ipydatagrid import DataGrid
    return DataGrid(table.to_pandas(), **datagrid_kwargs)
    # Voila!!

This returns an object that is very flexible and powerful, with basically just a few lines of astropy-maintained code. I think we should just do this for 7.0. The rest of a PR would just be handling this new optional dependency.

Does that mean we basically un-deprecate show_in_notebook (deprecated in v6.1) after 1 release? If so, maybe should PR soon given the impending feature freeze. Thanks! 🙏

Does that mean we basically un-deprecate show_in_notebook (deprecated in v6.1) after 1 release? If so, maybe should PR soon given the impending feature freeze. Thanks! 🙏

Yes and yes. 😄

#17165 🤞

Navigation Menu

Support showing interactive tables in Jupyter Notebooks (again!) #16702

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Support showing interactive tables in Jupyter Notebooks (again!) #16702

Description

Activity

github-actions commented on Jul 10, 2024

hamogu commented on Jul 11, 2024

exitflynn commented on Jul 11, 2024

pllim commented on Jul 12, 2024

exitflynn commented on Jul 12, 2024

exitflynn commented on Jul 12, 2024

pllim commented on Jul 15, 2024

pllim commented on Oct 9, 2024

taldcroft commented on Oct 10, 2024

pllim commented on Oct 10, 2024

taldcroft commented on Oct 11, 2024

pllim commented on Oct 11, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions