Cosmology read & write method for AAS MRT

@nstarman

The AAS MRT (https://docs.astropy.org/en/stable/io/unified.html#built-in-table-readers-writers) format would be a nice addition to cosmology I/O.

Originally posted by @nstarman in #14837 (comment)

@nstarman

@nstarman Before I start writing the write_mrt() method I feel like the code below should work first:

from astropy.cosmology import Planck18
from astropy.io import ascii
ascii.write(Planck18, "ascii_write_mrt_check.mrt", format="mrt")

However, it seems to be throwing me an error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[17], line 1
----> 1 ascii.write(Planck18, "ascii_write_mrt_check.mrt", format="mrt")

File ~/opt/anaconda3/envs/Astropy_Playground/lib/python3.10/site-packages/astropy/io/ascii/ui.py:981, in write(table, output, format, Writer, fast_writer, overwrite, **kwargs)
    978     writer.write(table, output)
    979     return
--> 981 lines = writer.write(table)
    983 # Write the lines to output
    984 outstr = os.linesep.join(lines)

File ~/opt/anaconda3/envs/Astropy_Playground/lib/python3.10/site-packages/astropy/io/ascii/mrt.py:699, in Mrt.write(self, table)
    696 # Create a copy of the ``table``, so that it the copy gets modified and
    697 # written to the file, while the original table remains as it is.
    698 table = table.copy()
--> 699 return super().write(table)

File ~/opt/anaconda3/envs/Astropy_Playground/lib/python3.10/site-packages/astropy/io/ascii/core.py:1567, in BaseReader.write(self, table)
   1563 table = self.update_table_data(table)
   1565 # Check that table column dimensions are supported by this format class.
   1566 # Most formats support only 1-d columns, but some like ECSV support N-d.
-> 1567 self._check_multidim_table(table)
   1569 # Now use altered columns
   1570 new_cols = list(table.columns.values())

File ~/opt/anaconda3/envs/Astropy_Playground/lib/python3.10/site-packages/astropy/io/ascii/core.py:1369, in BaseReader._check_multidim_table(self, table)
   1352 def _check_multidim_table(self, table):
   1353     """Check that the dimensions of columns in ``table`` are acceptable.
   1354 
   1355     The reader class attribute ``max_ndim`` defines the maximum dimension of
   (...)
   1367         If any column exceeds the number of allowed dimensions
   1368     """
-> 1369     _check_multidim_table(table, self.max_ndim)

File ~/opt/anaconda3/envs/Astropy_Playground/lib/python3.10/site-packages/astropy/io/ascii/core.py:55, in _check_multidim_table(table, max_ndim)
     53 nd_names = [col.info.name for col in table.itercols() if len(col.shape) > max_ndim]
     54 if nd_names:
---> 55     raise ValueError(
     56         f"column(s) with dimension > {max_ndim} "
     57         "cannot be be written with this format, try using 'ecsv' "
     58         "(Enhanced CSV) format"
     59     )

ValueError: column(s) with dimension > 1 cannot be be written with this format, try using 'ecsv' (Enhanced CSV) format

On the contrary, the latex implementation below seem to work just fine:

from astropy.cosmology import Planck18
from astropy.io import ascii
ascii.write(Planck18, "ascii_write_latex_check.tex", format="latex", overwrite=True)

I wouldn't expect that to work because Planck18 isn't a Table. Can you try?

from astropy.cosmology import Planck18
from astropy.io import ascii
ascii.write(Planck18.to_format("astropy.table"), "ascii_write_mrt_check.mrt", format="mrt")

I wouldn't expect that to work because Planck18 isn't a Table. Can you try?
from astropy.cosmology import Planck18
from astropy.io import ascii
ascii.write(Planck18.to_format("astropy.table"), "ascii_write_mrt_check.mrt", format="mrt")

It still gives me the same error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[11], line 3
      1 from astropy.cosmology import Planck18
      2 from astropy.io import ascii
----> 3 ascii.write(Planck18.to_format("astropy.table"), "ascii_write_mrt_check.mrt", format="mrt")

File ~/opt/anaconda3/envs/Astropy_Playground/lib/python3.10/site-packages/astropy/io/ascii/ui.py:981, in write(table, output, format, Writer, fast_writer, overwrite, **kwargs)
    978     writer.write(table, output)
    979     return
--> 981 lines = writer.write(table)
    983 # Write the lines to output
    984 outstr = os.linesep.join(lines)

File ~/opt/anaconda3/envs/Astropy_Playground/lib/python3.10/site-packages/astropy/io/ascii/mrt.py:699, in Mrt.write(self, table)
    696 # Create a copy of the ``table``, so that it the copy gets modified and
    697 # written to the file, while the original table remains as it is.
    698 table = table.copy()
--> 699 return super().write(table)

File ~/opt/anaconda3/envs/Astropy_Playground/lib/python3.10/site-packages/astropy/io/ascii/core.py:1567, in BaseReader.write(self, table)
   1563 table = self.update_table_data(table)
   1565 # Check that table column dimensions are supported by this format class.
   1566 # Most formats support only 1-d columns, but some like ECSV support N-d.
-> 1567 self._check_multidim_table(table)
   1569 # Now use altered columns
   1570 new_cols = list(table.columns.values())

File ~/opt/anaconda3/envs/Astropy_Playground/lib/python3.10/site-packages/astropy/io/ascii/core.py:1369, in BaseReader._check_multidim_table(self, table)
   1352 def _check_multidim_table(self, table):
   1353     """Check that the dimensions of columns in ``table`` are acceptable.
   1354 
   1355     The reader class attribute ``max_ndim`` defines the maximum dimension of
   (...)
   1367         If any column exceeds the number of allowed dimensions
   1368     """
-> 1369     _check_multidim_table(table, self.max_ndim)

File ~/opt/anaconda3/envs/Astropy_Playground/lib/python3.10/site-packages/astropy/io/ascii/core.py:55, in _check_multidim_table(table, max_ndim)
     53 nd_names = [col.info.name for col in table.itercols() if len(col.shape) > max_ndim]
     54 if nd_names:
---> 55     raise ValueError(
     56         f"column(s) with dimension > {max_ndim} "
     57         "cannot be be written with this format, try using 'ecsv' "
     58         "(Enhanced CSV) format"
     59     )

ValueError: column(s) with dimension > 1 cannot be be written with this format, try using 'ecsv' (Enhanced CSV) format

Ok. That means that the writer is actually successfully converting the cosmology object to a table. Neat! Unfortunately, the m_nu is a structured array, which MRT apparently doesn't support. The solution is to:

convert to a table
Replace the m_nu column with N columns, where N is the number of elements of m_nu
Save as an MRT

@nstarman

@nstarman Thanks for the suggestions. I followed your steps and the results overall look good to me.

from astropy.cosmology import Planck18
from astropy.io import ascii

# Step 1: Convert cosmology object to a table
table = Planck18.to_format("astropy.table")

# Step 2: Replace the m_nu column with N columns
m_nu = table['m_nu']  # Get the m_nu column
N = len(m_nu[0])  # Get the number of elements in m_nu

# Remove the m_nu column from the table
table.remove_column('m_nu')

# Add N new columns to the table with names 'm_nu_0', 'm_nu_1', 'm_nu_2', ...
for i in range(N):
    column_name = f'm_nu_{i}'
    column_data = m_nu[0][i]
    table.add_column(column_data, name=column_name, index=-2)
    
print(table)
# Step 3: Save as an MRT
ascii.write(table, "ascii_write_mrt_check.mrt", format="mrt", overwrite=True)

The output for print(table) looks promising:

  name        H0        Om0   Tcmb0   Neff m_nu_0 m_nu_1 m_nu_2   Ob0  
         km / (Mpc s)           K            eV     eV     eV          
-------- ------------ ------- ------ ----- ------ ------ ------ -------
Planck18        67.66 0.30966 2.7255 3.046    0.0    0.0   0.06 0.04897

However, the ascii_write_mrt_check.mrt file has Unknown labels in place of actual labels like H0, Tcmb0, etc. Is this because of units?

Title:
Authors:
Table:
================================================================================
Byte-by-byte Description of file: table.dat
--------------------------------------------------------------------------------
 Bytes Format Units  Label     Explanations
--------------------------------------------------------------------------------
 1- 8  A8           ---    name    Description of name
10-14  F5.2   km.Mpc-1.s-1 Unknown [67.66/67.66] Hubble constant as an
                            `~astropy.units.Quantity` at z=0.
16-22  F7.5         ---    Om0     [0.3/0.31] Omega matter; matter
                            density/critical density at z=0.
24-29  F6.4         K      Unknown [2.72/2.73] Temperature of the CMB as
                            `~astropy.units.Quantity` at z=0.
31-35  F5.3         ---    Neff    [3.04/3.05] Number of effective neutrino
                            species.
37-39  F3.1         eV     Unknown Description of Unknown
41-43  F3.1         eV     Unknown Description of Unknown
45-48  F4.2         eV     Unknown [0.06/0.06] Description of Unknown
50-56  F7.5         ---    Ob0     [0.04/0.05] Omega baryon; baryonic matter
                            density/critical density at z=0.
--------------------------------------------------------------------------------
Notes:
--------------------------------------------------------------------------------
Planck18 67.66 0.30966 2.7255 3.046 0.0 0.0 0.06 0.04897

Can you try getting rid of the underscores in the name?

Also, I haven't read the spec carefully. What's meant to be in the notes section; can we put the metadata there?

Can you try getting rid of the underscores in the name?

Do you mean changing m_nu_0, m_nu_1, m_nu_2 to m_nu0, m_nu1, m_nu2. I tried this but doesn’t help with the Unknown labels.

Also, I haven't read the spec carefully. What's meant to be in the notes section; can we put the metadata there?

Yes, we can but I believe ascii.write() method doesn’t support passing the metadata directly as a keyword argument. As the Mrt documentation says "Note that the metadata of the table, apart from units, column names and description, will not be written. These have to be filled in by hand later."

Well that makes it easier 🙂. Let's just not do the metadata for now. That's a good followup PR.

Do you mean changing m_nu_0, m_nu_1, m_nu_2 to m_nu0, m_nu1, m_nu2. I tried this but doesn’t help with the Unknown labels.

How about mnu0?

How about mnu0?

Again it looks like this when I do print(table):

  name        H0        Om0   Tcmb0   Neff mnu0 mnu1 mnu2   Ob0  
         km / (Mpc s)           K           eV   eV   eV         
-------- ------------ ------- ------ ----- ---- ---- ---- -------
Planck18        67.66 0.30966 2.7255 3.046  0.0  0.0 0.06 0.04897

However, the .mrt file still looks the same as before

@taldcroft

This file type has better I/O, so one PR is probably better, so that there can be round trip tests.

Thanks @taldcroft for the serialization help.

Hi,
I just read through this long issue and another one (#14837). I also checked PRs #14883 and #17279. It looks like the PRs have been closed while the issues remain open.

I just wanted to check—do we no longer need AAS MRT? If I understand correctly, we don’t need it anymore because we found a workaround. Should we close these issues to avoid confusion for others who might spend time working on them without knowing the current status? 😄

No, I still think it would be good to support this. The unresolved issue appears to be how best to write columns like m_nu.

@nstarman

Hi @nstarman

I noticed pull requests #11897 which is add MRT format writer to io.ascii is that going to be similar to this ?

take

It appears I can’t assign this to myself by simply typing "take"—it would make me look silly. anyway, I'm going to dive into this issues.

😆

We don't assign issues. Multiple people can propose multiple solutions. It is up to maintainers to decide which one to accept. :)

@nstarman

Hi @nstarman ,I'm working on this issue. After investigating, I believe the bottleneck is in the column m_nu. It appears that pull request #14883 attempts to split the m_nu column and generates the mrt file , which doesn’t seem ideal.

I also notice @taldcroft motioned we are using JSON-encoded string (e.g, [0.0,0.0,0.06]) in ecsv, which seems like a better approach compared to the current pull.

Looking at ecs.py in astropy/cosmology, the script simply does :

table = to_table(
        cosmology, cls=cls, cosmology_in_meta=cosmology_in_meta, rename=rename
    )

kwargs["format"] = "ascii.ecsv"
table.write(file, overwrite=overwrite, **kwargs)

if I understand correctly, the logic part that converts 0.0 ... 0.06 to [0.0,0.0,0.06] is implemented in astropy/io/ecsv.py , Please let me know if I've got right,

if yes, we might need to apply similar soultion in astropy/io/mrt.py , and I'd appreciate ur thoughts since I currently reviewing the astropy/io code

@taldcroft

I also notice @taldcroft motioned we are using JSON-encoded string (e.g, [0.0,0.0,0.06]) in ecsv, which seems like a better approach compared to the current pull.

Agreed!

if I understand correctly, the logic part that converts 0.0 ... 0.06 to [0.0,0.0,0.06] is implemented in astropy/io/ecsv.py , Please let me know if I've got right,

Yes. A potential solution would be to modify the table so that the m_nu column has an ndarray with an object dtype and the objects are the tuples.

@taldcroft

Thank you for clarifying. I'm currently investigating how ECSV works, and I came across this snippet from astropy/io/ascii:

if len(col.shape) > 1 or col.info.dtype.kind == "O":

                def format_col_item(idx):
                    obj = col[idx]
                    try:
                        obj = obj.tolist()
                    except AttributeError:
                        pass
                    return json.dumps(obj, separators=(",", ":"))

            else:

                def format_col_item(idx):
                    return str(col[idx])

            try:
                col.str_vals = [format_col_item(idx) for idx in range(len(col))]
            except TypeError as exc:
                raise TypeError(
                    f"could not convert column {col.info.name!r} to string: {exc}"
                ) from exc

I think this is part of code @taldcroft mentioned about use JSON for object-type or multidim values, let me know if I am wrong,

I also found function _check_multidim_table from astropy/io/ascii .
it seems like currently only ECSV supports multidim values, which is why the col m_nu got error in mrt.

so we get two solutions here,

we modify mrt.py in astropy/ascii/ to make mrt can deal with multidim values,
we change mrt.py in astropy/cosmlogy/_src/io/builtin change multidim values from table then feed "right" table to astropy/ascii/mrt.py

let me know what u think

@kangqiwang

@kangqiwang that looks like the right bit of code! For now definitely make the modifying changes in cosmology, not for all tables.

update mrt based on astropy#14871

update mrt based on #14871

Navigation Menu

Repository navigation

Cosmology read & write method for AAS MRT #14871

5 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Cosmology read & write method for AAS MRT #14871

Description

Activity

Mubin17 commented on May 26, 2023

nstarman commented on May 27, 2023

Mubin17 commented on May 27, 2023

nstarman commented on May 27, 2023

Mubin17 commented on May 27, 2023

nstarman commented on May 27, 2023

nstarman commented on May 27, 2023

Mubin17 commented on May 27, 2023

nstarman commented on May 27, 2023

nstarman commented on May 27, 2023

Mubin17 commented on May 27, 2023

5 remaining items

nstarman commented on May 28, 2023

kangqiwang commented on Mar 20, 2025

nstarman commented on Mar 21, 2025

kangqiwang commented on Mar 27, 2025

kangqiwang commented on Mar 27, 2025

kangqiwang commented on Mar 27, 2025

pllim commented on Mar 27, 2025

kangqiwang commented on Apr 2, 2025

nstarman commented on Apr 2, 2025

kangqiwang commented on Apr 3, 2025

nstarman commented on Apr 3, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions