Improve formatting of Table/Column.info() - #12022

👋 Thank you for your draft pull request! Do you know that you can use [ci skip] or [skip ci] in your commit messages to skip running continuous integration tests until you are ready?

I'm slightly ambivalent about this since you have functions like std where a reasonable precision for the values may not be reasonable for the std. E.g. if the data values have a mean of 1_000_000 but are very close with a std of 0.01.

What about formatting floating point outputs with 6g by default? I think in 99% of cases this is fine and will give a clean readable output.

Beyond that there are some technical problems with the current implementation.

• Integer values with a format like 05d
• You cannot really apply simple text-based logic like if fmt.startswith('{').

Perhaps the right thing (if we really want to inherit the column formatting, which I'm not convinced of) would be to mimic the pprint logic and do a try/except for each of these possibilities in order:

# From pprint.py
def _possible_string_format_functions(format_):
    """Iterate through possible string-derived format functions.

    A string can either be a format specifier for the format built-in,
    a new-style format string, or an old-style format string.
    """
    yield lambda format_, val: format(val, format_)
    yield lambda format_, val: format_.format(val)
    yield lambda format_, val: format_ % val

Then if all of those fail, fall through to no formatting. This will allow for integer formats that don't apply to floating point info values.

I have to start by clarifying that calling t.info() only prints a summary of the table, it's return value is always None. The code examples I had provided were misleading because they included print(t.info()) and I have corrected them.

I'm slightly ambivalent about this since you have functions like std where a reasonable precision for the values may not be reasonable for the std. E.g. if the data values have a mean of 1_000_000 but are very close with a std of 0.01.

If the user has specified a column format that makes the values indistinguishable then it should be expected that the standard deviation would appear to be zero. If the user actually wants to use the standard deviation for something then they can still compute it and the formatting does not affect the value they obtain.

What about formatting floating point outputs with 6g by default? I think in 99% of cases this is fine and will give a clean readable output.

I see no reason why the user shouldn't be able to specify the formatting of the stats and using the column format would be a very natural way of achieving that. That being said the mean and standard deviation of an integer column are floats, so the column format might not be applicable. In that case falling back to a sensible default like 6g would indeed be a good idea.

Beyond that there are some technical problems with the current implementation.

* Integer values with a format like `05d`

* You cannot really apply simple text-based logic like `if fmt.startswith('{')`.

Perhaps the right thing (if we really want to inherit the column formatting, which I'm not convinced of) would be to mimic the pprint logic and do a try/except for each of these possibilities in order:

# From pprint.py
def _possible_string_format_functions(format_):
    """Iterate through possible string-derived format functions.

    A string can either be a format specifier for the format built-in,
    a new-style format string, or an old-style format string.
    """
    yield lambda format_, val: format(val, format_)
    yield lambda format_, val: format_.format(val)
    yield lambda format_, val: format_ % val

I don't understand why it is necessary to try these different options, but maybe I'm just not familiar enough with string formatting. Can you provide an example that does not work with the text-based logic?

Then if all of those fail, fall through to no formatting. This will allow for integer formats that don't apply to floating point info values.

What I would propose is that the stats of a column should be formatted using the column format if it is specified. If the format is not applicable or it is not specified then applying a sensible default should be attempted. If that fails then then the code should fall back to current behavior.

I don't understand why it is necessary to try these different options, but maybe I'm just not familiar enough with string formatting. Can you provide an example that does not work with the text-based logic?

In [21]: t = simple_table()                                                                                                                          

In [22]: t['b'].info.format = '${:.2f}'                                                                                                              

In [23]: t                                                                                                                                           
Out[23]: 
<Table length=3>
  a      b     c  
int64 float64 str1
----- ------- ----
    1   $1.00    c
    2   $2.00    d
    3   $3.00    e

Another thing to remember is that this is user-customizable for any functions, so applying the column format won't necessarily be appropriate in those cases (as indeed it is already not appropriate for 'attributes').

I'm not too keen on the behavior of this simple convenience function getting complicated with the various fall-throughs (like going to 6g for int columns where the func(col) is float, but going to str(val) where that doesn't apply).

I also think this is a bit of the tail wagging the dog, since the PR was mostly driven by the need to contrive a doc example that looks tidy and passes tests. But the table in the doc example would not normally require format attributes to be set, so putting them in just so the stats are formatted is not something a user would really do.

If the user has specified a column format that makes the values indistinguishable then it should be expected that the standard deviation would appear to be zero.

An important thing in astropy is to understand that there are a huge world of use-cases and it is important to try avoiding making assumptions based on what you believe is sensible or expected. In particular if someone chooses to format floats to the nearest int .0f because the table needs to be compact, they may still want to see the std to full precision.

The worst case is to give an answer that appears to be definitive (e.g. std=0) but requires understanding of astropy details ("oh, the .0f format was applied to the real answer of 0.489232"). The user might have read the table in from an ECSV file they didn't create and not even be aware of the formatting.

There are two questions that need a definitive answer.

1. Should it be possible for the user to specify the formatting of the stats? This does not necessarily mean that the output of the 'stats' option would have to be formatted (although this PR originally proposed that it should be). It would also be possible to create a new 'formatted_stats' option so that the user would have full control over whether or not they want to apply column formats, and implementing a separate option for formatted output would not be any more complicated than changing the 'stats' option would be.
2. Should the default format for the stats be something different from what it is now? This is an example of how that might look like:

t.info('stats')    # with default format {:g}
<Table length=5>
name mean   std   min max
---- ---- ------- --- ---
   a    6 4.24264   0  12
   b    7 4.24264   1  13
   c    8 4.24264   2  14

If 1. is considered to be worthwhile then it can be implemented. 2. can be implemented with or without 1.

My opinion is that this is not something where we really need the flexibility / control implied by (1). For reference there is no way to configure the output format of pandas describe().

In [12]: tp.describe()                                                                                 
Out[12]: 
               a          b
count   3.000000   3.000000
mean   16.666667  16.666667
std    24.542480  24.542480
min     2.000000   2.000000
25%     2.500000   2.500000
50%     3.000000   3.000000
75%    24.000000  24.000000
max    45.000000  45.000000

So IMO just a simple change of formatting all float outputs with {:g} would be an improvement over the current behavior and would be sufficient for the goals of info().

Others may disagree and a free discussion here is a good thing.

Who is supposed to review this?

@eerovaher - I'll review, though not today since I am completely tied up. Sorry for the delay, I didn't realize that there was an update. (Probably a good idea to @ mention someone if you make some progress on a PR, new commits may not get the attention of reviewers).