Info "stats" looks horrible (since numpy 1.14)

@taldcroft

With numpy 1.14+ much of the formatting got much better, but info('stats') is not one of them:

t = QTable([[1.0, 1.1, 1.2]])
t.info('stats')
<QTable length=3>
name        mean                std         min max
---- ------------------ ------------------- --- ---
col0 1.0999999999999999 0.08164965809277258 1.0 1.2

where before it was

<QTable length=3>
name mean       std       min max
---- ---- --------------- --- ---
col0  1.1 0.0816496580928 1.0 1.2

Basically, the rounding rules changed. In the doctests, it was a change like in mean that made it look horrible, but in this example one sees it can be bad without too. I think we basically need to set some default format for columns that uses a sensible number of digits.

cc @taldcroft

I suppose something like '.6g' would work for float columns?

In [25]: '%.6g' % 1.099999999
Out[25]: '1.1'

In [13]: '%.6g' % 1.23456789e-3
Out[13]: '0.00123457'

In [14]: '%.6g' % 1.23456789e-4
Out[14]: '0.000123457'

In [15]: '%.6g' % 1.23456789e-5
Out[15]: '1.23457e-05'

In [19]: '%.6g' % 1.23456789
Out[19]: '1.23457'

In [20]: '%.6g' % 1.23456789e2
Out[20]: '123.457'

In [21]: '%.6g' % 1.23456789e3
Out[21]: '1234.57'

In [22]: '%.6g' % 1.23456789e4
Out[22]: '12345.7'

In [23]: '%.6g' % 1.23456789e5
Out[23]: '123457'

In [24]: '%.6g' % 1.23456789e6
Out[24]: '1.23457e+06'

@taldcroft

@taldcroft - the underlying problem is that numpy changed its behaviour for scalars - like Python ones, they now show the number of digits required to identify the value uniquely, instead of going through arrayprint, which has a default precision of 8. This combined with round-off problems in rounding (which, interestingly, statistics.mean() from the standard library does not have...), means the output looks ugly. The same will hold for any column that holds data with minor rounding errors; it is not specific to info('stats').

I think there are two ways around it: either we have a default format (as you suggest; for backwards compatibility, we could make this %.8g, which is the default precision for np.set_printoptions), or we could ensure that by default, we go through arrayprint (which does somewhat the right thing). The advantage of the latter is that the default format can be controlled via set_printoptions, and that, e.g., the view of a table column will look the same as doing column.data. In principle, we can even control this very simply, using a new legacy option to set_printoptions:

In [21]: np.set_printoptions(legacy='1.13')

In [22]: t.info('stats')
<QTable length=3>
name mean       std       min max
---- ---- --------------- --- ---
col0  1.1 0.0816496580928 1.0 1.2

Though I think it would be better to slightly adjust the printing routine, and use a proper format; we could still let the default format depend on numpy's arrayprint, by letting it calculate it.

Since today is supposed to be the ❄️ , I am moving the milestone. FYI.

Now that we only support numpy versions that have this change (as well as we can change this now in the new LTS more easily than waiting longer along the road when downstream should support multiple things), it would be nice to come up with a solution for this.

This issue was resolved by #12022, but it wasn't linked before merging so this issue must be closed manually.

Yes, good change! Closing...

Navigation Menu

Repository navigation

Info "stats" looks horrible (since numpy 1.14) #6962

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Info "stats" looks horrible (since numpy 1.14) #6962

Description

Activity

taldcroft commented on Dec 10, 2017

mhvk commented on Dec 10, 2017

pllim commented on Dec 23, 2017

bsipocz commented on Sep 26, 2019

eerovaher commented on Sep 27, 2021

mhvk commented on Sep 27, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions