Describe the bug
I hit this while feeding describe the output of an upstream filter that can
legitimately return zero rows. Instead of an empty/NaN summary, I get an error
coming from deep inside pandas/numpy, so at first it wasn't obvious the problem
was the empty input at all.
What makes it a bit worse is that the error depends on the column dtype, which
suggests there are actually two separate places where a 0-row frame isn't
handled:
- a numeric (e.g.
float) column raises
ValueError: Length of values (2) does not match length of index (1)
- a categorical column raises
KeyError: 2
Neither message mentions that the input is empty, which is the real cause.
For what it's worth, pandas.DataFrame.describe() accepts the same 0-row inputs
happily and just returns a frame of NaNs, so this felt like something
statsmodels could handle more gracefully too.
Code Sample, a copy-pastable example if possible
import pandas as pd
from statsmodels.stats.descriptivestats import describe
# 1) numeric, 0 rows
# -> ValueError: Length of values (2) does not match length of index (1)
describe(pd.DataFrame({"a": pd.Series([], dtype=float)}))
# 2) categorical, 0 rows
# -> KeyError: 2
describe(pd.DataFrame({"a": pd.Series([], dtype="category")}))
# For comparison, pandas handles both without error (returns a frame of NaN):
pd.DataFrame({"a": pd.Series([], dtype=float)}).describe()
pd.DataFrame({"a": pd.Series([], dtype="category")}).describe()
I'm on the released 0.14.6, but I also checked the current main and the
relevant code is unchanged, so this reproduces there as well.
Expected Output
Either a frame of NaN statistics (matching pandas.DataFrame.describe()), or a
clear ValueError telling me the input needs at least one observation. Right now
I just get an internal error that doesn't point at the actual problem.
Root cause (as far as I can tell)
In statsmodels/stats/descriptivestats.py, Description.__init__ validates the
selected column types, ntop, percentiles and alpha, but there's no check
for an empty (0-row) input. The empty frame then reaches the per-dtype
statistics, which assume at least one row:
- the numeric path breaks in the
Description.numeric property, where
df.apply(_mode) on a 0-row frame makes pandas build a Series from the
2-tuple returned by _mode against a length-1 index — hence the ValueError;
- the categorical path fails separately (the
KeyError), so it looks like the
cleanest place to fix this is a single guard in __init__, rather than
patching _mode.
Two ways to fix it
I'd be happy to open a PR, but I think it's a design call, so I'd rather let you
decide between:
- (a) fail fast — raise an informative
ValueError in Description.__init__
when there are zero observations (right next to the existing "empty DataFrame"
check that runs after select_dtypes);
- (b) graceful — return a frame of
NaNs, matching
pandas.DataFrame.describe().
Just let me know which you'd prefer and I'll send a PR with a test.
Small unrelated thing in the same spot
While reading that code I noticed the adjacent "empty DataFrame" error message is
missing a space and reads Selecting numericand categorical results in an empty DataFrame (should be numeric and categorical). It shows up with e.g.
Description(pd.DataFrame({"a": ["x", "y"]}), numeric=True, categorical=True).
Happy to fold the one-character fix into the same PR if that's alright.
Output of import statsmodels.api as sm; sm.show_versions()
Details
INSTALLED VERSIONS
------------------
Python: 3.13.12.final.0
statsmodels
===========
Installed: 0.14.6
Required Dependencies
=====================
cython: Not installed
numpy: 2.4.4
scipy: 1.17.1
pandas: 2.3.3
dateutil: 2.9.0.post0
patsy: 1.0.2
Optional Dependencies
=====================
matplotlib: 3.10.9 (backend: tkagg)
cvxopt: Not installed
joblib: 1.5.3
Developer Tools
================
IPython: Not installed
jinja2: 3.1.6
sphinx: Not installed
pygments: 2.19.2
pytest: 9.0.3
virtualenv: Not installed
Describe the bug
I hit this while feeding
describethe output of an upstream filter that canlegitimately return zero rows. Instead of an empty/NaN summary, I get an error
coming from deep inside pandas/numpy, so at first it wasn't obvious the problem
was the empty input at all.
What makes it a bit worse is that the error depends on the column dtype, which
suggests there are actually two separate places where a 0-row frame isn't
handled:
float) column raisesValueError: Length of values (2) does not match length of index (1)KeyError: 2Neither message mentions that the input is empty, which is the real cause.
For what it's worth,
pandas.DataFrame.describe()accepts the same 0-row inputshappily and just returns a frame of
NaNs, so this felt like somethingstatsmodelscould handle more gracefully too.Code Sample, a copy-pastable example if possible
I'm on the released 0.14.6, but I also checked the current
mainand therelevant code is unchanged, so this reproduces there as well.
Expected Output
Either a frame of
NaNstatistics (matchingpandas.DataFrame.describe()), or aclear
ValueErrortelling me the input needs at least one observation. Right nowI just get an internal error that doesn't point at the actual problem.
Root cause (as far as I can tell)
In
statsmodels/stats/descriptivestats.py,Description.__init__validates theselected column types,
ntop,percentilesandalpha, but there's no checkfor an empty (0-row) input. The empty frame then reaches the per-dtype
statistics, which assume at least one row:
Description.numericproperty, wheredf.apply(_mode)on a 0-row frame makes pandas build a Series from the2-tuple returned by
_modeagainst a length-1 index — hence theValueError;KeyError), so it looks like thecleanest place to fix this is a single guard in
__init__, rather thanpatching
_mode.Two ways to fix it
I'd be happy to open a PR, but I think it's a design call, so I'd rather let you
decide between:
ValueErrorinDescription.__init__when there are zero observations (right next to the existing "empty DataFrame"
check that runs after
select_dtypes);NaNs, matchingpandas.DataFrame.describe().Just let me know which you'd prefer and I'll send a PR with a test.
Small unrelated thing in the same spot
While reading that code I noticed the adjacent "empty DataFrame" error message is
missing a space and reads
Selecting numericand categorical results in an empty DataFrame(should benumeric and categorical). It shows up with e.g.Description(pd.DataFrame({"a": ["x", "y"]}), numeric=True, categorical=True).Happy to fold the one-character fix into the same PR if that's alright.
Output of
import statsmodels.api as sm; sm.show_versions()Details