Skip to content

BUG: describe/Description raises unhelpful error on empty (0-row) input #9891

Description

@maxime2476

Describe the bug

I hit this while feeding describe the output of an upstream filter that can
legitimately return zero rows. Instead of an empty/NaN summary, I get an error
coming from deep inside pandas/numpy, so at first it wasn't obvious the problem
was the empty input at all.

What makes it a bit worse is that the error depends on the column dtype, which
suggests there are actually two separate places where a 0-row frame isn't
handled:

  • a numeric (e.g. float) column raises
    ValueError: Length of values (2) does not match length of index (1)
  • a categorical column raises KeyError: 2

Neither message mentions that the input is empty, which is the real cause.

For what it's worth, pandas.DataFrame.describe() accepts the same 0-row inputs
happily and just returns a frame of NaNs, so this felt like something
statsmodels could handle more gracefully too.

Code Sample, a copy-pastable example if possible

import pandas as pd
from statsmodels.stats.descriptivestats import describe

# 1) numeric, 0 rows
#    -> ValueError: Length of values (2) does not match length of index (1)
describe(pd.DataFrame({"a": pd.Series([], dtype=float)}))

# 2) categorical, 0 rows
#    -> KeyError: 2
describe(pd.DataFrame({"a": pd.Series([], dtype="category")}))

# For comparison, pandas handles both without error (returns a frame of NaN):
pd.DataFrame({"a": pd.Series([], dtype=float)}).describe()
pd.DataFrame({"a": pd.Series([], dtype="category")}).describe()

I'm on the released 0.14.6, but I also checked the current main and the
relevant code is unchanged, so this reproduces there as well.

Expected Output

Either a frame of NaN statistics (matching pandas.DataFrame.describe()), or a
clear ValueError telling me the input needs at least one observation. Right now
I just get an internal error that doesn't point at the actual problem.

Root cause (as far as I can tell)

In statsmodels/stats/descriptivestats.py, Description.__init__ validates the
selected column types, ntop, percentiles and alpha, but there's no check
for an empty (0-row) input. The empty frame then reaches the per-dtype
statistics, which assume at least one row:

  • the numeric path breaks in the Description.numeric property, where
    df.apply(_mode) on a 0-row frame makes pandas build a Series from the
    2-tuple returned by _mode against a length-1 index — hence the ValueError;
  • the categorical path fails separately (the KeyError), so it looks like the
    cleanest place to fix this is a single guard in __init__, rather than
    patching _mode.

Two ways to fix it

I'd be happy to open a PR, but I think it's a design call, so I'd rather let you
decide between:

  • (a) fail fast — raise an informative ValueError in Description.__init__
    when there are zero observations (right next to the existing "empty DataFrame"
    check that runs after select_dtypes);
  • (b) graceful — return a frame of NaNs, matching
    pandas.DataFrame.describe().

Just let me know which you'd prefer and I'll send a PR with a test.

Small unrelated thing in the same spot

While reading that code I noticed the adjacent "empty DataFrame" error message is
missing a space and reads Selecting numericand categorical results in an empty DataFrame (should be numeric and categorical). It shows up with e.g.
Description(pd.DataFrame({"a": ["x", "y"]}), numeric=True, categorical=True).
Happy to fold the one-character fix into the same PR if that's alright.

Output of import statsmodels.api as sm; sm.show_versions()

Details
INSTALLED VERSIONS
------------------
Python: 3.13.12.final.0

statsmodels
===========
Installed: 0.14.6

Required Dependencies
=====================
cython: Not installed
numpy: 2.4.4
scipy: 1.17.1
pandas: 2.3.3
    dateutil: 2.9.0.post0
patsy: 1.0.2

Optional Dependencies
=====================
matplotlib: 3.10.9 (backend: tkagg)
cvxopt: Not installed
joblib: 1.5.3

Developer Tools
================
IPython: Not installed
    jinja2: 3.1.6
sphinx: Not installed
    pygments: 2.19.2
pytest: 9.0.3
virtualenv: Not installed

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions