Table.from_df and Table.from_pandas give different results #20807

Description

Table.from_df (narwhals) and Table.from_pandas are documented as two routes for the same conversion, but for several common pandas column types they give different tables for the same DataFrame. #20805 fixes the differences for string-like columns with missing values and this issue collects the ones that remain. Each case below is a plain pd.DataFrame with one column; there is no mixin or unit involved.

With these fixed, the two methods would agree on every one of the 23 common pandas column types I tried (numeric and nullable numeric, pyarrow-backed numeric, naive datetime and timedelta with and without missing values, string, object with mixed types, lists, dicts, complex).

Raised by @cgobat on #20805, who had assumed the two methods were interchangeable for pandas input. Whether from_pandas should eventually become a wrapper around from_df is a separate question; this issue is only about making them agree.

How to Reproduce

Run on #20805 (the string cases are already fixed there):

import pandas as pd
from astropy.table import Table

cases = {
    "boolean": pd.Series([True, None], dtype="boolean"),
    "bool[pyarrow]": pd.Series([True, None], dtype="bool[pyarrow]"),
    "category": pd.Categorical(["a", None]),
    "datetime64[ns, US/Eastern]": pd.Series(pd.to_datetime(["2020-01-01", None])).dt.tz_localize("US/Eastern"),
}
for label, s in cases.items():
    df = pd.DataFrame({"c": s})
    print(f"\n# {label}")
    for m in ("from_df", "from_pandas"):
        c = getattr(Table, m)(df)["c"]
        print(f"{m:12s} {type(c).__name__:13s} {str(getattr(c, 'dtype', '')):8s} {list(c)}")

print("\n# index=True with an unnamed index")
df = pd.DataFrame({"x": [1, 2]})
for m in ("from_df", "from_pandas"):
    try:
        print(f"{m:12s} {getattr(Table, m)(df, index=True).colnames}")
    except Exception as e:
        print(f"{m:12s} {type(e).__name__}: {e}")

# boolean
from_df      MaskedColumn  object   [True, masked]
from_pandas  MaskedColumn  bool     [np.True_, masked]

# bool[pyarrow]
from_df      MaskedColumn  object   [True, masked]
from_pandas  MaskedColumn  bool     [np.True_, masked]

# category
from_df      MaskedColumn  object   ['a', masked]
from_pandas  MaskedColumn  <U1      [np.str_('a'), masked]

# datetime64[ns, US/Eastern]
from_df      Time          Time     [<Time object: scale='utc' format='isot' value=2020-01-01T05:00:00.000>, <Time object: scale='utc' format='isot' value=                    ———>]
from_pandas  MaskedColumn  object   [Timestamp('2020-01-01 00:00:00-0500', tz='US/Eastern'), masked]

# index=True with an unnamed index
from_df      KeyError: 'Requested level (index) does not match index name (None)'
from_pandas  ['index', 'x']

Versions

astropy 8.1.0.dev506 (branch of #20805 on main at 68e0005), pandas 2.3.3, narwhals 2.20.0, polars 1.32.3, numpy 2.2.3, Python 3.13, macOS arm64.