Skip to content

Make from_df and from_pandas agree on missing values in string columns - #20805

Draft
taldcroft wants to merge 3 commits into
astropy:mainfrom
taldcroft:table-20663-from-df-pd-na-strings
Draft

Make from_df and from_pandas agree on missing values in string columns#20805
taldcroft wants to merge 3 commits into
astropy:mainfrom
taldcroft:table-20663-from-df-pd-na-strings

Conversation

@taldcroft

@taldcroft taldcroft commented Nov 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Summary

Table.from_df raises TypeError on any DataFrame with a string-like column that holds pd.NA or pd.NaT. That includes the most common case: a pandas string dtype column (python or pyarrow storage) with a missing entry, since pandas stores the missing value as pd.NA. Table.from_pandas does not raise, but it treats only np.nan as missing, so the same column comes back as an object column with pd.NA stored under the mask.

>>> df = pd.DataFrame({"s": ["a", None]}, dtype="string")
>>> Table.from_df(df)["s"].dtype
TypeError: must be real number, not NAType    # main
dtype('<U1')                                  # this branch, entry 1 masked
>>> Table.from_pandas(df)["s"].dtype
dtype('O')                                    # main
dtype('<U1')                                  # this branch, entry 1 masked

The two methods each decided on their own which values were missing: from_df called isnan on every non-string value, and from_pandas accepted only the np.nan object. (On main this check lives in the _get_string_fill_value helper, where #20599 moved it without changing it; the behavior is the same in v8.0.2.) This PR passes in the null mask that each caller already computes (narwhals is_null() in from_df, pandas isnull() in from_pandas), and the helper skips masked entries whatever their value. Both methods now give the same result for every case below. Columns that are not all strings still stay object, as before.

Fixes #20663.

AI disclosure

This PR includes AI-generated content using Claude Opus 5.5. I have carefully reviewed the content and fully understand the changes.

  • I certify that I am human and take responsibility for the code and interactions with reviewers.

Details

Click to expand

Use the caller's null mask to decide which entries of a string column are missing

1. Results before and after

Measured on main (68e0005) and on this branch with pandas 2.3.3. The main results are identical with the _dataframes.py of v8.0.2, the latest release: #20599 moved this check into _get_string_fill_value without changing its behavior, so these are all changes since the last release. The two methods disagree on 9 of the 12 rows on main.

DataFrame column from_df main from_pandas main both, this branch
dtype="string", None (stored as pd.NA) TypeError object MaskedColumn <U1
dtype="string[pyarrow]", None TypeError object MaskedColumn <U1
dtype=object, pd.NA TypeError object MaskedColumn <U1
dtype=object, pd.NaT TypeError object MaskedColumn <U1
dtype=object, None <U1 object MaskedColumn <U1
dtype=object, np.nan <U1 <U1 MaskedColumn <U1
dtype=object, [b"a", None] S1 object MaskedColumn S1
dtype=object, ["a", b"b", None] <U1 object MaskedColumn <U1
dtype=object, all None S1 object MaskedColumn object, all masked
dtype="string", all None TypeError object MaskedColumn <U1, all masked
dtype=object, ["a", {"x": 1}] TypeError object Column object
dtype=object, ["a", b"\xff"] UnicodeDecodeError UnicodeDecodeError Column object

2. Mixed str and bytes

A column that mixes text with ASCII bytes still converts to <U, using NumPy's ASCII decode, which is the convention _get_string_dtype already documents. Before this PR that worked in from_df, and in from_pandas when nothing was missing. If the text is mixed with bytes that are not ASCII, NumPy raises UnicodeDecodeError, so the helper now returns None and the column stays object. Keeping every str/bytes mix as object would have been simpler, but it would change a case that works today.

3. All-missing columns

When every entry is masked there is no value to take the string type from. On main, from_df produced a fully masked S1 (bytes) column for an all-None object or polars String column, whatever the dtype said, and raised for an all-missing pandas string column, while from_pandas left an object column. The helper now takes a default placeholder that the caller picks from the dataframe dtype: "" for a pandas string dtype or a narwhals String, b"" for a narwhals Binary, and None (leave the column as object) otherwise. A pandas object column says nothing about its type, so an all-missing one stays a fully masked object column from both methods. Narwhals reports an all-None pandas object column as String, so from_df also checks the native dtype to tell the two apart.

4. Implementation

  • astropy/table/_column_mixins.pyx, _get_string_fill_value(data, mask, default=None): the legacy_pandas keyword and the libc.math.isnan cimport are removed. The function loops over zip(data, mask) and skips masked entries. It returns None (keep the column as object) on the first unmasked value that is not str or bytes, or if text is mixed with non-ASCII bytes. Otherwise it returns "" or b"", following the first unmasked value, and default when every entry is masked.
  • astropy/table/_dataframes.py, from_df: computes nw_mask.to_numpy() once and passes it to the helper and the fill comprehension. default is b"" for nw.Binary, "" for nw.String unless the native column dtype is NumPy object, else None. from_pandas: passes its existing mask, with default="" for a pd.StringDtype column and None otherwise.

5. Tests

Three new parametrized tests in astropy/table/tests/test_df.py, each run for from_df and from_pandas (from_df is skipped without narwhals), plus one polars test:

  • test_from_dataframe_string_missing_values: string, string[pyarrow], and object columns with None, np.nan, pd.NA, pd.NaT, bytes, and mixed str/ASCII bytes, for Table and QTable.
  • test_from_dataframe_string_all_missing: all-missing string and string[pyarrow] columns give a fully masked <U1 column; an all-missing object column stays a fully masked object column.
  • test_from_dataframe_object_column_kept: dicts, lists, and text mixed with non-ASCII bytes stay object, with and without a missing value.
  • test_from_df_polars_string_missing_values: polars String and Binary columns with a null, and all-null, give <U1 and S1 masked columns.

pd.NA and pd.NaT appear in the parametrize lists as the strings "pd.NA" / "pd.NaT" and are mapped back inside the tests, because pandas is imported lazily in this module. With the two source files reverted to main, 49 of the 70 new cases fail. The other 21 (for example object + np.nan, or an all-missing object column in from_pandas) already pass on main and are kept as regression guards.

python -m pytest astropy/table/tests/test_df.py astropy/table/tests/test_column.py astropy/table/tests/test_pickle.py docs/table: 414 passed, 53 skipped, 2 xfailed.

6. Backwards compatibility

No API change; _get_string_fill_value is private. The from_df cases that raised now return a column. In from_pandas, string columns whose missing value is None, pd.NA or pd.NaT now become native U/S MaskedColumns instead of object columns. That matches what from_pandas already did for np.nan, and what from_df did for None. An all-missing column follows the dataframe dtype from both methods: a pandas string or polars String column becomes a fully masked <U1 column, and a pandas object column stays a fully masked object column. On main, from_df gave S1 for the object column and raised for the string column, and from_pandas gave object for both. Columns with no missing values, and those holding non-string objects, are converted as before.

One more case changes, in from_pandas(df, index=True). The index gets an all-False mask, so a NaN in an object-dtype index is never treated as missing. On main the helper counted that NaN as missing anyway and the index became a <U32 column holding the string 'nan'; now the NaN is an unmasked non-string value, so the index stays an object column holding the float NaN. Masking the index properly is a separate change.

Changelog: docs/changes/table/20805.bugfix.rst.

🤖 Generated with Claude Code

@taldcroft @claude
Fix from_df and from_pandas handling of missing values in string columns
0bfe85a 
Table.from_df raised TypeError when a string-like object column held
pd.NA or pd.NaT, because _get_string_fill_value called libc isnan on
every non-string value. Table.from_pandas treated only np.nan as missing,
so columns using None, pd.NA or NaT stayed as object columns.

_get_string_fill_value now takes the caller's null mask and skips masked
entries whatever their value. It returns None, keeping the object column,
for any unmasked value that is not str or bytes, and for text mixed with
non-ASCII bytes, which would otherwise raise UnicodeDecodeError. Both
from_df and from_pandas pass their existing null masks.

Fix astropy#20663

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@github-actions github-actions Bot added the table label Nov 8, 2026
@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.

  • Do the proposed changes actually accomplish desired goals?
  • Do the proposed changes follow the Astropy coding guidelines?
  • Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
  • Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
  • Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see instructions for rebase and squash.
  • Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the "Extra CI" label. Codestyle issues can be fixed by the bot.
  • Is a change log needed? If yes, did the change log check pass? If no, add the "no-changelog-entry-needed" label. If this is a manual backport, use the "skip-changelog-checks" label unless special changelog handling is necessary.
  • Is this a big PR that makes a "What's new?" entry worthwhile and if so, is (1) a "what's new" entry included in this PR and (2) the "whatsnew-needed" label applied?
  • At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate "backport-X.Y.x" label(s) before merge.

@taldcroft @claude
Make an all-missing string-like column follow the dataframe dtype
c5a90be 
_get_string_fill_value takes a default placeholder for the case where
every entry is masked and there is no value to take the type from. The
callers pick it from the dataframe dtype: "" for a pandas string dtype or
narwhals String, b"" for narwhals Binary, and None otherwise, which leaves
a pandas object column as a fully masked object column from both methods.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
@taldcroft
taldcroft force-pushed the table-20663-from-df-pd-na-strings branch from ab7ad43 to c5a90be Compare November 8, 2026 11:18
@taldcroft taldcroft added this to the v8.1.0 milestone Nov 8, 2026
@taldcroft taldcroft added the Bug label Nov 8, 2026
@taldcroft taldcroft changed the title Fix from_df TypeError and from_pandas object columns for pd.NA in string columns Make from_df and from_pandas agree on missing values in string columns Nov 8, 2026
@cgobat

cgobat commented Nov 8, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

I guess this is on me for not reading the docs more closely, but I have always thought/assumed that Table.from_df() and Table.from_pandas() were interchangeable for pandas DataFrames, so I'm very surprised to see that not only are they different implementations, but that they have different behavior. Is there a reason that .from_df doesn't just delegate directly to .from_pandas for pandas DataFrames?

@taldcroft

taldcroft commented Nov 9, 2026

Copy link
Copy Markdown
Member Author

@cgobat - that's a good question. #18435 introduced from_df and the intent was a conversion code path that serves all the backends consistently through narwhals. Having two code paths doing basically the same thing is not ideal, but we don't want to drop from_pandas nor make it depend on narwhals. Having from_df delegate to from_pandas is perhaps something to consider but this then breaks the intent to uniformly use narwhals by special-casing pandas.

There are no great answers, but the good news is that AI is really good at finding inconsistencies and then helping to fix them.

@taldcroft taldcroft mentioned this pull request Nov 9, 2026
Open

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

Bug table

Projects

None yet

Milestone

v8.1.0

Development

Successfully merging this pull request may close these issues.

Table.from_df raises TypeError for pandas string columns with missing values (pd.NA)

2 participants

@taldcroft @cgobat