Repository navigation
ad.concat type unification fails for object + categorical in AnnData >= 0.10 #1170
Description
Please make sure these conditions are met
- I have checked that this issue has not already been reported.
- I have confirmed this bug exists on the latest version of anndata.
- (optional) I have confirmed this bug exists on the master branch of anndata.
Report
AnnData's 0.10 update changes dtype promotion of concat, which may represent an undesired regression (or a desired sanity check). Concatenating object and category would automatically be promoted to object up until 0.9.2, and fails with below traceback from 0.10. This is also true for the current anndata-0.11.0.dev0 and was presumably introduced in #992. This could be desired behavior, although a gentle error message could provide a better pointer on how to fix the issue.
Code:
import anndata as ad
import pandas as pd
import numpy as np
one = ad.AnnData(X=np.array([[1]]))
one.obs = pd.DataFrame(["a"], index=['0'], columns=["col"])
two = ad.AnnData(X=np.array([[2]]))
two.obs = pd.DataFrame(["b"], index=['0'], columns=["col"])
two.obs["col"] = two.obs["col"].astype("category")
ad.concat([one,two])Traceback:
File ".../lib/python3.11/site-packages/anndata/_core/merge.py", line 1255, in concat
unify_dtypes([getattr(a, dim) for a in adatas]),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../lib/python3.11/site-packages/anndata/_core/merge.py", line 238, in unify_dtypes
df[col] = df[col].astype(dtype)
^^^^^^^^^^^^^^^^^^^^^
File ".../lib/python3.11/site-packages/pandas/core/generic.py", line 6534, in astype
new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../lib/python3.11/site-packages/pandas/core/internals/managers.py", line 414, in astype
return self.apply(
^^^^^^^^^^^
File ".../lib/python3.11/site-packages/pandas/core/internals/managers.py", line 354, in apply
applied = getattr(b, f)(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^
File ".../lib/python3.11/site-packages/pandas/core/internals/blocks.py", line 616, in astype
new_values = astype_array_safe(values, dtype, copy=copy, errors=errors)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File ".../lib/python3.11/site-packages/pandas/core/dtypes/astype.py", line 232, in astype_array_safe
dtype = pandas_dtype(dtype)
^^^^^^^^^^^^^^^^^^^
File ".../lib/python3.11/site-packages/pandas/core/dtypes/common.py", line 1636, in pandas_dtype
npdtype = np.dtype(dtype)
^^^^^^^^^^^^^^^
TypeError: Cannot interpret 'False' as a data typeVersions
anndata 0.10.0
session_info 1.0.0
cython_runtime NA
dateutil 2.8.2
h5py 3.9.0
natsort 8.4.0
numpy 1.24.4
packaging 23.2
pandas 2.1.1
pytz 2023.3.post1
scipy 1.11.3
six 1.16.0
Python 3.11.6 | packaged by conda-forge | (main, Oct 3 2023, 10:40:35) [GCC 12.3.0]
Linux-<redacted>-x86_64-x86_64
Session information updated at 2023-10-08 11:32
Reacted by LucaMarconato
Activity
LucaMarconato commentedon Oct 8, 2023
I also have the same problem, one of the tests fails because of this https://github.com/scverse/spatialdata/actions/runs/6448303558/job/17505424093
ivirshup commentedon Oct 8, 2023
Thanks for the report! The issue was more of a typo, we were returning False from
try_unifying_dtypewhen we should be returningNone. PR to fix: 1171ivirshup commentedon Oct 8, 2023
It could be nice for concatenation to automatically upcast appropriate types to categorical, but that's not the scope of the original feature so I've just fixed this error.
If you ran into this error via a test suite you have, it'd be great if you could add a job to your CI testing against pre-release candidates of upstream packages. That way we can find and fix these errors before they go out into a release.
Here's an example of a github actions workflow from the scverse cookie cutter template that does this:
ivirshup commentedon Oct 8, 2023
Bug fix release made
LucaMarconato commentedon Oct 8, 2023
It fixed the issue, thanks! I will add the test against pre-release packages.