Skip to content

Commit 7944486

Browse files
authored
fix: preserve column order in PointsModel.parse() (#1192)
`PointsModel.parse()` appends the non-coordinate columns by iterating `set(data.columns) - {...}`, so their order in the returned element follows string hash order and varies with `PYTHONHASHSEED`. The reordering safeguard at the end of `parse()` only fires when the column set is unchanged, which hides this on the common path. Passing `coordinates=` to rename coordinate columns changes that set, which skips the safeguard and leaves the order shuffled. The `np.ndarray` path has no safeguard. This looks like the root cause of #486, closed as not reproducible and suspected to be a dask bug. Column order is not load-bearing for correctness, but it lands in the written Zarr, so two runs of one pipeline produce byte-different stores. Both loops now walk `data.columns` / `annotation.columns` in order. I left the safeguard in place because it normalizes the case where the set is unchanged but the coordinate columns were reordered. Sorting the extra columns alphabetically would also be deterministic. I did not, because it discards input order and would change output on the path the safeguard covers. Measured on the pandas path with six extra columns: `PYTHONHASHSEED` 0 through 5 gave six distinct column orders before, one order after. The new test fails on 30 of 30 seeds with the source change reverted. Stores written before this keep whatever order they were written with. Co-authored-by: Dylan Pulver <[email protected]>
1 parent 908ae5e commit 7944486

2 files changed

Lines changed: 48 additions & 5 deletions

File tree

‎src/spatialdata/models/models.py‎

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -790,8 +790,10 @@ def _(
790790
df_dict[instance_key] = annotation[instance_key]
791791
if Z not in axes and Z in annotation.columns:
792792
logger.info(f"Column `{Z}` in `annotation` will be ignored since the data is 2D.")
793-
for c in set(annotation.columns) - {feature_key, instance_key, X, Y, Z}:
794-
df_dict[c] = annotation[c]
793+
handled_columns = {feature_key, instance_key, X, Y, Z}
794+
for c in annotation.columns:
795+
if c not in handled_columns:
796+
df_dict[c] = annotation[c]
795797

796798
table: DaskDataFrame = dd.from_pandas(pd.DataFrame(**df_kwargs), **kwargs)
797799
return cls._add_metadata_and_validate(
@@ -881,15 +883,17 @@ def _(
881883
)
882884
if Z not in axes and Z in data.columns:
883885
logger.info(f"Column `{Z}` in `data` will be ignored since the data is 2D.")
884-
for c in set(data.columns) - {
886+
handled_columns = {
885887
feature_key,
886888
instance_key,
887889
*coordinates.values(),
888890
X,
889891
Y,
890892
Z,
891-
}:
892-
table[c] = data[c]
893+
}
894+
for c in data.columns:
895+
if c not in handled_columns:
896+
table[c] = data[c]
893897

894898
validated = cls._add_metadata_and_validate(
895899
table,

‎tests/models/test_models.py‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -814,6 +814,45 @@ def test_force2d():
814814
assert_elements_are_identical(multipolygons_3d, expected_multipolygons_2d)
815815

816816

817+
def test_points_model_preserves_column_order():
818+
# the extra columns used to be added iterating over a set, so their order in the parsed element depended on
819+
# PYTHONHASHSEED. Renaming the coordinate columns changes the set of column names, which used to bypass the
820+
# reordering safeguard at the end of PointsModel.parse().
821+
extra_columns = ["qv", "intensity", "radius", "z_score", "nucleus_distance", "codeword_index"]
822+
n = 10
823+
data = pd.DataFrame(
824+
{
825+
"my_x": np.arange(n, dtype=float),
826+
"my_y": np.arange(n, dtype=float),
827+
"target": pd.Categorical(["a", "b"] * (n // 2)),
828+
"cell_id": np.arange(n),
829+
**{c: np.arange(n, dtype=float) for c in extra_columns},
830+
}
831+
)
832+
expected = ["x", "y", "target", "cell_id", *extra_columns]
833+
834+
from_pandas = PointsModel.parse(
835+
data, coordinates={"x": "my_x", "y": "my_y"}, feature_key="target", instance_key="cell_id"
836+
)
837+
assert list(from_pandas.columns) == expected
838+
839+
from_dask = PointsModel.parse(
840+
dd.from_pandas(data, npartitions=2),
841+
coordinates={"x": "my_x", "y": "my_y"},
842+
feature_key="target",
843+
instance_key="cell_id",
844+
)
845+
assert list(from_dask.columns) == expected
846+
847+
from_numpy = PointsModel.parse(
848+
data[["my_x", "my_y"]].to_numpy(),
849+
annotation=data.drop(columns=["my_x", "my_y"]),
850+
feature_key="target",
851+
instance_key="cell_id",
852+
)
853+
assert list(from_numpy.columns) == expected
854+
855+
817856
def test_dask_points_unsorted_index_with_warning(points):
818857
chunksize = 300
819858
element = points["points_0"]

0 commit comments

Comments
 (0)