Repository navigation
Commit 7944486
authored
fix: preserve column order in PointsModel.parse() (#1192)
`PointsModel.parse()` appends the non-coordinate columns by iterating
`set(data.columns) - {...}`, so their order in the returned element
follows string hash order and varies with `PYTHONHASHSEED`.
The reordering safeguard at the end of `parse()` only fires when the
column set is unchanged, which hides this on the common path. Passing
`coordinates=` to rename coordinate columns changes that set, which
skips the safeguard and leaves the order shuffled. The `np.ndarray` path
has no safeguard.
This looks like the root cause of #486, closed as not reproducible and
suspected to be a dask bug. Column order is not load-bearing for
correctness, but it lands in the written Zarr, so two runs of one
pipeline produce byte-different stores.
Both loops now walk `data.columns` / `annotation.columns` in order. I
left the safeguard in place because it normalizes the case where the set
is unchanged but the coordinate columns were reordered.
Sorting the extra columns alphabetically would also be deterministic. I
did not, because it discards input order and would change output on the
path the safeguard covers.
Measured on the pandas path with six extra columns: `PYTHONHASHSEED` 0
through 5 gave six distinct column orders before, one order after. The
new test fails on 30 of 30 seeds with the source change reverted.
Stores written before this keep whatever order they were written with.
Co-authored-by: Dylan Pulver <[email protected]>1 parent 908ae5e commit 7944486
2 files changed
Lines changed: 48 additions & 5 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
790 | 790 | | |
791 | 791 | | |
792 | 792 | | |
793 | | - | |
794 | | - | |
| 793 | + | |
| 794 | + | |
| 795 | + | |
| 796 | + | |
795 | 797 | | |
796 | 798 | | |
797 | 799 | | |
| |||
881 | 883 | | |
882 | 884 | | |
883 | 885 | | |
884 | | - | |
| 886 | + | |
885 | 887 | | |
886 | 888 | | |
887 | 889 | | |
888 | 890 | | |
889 | 891 | | |
890 | 892 | | |
891 | | - | |
892 | | - | |
| 893 | + | |
| 894 | + | |
| 895 | + | |
| 896 | + | |
893 | 897 | | |
894 | 898 | | |
895 | 899 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
814 | 814 | | |
815 | 815 | | |
816 | 816 | | |
| 817 | + | |
| 818 | + | |
| 819 | + | |
| 820 | + | |
| 821 | + | |
| 822 | + | |
| 823 | + | |
| 824 | + | |
| 825 | + | |
| 826 | + | |
| 827 | + | |
| 828 | + | |
| 829 | + | |
| 830 | + | |
| 831 | + | |
| 832 | + | |
| 833 | + | |
| 834 | + | |
| 835 | + | |
| 836 | + | |
| 837 | + | |
| 838 | + | |
| 839 | + | |
| 840 | + | |
| 841 | + | |
| 842 | + | |
| 843 | + | |
| 844 | + | |
| 845 | + | |
| 846 | + | |
| 847 | + | |
| 848 | + | |
| 849 | + | |
| 850 | + | |
| 851 | + | |
| 852 | + | |
| 853 | + | |
| 854 | + | |
| 855 | + | |
817 | 856 | | |
818 | 857 | | |
819 | 858 | | |
| |||
0 commit comments