Skip to content

dask: Data.unique#391

Merged
davidhassell merged 5 commits intoNCAS-CMS:lama-to-daskfrom
davidhassell:dask-unique
Apr 27, 2022
Merged

dask: Data.unique#391
davidhassell merged 5 commits intoNCAS-CMS:lama-to-daskfrom
davidhassell:dask-unique

Conversation

@davidhassell
Copy link
Copy Markdown
Collaborator

@davidhassell davidhassell commented Apr 25, 2022

No description provided.

@davidhassell davidhassell added the dask Relating to the use of Dask label Apr 25, 2022
@davidhassell davidhassell mentioned this pull request Apr 25, 2022
Closed
sadielbartholomew
sadielbartholomew approved these changes Apr 27, 2022
View reviewed changes
Copy link
Copy Markdown
Member

@sadielbartholomew sadielbartholomew left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All good and ready to merge, with a nice extension to the unit test, though if I was to be harsh I might suggest first including also in the unit test a small block to cover the various data type possibilities ("fibUS" as set here) and in particular the case of a string-type data array, an example I checked manually in reviewing, since really we are only checking integer arrays at present?).

@davidhassell
Copy link
Copy Markdown
Collaborator Author

davidhassell commented Apr 27, 2022

Harsh is fine! Data type unit tests: ae13c20

sadielbartholomew
sadielbartholomew approved these changes Apr 27, 2022
View reviewed changes
Copy link
Copy Markdown
Member

@sadielbartholomew sadielbartholomew left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bonus points for adding the new commit, which I have sanity checked and has added my suggestion in an elegant way. All good to merge.

@davidhassell davidhassell merged commit bcbd03a into NCAS-CMS:lama-to-dask Apr 27, 2022
@davidhassell
Copy link
Copy Markdown
Collaborator Author

davidhassell commented Apr 27, 2022

For future reference - the da.unique function was not used in this PR, we used da.reduction instead, because the former does not deal very well with missing data. I had a look at modifying da.unique to cope, but a) it was not trivial; b) cf.Data doesn't need (yet) the return_* keywords; and c) the dask folk might change it to be a reduction at some stage (dask/dask#2851). For these three reasons, it was preferable at this time to implement a missing-data-friendly "unique", without the return_* keywords as a standard cf.data.Collapse method.

@davidhassell davidhassell deleted the dask-unique branch November 15, 2022 09:27
@davidhassell davidhassell added this to the 3.14.0 milestone Nov 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sadielbartholomew sadielbartholomew sadielbartholomew approved these changes

Assignees

No one assigned

Labels

dask Relating to the use of Dask

Projects

None yet

Milestone

3.14.0

Development

Successfully merging this pull request may close these issues.

2 participants

@davidhassell @sadielbartholomew