Skip to content

Data container reductions - #1515

Merged
hrobarts merged 43 commits into
masterfrom
DataContainer_reductions
Nov 8, 2023
Merged

Data container reductions#1515
hrobarts merged 43 commits into
masterfrom
DataContainer_reductions

Conversation

@hrobarts

@hrobarts hrobarts commented Oct 2, 2023

Copy link
Copy Markdown
Contributor

Describe your changes

Updated reduction functions on the DataContainer class to accept a direction argument as a string or tuple of strings which match values in dimension_labels
Added a function _directional_reduction_unary() which:

  • checks whether the direction argument is specified and whether the direction values exist in dimension_labels
  • checks and raises an error if direction argument and numpy argument axis are both specified
  • passes the arguments to a specified function
  • checks if the function returns a numpy array, and if so returns the output as a DataContainer with the correct dimension_labels after reduction

The _directional_rediction_unary function is called by DataContainer.mean(), DataContainer.min(), DataContainer.max() and DataContainer.sum()

Describe any testing you have performed

Added tests passing different direction arguments to DataContainer.mean(), DataContainer.min(), DataContainer.max() and DataContainer.sum()

Link relevant issues

#1507

Checklist when you are ready to request a review

  • I have performed a self-review of my code
  • I have added docstrings in line with the guidance in the developer guide
  • I have implemented unit tests that cover any new or modified functionality
  • CHANGELOG.md has been updated with any functionality change
  • Request review from all relevant developers
  • Change pull request label to 'Waiting for review'

Contribution Notes

Please read and adhere to the developer guide and local patterns and conventions.

  • The content of this Pull Request (the Contribution) is intentionally submitted for inclusion in CIL (the Work) under the terms and conditions of the Apache-2.0 License.
  • I confirm that the contribution does not violate any intellectual property rights of third parties

@gfardell

gfardell commented Oct 6, 2023

Copy link
Copy Markdown
Member

I had a chat with @paskino about this. We decided that:

  1. The accumulation should always happen at 64bit, we can remove this as an option for the user.
  2. The return data container dtype should be the same as the input dtype. Again this is not something we should expose as an argument.
  3. The reductions can return an AcquisitionData or ImageData when this is passed to out. In this instance we will check that the array is the expected size and dtype, we will not check or update the geometry. In practise this will mean doing the reduction, getting the numpy array and then using out.fill(), we cannot use out.aray directly as it will (probably) be the wrong dtype to accumulate in to.

@hrobarts
hrobarts marked this pull request as ready for review October 6, 2023 15:49
@hrobarts

hrobarts commented Oct 9, 2023

Copy link
Copy Markdown
Contributor Author

Hello @gfardell and @paskino, just thinking about point 2:

  1. The return data container dtype should be the same as the input dtype. Again this is not something we should expose as an argument.

do you think this should also be the case for ints? Especially for mean that might not be intuitive

@paskino paskino left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks almost good to me: small changes and additions to docstrings

@gfardell gfardell left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's looking very neat now! A few suggestions for unit tests and some areas you can reuse code, but we can discuss in person too.

Comment thread CHANGELOG.md Outdated
Comment thread Wrappers/Python/cil/framework/framework.py Outdated
Comment thread Wrappers/Python/cil/framework/framework.py Outdated
Comment thread Wrappers/Python/cil/framework/framework.py Outdated
Comment thread Wrappers/Python/cil/framework/framework.py Outdated
Comment thread Wrappers/Python/cil/framework/framework.py Outdated
Comment thread Wrappers/Python/test/test_DataContainer.py Outdated
Comment thread Wrappers/Python/test/test_DataContainer.py Outdated
Comment thread Wrappers/Python/test/test_DataContainer.py Outdated
@hrobarts
hrobarts requested a review from gfardell October 27, 2023 08:56
Comment thread Wrappers/Python/cil/framework/framework.py Outdated
Comment thread Wrappers/Python/cil/framework/framework.py Outdated
Comment thread Wrappers/Python/test/test_DataContainer.py
Co-authored-by: Gemma Fardell <[email protected]>
Signed-off-by: Hannah Robarts <[email protected]>
@gfardell

gfardell commented Nov 6, 2023

Copy link
Copy Markdown
Member

image
The link hasn't rendered nicely in the documentation - but I'm not sure it's necessary or adds much. Maybe for min and max should enforce the same data type as the input. For mean I think you could just say the accumulated 64bit data will be cast to the output data type.

@hrobarts

hrobarts commented Nov 6, 2023

Copy link
Copy Markdown
Contributor Author

image The link hasn't rendered nicely in the documentation - but I'm not sure it's necessary or adds much. Maybe for min and max should enforce the same data type as the input. For mean I think you could just say the accumulated 64bit data will be cast to the output data type.

This should be rendering correctly now. I think it makes sense to keep the same behaviour for min and max because we might want to be able to pass a different data type to out, so we don't want to enforce the same data type as the input.

@paskino paskino left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See suggestion, otherwise it looks good to me.

Comment thread Wrappers/Python/cil/framework/framework.py Outdated
Co-authored-by: Edoardo Pasca <[email protected]>
Signed-off-by: Hannah Robarts <[email protected]>
@hrobarts
hrobarts merged commit b7d05fa into master Nov 8, 2023
@hrobarts
hrobarts deleted the DataContainer_reductions branch November 8, 2023 11:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants