Influence on the order of axes when declaring different affine transformations

@ivirshup

In short: specifying an affine transformation manually is not intuitive and requires an understanding of the underlining implicit and explicit coordinate spaces. Here I describe the problem and I propose one solution to simplify the exposure to this complexity to the user.

I tag @ivirshup, we started discussing this in the last spatial data call and you asked me to write an issue about it. I would also like to know the opinion of @bogovicj, thanks 😊

To illustrate this issue I start with a real example. Here is the .zattr for some 2D Points (left) and for some 2D Labels (right).

Let's look for example at the affine transformation which maps the Points (left) to from the implicit (xy) to the target (cyx) coordinate space.
This affine transformation is the following matrix (the NGFF specs says to drop the last row when saving). I pad the matrix with the coordinate spaces for extra clarify.

  x y <- source space
c 0 0 0
y 0 1 0
x 1 0 0
  0 0 1
^
target space

As you can see this matrix is different from the one specified for Labels, since the implicit coordinate space for labels is yx and not xy like for point.
So far so good. The problem is when the user wants to specify an affine transformation explicitly things could get confusing. For instance if the user wants to translate the points by the vector (x, y) = (a, b), the affine transformation would be

If the user wanted to translate the image, the affine transformation would be this one:

While if the user wanted to translate the points between the coordinate systems (x, y) -> (x, y), the affine transformation would simply be

  x y
x 1 0 a
y 0 1 b
  0 0 1

This last form of affine transformation is the most common one (for instance ihmo it is the most used in math classes and among softwares, for instance napari uses it or Apple (minus a transposition)).

What I propose is that by default the user will not have to specify the final form of the affine transformation that is written to disk, but that the user will only write an affine transformation of the form (for the 2D case):

  x y
x 1 0 a
y 0 1 b
  0 0 1

and the analogous for the 3D case.
In particular I think that the user will not be interested in transforming channels that are not x, y, z (so no c, t, ... I believe n-dimensional tensors will have just higher dimensional channels). In this way we can relieve the user from the burden to know the final form of the affine transformation that needs to take into account for which axes are present in input and output, and in which order they are specified.

@LucaMarconato

Thanks for the write up @LucaMarconato !

I've only read through this quickly just now, and will need to revisit again more carefully later, but my first thought is that you could re-write some of these transformations using

{ "type": "mapAxis",  "mapAxis": { "x":"x", "y":"y" } }

and maybe this makes it easier to do manually (?).
Let me know what you think. (Also I WILL read this again tomorrow) :)

Edit:
Follow up, this idea really just moves the complexity. The permutation has to get handled by the implementation, rather than the person / code representing the transformation with the spec

@bogovicj

@bogovicj I have an update on this. I realized that mapAxis, which can indeed been used to map an input cyx system to an output yx one (thanks for pointing this out), cannot unfortunately be used on the other direction, unless we make the choice c = x or c = y.
Similarly, the ByDimension transformation can't be used elegantly for that case. Therefore, I ended up using affine transformations.

I believe that the following can be done to cover the needs of the users:

for "low level" users, they can just define affine transformations, or some simple helper functions can be written (I did that), to create an affine transformation for them.
for "higher level" users, which wants to operate on xy(z), the code they have to write could be simplified by introducing in the implementation a Affine2D class, which uses the NGFF Affine class as a backend and for io operations. This would remove the ambiguity from the user when for instance applying the same affine transformation to a set of points in the xy space or a multi-channel image in the cyx space.

@giovp

cc @giovp, we may want to consider to implement a class Affine2D as in the point 2 above.

If we go with extra classes with we use xarray to annotate spatial dimensions are described here: #47.

An idea could be to keep the BaseTransformation objects "hidden" to the user and use them only to interface to the io to NGFF, while having more intuitive classes in memory.

In particular (just to recap the sum up the discussion above), at the moment if we have a transformation for points (xy) and we copy it inside an image (yx), the transformation will be different unless we wrap it inside a pari of MapAxis({'x': 'x', 'y': 'y'}) or equivalent. If we copy it inside a (cyx) image we need also a ByDimension transformation (see real code implementing this, it's a bit intricate). Things could be confusing for the user and an extra set of transformation classes could make transformations more ergonomics.

More details on the code linked above.
The code is automagically making transformations like Scale([2., 3.]) work on xy points, yx labels, cyx images etc at construction time, and this does not work when copying a transformation inside another object. I would prefer to remove that code and use a cleaner approach:

extra classes as mentioned
helper functions like adapt_transformation(transformation, new_input_coordianate_system, new_output_coordinate_syestem)

replying here but related to #47

Very non-useful comment but I honestly don't know what would be a better API for this, on one way transformation classes that are elements specific (which essentially build the mapper transformation sequence linked) sounds it could be clean yet it might lead to an explosion of classes instances. Using xarray would indeed simplifying some of the complexity (ambiguity) of applying transformations.

I'll play around with it a bit in the next weeks

@giovp

Thanks @giovp! I think for the moment I will add some helper functions so we don't have to introduce new classes.

@LucaMarconato

@LucaMarconato, what do you mean for ByDimension here?

I realized that mapAxis, which can indeed been used to map an input cyx system to an output yx one (thanks for pointing this out), cannot unfortunately be used on the other direction, unless we make the choice c = x or c = y.
Similarly, the ByDimension transformation can't be used elegantly for that case. Therefore, I ended up using affine transformations.

I feel like this seems fine:

coordinate_systems = [
    CoordinateSystem("space", [Axis("X", "space"), Axis("Y", "space")]),
    CoordinateSystem("image", [Axis("X", "array"), Axis("Y", "array"), Axis("C", "array")]),
]

transform = ByDimension("image", "space", transformations=[Scale([.42, .13], ["X", "Y"], ["X", "Y"])])

Then I think it would be quite reasonable to have a helper that would create a new ByDimension transformation padded with Identity transformations for extra channels.

@bogovicj

I mean the use of ByDimension that you suggested.

Your syntax would indeed work nice if one didn't have to specify also the explicit padding with the affine transformation. This is required by the specs:

Every axis name in the parent byDimension's output MUST appear in exactly one of its child transformations' output.

@bogovicj I think that relaxing this constraints could make the transformation byDimension applicable to more contexts and help with our problem. What do you think?

I think the current specification works fine, and I like being able to remove dimensions with it. I think we could just have a function like:

def pass_axes_through(input: CoordinateSpace, transform: ByDimension) -> ByDimension:
    output_transformations = deepcopy(transform.transformations)

    transformed_dims = []
    for sub_t in transform.transformations:
        transformed_dims.extend(sub_t.input_coordinate_space)
    untransformed = list(set(input.axes_names).difference(transformed_dims))
    output_transformations.append(Identity(untransformed, untransformed))

    output_coordinate_space = ... # compute from ByDimensions

    return ByDimension(
        input_coordinate_space=input,
        output_coordinate_space=output_coordinate_space,
        transforms=output_transformations,
    )

This has some issues, but I think surmountable ones:

Transforms within ByDimensions allow list of axis names instead of coordinate spaces, but I think we can work around this pretty easily
Coordinate systems are required to have names in the OME-NGFF spec. It's probably bad to be generating things we can't write, so we may have to either figure out a reasonable way to generate names (at write time?) or force naming things all the time.

Ok thanks Isaac, I'll try an implementation and think about the issues.

The new transformations pr is now merged in main and this issue is solved.

Navigation Menu

Repository navigation

Influence on the order of axes when declaring different affine transformations #39

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Influence on the order of axes when declaring different affine transformations #39

Description

Activity

bogovicj commented on Oct 20, 2022

LucaMarconato commented on Nov 8, 2022

LucaMarconato commented on Nov 8, 2022

LucaMarconato commented on Dec 12, 2022

giovp commented on Dec 13, 2022

LucaMarconato commented on Dec 13, 2022

ivirshup commented on Dec 13, 2022

LucaMarconato commented on Dec 13, 2022

ivirshup commented on Dec 13, 2022

LucaMarconato commented on Dec 13, 2022

LucaMarconato commented on Feb 13, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions