Inconsistent shape manipulation in operator unique

Hi! I recently write a mxnet version numpy unique, and find there may be something wrong.

Reproducing code example:

import numpy as np
x = np.array([[[-4, -4, -2, -1],
               [ 7,  5, -1,  5],
               [ 2,  5,  0,  5]],

              [[-3,  7,  0,  0],
               [-3,  1, -2,  4],
               [-6,  6,  0,  0]],

              [[-6,  4, -7, -2],
               [ 0, -2,  0,  1],
               [ 5,  0, -1,  2]],

              [[ 5, -2, -7,  0],
               [-7,  0, -2,  0],
               [-4, -3,  6,  0]],

              [[-6,  1, -5,  5],
               [ 0,  5,  1,  5],
               [-7, -7,  5, -1]]])
np.unique(x, axis=-1)

The result is:

       [[[-4, -4, -2, -1],
         [ 7,  5, -1,  5],
         [ 2,  5,  0,  5]],
 
        [[-3,  7,  0,  0],
         [-3,  1, -2,  4],
         [-6,  6,  0,  0]],
 
        [[-6,  4, -7, -2],
         [ 0, -2,  0,  1],
         [ 5,  0, -1,  2]],
 
        [[ 5, -2, -7,  0],
         [-7,  0, -2,  0],
         [-4, -3,  6,  0]],
 
        [[-6,  1, -5,  5],
         [ 0,  5,  1,  5],
         [-7, -7,  5, -1]]])

Description

In this example, the first two 'column' does not swap.

I read the source of numpy, and find it simply swap the axis before view.
In this example, for argument axis is -1 (2), the shape goes:
(5, 3, 4)->(4, 3, 5)->(4, 15)->use view->(4, )
The sequence of '5' and '3' is reversed, I think this is wrong.

If argument axis is 1, it goes:
(5, 3, 4)->(3, 5, 4)->(3, )
It's OK. So, It is obvious that when axis is -1, it should always get a wrong result.

In my implementation, the shape goes:
(5, 3, 4)->(15, 4, 1)->(4, 15, 1)->(4, 15)->use view->(4, )

I hand-write this procedure, get the shape manipulation result:

numpy version:
array([[-4, -3, -6,  5, -6,  7, -3,  0, -7,  0,  2, -6,  5, -4, -7],
       [-4,  7,  4, -2,  1,  5,  1, -2,  0,  5,  5,  6,  0, -3, -7],
       [-2,  0, -7, -7, -5, -1, -2,  0, -2,  1,  0,  0, -1,  6,  5],
       [-1,  0, -2,  0,  5,  5,  4,  1,  0,  5,  5,  0,  2,  0, -1]])

correct version:
array([[-4,  7,  2, -3, -3, -6, -6,  0,  5,  5, -7, -4, -6,  0, -7],
       [-4,  5,  5,  7,  1,  6,  4, -2,  0, -2,  0, -3,  1,  5, -7],
       [-2, -1,  0,  0, -2,  0, -7,  0, -1, -7, -2,  6, -5,  1,  5],
       [-1,  5,  5,  0,  4,  0, -2,  1,  2,  0,  0,  0,  5,  5, -1]])

Numpy/Python version information:

numpy : 1.16
OS : MacOS

This seems all correct to me. The indexed axis is replaced by the unique/sorted one:

import numpy as np
x = np.array([[[-4, -4, -2, -1],
               [ 7,  5, -1,  5],
               [ 2,  5,  0,  5]],

              [[-3,  7,  0,  0],
               [-3,  1, -2,  4],
               [-6,  6,  0,  0]],

              [[-6,  4, -7, -2],
               [ 0, -2,  0,  1],
               [ 5,  0, -1,  2]],

              [[ 5, -2, -7,  0],
               [-7,  0, -2,  0],
               [-4, -3,  6,  0]],

              [[-6,  1, -5,  5],
               [ 0,  5,  1,  5],
               [-7, -7,  5, -1]]])
for ax in range(3):
    print(ax)
    ux, i = np.unique(x, axis=ax, return_index=True)
    print(np.array_equal(ux, x.take(i, axis=ax)))

## Printing:
# 0
# True
# 1
# True
# 2
# True

Please do double check if I got something wrong, but I think all is in order here. Maybe we should improve the documentation on it?

@seberg

@seberg In your check method, its result will always true.
You don't understand well where is wrong.

I'll try to show my points by example.

np.take(x, [0, 1], axis=2) # x -> shape (5, 3, 4)

result is

array([[[-4, -4],
        [ 7,  5],
        [ 2,  5]],

       [[-3,  7],
        [-3,  1],
        [-6,  6]],

       [[-6,  4],
        [ 0, -2],
        [ 5,  0]],

       [[ 5, -2],
        [-7,  0],
        [-4, -3]],

       [[-6,  1],
        [ 0,  5],
        [-7, -7]]])

After swap axis:

x = np.swapaxes(x, 0, 2)
np.take(x, [0, 1], axis=0) # x -> shape (4, 3, 5)

result is

array([[[-4, -3, -6,  5, -6],
        [ 7, -3,  0, -7,  0],
        [ 2, -6,  5, -4, -7]],

       [[-4,  7,  4, -2,  1],
        [ 5,  1, -2,  0,  5],
        [ 5,  6,  0, -3, -7]]])

In unique, sort is used. Then how a matrix or a tensor be serialized is important.
At the example I used, we compare whole flattened array with size 15.
-4 == -4, while 7 > 5, so the two subarrays should be swapped.
While after swap axis, we take the transposed subarrays. It just like from row major to column major.
In this case, -4 == -4, -3 < 7, the two subarrays will not be swapped.

I think when input's ndim >= 3, argument axis >=2, the sequence of axes except the chosen one will be broken. In fact, we should keep this sequence, that's why there is inconsistent behavior in unique.

OK, so the sort order is not correct, the unique part works fine. Sorry, but I read the "swapping" as transpose operation of the axis. So what you correctly say is that is that the swapaxis causes the sort order to change, when a moveaxis would not.

We may indeed want to change that, although it is a bit tricky since the order changes, so it would need at least release notes. Do you want to give it a try?

OK, I'll have a try.

Closed by gh-14255

Navigation Menu

Repository navigation

Inconsistent shape manipulation in operator `unique` #14244

Reproducing code example:

Description

Numpy/Python version information:

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Inconsistent shape manipulation in operator unique #14244

Description

Reproducing code example:

Description

Numpy/Python version information:

Activity

seberg commented on Aug 11, 2019

hgt312 commented on Aug 11, 2019

seberg commented on Aug 11, 2019

hgt312 commented on Aug 11, 2019

seberg commented on Sep 12, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Inconsistent shape manipulation in operator `unique` #14244