Add memory format support to `resize_as_` operator #27979

VitalyFedyunin · 2019-10-15T14:31:21Z

Rationale to add optional memory format to resize_as_ and resize_ operators is the fact that they are frequently used as inplace empty_like and empty operators inside of our code base. So having them to accept memory format similarly to empty_like and empty seems to be logical.

We could also add two new operators, but it will be more confusing (taking into account that renaming existing operators in not an option).

Stack from ghstack:

[WIP] Experiments #29461 [WIP] Experiments
[WIP] Switch default of clone, to, *_like ops #29460 [WIP] Switch default of clone, to, *_like ops
[WIP] Prepare copy op to switch memory format preservation defaults #29459 [WIP] Prepare copy op to switch memory format preservation defaults
Adding function to convert Module to channels last #28991 Adding function to convert Module to channels last
Add torch.memory_format support to the TorchScript #28544 Add torch.memory_format support to the TorchScript
Add memory format support to the resize_ op. #28292 Add memory format support to the resize_ op.
TensroIterator preserve format for binary, ternary operators. #28291 TensroIterator preserve format for binary, ternary operators.
Add channels last support to cuda.comm.scatter and gather #28077 Add channels last support to cuda.comm.scatter and gather
Kill operator== of TensorOptions as confusing one #28076 Kill operator== of TensorOptions as confusing one
Add memory format support to resize_as_ operator #27979 Add memory format support to resize_as_ operator

Adds memory_format keyword argument (positional for cpp).

'Preserve' behavior now follows next rules:

If tensor is non-overlapping and dense - output tensor will have the same strides as input tensor.
If not (1) and tensor is stored in the channels last format, output tensor going to have channels last format.
Output tensor is going to be contiguous in all other cases.

Dense tensor is the tensor that store values in a contiguous block of memory.
Non-overlapping tensor is the tensor in which elements occupy individual non-repetitive memory.

Differential Revision: D17980311

Adds memory_format keyword argument (positional for cpp). 'Preserve' behavior now follows next rules: 1) If tensor is non-overlapping and dense - output tensor will have the same strides as input tensor. 2) If not (1) and tensor is stored in the channels last format, output tensor going to have channels last format. 3) Output tensor is going to be contiguous in all other cases. --- Dense tensor is the tensor that store values in a contiguous block of memory. Non-overlapping tensor is the tensor in which elements occupy individual non-repetitive memory. [ghstack-poisoned]

Adds memory_format keyword argument (positional for cpp). 'Preserve' behavior now follows next rules: 1) If tensor is non-overlapping and dense - output tensor will have the same strides as input tensor. 2) If not (1) and tensor is stored in the channels last format, output tensor going to have channels last format. 3) Output tensor is going to be contiguous in all other cases. --- Dense tensor is the tensor that store values in a contiguous block of memory. Non-overlapping tensor is the tensor in which elements occupy individual non-repetitive memory. ghstack-source-id: e6b4b32 Pull Request resolved: #27979

aten/src/ATen/native/Resize.cpp

… operator" Adds memory_format keyword argument (positional for cpp). 'Preserve' behavior now follows next rules: 1) If tensor is non-overlapping and dense - output tensor will have the same strides as input tensor. 2) If not (1) and tensor is stored in the channels last format, output tensor going to have channels last format. 3) Output tensor is going to be contiguous in all other cases. --- Dense tensor is the tensor that store values in a contiguous block of memory. Non-overlapping tensor is the tensor in which elements occupy individual non-repetitive memory. [ghstack-poisoned]

…_as_` operator" Adds memory_format keyword argument (positional for cpp). 'Preserve' behavior now follows next rules: 1) If tensor is non-overlapping and dense - output tensor will have the same strides as input tensor. 2) If not (1) and tensor is stored in the channels last format, output tensor going to have channels last format. 3) Output tensor is going to be contiguous in all other cases. --- Dense tensor is the tensor that store values in a contiguous block of memory. Non-overlapping tensor is the tensor in which elements occupy individual non-repetitive memory. [ghstack-poisoned]

…ator" Adds memory_format keyword argument (positional for cpp). 'Preserve' behavior now follows next rules: 1) If tensor is non-overlapping and dense - output tensor will have the same strides as input tensor. 2) If not (1) and tensor is stored in the channels last format, output tensor going to have channels last format. 3) Output tensor is going to be contiguous in all other cases. --- Dense tensor is the tensor that store values in a contiguous block of memory. Non-overlapping tensor is the tensor in which elements occupy individual non-repetitive memory. [ghstack-poisoned]

…ator" Adds memory_format keyword argument (positional for cpp). 'Preserve' behavior now follows next rules: 1) If tensor is non-overlapping and dense - output tensor will have the same strides as input tensor. 2) If not (1) and tensor is stored in the channels last format, output tensor going to have channels last format. 3) Output tensor is going to be contiguous in all other cases. --- Dense tensor is the tensor that store values in a contiguous block of memory. Non-overlapping tensor is the tensor in which elements occupy individual non-repetitive memory. Differential Revision: [D17980311](https://our.internmc.facebook.com/intern/diff/D17980311) [ghstack-poisoned]

Adds memory_format keyword argument (positional for cpp). 'Preserve' behavior now follows next rules: 1) If tensor is non-overlapping and dense - output tensor will have the same strides as input tensor. 2) If not (1) and tensor is stored in the channels last format, output tensor going to have channels last format. 3) Output tensor is going to be contiguous in all other cases. --- Dense tensor is the tensor that store values in a contiguous block of memory. Non-overlapping tensor is the tensor in which elements occupy individual non-repetitive memory. Differential Revision: [D17980311](https://our.internmc.facebook.com/intern/diff/D17980311) [ghstack-poisoned]

VitalyFedyunin · 2019-10-18T17:24:06Z

@ailzhang ideally I want to land this stack (up to this PR) Monday morning

…ator" Adds memory_format keyword argument (positional for cpp). 'Preserve' behavior now follows next rules: 1) If tensor is non-overlapping and dense - output tensor will have the same strides as input tensor. 2) If not (1) and tensor is stored in the channels last format, output tensor going to have channels last format. 3) Output tensor is going to be contiguous in all other cases. --- Dense tensor is the tensor that store values in a contiguous block of memory. Non-overlapping tensor is the tensor in which elements occupy individual non-repetitive memory. Differential Revision: [D17980311](https://our.internmc.facebook.com/intern/diff/D17980311) [ghstack-poisoned]

…perator" Adds memory_format keyword argument (positional for cpp). 'Preserve' behavior now follows next rules: 1) If tensor is non-overlapping and dense - output tensor will have the same strides as input tensor. 2) If not (1) and tensor is stored in the channels last format, output tensor going to have channels last format. 3) Output tensor is going to be contiguous in all other cases. --- Dense tensor is the tensor that store values in a contiguous block of memory. Non-overlapping tensor is the tensor in which elements occupy individual non-repetitive memory. Differential Revision: [D17980311](https://our.internmc.facebook.com/intern/diff/D17980311) [ghstack-poisoned]

Adds memory_format keyword argument (positional for cpp). 'Preserve' behavior now follows next rules: 1) If tensor is non-overlapping and dense - output tensor will have the same strides as input tensor. 2) If not (1) and tensor is stored in the channels last format, output tensor going to have channels last format. 3) Output tensor is going to be contiguous in all other cases. --- Dense tensor is the tensor that store values in a contiguous block of memory. Non-overlapping tensor is the tensor in which elements occupy individual non-repetitive memory. Differential Revision: [D17980311](https://our.internmc.facebook.com/intern/diff/D17980311) [ghstack-poisoned]

…` operator" Adds memory_format keyword argument (positional for cpp). 'Preserve' behavior now follows next rules: 1) If tensor is non-overlapping and dense - output tensor will have the same strides as input tensor. 2) If not (1) and tensor is stored in the channels last format, output tensor going to have channels last format. 3) Output tensor is going to be contiguous in all other cases. --- Dense tensor is the tensor that store values in a contiguous block of memory. Non-overlapping tensor is the tensor in which elements occupy individual non-repetitive memory. Differential Revision: [D17980311](https://our.internmc.facebook.com/intern/diff/D17980311) [ghstack-poisoned]

Adds memory_format keyword argument (positional for cpp). 'Preserve' behavior now follows next rules: 1) If tensor is non-overlapping and dense - output tensor will have the same strides as input tensor. 2) If not (1) and tensor is stored in the channels last format, output tensor going to have channels last format. 3) Output tensor is going to be contiguous in all other cases. --- Dense tensor is the tensor that store values in a contiguous block of memory. Non-overlapping tensor is the tensor in which elements occupy individual non-repetitive memory. Differential Revision: [D17980311](https://our.internmc.facebook.com/intern/diff/D17980311) [ghstack-poisoned]

…perator" Rationale to add optional memory format to `resize_as_` and `resize_` operators is the fact that they are frequently used as inplace `empty_like` and `empty` operators inside of our code base. So having them to accept memory format similarly to `empty_like` and `empty` seems to be logical. We could also add two new operators, but it will be more confusing (taking into account that renaming existing operators in not an option). --- Adds memory_format keyword argument (positional for cpp). 'Preserve' behavior now follows next rules: 1) If tensor is non-overlapping and dense - output tensor will have the same strides as input tensor. 2) If not (1) and tensor is stored in the channels last format, output tensor going to have channels last format. 3) Output tensor is going to be contiguous in all other cases. --- Dense tensor is the tensor that store values in a contiguous block of memory. Non-overlapping tensor is the tensor in which elements occupy individual non-repetitive memory. Differential Revision: [D17980311](https://our.internmc.facebook.com/intern/diff/D17980311) [ghstack-poisoned]

Rationale to add optional memory format to `resize_as_` and `resize_` operators is the fact that they are frequently used as inplace `empty_like` and `empty` operators inside of our code base. So having them to accept memory format similarly to `empty_like` and `empty` seems to be logical. We could also add two new operators, but it will be more confusing (taking into account that renaming existing operators in not an option). --- Adds memory_format keyword argument (positional for cpp). 'Preserve' behavior now follows next rules: 1) If tensor is non-overlapping and dense - output tensor will have the same strides as input tensor. 2) If not (1) and tensor is stored in the channels last format, output tensor going to have channels last format. 3) Output tensor is going to be contiguous in all other cases. --- Dense tensor is the tensor that store values in a contiguous block of memory. Non-overlapping tensor is the tensor in which elements occupy individual non-repetitive memory. Differential Revision: [D17980311](https://our.internmc.facebook.com/intern/diff/D17980311) [ghstack-poisoned]

gchanan · 2019-11-11T23:09:40Z

aten/src/ATen/native/Resize.cpp

+    TORCH_CHECK(
+        !optional_memory_format.has_value(),
+        "Unsupported memory format for sparse tensor resize_as_ :",
+        optional_memory_format.value_or(MemoryFormat::Contiguous));


can't you use optional_memory_format.value() or *optional_memory_format, because you know it has a value?

gchanan · 2019-11-11T23:15:01Z

aten/src/ATen/native/Resize.cpp

+  if (memory_format == MemoryFormat::Preserve) {
+    memory_format = the_template.suggest_memory_format();
+  }
+  self.unsafeGetTensorImpl()->empty_tensor_restride(memory_format);


doesn't this change the behavior, even in cases where there's no channel_last tensors? i.e. non_contig.resize_as_(contig.shape) will make a tensor contiguous when it didn't previously.

Is that what you want?

Generally line 38 above Tensor& result = self.resize_(the_template.sizes()); was already setting up strides to contiguous.

But thank to this comment I found the only case when it is not intended and might be BC breaking:

x = torch.empty(2,3).t() x.resize_as_(x) # x.is_contiguous() == True but supposed to keep original strides.

Gonna fix it now and test cover (we had no such test).

…_as_` operator" Rationale to add optional memory format to `resize_as_` and `resize_` operators is the fact that they are frequently used as inplace `empty_like` and `empty` operators inside of our code base. So having them to accept memory format similarly to `empty_like` and `empty` seems to be logical. We could also add two new operators, but it will be more confusing (taking into account that renaming existing operators in not an option). --- Adds memory_format keyword argument (positional for cpp). 'Preserve' behavior now follows next rules: 1) If tensor is non-overlapping and dense - output tensor will have the same strides as input tensor. 2) If not (1) and tensor is stored in the channels last format, output tensor going to have channels last format. 3) Output tensor is going to be contiguous in all other cases. --- Dense tensor is the tensor that store values in a contiguous block of memory. Non-overlapping tensor is the tensor in which elements occupy individual non-repetitive memory. Differential Revision: [D17980311](https://our.internmc.facebook.com/intern/diff/D17980311) [ghstack-poisoned]

Rationale to add optional memory format to `resize_as_` and `resize_` operators is the fact that they are frequently used as inplace `empty_like` and `empty` operators inside of our code base. So having them to accept memory format similarly to `empty_like` and `empty` seems to be logical. We could also add two new operators, but it will be more confusing (taking into account that renaming existing operators in not an option). --- Adds memory_format keyword argument (positional for cpp). 'Preserve' behavior now follows next rules: 1) If tensor is non-overlapping and dense - output tensor will have the same strides as input tensor. 2) If not (1) and tensor is stored in the channels last format, output tensor going to have channels last format. 3) Output tensor is going to be contiguous in all other cases. --- Dense tensor is the tensor that store values in a contiguous block of memory. Non-overlapping tensor is the tensor in which elements occupy individual non-repetitive memory. Differential Revision: [D17980311](https://our.internmc.facebook.com/intern/diff/D17980311) [ghstack-poisoned]

gchanan

the behavior looks correct but this badly needs documentation.

…rator" Rationale to add optional memory format to `resize_as_` and `resize_` operators is the fact that they are frequently used as inplace `empty_like` and `empty` operators inside of our code base. So having them to accept memory format similarly to `empty_like` and `empty` seems to be logical. We could also add two new operators, but it will be more confusing (taking into account that renaming existing operators in not an option). --- Adds memory_format keyword argument (positional for cpp). 'Preserve' behavior now follows next rules: 1) If tensor is non-overlapping and dense - output tensor will have the same strides as input tensor. 2) If not (1) and tensor is stored in the channels last format, output tensor going to have channels last format. 3) Output tensor is going to be contiguous in all other cases. --- Dense tensor is the tensor that store values in a contiguous block of memory. Non-overlapping tensor is the tensor in which elements occupy individual non-repetitive memory. Differential Revision: [D17980311](https://our.internmc.facebook.com/intern/diff/D17980311) [ghstack-poisoned]

VitalyFedyunin · 2019-11-14T18:21:13Z

Docs added

gchanan · 2019-11-15T00:57:31Z

aten/src/ATen/native/Resize.cpp

+// Strides of the output tensor of `resize_as_` operator is defined by input
+// tensor strides and the value of memory_format argument.
+//
+// If memory_format is not defined or equals to MemoryFormat::Contiguous


I don't think this is correct.

Don't we not restride if memory_format is not set and the sizes match?

…rator" Rationale to add optional memory format to `resize_as_` and `resize_` operators is the fact that they are frequently used as inplace `empty_like` and `empty` operators inside of our code base. So having them to accept memory format similarly to `empty_like` and `empty` seems to be logical. We could also add two new operators, but it will be more confusing (taking into account that renaming existing operators in not an option). --- Adds memory_format keyword argument (positional for cpp). 'Preserve' behavior now follows next rules: 1) If tensor is non-overlapping and dense - output tensor will have the same strides as input tensor. 2) If not (1) and tensor is stored in the channels last format, output tensor going to have channels last format. 3) Output tensor is going to be contiguous in all other cases. --- Dense tensor is the tensor that store values in a contiguous block of memory. Non-overlapping tensor is the tensor in which elements occupy individual non-repetitive memory. Differential Revision: [D17980311](https://our.internmc.facebook.com/intern/diff/D17980311) [ghstack-poisoned]

gchanan · 2019-11-15T23:58:34Z

aten/src/ATen/native/Resize.cpp

+// tensor strides and the value of memory_format argument.
+//
+// If memory_format is omitted and input tensor have the same shape as output
+// tensor, strides of the output will remain unchanged. Strides going to be


nit: strides are going to be...

gchanan · 2019-11-16T00:00:31Z

test/test_torch.py

+        x.resize_as_(x)
+        self.assertEqual(x.stride(), old_strides)
+
+    def test_memory_format_resize_as(self, device):


we should have tests for all the cases, i.e. add a:

explicit contiguous test

explicit channels_last test.

facebook-github-bot · 2019-11-18T14:03:38Z

@VitalyFedyunin merged this pull request in 9f3b347.

Summary: Pull Request resolved: pytorch/pytorch#27979 Adds memory_format keyword argument (positional for cpp). 'Preserve' behavior now follows next rules: 1) If tensor is non-overlapping and dense - output tensor will have the same strides as input tensor. 2) If not (1) and tensor is stored in the channels last format, output tensor going to have channels last format. 3) Output tensor is going to be contiguous in all other cases. --- Dense tensor is the tensor that store values in a contiguous block of memory. Non-overlapping tensor is the tensor in which elements occupy individual non-repetitive memory. Test Plan: Imported from OSS Differential Revision: D17980311 Pulled By: VitalyFedyunin fbshipit-source-id: 12d013521091fcc9c045833577f6dc78d7b1e68f

VitalyFedyunin requested a review from apaszke as a code owner October 15, 2019 14:31

zou3519 reviewed Oct 15, 2019

View reviewed changes

aten/src/ATen/native/Resize.cpp Show resolved Hide resolved

This was referenced Oct 16, 2019

Kill operator== of TensorOptions as confusing one #28076

Closed

Add channels last support to cuda.comm.scatter and gather #28077

Closed

VitalyFedyunin added 3 commits October 17, 2019 07:59

This was referenced Oct 18, 2019

TensroIterator preserve format for binary, ternary operators. #28291

Closed

Add memory format support to the resize_ op. #28292

Closed

VitalyFedyunin requested a review from ezyang October 18, 2019 17:23

VitalyFedyunin requested a review from zou3519 October 21, 2019 15:12

ailzhang approved these changes Oct 21, 2019

View reviewed changes

VitalyFedyunin mentioned this pull request Oct 31, 2019

Adding function to convert Module to channels last #28991

Closed

VitalyFedyunin added 2 commits November 4, 2019 12:00

This was referenced Nov 8, 2019

[WIP] Prepare copy op to switch memory format preservation defaults #29459

Closed

[WIP] Switch default of clone, to, *_like ops #29460

Closed

[WIP] Experiments #29461

Closed

VitalyFedyunin added 2 commits November 8, 2019 10:23

gchanan added the module: safe resize label Nov 11, 2019

gchanan reviewed Nov 11, 2019

View reviewed changes

VitalyFedyunin requested a review from gchanan November 12, 2019 00:39

gchanan requested changes Nov 13, 2019

View reviewed changes

VitalyFedyunin requested a review from gchanan November 14, 2019 18:21

gchanan reviewed Nov 15, 2019

View reviewed changes

VitalyFedyunin added 2 commits November 14, 2019 19:30

VitalyFedyunin requested a review from gchanan November 15, 2019 03:38

gchanan approved these changes Nov 16, 2019

View reviewed changes

facebook-github-bot closed this in 9f3b347 Nov 18, 2019

facebook-github-bot added the merged label Nov 18, 2019

facebook-github-bot deleted the gh/VitalyFedyunin/14/head branch November 21, 2019 15:16

mruberry added the Merged label Oct 28, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add memory format support to `resize_as_` operator #27979

Add memory format support to `resize_as_` operator #27979

Uh oh!

VitalyFedyunin commented Oct 15, 2019 •

edited

Loading

Uh oh!

Uh oh!

VitalyFedyunin commented Oct 18, 2019 •

edited

Loading

Uh oh!

gchanan Nov 11, 2019

Uh oh!

gchanan Nov 11, 2019

Uh oh!

VitalyFedyunin Nov 11, 2019

Uh oh!

gchanan left a comment

Uh oh!

VitalyFedyunin commented Nov 14, 2019

Uh oh!

gchanan Nov 15, 2019

Uh oh!

VitalyFedyunin Nov 15, 2019

Uh oh!

gchanan Nov 15, 2019

Uh oh!

gchanan Nov 16, 2019

Uh oh!

facebook-github-bot commented Nov 18, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Add memory format support to resize_as_ operator #27979

Add memory format support to resize_as_ operator #27979

Uh oh!

Conversation

VitalyFedyunin commented Oct 15, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

VitalyFedyunin commented Oct 18, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gchanan Nov 11, 2019

Choose a reason for hiding this comment

Uh oh!

gchanan Nov 11, 2019

Choose a reason for hiding this comment

Uh oh!

VitalyFedyunin Nov 11, 2019

Choose a reason for hiding this comment

Uh oh!

gchanan left a comment

Choose a reason for hiding this comment

Uh oh!

VitalyFedyunin commented Nov 14, 2019

Uh oh!

gchanan Nov 15, 2019

Choose a reason for hiding this comment

Uh oh!

VitalyFedyunin Nov 15, 2019

Choose a reason for hiding this comment

Uh oh!

gchanan Nov 15, 2019

Choose a reason for hiding this comment

Uh oh!

gchanan Nov 16, 2019

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Nov 18, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Add memory format support to `resize_as_` operator #27979

Add memory format support to `resize_as_` operator #27979

VitalyFedyunin commented Oct 15, 2019 •

edited

Loading

VitalyFedyunin commented Oct 18, 2019 •

edited

Loading