Prune python tensor #5596

daquexian · 2021-07-26T03:05:25Z

将 python 的 Tensor 类删除，逻辑转移到 c++

>>> import oneflow as flow
>>> x=flow.Tensor(1,2)
>>> type(x)
<class 'oneflow.Tensor'>
>>> y=x+x
>>> type(y)
<class 'oneflow.Tensor'>
>>>

Signed-off-by: daquexian <[email protected]>

daquexian · 2021-07-26T03:11:36Z

oneflow/compatible_single_client_python/framework/tensor.py

 See the License for the specific language governing permissions and
 limitations under the License.
 """
-from oneflow.compatible.single_client.core.job import (


用 oneflow/python/framework/tensor.py 的内容覆盖了这个文件的原内容。因为它们共用了一个 _oneflow_internal

daquexian · 2021-07-26T03:12:29Z

oneflow/extension/python/py_compute.cpp

 namespace {
 static PyObject* py_kernels_dic = nullptr;

-void OFDataTypeToNumpyType(DataType of_data_type, int* out_numpy_type) {


这两个方法移动到了 oneflow/extension/python/numpy.h 里并做了修改

Signed-off-by: daquexian <[email protected]>

daquexian · 2021-07-26T03:41:22Z

oneflow/python/framework/tensor.py

+    Tensor.normal_ = _normal
+    Tensor.fill_ = _fill
+    Tensor._placement_scope = _placement_scope
+    Tensor.copy_ = _copy


c++ tensor 类的一些方法在 python 层注入，暂时不着急全部重写到 c++

Signed-off-by: daquexian <[email protected]>

oneflow/api/python/framework/device.cpp

oneflow/api/python/framework/tensor.cpp

Signed-off-by: daquexian <[email protected]>

daquexian · 2021-07-26T05:59:11Z

cmake/pybind11.cmake

@@ -1,7 +1,7 @@
 include(FetchContent)
-set(PYBIND11_TAR_URL https://github.com/pybind/pybind11/archive/v2.6.0.zip)
+set(PYBIND11_TAR_URL https://github.com/pybind/pybind11/archive/v2.7.0.zip)


升级 pybind11 把这个修复 gcc 4 编译错误的 pr 包含进来 pybind/pybind11#2956

wyg1997

Parameter的构造函数里是不是忘处理了？

wyg1997 · 2021-07-26T10:32:00Z

oneflow/api/python/framework/tensor.cpp

+    } else {
+      return MakeLocalTensorByNumpy(arg, desired_dtype, device, requires_grad);
+    }
+  } else {


这里需要和Pytorch那样，如果参数检察不正确，把所有的合法参数都打印出来给个提示吗？

感觉可以 😂

Signed-off-by: daquexian <[email protected]>

lixinqi · 2021-07-27T11:20:48Z

oneflow/api/python/framework/device.cpp

+  std::string type;
+  int device_id = -1;
+  ParsingDeviceTag(type_and_id, &type, &device_id).GetOrThrow();
+  if (device_id == -1) { device_id = 0; }
+  return MakeDevice(type, device_id);


记得新开一个issue，让新手对这里做优化。用thread_local作缓存。

lixinqi · 2021-07-27T11:22:38Z

oneflow/core/framework/tensor.h

+  }
+
+  bool requires_grad() const override { return tensor_->requires_grad(); }
+  bool is_leaf() const override { return tensor_->is_leaf(); }


是不是应该直接返回true呢？

是的，已修改，同时通过让 Parameter 继承 TensorIf，也实现了 Parameter 有独立的 grad_fn

lixinqi · 2021-07-27T11:23:51Z

oneflow/core/framework/tensor.h

+  user_op::TensorDesc* mut_tensor_meta() override { return tensor_->mut_tensor_meta(); }
+
+  Maybe<MirroredTensor> AsMirroredTensor() override {
+    if (const auto& mirrored_tensor = std::dynamic_pointer_cast<MirroredTensor>(tensor_)) {


这语法是c++11所能支持的吗？

是的，这个和 c++17 的 if (int x = 0; x < 1) 不是同一个语法

哦哦。我才注意到两者的区别，那我还把很多的宏写复杂了。

oneflow/core/framework/tensor.h

oneflow/api/python/framework/tensor.cpp

…d missing get_device method, remove out-dated methods in docs Signed-off-by: daquexian <[email protected]>

Signed-off-by: daquexian <[email protected]>

…into prune_python_tensor

Signed-off-by: daquexian <[email protected]>

github-actions · 2021-07-28T01:46:59Z

Speed stats:

GPU Name: GeForce GTX 1080 

PyTorch resnet50 time: 141.0ms (= 7050.6ms / 50, input_shape=[16, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 126.3ms (= 6313.8ms / 50, input_shape=[16, 3, 224, 224], backward is enabled)
Relative speed: 1.12 (= 141.0ms / 126.3ms)

PyTorch resnet50 time: 84.7ms (= 4236.8ms / 50, input_shape=[8, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 73.7ms (= 3683.9ms / 50, input_shape=[8, 3, 224, 224], backward is enabled)
Relative speed: 1.15 (= 84.7ms / 73.7ms)

PyTorch resnet50 time: 60.3ms (= 3016.7ms / 50, input_shape=[4, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 49.2ms (= 2459.6ms / 50, input_shape=[4, 3, 224, 224], backward is enabled)
Relative speed: 1.23 (= 60.3ms / 49.2ms)

PyTorch resnet50 time: 51.2ms (= 2558.2ms / 50, input_shape=[2, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 45.7ms (= 2286.1ms / 50, input_shape=[2, 3, 224, 224], backward is enabled)
Relative speed: 1.12 (= 51.2ms / 45.7ms)

PyTorch resnet50 time: 44.0ms (= 2199.3ms / 50, input_shape=[1, 3, 224, 224], backward is enabled)
OneFlow resnet50 time: 41.9ms (= 2094.6ms / 50, input_shape=[1, 3, 224, 224], backward is enabled)
Relative speed: 1.05 (= 44.0ms / 41.9ms)

daquexian added 3 commits July 26, 2021 10:23

move tensor to c++

5da3203

Signed-off-by: daquexian <[email protected]>

Merge remote-tracking branch 'origin/master' into prune_python_tensor

0765924

Signed-off-by: daquexian <[email protected]>

align with master and fix export error

2369658

Signed-off-by: daquexian <[email protected]>

daquexian added enhancement system labels Jul 26, 2021

reformat

9b0d522

Signed-off-by: daquexian <[email protected]>

daquexian commented Jul 26, 2021

View reviewed changes

rename some functions

08e4409

Signed-off-by: daquexian <[email protected]>

daquexian requested review from oneflow-ci-bot and removed request for oneflow-ci-bot July 26, 2021 03:19

daquexian force-pushed the prune_python_tensor branch from f423439 to 003787a Compare July 26, 2021 03:38

refine

ffa1047

Signed-off-by: daquexian <[email protected]>

daquexian force-pushed the prune_python_tensor branch from 003787a to ffa1047 Compare July 26, 2021 03:39

daquexian commented Jul 26, 2021

View reviewed changes

remove unused header

7a93ba0

Signed-off-by: daquexian <[email protected]>

daquexian requested review from oneflow-ci-bot and removed request for oneflow-ci-bot July 26, 2021 03:53

oneflow-ci-bot removed their request for review July 26, 2021 05:16

lixinqi reviewed Jul 26, 2021

View reviewed changes

oneflow/api/python/framework/device.cpp Outdated Show resolved Hide resolved

oneflow/api/python/framework/device.cpp Outdated Show resolved Hide resolved

oneflow/api/python/framework/tensor.cpp Outdated Show resolved Hide resolved

daquexian added 2 commits July 26, 2021 13:37

restore device repr and str

0c7a413

Signed-off-by: daquexian <[email protected]>

upgrade pybind11 to 2.7.0 for pybind/pybind11#2956

9596fd8

Signed-off-by: daquexian <[email protected]>

daquexian commented Jul 26, 2021

View reviewed changes

daquexian requested a review from oneflow-ci-bot July 26, 2021 06:00

lixinqi approved these changes Jul 26, 2021

View reviewed changes

oneflow-ci-bot removed their request for review July 26, 2021 07:09

wyg1997 reviewed Jul 26, 2021

View reviewed changes

Merge remote-tracking branch 'origin/master' into prune_python_tensor

a87b836

Signed-off-by: daquexian <[email protected]>

daquexian force-pushed the prune_python_tensor branch from 0e80934 to a87b836 Compare July 27, 2021 01:19

impl parameter in c++, remove determine() call

11ebffd

Signed-off-by: daquexian <[email protected]>

oneflow-ci-bot removed their request for review July 27, 2021 11:15

lixinqi reviewed Jul 27, 2021

View reviewed changes

hjchen2 reviewed Jul 27, 2021

View reviewed changes

oneflow/core/framework/tensor.h Show resolved Hide resolved

hjchen2 reviewed Jul 27, 2021

View reviewed changes

oneflow/api/python/framework/tensor.cpp Outdated Show resolved Hide resolved

oneflow/api/python/framework/tensor.cpp Outdated Show resolved Hide resolved

daquexian added 4 commits July 27, 2021 20:57

let parameter derive from TensorIf to own a separate grad_fn_node, ad…

2f2a6e0

…d missing get_device method, remove out-dated methods in docs Signed-off-by: daquexian <[email protected]>

Merge remote-tracking branch 'origin/master' into prune_python_tensor

72a92ba

Merge remote-tracking branch 'origin/master' into prune_python_tensor

f654490

add test_parameter.py, fix requires_grad_ and get_device

1d43c60

Signed-off-by: daquexian <[email protected]>

daquexian requested a review from oneflow-ci-bot July 27, 2021 14:06

use oneflow cast instead of numpy cast

1ecdd29

Signed-off-by: daquexian <[email protected]>

daquexian added automerge and removed automerge labels Jul 27, 2021

restore format changes

63d683b

Signed-off-by: daquexian <[email protected]>

daquexian requested review from oneflow-ci-bot and removed request for oneflow-ci-bot July 27, 2021 14:34

auto format by CI

0d52ea8

daquexian requested review from oneflow-ci-bot and removed request for oneflow-ci-bot July 27, 2021 14:37

daquexian added the automerge label Jul 27, 2021

daquexian mentioned this pull request Jul 27, 2021

ddp #5008

Merged

oneflow-ci-bot removed their request for review July 27, 2021 16:02

daquexian added 3 commits July 28, 2021 09:00

fix tests involving _local_or_consistent_tensor

6a5b27b

Signed-off-by: daquexian <[email protected]>

Merge branch 'prune_python_tensor' of github.com:Oneflow-Inc/oneflow …

6080c68

…into prune_python_tensor

Merge remote-tracking branch 'origin/master' into prune_python_tensor

03fd902

Signed-off-by: daquexian <[email protected]>

daquexian requested a review from oneflow-ci-bot July 28, 2021 01:03

oneflow-ci-bot removed their request for review July 28, 2021 02:04

oneflow-ci-bot merged commit 3a2b338 into master Jul 28, 2021

oneflow-ci-bot deleted the prune_python_tensor branch July 28, 2021 02:06

Prune python tensor #5596

Prune python tensor #5596

Uh oh!

Conversation

daquexian commented Jul 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wyg1997 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Jul 28, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

daquexian commented Jul 26, 2021 •

edited

Loading