This repository was archived by the owner on Nov 17, 2023. It is now read-only.
This repository was archived by the owner on Nov 17, 2023. It is now read-only.
It seems nd.array or NDArrayIter not support too large array #9207
Description
Description
I have a large array, cannot be transformed into nd.array
Environment info (Required)
CentOS and MacOS
Package used (Python/R/Scala/Julia):
Python
Build info (Required if built from source)
install by pip, mxnet=1.0.0
Error Message:
15:31:56] /Users/travis/build/dmlc/mxnet-distro/mxnet-build/dmlc-core/include/dmlc/logging.h:308: [15:31:56] include/mxnet/./tensor_blob.h:275: Check failed: this->shape_.Size() == shape.Size() (6553600000 vs. 2258632704) TBlob.get_with_shape: new and old shape do not match total elements
Stack trace returned 7 entries:
[bt] (0) 0 libmxnet.so 0x0000000d378eaad8 _ZN4dmlc15LogMessageFatalD2Ev + 40
[bt] (1) 1 libmxnet.so 0x0000000d3790cae9 _ZNK5mxnet5TBlob14get_with_shapeIN7mshadow3cpuELi1EfEENS2_6TensorIT_XT0_ET1_EERKNS2_5ShapeIXT0_EEEPNS2_6StreamIS5_EE + 777
[bt] (2) 2 libmxnet.so 0x0000000d380fa0be _ZN5mxnet7ndarray4CopyIN7mshadow3cpuES3_EEvRKNS_5TBlobEPS4_NS_7ContextES8_NS_10RunContextE + 14382
[bt] (3) 3 libmxnet.so 0x0000000d380d9673 _ZNK5mxnet7NDArray15SyncCopyFromCPUEPKvm + 1139
[bt] (4) 4 libmxnet.so 0x0000000d37fcc1fd MXNDArraySyncCopyFromCPU + 13
[bt] (5) 5 _ctypes.cpython-36m-darwin.so 0x0000000101eb742f ffi_call_unix64 + 79
[bt] (6) 6 ??? 0x00007fff5e83a820 0x0 + 140734779074592
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.6/site-packages/mxnet/ndarray.py", line 1295, in array
arr[:] = source_array
File "/usr/local/lib/python3.6/site-packages/mxnet/ndarray.py", line 386, in __setitem__
self._sync_copyfrom(value)
File "/usr/local/lib/python3.6/site-packages/mxnet/ndarray.py", line 560, in _sync_copyfrom
ctypes.c_size_t(source_array.size)))
File "/usr/local/lib/python3.6/site-packages/mxnet/base.py", line 129, in check_call
raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [15:31:56] include/mxnet/./tensor_blob.h:275: Check failed: this->shape_.Size() == shape.Size() (6553600000 vs. 2258632704) TBlob.get_with_shape: new and old shape do not match total elements
Stack trace returned 7 entries:
[bt] (0) 0 libmxnet.so 0x0000000d378eaad8 _ZN4dmlc15LogMessageFatalD2Ev + 40
[bt] (1) 1 libmxnet.so 0x0000000d3790cae9 _ZNK5mxnet5TBlob14get_with_shapeIN7mshadow3cpuELi1EfEENS2_6TensorIT_XT0_ET1_EERKNS2_5ShapeIXT0_EEEPNS2_6StreamIS5_EE + 777
[bt] (2) 2 libmxnet.so 0x0000000d380fa0be _ZN5mxnet7ndarray4CopyIN7mshadow3cpuES3_EEvRKNS_5TBlobEPS4_NS_7ContextES8_NS_10RunContextE + 14382
[bt] (3) 3 libmxnet.so 0x0000000d380d9673 _ZNK5mxnet7NDArray15SyncCopyFromCPUEPKvm + 1139
[bt] (4) 4 libmxnet.so 0x0000000d37fcc1fd MXNDArraySyncCopyFromCPU + 13
[bt] (5) 5 _ctypes.cpython-36m-darwin.so 0x0000000101eb742f ffi_call_unix64 + 79
[bt] (6) 6 ??? 0x00007fff5e83a820 0x0 + 140734779074592
Minimum reproducible example
import numpy as np
import mxnet as mx
X = np.zeros((20000, 32768), dtypes=np.float32)
mx.nd.array(X)
Steps to reproduce
just run the code above