@bddppq Hi, the backward integration is done and please review the code.
Feel free to invite more reviewers.

@uyongw, @Jianhui-Li, @jgong5.

@dzhulgakov so what is the proposed system config for training? double sockets or single socket?
@XiaobingSuper post benchmark numbers here.

However, we have a known issue here that memory buffer is not cached (output tensors need to be allocated everytime), this is going to add significant overhead for the training scenario.

Customized memory allocator is a solution but seems to be against the current design. Suggestions?

However, we have a known issue here that memory buffer is not cached (output tensors need to be allocated everytime), this is going to add significant overhead for the training scenario.

Customized memory allocator is a solution but seems to be against the current design. Suggestions?

@dzhulgakov The performance issue @mingfeima pointed out is caused by clear_page overhead on large buffer allocation with malloc. This happens when PyTorch allocates large activation buffer which is a typical case during training since the batch size is usually large. As you know, whether clear_page is triggered is controlled by M_MMAP_THRESHOLD (http://man7.org/linux/man-pages/man3/mallopt.3.html) which default to 128KB and on 64-bit system, the upper bound is 32MB which is not a big number w.r.t. typical training workloads. I notice that PyTorch has caching allocator in the GPU path and probably CPU can use similar idea to avoid calling malloc again and again for output tensors. Currently, we have a quick implementation of caching allocator inside ideep and @XiaobingSuper will share the training performance numbers with and without caching allocator FYI.

Runing this benchmark on SKX-6148 with 2 sockets, we can get 2X performance improvement using mkldnn compared to native path:

The following is the details logs:

• Native path:

### using OMP_NUM_THREADS=40
### using KMP_AFFINITY=granularity=fine,compact,1,0
### using KMP_BLOCKTIME=1

Running on device:  Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
Running on torch: 1.2.0a0+69e3229
Running on torchvision: 0.3.0a0+8837e0e

ModelType: resnet50, Kernels: nn Input shape: 128x3x224x224
nn                              :forward:    1803.12 (ms)      70.99 (imgs/s)
nn                             :backward:    2725.13 (ms)
nn                               :update:       4.72 (ms)
nn                                :total:    4528.25 (ms)      28.27 (imgs/s)
ModelType: resnext101_32x8d, Kernels: nn Input shape: 128x3x224x224
nn                              :forward:    6042.85 (ms)      21.18 (imgs/s)
nn                             :backward:    8624.28 (ms)
nn                               :update:      13.07 (ms)
nn                                :total:   14667.13 (ms)       8.73 (imgs/s)

• MKLDNN path:

### using OMP_NUM_THREADS=40
### using KMP_AFFINITY=granularity=fine,compact,1,0
### using KMP_BLOCKTIME=1

Running on device:  Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
Running on torch: 1.2.0a0+69e3229
Running on torchvision: 0.3.0a0+8837e0e

ModelType: resnet50, Kernels: nn Input shape: 128x3x224x224
nn                              :forward:     659.40 (ms)     194.12 (imgs/s)
nn                             :backward:    1228.16 (ms)
nn                               :update:       4.69 (ms)
nn                                :total:    1887.55 (ms)      67.81 (imgs/s)
ModelType: resnext101_32x8d, Kernels: nn Input shape: 128x3x224x224
nn                              :forward:    2259.24 (ms)      56.66 (imgs/s)
nn                             :backward:    4234.59 (ms)
nn                               :update:      13.00 (ms)
nn                                :total:    6493.83 (ms)      19.71 (imgs/s)

Next step, I will share the performance using caching allocator. Thanks!

Adding the performance using idepp caching allocator on SKX-6148 with 2 sockets, there has 1.36x performance improvement at least for inference with large batch size and has 1.32x performance improvement at least for training.

1. For inference using MKLDNN with large batch size:

2. For training using MKLDNN:

Adding CPU caching allocator is something we also discussed. The problem is mostly alleviated by using better malloc - e.g. using jemalloc (http://jemalloc.net/jemalloc.3.html). Unfortunately, I don't think there's a safe way to package it with prebuilt pytorch binaries - it has to be preloaded for the entire Python process.

Overall, this PR looks pretty good to me, also cc @zheng-xq to take a look

@zheng-xq , can you help review it?

@bddppq, can you help review this code, perharps we can first merge this PR which can unify the mkldnn convolution code. Thanks!

@pytorchbot rebase this please

@VitalyFedyunin, I rebase the code again, can you help review it when you have free time? Thanks!

@VitalyFedyunin, the test failed cases are not related to this PR. Thanks!

@VitalyFedyunin

@dzhulgakov regarding the jemalloc. We tried out Jemalloc and TCmalloc, but both creates extra dependencies and doesn’t support NUMA. TF expose NUMA as subdevice and user can do NUMA-aware memory allocation. The memory allocator makes big difference when running trainning or offline inference with large batch size, since malloc may trigger large overhead since the malloc of large chunk memory causes clear page overhead. We observed +30% benefit on throughput. Are there any plan for Pytorch to support similar capability, if not, do you think it would work for us to implement one and PR it?

This PR reopened in #36121.