mingfeima

PyTorch Performance Optimization on CPU

1. pytorch mkldnn integration prototype design

• mkldnn conv integration
• conv3d parallelization: vol2col, col2vol
• LSTM optimization non-fused: tanh/sigmoid parallelization

2. Create MKLDNN conda channel

3. MKLDNN tensor type

• create lib/THMKL?

mingfeima

Hotspot analysis:

/opt/intel/vtune_amplifier/bin64/amplxe-cl -collect hotspots -knob analyze-openmp=true -knob sampling-interval=10 --resume-after 5 -d 20 \
  -- /home/mingfeim/pytorch/unit_tests/run.sh
/opt/intel/vtune_amplifier/bin64/amplxe-cl -archive -r $1

Interpret vtune log function names: e.g.

mingfeima

nn.Embedding()

ref: TensorFlow

mingfeima

backups for PR19736 of topk() performance optimization on CPU.

---

description

Suppose input tensor has shape of [N, C], performance input.topk(K, sorted=Sorted) for the followings scenarios:

1. C = 10000, 40000, 320000
2. K = 10, 50, 100, C/10, C/2, C-5
3. Test with 20 threads and 1 thread
4. Test with Sorted=True and Sorted=False

mingfeima

trace #30806 of torch.cat() performance regression.

benchmark_all_test result, command line:

python -m benchmark_all_test --operators cat --tag_filter all

with commit 7b50e76255aebbbcdae702ee1f00d07d86b0112b

(pytorch-mingfei) [mingfeim@mlt-skx090 operator_benchmark]$ python -m benchmark_all_test --operators cat --tag_filter all

mingfeima

MKL-DNN Integration Plan

The purpose is to further improve PyTorch CPU performance on both imperative path and jit path. MKLDNN requires to reorder memory from plain layout to blocked layout to achieve optimal performance on CPU, e.g. from nchw to nChw16c, etc. At this moment on PyTorch, MKLDNN operators reuse CPU tensor, which means for each MKLDNN operator, it takes three steps to finish the computation:

input_reorder(plain_layout, blocked_layout)
mkldnn_computation()
output_reorder(blocked_layout, plain_layout)

These reorders takes about 50% of time on a typical ImageNet topology, e.g. ResNet50. Also MKLDNN chose different blocked format according to different input config from Convolution, with nn.Conv2d always output in plain layout, subsequent layers (BatchNorm, Pooling) would only execute on plain layout and this is the slow path for MKLDNN. With these problems solved, the CNN models would have 3~4x speedup v.s. current performance.

mingfeima

This Gist records optimization effort of DLRM on PyTorch CPU path.

Branch on track: dlrm

Task list:

• LAMB fused optimizer (fp32)
• Adagrad fused optimier (fp32)
• Split-SGD (bf16)

mingfeima

Part IV: BFloat16 Kernel Optimization

(Training material on pytorch CPU performance optimization)

• Part I: Memory Formats and Channels Last Optimization
• Part II: Parallelization Techniques
• Part III: Vectorization Techniques

Chinese version for this chapter, link.

This section contains the following subjects:

mingfeima

BKMs to check whether mkl or mkldnn is enabled on PyTorch

PyTorch can be installed via different channels: conda, pip, docker, source code...

By default, mkl and mkl-dnn are enabled; But this might not always be true, so it is still useful to learn how to check this by yourself:

1. How to check whether mkl is enabled?

### check where your torch is installed
python -c 'import torch; print(torch.__path__)'

mingfeima

benchmark

Based on huggingface repo for performance evaluation, actual benchmark run script placed at repo. How to reproduce performance:

1. prepare dataset according to link.
2. update GLUE_DIR to actual dataset path in run_inference.sh.
3. change env settings, the default setting is using 20 cores;

MKL v.s. MKLDNN

Inference performance result on Xeon 6148 (2x20 cores), single socket and single thread.