Pytorch acceleration tips

pytorch_acceleration.md

Summry of the following guide https://nvlabs.github.io/eccv2020-mixed-precision-tutorial/files/szymon_migacz-pytorch-performance-tuning-guide.pdf

1. In the dataset class

pin_memory=True

2. Enable for device specific CNN acceleration

torch.backends.cudnn.benchmark = True

3. Increase the batch size to max out GPU memory. SGD modification for large batch: LARS

4. Disable bias for convlutaion if followed firectly by batch norm to reduce paramters. Instead

model.zero_grad()

use

for param in model.parameters():
    param.grad = None

5. Add jit decorator to fuse cuda kernels

@torch.jit.script decorator to fuse cuda kernels