Skip to content

Instantly share code, notes, and snippets.

@salmedina

salmedina/pytorch_hacks.md

Created October 19, 2022 15:41
Show Gist options
  • Save salmedina/32c9ec91a5bb1288df03ec73fa2bbf95 to your computer and use it in GitHub Desktop.
Save salmedina/32c9ec91a5bb1288df03ec73fa2bbf95 to your computer and use it in GitHub Desktop.
Download ZIP
PyTorch Training Hacks
Raw
pytorch_hacks.md
  1. For DDP find_unused_parameters=False
  2. For optimization while training Optimizer.zero_grad(set_to_none=False)
  3. CUDNN flags:
    • torch.backend.cudnn.enabled=True
    • torch.backend.cudnn.deterministic=False
    • torch.backend.cudnn.benchmark=True
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment