Skip to content

Instantly share code, notes, and snippets.

@dasayan05
Created February 27, 2019 10:20
Show Gist options
  • Save dasayan05/9b9cdc4a272500ecbfa648dd54d5fbde to your computer and use it in GitHub Desktop.
Save dasayan05/9b9cdc4a272500ecbfa648dd54d5fbde to your computer and use it in GitHub Desktop.
Synchronize gradients
def sync_gradients(model, rank, world_size):
for param in model.parameters():
dist.all_reduce(param.grad.data, op=dist.reduce_op.SUM)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment