TengdaHan

Multi-node-training on slurm with PyTorch

What's this?

• A simple note for how to start multi-node-training on slurm scheduler with PyTorch.
• Useful especially when scheduler is too busy that you cannot get multiple GPUs allocated, or you need more than 4 GPUs for a single job.
• Requirement: Have to use PyTorch DistributedDataParallel(DDP) for this purpose.
• Warning: might need to re-factor your own code.
• Warning: might be secretly condemned by your colleagues because using too many GPUs.

lantiga

from transformers import BertForQuestionAnswering
import torch

bert_name = "bert-large-uncased-whole-word-masking-finetuned-squad"

model = BertForQuestionAnswering.from_pretrained(bert_name, torchscript=True)
model.eval()

inputs = [torch.ones(1, 2, dtype=torch.int64),
          torch.ones(1, 2, dtype=torch.int64),

mohanpedala

Table of Contents

• set -e, -u, -x, -o pipefail
• set -e
• set -x
• set -u
• set -o pipefail
• Setting IFS
• Original Reference

set -e, -u, -x, -o pipefail