ultrons

Build Instructions:
===================

We follow the "Building with Docker" instructions (with modifications) as described here: 
https://github.com/tensorflow/serving/blob/master/tensorflow_serving/g3doc/building_with_docker.md#building-a-container-from-a-dockerfile

Modfification: 
---------------
Add xla_gpu_jit to the dependencies in the BUILD file and modify the dockerfile to use the change:

ultrons

diff --git a/tests/pytorch/nightly/wav2vec2.libsonnet b/tests/pytorch/nightly/wav2vec2.libsonnet
index 5c6cd8e..a3428ac 100644
--- a/tests/pytorch/nightly/wav2vec2.libsonnet
+++ b/tests/pytorch/nightly/wav2vec2.libsonnet
@@ -26,6 +26,8 @@ local utils = import 'templates/utils.libsonnet';
     export OMP_NUM_THREADS=1
     fairseq-hydra-train \
     task.data=/datasets/w2v2-librispeech-100hrs/w2v/manifest \
+    optimization.max_update %d \
+    dataset.batch_size 4 \

ultrons

pytorch.org,https://pytorch.org/xla/release/1.6/index.html
Cloud TPU Tools,https://cloud.google.com/tpu/docs/cloud-tpu-tools
PyTorch / XLA GitHub,https://github.com/pytorch/xla
Official Tutorials,"Fairseq Transformer: 
Fairseq RoBERTa: 
ResNet-50: 
PyTorch on Cloud TPU Pods:"
Kaggle Kernels,"Super Clean PyTorch TPU Training Kernel
Accelerator Power Hour (PyTorch + TPU)
The Ultimate PyTorch+TPU Tutorial (Jigsaw XLM-R)

ultrons

import torch
import torch.nn as nn
import torch.multiprocessing as mp
import torch.distributed as dist

ultrons

diff --git a/examples/wav2vec/train-vq.sh b/examples/wav2vec/train-vq.sh
index e2065b6..cc48fba 100644
--- a/examples/wav2vec/train-vq.sh
+++ b/examples/wav2vec/train-vq.sh
@@ -6,7 +6,7 @@ python \
         --tpu \
         --bf16 \
         --distributed-world-size 8 \
---max-sentences 1 \
+--max-sentences 8 \

ultrons

diff --git a/fairseq/trainer.py b/fairseq/trainer.py
index 377ce99b..22756df6 100644
--- a/fairseq/trainer.py
+++ b/fairseq/trainer.py
@@ -20,6 +20,7 @@ from fairseq.file_io import PathManager
 from fairseq.logging import meters, metrics
 from fairseq.nan_detector import NanDetector
 from fairseq.optim import lr_scheduler
+from fairseq.metsumm import metsumm
 logger = logging.getLogger(__name__)

ultrons

diff --git a/fairseq/criterions/binary_cross_entropy.py b/fairseq/criterions/binary_cross_entropy.py
index 557f50bd..3b2d0d0f 100644
--- a/fairseq/criterions/binary_cross_entropy.py
+++ b/fairseq/criterions/binary_cross_entropy.py
@@ -8,8 +8,9 @@ import math
 import torch
 import torch.nn.functional as F
 
-from fairseq import utils
+from fairseq import utils, metrics

ultrons

tmux shortcuts & cheatsheet

start new:

tmux

start new with session name:

tmux new -s myname