Skip to content

Instantly share code, notes, and snippets.

View gist:e6bd5cb10e1955f17336c7d9f806fc2a
This file has been truncated, but you can view the full file.

Deep Learning InfraOps without Tears

This is an introduction to managing infrastructure (Infra) and system administration operations (Ops) particularly for deep learning applications.

Why do I have to learn this?

Q: I have my Jupyter notebooks and virtual machines that comes with all batteries included. Why do I have to handle bare-metal machines?

A: You don't (if you are not interested). In most cases, the normal virtual machines with everything included, e.g. AWS Sagemaker or Azures Machine Learning Studio. But these are technically "Managed" Virtual Machines and often fixing something that goes wrong after the VM has been spun up for a couple of months will lead to the same issues that requires the knowledge of infra-ops to handle/fix the issue.

from functools import lru_cache
from itertools import product
def ld(s, t):
Levenshtein distance memoized implementation from Rosetta code:
if not s: return len(t)
import requests
from io import StringIO
response = requests.get('')
with open('emoji.txt', 'w') as fout:
with StringIO(response.content.decode('utf8')) as fin:
for line in fin:
if line.strip() and not line.startswith('#'):
hexa = line.split(';')[0].split('..')
View paracrawl-3-human-eval.ipynb
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

What the BookCorpus?

So in the midst of all these Sesame Streets characters and robots transforming automobile era of "contextualize" language models, there is this "Toronto Book Corpus" that points to this kinda recently influential paper:

Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. "Aligning books and movies: Towards story-like visual explanations by watching movies and reading books." In Proceedings of the IEEE international conference on computer vision, pp. 19-27.

Why do I even care, there's no translations there?

Some might know my personal pet peeve on collecting translation datasets but this BookCorpus has no translations, so why do I even care about it?

View make-15AUG19.log
$ make -j $(nproc)
Scanning dependencies of target nccl_install
Scanning dependencies of target marian_version
Scanning dependencies of target pathie-cpp
Scanning dependencies of target SQLiteCpp
Scanning dependencies of target libyaml-cpp
Scanning dependencies of target zlib
[ 0%] Running cpp protocol buffer compiler on sentencepiece_model.proto
[ 1%] Running cpp protocol buffer compiler on sentencepiece.proto
[ 2%] Running cpp protocol buffer compiler on sentencepiece_model.proto
View train-15AUG19.log
[2019-08-15 08:31:02] [marian] Marian v1.7.8 c65c26d6 2019-08-11 18:27:00 +0100
[2019-08-15 08:31:02] [marian] Running on walle3 as process 24138 with command line:
[2019-08-15 08:31:02] [marian] /home/xyz/marian-dev/build/marian --model /disk2/models/xx-yy-r0/model.npz --type transformer --train-sets /disk2/data/xx-yy/ /disk2/data/xx-yy/train.en --vocabs /disk2/models/xx-yy-r0/vocab.src.spm /disk2/models/xx-yy-r0/vocab.trg.spm --dim-vocabs 32000 32000 --mini-batch-fit --mini-batch 1000 --maxi-batch 1000 --valid-freq 10000 --save-freq 10000 --disp-freq 500 --valid-metrics ce-mean-words perplexity bleu-detok --valid-sets /disk2/data/xx-yy/ /disk2/data/xx-yy/valid.en --quiet-translation --beam-size 6 --normalize=0.6 --valid-mini-batch 16 --early-stopping 5 --cost-type=ce-mean-words --log /disk2/models/xx-yy-r0/train.log --valid-log /disk2/models/xx-yy-r0/valid.log --enc-depth 6 --dec-depth 6 --transformer-preprocess n --transformer-postprocess da --tied-embeddings-all --dim-emb 1024 --transforme
View big.txt
This file has been truncated, but you can view the full file.
The Project Gutenberg EBook of The Adventures of Sherlock Holmes
by Sir Arthur Conan Doyle
(#15 in our series by Sir Arthur Conan Doyle)
Copyright laws are changing all over the world. Be sure to check the
copyright laws for your country before downloading or redistributing
this or any other Project Gutenberg eBook.
This header should be the first thing seen when viewing this Project