CiaoHe

import zipfile
path_to_zip_file = '...'
directory_to_extract_to = './'
with zipfile.ZipFile(path_to_zip_file, 'r') as zip_ref:
    zip_ref.extractall(directory_to_extract_to)

CiaoHe

1. create .env under src(source code) folder
2. Edit .env as follows: PYTHONPATH=path_a:path_b , (path_a better be abosulte path)

refer to https://stackoverflow.com/a/41668211/15714186

CiaoHe

1. submit sbatch jupyter.sh

example of jupyter.sh

#!/bin/bash

#SBATCH --nodes=1
#SBATCH --job-name=jupyter
#SBATCH --gres=gpu:1
#SBATCH --time=2-00:00:00

CiaoHe

Multi-node-training on slurm with PyTorch

What's this?

• A simple note for how to start multi-node-training on slurm scheduler with PyTorch.
• Useful especially when scheduler is too busy that you cannot get multiple GPUs allocated, or you need more than 4 GPUs for a single job.
• Requirement: Have to use PyTorch DistributedDataParallel(DDP) for this purpose.
• Warning: might need to re-factor your own code.
• Warning: might be secretly condemned by your colleagues because using too many GPUs.

CiaoHe

import torch as th
import numpy as np
class AutoClipGradient(object):
    def __init__(self, max_history=1000, clip_percentile=99.9, max_grad_norm=0.5):
        self.max_history = max_history
        self.clip_percentile = clip_percentile
        self.history = []
        self.max_grad_norm = max_grad_norm
        
    def _compute_grad_norms(self, params, grad_scale=1.0):

CiaoHe

In [1]: from huggingface_hub import snapshot_download

In [2]: snapshot_download(repo_id="bert-base-chinese", ignore_regex=["*.h5", "*.ot", "*.msgpack"], cache_dir='MODEL WHERE STORE')

ignore_regex：忽略其他非必要文件

CiaoHe

n02119789 1 kit_fox
n02100735 2 English_setter
n02110185 3 Siberian_husky
n02096294 4 Australian_terrier
n02102040 5 English_springer
n02066245 6 grey_whale
n02509815 7 lesser_panda
n02124075 8 Egyptian_cat
n02417914 9 ibex
n02123394 10 Persian_cat

CiaoHe

"""
Simply use peek_model_from_ckpt(CKPT_PATH)
"""
import torch
from typing import Dict
class TrieNode:
    def __init__(self):
        self.is_end = False
        self.children = {}

CiaoHe

import torch
import contextlib
print('Pytorch version\t:', torch.__version__)
print('CUDA version\t:', torch.version.cuda)

for i in range(torch.cuda.device_count()):
    print(f'GPU{i}\t\t:',torch.cuda.get_device_name(i))

import os
import re

CiaoHe

tokenization

BPE Encoding (Byte-Pair Encoding)

• A clear demonstration : Byte-Pair Encoding: Subword-based tokenization algorithm