LiutongZhou

Memory Efficient Training

Optimizers

Short Name	Full Name	URL
CAME	Confidence-guided Adaptive Memory Efficient Optimization (ACL23 outstanding paper award)	https://github.com/huawei-noah/Pretrained-Language-Model/blob/master/CAME/came.py
LOMO	LOw-Memory Optimization	https://github.com/OpenLMLab/LOMO/blob/main/src/lomo.py

Low Rank Decomposition

LORA (Low Rank Adaption): https://github.com/microsoft/LoRA or https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora.py

LiutongZhou

"""Data Strutures that extend OrderedDict"""
from collections import Counter, OrderedDict
from typing import Any, Hashable, Optional, Tuple, List

from hypothesis import given, strategies as st

__all__ = ["OrderedDefaultDict", "MinMaxCounter"]


class OrderedDefaultDict(OrderedDict):

LiutongZhou

Docker Tips

Move default docker storage to another location

nano /etc/docker/daemon.json

## add this config
{
"data-root": "/newlocation"
}

LiutongZhou

How to document a project

How to update the docs and publish to the Home Page?

Prerequisites

One-time installation of dependencies

python3 -m pip install -U jupyter-book ghp-import

LiutongZhou

Large-Scale Distributed Data and Model Parallel Training

Data Streaming

FastFile Mode

sagemaker.inputs.TrainingInput(S3_INPUT_FOLDER, input_mode='FastFile') 

LiutongZhou

"""BitMask"""

class BitMask:
    __slots__ = ("size", "mask")
    
    def __init__(self, size: int = 16):
        """Create a bit mask to store (size) 0/1 status"""
        self.size = size
        self.mask = 1 << size
        

LiutongZhou

"""MinHeap and MaxHeap (Optimized Implementation)"""
from abc import ABC, abstractmethod
from collections import Counter, UserList
from functools import singledispatchmethod
from heapq import (
    _heapify_max,
    _heappop_max,
    _heapreplace_max,
    _siftdown,
    _siftdown_max,

LiutongZhou

"""UnionFind (Disjoint Sets)"""

from typing import Optional, Iterable, Hashable, Any


class UnionFind:
    def __init__(
        self, initial_disjoint_items: Optional[Iterable[Hashable]] = None
    ):
        """Initialize a UnionFind of disjoint sets"""

LiutongZhou

""" Time-and-Space-Efficient Sampling Methods"""
from typing import Iterable
from random import random, randint, seed
from math import exp, log, floor
from itertools import islice

__author__ = "Liutong Zhou"


def take(iterable, n: int) -> list:

LiutongZhou

Setup Cloud9 on EC2 and remote Jupyter

Create an EC2 instance on AWS

• Launch: t3.2xlarge ($0.33/h) / m5d.4xlarge ($0.904) / g4dn.4xlarge ($1.2/h) / p3.2xlarge ($3.02/h)
• Image: Deep Learning AMI (Ubuntu 22.04)
• Configure Security Group:
  • open custom TCP and port 9999
  • open HTTPS, HTTP to anywhere
• Attach an Elastic IP to the instance

ssh into EC2 from MobaXterm and run