LiutongZhou

Docker Tips

Move default docker storage to another location

nano /etc/docker/daemon.json

## add this config
{
"data-root": "/newlocation"
}

LiutongZhou

Setup Cloud9 on EC2 and remote Jupyter

Create an EC2 instance on AWS

• Launch: t3.2xlarge ($0.33/h) / m5d.4xlarge ($0.904) / g4dn.4xlarge ($1.2/h) / p3.2xlarge ($3.02/h)
• Image: Deep Learning AMI (Ubuntu 22.04)
• Configure Security Group:
  • open custom TCP and port 9999
  • open HTTPS, HTTP to anywhere
• Attach an Elastic IP to the instance

ssh into EC2 from MobaXterm and run

LiutongZhou

1. Squash commits into a single commit and rebase feature branch onto upstream/develop

   git fetch upstream && git rebase -i $(git merge-base feature_name upstream/develop)

2. Cleanup git repository aggressively

   use bfg https://rtyley.github.io/bfg-repo-cleaner/

   java -jar bfg.jar --delete-files your_unwanted_files

LiutongZhou

import functools
from typing import Any, Mapping


def singleton(cls):
    '''a class decorator that wraps class definition so that only one class instance can exist'''
    existing_instances = dict() 
    @functools.wraps(cls)
    def singleton_class(*args, **kwargs): 
        if cls not in existing_instances:

LiutongZhou

Memory Efficient Training

Optimizers

Short Name	Full Name	URL
CAME	Confidence-guided Adaptive Memory Efficient Optimization (ACL23 outstanding paper award)	https://github.com/huawei-noah/Pretrained-Language-Model/blob/master/CAME/came.py
LOMO	LOw-Memory Optimization	https://github.com/OpenLMLab/LOMO/blob/main/src/lomo.py

Low Rank Decomposition

LORA (Low Rank Adaption): https://github.com/microsoft/LoRA or https://github.com/huggingface/peft/blob/main/src/peft/tuners/lora.py

LiutongZhou

"""Data Strutures that extend OrderedDict"""
from collections import Counter, OrderedDict
from typing import Any, Hashable, Optional, Tuple, List

from hypothesis import given, strategies as st

__all__ = ["OrderedDefaultDict", "MinMaxCounter"]


class OrderedDefaultDict(OrderedDict):

LiutongZhou

Large-Scale Distributed Data and Model Parallel Training

Data Streaming

FastFile Mode

sagemaker.inputs.TrainingInput(S3_INPUT_FOLDER, input_mode='FastFile') 

LiutongZhou

How to document a project

How to update the docs and publish to the Home Page?

Prerequisites

One-time installation of dependencies

python3 -m pip install -U jupyter-book ghp-import

LiutongZhou

"""MinHeap and MaxHeap (Optimized Implementation)"""
from abc import ABC, abstractmethod
from collections import Counter, UserList
from functools import singledispatchmethod
from heapq import (
    _heapify_max,
    _heappop_max,
    _heapreplace_max,
    _siftdown,
    _siftdown_max,

LiutongZhou

"""UnionFind (Disjoint Sets)"""

from typing import Optional, Iterable, Hashable, Any


class UnionFind:
    def __init__(
        self, initial_disjoint_items: Optional[Iterable[Hashable]] = None
    ):
        """Initialize a UnionFind of disjoint sets"""