wolfecameron

ResNet(
  (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
  (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (relu): ReLU(inplace=True)
  (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
  (layer1): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)

wolfecameron

Learning Rate Scheduling:
- Overview of useful LR schedules: https://cameronrwolfe.substack.com/p/the-best-learning-rate-schedules
- REX Paper: https://arxiv.org/abs/2107.04197

Precision Scheduling:
- Overview of Low Precision Training Techniques: https://cameronrwolfe.substack.com/p/quantized-training-with-deep-networks-82ea7f516dc6
- CPT Paper: https://arxiv.org/abs/2101.09868

Video Batch Size Scheduling:
- Overview of Video Deep Learning (Part One): https://cameronrwolfe.substack.com/p/deep-learning-on-video-part-one-the-early-days-8a3632ed47d4

wolfecameron

import torch

class FFNN(torch.nn.Module):
  def __init__(self, input_size, hidden_size, output_size, num_layers):
    super().__init__()
    self.input_size = input_size
    self.hidden_size = hidden_size
    self.output_size = output_size
    self.num_layers = num_layers

wolfecameron

import matplotlib.pyplot as plt

def calc_demon_decay(total_iter, curr_iter, min_val, max_val):
    z = float(total_iter - curr_iter) / total_iter
    return min_val + float(max_val - min_val) * (z / (1 - 0.9 + 0.9*z))
  
train_iters = 100
max_mom = 0.9
min_mom = 0.0
plt.title('Demon Decay')