A living document with links and descriptions for the RC ARENA study group starting with the Summer 1 batch

AREADME.md

Hype music | Office hours schedule

• Week 1: Prereqs (<6hr/day):
  • Mon: Linear algebra prereqs
  • Tues: Tensor manipulation
  • Wed: Einops
    • Einops introduction
    • ARENA prereqs
  • Thurs: Neural networks from scratch
  • Fri: CNNs
• Week 2: (6+ hr/day): 
  • ONE FLEX DAY, USE WHERE NEEDED
  • Mon:GANs and VAEs
  • Tues: Impossible Day
  • Optimizers
  • Backpropogation
• Week 3 (6+ hr/day):
  • Mon-Tues: Transformers from scratch
  • Wed-Thurs: Transformer interpretability
  • Thurs-Fri: Balanced bracket classifier

tech_setup.md

ARENA Tech tips

Welcome to ARENA! I want to make the course accessible to everyone regardless of their Python and hosting familiarity. As such, I recommend one of three options for running the exercises:

1. In Google Colab
2. Locally in a Jupyter Notebook
3. Locally in a Python script with VSCode's Jupyter integration

Google Colab

Pros

• The simplest setup
• GPU support at the click of a button

Cons

• Sessions get interrupted
• Interface can be challenging for new users
• Dependencies re-installed every time the session restarts
• Pro costs $10 USD per month
• Poor git integration (have to download code once you're done for the day and then commit it)

Setup

• (If necessary) Log into Google in the Chrome browser (see gotchas)
• Navigate to the Streamlit app for the appropriate chapter. See the schedule above to find it
• Click the “exercises” link at the top of the page. You are now in a Colab notebook, but you don’t want to run this one
• Click the “File” navbar and “Save a copy in Drive”. This copies the notebook to your Google Drive and it opens that copy in a new tab
• This new notebook will be your new runtime

Gotchas

• Colab is only tested in Chrome and has strange edge cases with Safari. It’s best to use Chrome with it
• The free tier of Colab interrupts your kernel on laptop sleep. This is annoying but manageable for a while, but by week two, this becomes enough of a hassle that everyone using Colab paid for Pro ($10 USD per month)
• If you are using Pro, make sure to only run on a A4 GPU when you need one. In practice, you will likely only need a T4 GPU or CPU. Basically if you are leaving your tab for any time, double check you aren't using an A4. You can burn through your credits pretty quickly

Running ARENA exericses locally in Jupyter notebooks

Advantages

• Free!
• Fewer session interruptions
• Easier access to the code you write
• Dependencies only install once
• For those using Apple Silicon laptops, MPS support with PyTorch

Disadvantages

• More initial setup
• More setup ahead of running each section
• Poor git integration (the usual for Jupyter notebooks if you've used them before)
• Unlikely GPU support

Setup

Initial setup

• Clone the ARENA repo
• Set up a virtual environment using their requirements.txt. One way to do this using pyenv:
  • Install pyenv-virtualenv
  • From the ARENA repo, run pyenv virtualenv arena and pyenv activate arena
  • Install requirements with pip install -r requirements.txt
• Add a new Jupyter kernel to run this virtualenv, see this link if you have questions. For me, this looked like:
  • Make sure you're in the arena virtualenv (pyenv activate arena)
  • Run python3 -m ipykernel install --user --name arena

Chapter setup

• Navigate to the chapter of ARENA in streamlit
• Click the “exercises” link
• Click the “File” header, and navigate to “Download -> Download .ipynb”
• Move the downloaded ipynb to the corresponding exercise. For our example, we’d move the downloaded file from ~/Downloads/[0.5]… to …/ARENA_3.0/chapter0_fundamentals/exercises/part5_gans_and_vaes/exercises.ipynb
• Run juptyer notebook and select arena from the kernel list
• In the second code cell, we need to modify the PATH so Python can find the relative imports. That is, we want to change the following:

- # Get file paths to this set of exercises
- exercises_dir = Path("chapter0_fundamentals/exercises")
- section_dir = exercises_dir / "part5_gans_and_vaes"
+ import os
+ section_dir = Path(os.getcwd())
+ exercises_dir = Path(os.getcwd()).parent
+ sys.path.append(str(exercises_dir))

You can verify this is correct when this cell executes. If you are seeing ModuleNotFoundError: No module named 'plotly_utils', yo uhave not modified your PATH correctly.

Running ARENA exericses locally in VSCode + Jupyter

Advantages

• All of the advantages of running locally with Jupyter notebooks
• Great git integration. Each line can actually be version-controlled reliably
• Integration with all of your VSCode plugins and configuration (if applicable)

Disadvantages

• Cells have to be created manually
• Unlikely to have GPU support

Initial setup

• Same as notebook initial setup

Chapter setup

• Download the notebook as .py instead of .ipynb
• Move it to the same location (e.g. ARENA_3.0/chapter0_fundamentals/exercises/part5_gans_and_vaes/exercises.py)
• Open VSCode
• Open the Command Palette (on Mac, CMD + Shift + P)
• Search Jupyter: Select Kernel and choose arena
• You are now running a Python script block-by-block as if it is a Jupyter notebook. Create virtual Jupyter cells using # %% to demarcate sections. This is a little challnging getting started with, but it's pretty rewarding