jon-tow

command_file: null 
commands: null 
compute_environment: LOCAL_MACHINE 
deepspeed_config: 
  gradient_accumulation_steps: 1 
  gradient_clipping: 1.0 
  offload_optimizer_device: none 
  offload_param_device: none 
  zero3_init_flag: true 
  zero_stage: 2 

jon-tow

Command:

accelerate launch --num_processes 7 --config_file ../../configs/accelerate/zero2-bf16.yaml ppo_hh.py

Traceback:

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ trlx/examples/hh/ppo_hh.py:129 in <module>                                        │
│                                                                                                  │

jon-tow

import argparse
import yaml
import trlx
from trlx.data.configs import TRLConfig


parser = argparse.ArgumentParser()
parser.add_argument("--config", type=str, default="configs/ilql_config.yml")
parser.add_argument("--checkpoint", type=str, default="ckpts")
args = parser.parse_args()

jon-tow

"""This scripts shows how to convert a `accelerate` checkpoint to a `hf` model
that can be used for inference.

NOTE: You may need to call this script with `accelerate launch` (or the proper distributed launcher)
to run it on multiple GPUs and load the model properly (e.g. when the model was trained with `deepspeed`). 
"""
import argparse
import yaml
import trlx
from trlx.data.configs import TRLConfig

jon-tow

# Generates positive movie reviews by tuning a pretrained model on IMDB dataset
# with a sentiment reward function
import os
from typing import List
import torch
import yaml
from datasets import load_dataset
from transformers import pipeline
import trlx
from trlx.data.configs import TRLConfig

jon-tow

0. Create your python env with conda or micromamba and intall mpi / mpi4py (mpi is not available on the CW cluster)

micromamba install -c conda-forge mpi
micromamba install -c conda-forge mpi4py

1. Copy both slurm_<_>.sh scripts into your trlx clone.
2. Update your accelerate config with the provided yaml.
3. sbatch slurm_ddp.sh

jon-tow

This gist contains config setups for PPO and ILQL methods. NOTE: Hyper-parameters were optimized for opt/gpt-j models at the 6B parameter scale.

jon-tow

#!/bin/bash
# Decompress the raw Pile from `the-eye`.
#
# Takes an input dir of the form:
#
# train/
#   {00-29}.jsonl.zst
# val.jsonl.zst
# test.jsonl.zst
# SHA256SUMS.txt

jon-tow

# Random and Human Baselines for common NLP tasks/benchmarks.

# TDOO: Maybe add metadata containing citations, repos, etc.

arc_challenge:
    # https://leaderboard.allenai.org/arc/submissions/public
    # NOTE: For for random scores the .02% score difference from an ideal 25% is due
    # to some questions having fewer or more than four choices.
    acc:
        random: 0.2502