Skip to content

Instantly share code, notes, and snippets.

@jimmyahacker
Last active Jun 29, 2021
Embed
What would you like to do?
Configure StarCraft II environment for DeepMind challenge on NYU Prince Cluster

Prerequisites

  1. Load singularity module:
module load singularity/2.4.4
  1. Download and unzip StarCraft II for Linux: here I take StarCraft II 3.16 for example, you could replace the link and zip file name according to SC2 Official Linux Downloads

    1. Download zip
    wget http://blzdistsc2-a.akamaihd.net/Linux/SC2.3.16.1.zip
    1. Unzip
    unzip SC2.3.16.1.zip
    1. Prompt the password iagreetotheeula
    2. Move StarCraft II files
    mv ./StarCraftII/ $SCRATCH/StarCraftII
  2. Download and unzip StarCraft II mini maps

    1. Download mini game maps
    wget https://github.com/deepmind/pysc2/releases/download/v1.2/mini_games.zip
    1. Unzip
    unzip mini_games.zip
    1. Move mini game maps
    mv ./mini_games $SCRATCH/StarCraftII/Maps/
  3. Start the Ubuntu container

singularity shell --nv /beegfs/work/public/singularity/ubuntu-17.10.img
  1. Create and activate virtual environment

    1. Create virtual environment named py3tf
    virtualenv -p python3 $SCRATCH/py3tf
    1. Activate virtual environment py3tf
    source $SCRATCH/py3tf/bin/activate
  2. Add python dependencies in virtual environment

    1. Install tensorflow-gpu
    pip install tensorflow-gpu
    1. Install pysc2
    pip install pysc2

    3. (Optional) Install other dependencies defined in requirements.txt

    pip install -r requirements.txt
  3. Deactivate virtual environment, exit the container and return to login node

    1. Deactivate virtual environment
    deactivate
    1. Exit container
    exit

Batch mode(Preferred)

  1. Download the model you wanna train, I take this one for example
    1. Download
    git clone https://github.com/simonmeister/pysc2-rl-agents
    1. Move
    mv pysc2-rl-agents $SCRATCH/
  2. Download slurm scripts
    1. Download and move control code(you should modify the notification email address): modify code as you see fit(references1,2)
    wget https://gist.github.com/jimmyahacker/1bb8b99b58a9d2af1ca8582f9f8d14dc/raw/d5e5d2a71ebfaa3a18e938c6e58baf127ad57bca/job.s;mv job.s $SCRATCH/
    1. Download and move in-container code
    wget https://gist.github.com/jimmyahacker/1bb8b99b58a9d2af1ca8582f9f8d14dc/raw/d5e5d2a71ebfaa3a18e938c6e58baf127ad57bca/in_container.sh; mv in_container.sh $SCRATCH/
  3. Run in batch mode
cd $SCRATCH;sbatch job.s
  1. See job status
squeue -u $USER

Interactive mode(ONLY FOR DEBUGGING USE)

  1. Download allocate_gpus.sh written below

    1. Download
    wget https://gist.github.com/jimmyahacker/1bb8b99b58a9d2af1ca8582f9f8d14dc/raw/d11fa4148bb506103b08a2b44fe06247fb1f472b/allocate_gpus.sh
    1. Move to home directory
    mv ./allocate_gpus.sh $SCRATCH/allocate_gpus.sh
  2. Load packages and allocate GPUs: allocate 1 GPU(you could replace 1 with any number of GPUs you wanna allocate)

source $SCRATCH/allocate_gpus.sh 1
  1. Start the Ubuntu container
singularity shell --nv /beegfs/work/public/singularity/ubuntu-17.10.img
  1. Run virtual environment py3tf
source $SCRATCH/py3tf/bin/activate
  1. Set StarCraftII environment variable
export SC2PATH="$SCRATCH/StarCraftII"
#!/bin/bash
module purge
module load cuda/9.0.176 cudnn/9.0v7.0.5 singularity/2.4.4
export MY_LD_LIBRARY_PATH=$LD_LIBRARY_PATH
srun --gres=gpu:$1 --pty /bin/bash
#!/bin/bash
cd $SCRATCH
source py3tf/bin/activate
cd pysc2-rl-agents
python run.py my_experiment --map MoveToBeacon --envs 1 --iters 1
#!/bin/bash
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --gres=gpu:1
#SBATCH --time=10:00:00
#SBATCH --mem=3GB
#SBATCH --job-name=GPU-SC2-Demo
#SBATCH --mail-type=ALL
#SBATCH --mail-user=@nyu.edu
#SBATCH --output=slurm_%j.out
module purge
module load cuda/9.0.176 cudnn/9.0v7.0.5 singularity/2.4.4
export MY_LD_LIBRARY_PATH=$LD_LIBRARY_PATH
export SC2PATH=$SCRATCH/StarCraftII
singularity exec --nv /beegfs/work/public/singularity/ubuntu-17.10.img bash in_container.sh
@Willian-Zhang

This comment has been minimized.

Copy link

@Willian-Zhang Willian-Zhang commented Apr 9, 2018

Requirements

PySC2 Old version

used in here

absl-py==0.1.13
astor==0.6.2
bleach==1.5.0
enum34==1.1.6
future==0.16.0
futures==3.1.1
gast==0.2.0
google-apputils==0.4.2
grpcio==1.10.1
html5lib==0.9999999
Markdown==2.6.11
mock==2.0.0
numpy==1.14.2
pbr==4.0.1
portpicker==1.2.0
protobuf==3.5.2.post1
pygame==1.9.3
PySC2==1.1
python-dateutil==2.7.2
python-gflags==3.1.2
pytz==2018.3
s2clientprotocol==1.1
six==1.11.0
tensorboard==1.7.0
tensorflow-gpu==1.7.0
termcolor==1.1.0
websocket-client==0.47.0
Werkzeug==0.14.1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment