REALM Manual for MIT SuperCloud

realm_manual_supercloud.md

REALM Manual for MIT SuperCloud

Reference: https://supercloud.mit.edu/

• SuperCloud: Research for MIT Lincoln Laboratory (224 GPU nodes with 2 Nvidia Volta V100 each, and 480 CPU nodes)
• Account: Fill out the form and complete HPC course for full access
• Help: go https://supercloud.mit.edu/getting-help or contact supercloud@mit.edu
• Office hours: Fridays 2-3pm EST (https://mit.zoom.us/j/173993735)
• Monthly downtimes: Third Thursday of each month

Directories

Reference: https://supercloud.mit.edu/best-practices-and-performance-tips

• User directory: /home/gridsan/USERNAME
• Aim for less than ~1000 files per directory

Environment Setup

Reference: https://supercloud.mit.edu/software-and-package-management

Submit an Interactive Job in MIT SuperCloud

Reference: https://supercloud.mit.edu/submitting-jobs#interactive

0. Log into the supercloud login node: ssh USERNAME@txe1-login.mit.edu
1. Submit an interactive job (with one gpu) using: LLsub -i -s 20 -g volta:1
2. Once available, you will automatically login to that node. If you exit then, the node will be terminated.

Submit a Serial Job in MIT SuperCloud

Reference: https://supercloud.mit.edu/submitting-jobs#serial

0. Log into the supercloud login node: ssh USERNAME@txe1-login.mit.edu
1. Write the following to a file called myScript.sh

#!/bin/bash
#SBATCH -J USERNAME
#SBATCH -o %j.stdout
#SBATCH -e %j.stderr
#SBATCH -c 20
#SBATCH --gres=gpu:volta:1
#SBATCH --time=24:00:00

# Write your commands here
# Write this line to use conda
source /state/partition1/llgrid/pkg/anaconda/anaconda3-2022a/etc/profile.d/conda.sh
conda activate nn_pde_new
cd /home/gridsan/ymeng/mit/hybrid_clf/DeepRL_Algorithms
python global_patch.py --algor sac --exp_name bp --gpus 0 --env_id BpEnv-v0 --random_seed 20221007

2. Then submit the job: LLsub myScript.sh (or: sbatch myScript.sh)
3. Check the job status using: LLstat
4. If the job is assigned with "NODELIST=d-14-7-2", you can login the compute node using: ssh d-14-7-2
5. (In the login node) You can stop a job manually by: LLkill JOBID
6. You can view the stdout (or stderr) files: cat JOBID.stdout (or: cat JOBID.stderr)
7. You can also append the following to your .bashrc file and source ~/.bashrc, and then you can run like: run_job python global_patch.py --algor sac --exp_name bp --gpus 0 --env_id BpEnv-v0 --random_seed 20221007 and check stdout/stderr by cato JOBID (cate JOBID)

function run_job {
echo "#!/bin/bash
#SBATCH -J USERNAME
#SBATCH -o %j.stdout
#SBATCH -e %j.stderr
#SBATCH -c 20
#SBATCH --gres=gpu:volta:1
#SBATCH --time=24:00:00

# write your commands here
source /state/partition1/llgrid/pkg/anaconda/anaconda3-2022a/etc/profile.d/conda.sh
conda activate nn_pde_new
cd /home/gridsan/ymeng/mit/hybrid_clf/DeepRL_Algorithms
$@
" > ~/tmp.slurm
mkdir -p ~/.lsf/
cd ~/.lsf/
sbatch ~/tmp.slurm
cd -
}

function cato {
cat ~/.lsf/$1.stdout
}

function cate {
cat ~/.lsf/$1.stderr
}