Skip to content

Instantly share code, notes, and snippets.

@meichen1

meichen1/cc.md

Last active July 3, 2025 22:49
Show Gist options
  • Save meichen1/c003c60a23baadccc3962c482818ecc8 to your computer and use it in GitHub Desktop.
Save meichen1/c003c60a23baadccc3962c482818ecc8 to your computer and use it in GitHub Desktop.
Download ZIP
ComputeCanada
Raw
cc.md

$HOME/projects/def-lkong/ciel5577 pip install --no-index --upgrade pip pip install setuptools --upgrade pip install --no-index -r requirements.txt

pip install -r requirements.txt

Compute Canada username: ciel5577 pass: Lmcsd987123$$

ssh meichen@129.128.207.253 conda activate BIAS ctrl c >>>> y >>>>>exit

ls -l

ssh -L 9988:localhost:8889 meichen@129.128.207.253 ssh -L 9988:127.0.0.1:8889 meichen@129.128.207.253

exit

  • forward multiple ports, use -L multiple times

https://wgpages.netlify.app/bashfeatures/

wandb hpc offline:

wandb/wandb#2810 (comment) https://docs.wandb.ai/support/run_sweeps_slurm/

computecanada tutorial

https://prashp.gitlab.io/post/compute-canada-tut/

https://docs.alliancecan.ca/wiki/Visual_Studio_Code

rsync -avzP $SCRATCH/ma-pvi-nia/ ciel5577@narval.alliancecan.ca:~

niagara & mist (gpu)

https://docs.scinet.utoronto.ca/index.php/Mist#Testing_and_debugging

ssh -Y mist-login01 (through nia) salloc --nodes=1 --gpus-per-node=1 --time=2:00:00

niagara

salloc -p debug --nodes 1 --time=1:00:00

Narval

srun --nodes=1 tasks=2 --mem=8G --pty bash

salloc --time=3:0:0 --gres=gpu:a100_2g.10gb:1 salloc --time=0-1:00 --mem-per-cpu=4G --ntasks=4

'narval': source ~/cyberDL/bin/activate salloc --gpus-per-node=1 --time=1:00:00 --mem=16G --cpus-per-task=6 nvidia-smi

Apptainer

apptainer(can build from docker)/docker/podman(similar to docker) daemons are servers

When building images on our clusters unset the APPTAINER_BIND environment variable before building the image: • module load apptainer • unset APPTAINER_BIND • apptainer build apptainer build rocky8.sif docker://rockylinux:8 • ... • and run module load apptainer to set unset APPTAINER_BIND again.

There are a number of ways to use Apptainer:

  1. Run a single command which executes and then stops running. • apptainer run --nv rapids.sif my_script.sh
  2. Run many commands in an interactive session. • apptainer shell --nv rapids.sif
  3. Run a container instance to run daemons and have backgrounded processes. • apptainer instance start image.sif a_name • apptainer run image.sif instance://a_name ps -eaf • apptainer run image.sif instance://a_name nohup find / -type d ›dump. txt • apptainer run image.sif instance://a_name ps -eaf • apptainer instance stop image.sif a_name

When using the run, shell instance, and exec Apptainer commands: -C, -c, or -e options. • -C: hides filesystems, PID, IPC, and environment • Prefer this option. • Requires explicitly bind mounting all paths needed and explicitly passing desired environment variables to the container (or they cannot be seen). • -c: uses minimal /dev, shared-with-host directories will appear empty, e.g., / tmp, unless explicitly bind mounted • -e: clean environment before running container NOTE: You don't want things from your shell's environment to be present when using your container: many things are not inside your container and therefore cannot be accessed.

narval-apptainer

module load scipy-stack/2024b apptainer-suid apptainer

unset APPTAINER_BIND export APPTAINER_CACHEDIR=~/scratch/a_dir

apptainer build nvidia-pytorch.sif docker://nvcr.io/nvidia/pytorch:24.06-py3

vim shortcut:

Insert Mode:

i: Insert before the cursor. a: Insert after the cursor. o: Open a new line below the current line and enter insert mode. Navigation:

h: Move cursor left. j: Move cursor down. k: Move cursor up. l: Move cursor right. w: Move to the beginning of the next word. b: Move to the beginning of the previous word. gg: Go to the beginning of the file. G: Go to the end of the file. Editing:

dd: Delete the current line. yy: Yank (copy) the current line. p: Paste after the cursor. u: Undo the last change. Ctrl + r: Redo the last undone change. Search:

/pattern: Search for pattern in the file. n: Move to the next occurrence of the search pattern. N: Move to the previous occurrence of the search pattern. Save and Quit:

:w: Save the file. :q: Quit Vim. :wq: Save and quit Vim. :q!: Quit without saving. Visual Mode:

v: Enter visual mode to select text. V: Enter visual line mode to select whole lines. Ctrl + v: Enter visual block mode to select a block of text. Replace:

:%s/old/new/g: Replace all occurrences of old with new in the file.

Indentation:

>>: Indent the current line.
<<: Unindent the current line.

Buffers:

:bn: Go to the next buffer. :bp: Go to the previous buffer. :bd: Delete the current buffer. Splits:

:split: Split the window horizontally. :vsplit: Split the window vertically. Ctrl + w, w: Switch between splits.

References:

'niagara':

#!/bin/bash
#SBATCH --nodes=1
#SBATCH --gpus-per-node=1
#SBATCH --time=1:00:0
#SBATCH --account=soscip-<SOSCIP_PROJECT_ID> #For SOSCIP projects only

module load anaconda3
source activate conda_env
python code.py ...

ssh -Y mist-login01

debugjob ## 120min 1gpu salloc --nodes=1 --time=3:00:00 salloc --nodes=1 --gpus-per-node=1 --time=1:00:00 ## only for Mist cluster

module load NiaEnv/2022a gcc/11.3.0 openssl/1.1.1k python/3.11.5 source ~/ecg_venv/bin/activate

export PATH=$PATH:$(pwd)/.local/bin echo $PATH

lmql serve-model ProbeMedicalYonseiMAILab/medllama3-v20 --trust_remote_code True

export PATH=$PATH:$(pwd)/.local/bin echo $PATH

Mist gpu

  • can only write file on login node

nvidia-smi

module load anaconda3 cuda source ~/ecg_venv/bin/activate

ssh -i ~/sshkey/mac230806 ciel5577@narval.alliancecan.ca ssh -i ~/sshkey/mac230806 ciel5577@beluga.alliancecan.ca ssh -i ~/sshkey/mac230806 ciel5577@niagara.alliancecan.ca

salloc --time=2:0:0 --mem=8G

sshfs cielliu@Ciels-MacBook-Air:Google Drive/My Drive/Case_func/unos_data /scratch/mnt -o allow_other,default_permissions

scp -i ~/sshkey/mac230806 -r case-nia ciel5577@niagara.alliancecan.ca:$SCRATCH/case-real

#!/bin/bash

#SBATCH --cpus-per-task=16 # number of processes #SBATCH --mem-per-cpu=4G #SBATCH --time=6:59:59 # time (HH:MM:SS)

#SBATCH --job-name="FNMEM" #SBATCH --output=FNMEM_%j.txt #SBATCH --mail-user=meichen1@ualberta.ca #SBATCH --mail-type=ALL

#SBATCH --array=1

module load StdEnv/2020 gcc/9.3.0 r/4.1.2

cd $SCRATCH/FNMEM/1

export PATH=“/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/usr/local/share/dotnet:/Users/cielliu/.dotnet/tools:/opt/homebrew/bin:/opt/homebrew/bin”

module load CCEnv
module load StdEnv/2023 scipy-stack/2023b
module load cudacore/.12.2.2  cudnn/8.9.5.29
module load StdEnv/2023 scipy-stack/2023b  cudacore/.12.2.2  cudnn/8.9.5.29

salloc --gpus-per-node=1 --time=3:00:00 --mem=16G --cpus-per-task=6

salloc --gpus-per-node=1 --time=1:00:00 srun $VIRTUAL_ENV/bin/notebook.sh

module load python/3.10.2 scipy-stack/2023b

source cyberDL/bin/activate

salloc --time=1:0:0 --ntasks=1 --cpus-per-task=4 --mem-per-cpu=2048M srun $VIRTUAL_ENV/bin/notebook.sh

windows:

ssh -L 8888:ng11105.narval.calcul.qubec:8888 ciel5577@narval.alliancecan.ca -i .ssh/mac230806

whoami

pwd

ls -a (flags: all) wget https://git.io/HSS-SHELL.zip

man ls: q to quit

exit

cat

head -n 2 file.csv

tail -n +1 (offset include the first line) EPLABB.csv

tail -n +1 EPLABB.csv | head -n 1

tail -n +2 EPLABB.csv | cut -d , -f (field) 4

tail -n +2 EPLABB.csv | cut -d / -f 1 | cut -d , -f 4- | sed ‘search/search for/replace with/’

tail -n +2 EPLABB.csv | cut -d / -f 1 | cut -d , -f 4- | sed 's/"//' | sort | uniq -c |sort -nr| head -n 10

nano top10.sh

bash top10.sh

CTRL C: kill current process

zip -r

When using the Windows Subsystem for Linux (WSL), there are several useful commands that can help you navigate and work within the Linux environment. Here are 10 commonly used commands:

  1. ls: cd: pwd: mkdir:
  2. rm: This command is used to remove files and directories.
  3. cp: It is used to copy files and directories.
  4. mv: This command is used to move or rename files and directories.
  5. cat: It is used to display the contents of a file.
  6. grep: This command is used for searching patterns within files.
  7. sudo: It is used to execute commands with administrative privileges.

[1]: DigitalOcean - Top 50+ Linux Commands You MUST Know [2]: Integrate Linux Commands into Windows with PowerShell and the Windows Subsystem for Linux [3]: Most Useful Linux Commands You Can Run in Windows 10 [4]: GeeksforGeeks - Linux vs Windows Commands

manual

exit connection

tail -n +2 EPLABB.csv | cut -d / -f 1 | cut -d , -f 4- | sed ‘s/”//’

sed: string editor

nano ; vim; emacs

$ variable

tail -n +2 $1 | cut -d / -f 1 | cut -d , -f 4- | sed 's/"//' | sort | uniq -c >

pipeline |

[root@ip-172-31-21-209 ec2-user]# v2ray url

---------- V2Ray vmess URL / V2RayNG v0.4.1+ / V2RayN v2.1+ / 仅适合部分客户端 -------------

vmess://ewoidiI6ICIyIiwKInBzIjogIjIzM3YyLmNvbV8zNS4xODMuMTExLjIwOCIsCiJhZGQiOiAiMzUuMTgzLjExMS4yMDgiLAoicG9ydCI6ICIyNzQ3OSIsCiJpZCI6ICI3OGIyNDVkNi0zMjI0LTQxYjMtYWYyOC1kYmM0NWZjMWQyZTQiLAoiYWlkIjogIjAiLAoibmV0IjogInRjcCIsCiJ0eXBlIjogIm5vbmUiLAoiaG9zdCI6ICIiLAoicGF0aCI6ICIiLAoidGxzIjogIiIKfQo=

Raw
tmux.md

tutorial:

https://docs-dev.alliancecan.ca/wiki/Tmux

https://hyperpolyglot.org/multiplexers

Youtube: Tmux has forever changed the way I write code.

https://store.dreamsofcode.io/

  • sessions: one active window*, many windows
  • windows
  • panes: splits in the window

tmux list-keys | sed -n '10,20p'

prefix key: ctrl b navigate next or previous: ctrl b + n/p close: ctrl b + &

Here are the steps to use tmux and some of the most useful commands:

Starting tmux

  • Start a new tmux session:

tmux

  • Start a new tmux session with a name:

tmux new -s session_name

  • Detach from the current session:

Press Ctrl + b, then d.

  • List all sessions: tmux ls

  • Attach to a session: tmux attach -t session_name

Kill a session: tmux kill-session -t session_name

  • Create a new window:

Press Ctrl + b, then c.

  • Switch between windows:

Press Ctrl + b, then the window number (e.g., 0, 1, 2, etc.).

  • Rename the current window:

Press Ctrl + b, then , (comma), type the new name, and press Enter.

  • Split the window horizontally:

Press Ctrl + b, then " (double quote).

  • Split the window vertically:

Press Ctrl + b, then %.

  • Navigate between panes:

Press Ctrl + b, then use the arrow keys.

To quit a tmux session, you can use the following commands:

Exit the Shell:

Simply type exit or press Ctrl + d in the shell of each window/pane. This will close the shell and, if it's the last window, it will terminate the tmux session.

Kill the Session:

You can kill the entire tmux session from within tmux by typing: tmux kill-session Or, from outside tmux, you can specify the session name: tmux kill-session -t session_name

Detach and Kill:

Detach from the session using Ctrl + b, then d, and then kill the session from another terminal: tmux kill-session -t session_name

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment