Skip to content

Instantly share code, notes, and snippets.

@bryanlimy

bryanlimy/setup.md

Last active November 8, 2021 10:34
Show Gist options
  • Save bryanlimy/e02a2b71fea3b8d7aa82db1705ea218e to your computer and use it in GitHub Desktop.
Save bryanlimy/e02a2b71fea3b8d7aa82db1705ea218e to your computer and use it in GitHub Desktop.
Download ZIP
School of Informatics Setup
Raw
setup.md

Table of content

School of Informatics OpenVPN

It is much simpler and faster to access the various school's servers when connected to the School of Informatics OpenVPN.

  1. Download the OpenVPN configuration file from https://groups.inf.ed.ac.uk/inf-unit/OpenVPN/Config/Generic/
    • I recommend using Informatics-EdLAN-Forum.opvn
  2. Download and install OpenVPN client on your local system.
  3. Connect the Informatics OpenVPN with your student ID and password.
  4. Once connected to the Informatics VPN, you should be able to ping the school servers. For instance: 
    (base) ➜  ~ ping waterloo.inf.ed.ac.uk
PING waterloo.inf.ed.ac.uk (129.215.33.27): 56 data bytes
64 bytes from 129.215.33.27: icmp_seq=0 ttl=62 time=35.598 ms
64 bytes from 129.215.33.27: icmp_seq=1 ttl=63 time=69.142 ms
64 bytes from 129.215.33.27: icmp_seq=2 ttl=63 time=35.282 ms

After connected to the Informatics Forum using OpenVPN, you don't need to ssh into staff.ssh.inf.ed.ac.uk then ssh to your allocated desktop. Instead, you can ssh into your allocated desktop directly.

The rest of the tutorial assumes you are connected to the school's OpenVPN.

SSH Config File

Normally, when you connect to the school server, let say staff.ssh.inf.ed.ac.uk, you should have to type in ssh s1873872@staff.ssh.inf.ed.ac.uk every single time, which is annoying. The ssh config file can save you lots of time from remembering and typing the full hostname of various servers that you connect to.

  • By default, the config file should be located at ~/.ssh/config. You have to create the file on your own if this is your first time setting it up.

  • You can paste the following into your config file with vim or nano

    Host inf
    HostName staff.ssh.inf.ed.ac.uk
    User s1873872
    GSSAPIAuthentication yes
    GSSAPIDelegateCredentials yes
Host chania
    HostName %h.inf.ed.ac.uk
    User s1873872
    GSSAPIAuthentication yes
    GSSAPIDelegateCredentials yes

    and you can access staff.ssh.inf.ed.ac.uk by simply typing ssh inf or accessing my allocated desktop with ssh chania.

    Note: Change chania with the name of your allocated desktop.

  • Often times, server have a time limit where if no action was made, it would automatically disconnect your connection, we can also tell ssh to periodically send a signal to the server by specifying ServerAliveInterval. For instance:

    Host *
    ServerAliveInterval 240

Host inf
    HostName staff.ssh.inf.ed.ac.uk
    User s1873872
    GSSAPIAuthentication yes
    GSSAPIDelegateCredentials yes
Host chania
    HostName %h.inf.ed.ac.uk
    User s1873872
    GSSAPIAuthentication yes
    GSSAPIDelegateCredentials yes

Understanding the DICE file system

Regardless of which school servers you are connected to, including your allocated desktop, your home directory ~/ is always part of the school's AFS file system, which has limited space and usually relatively slow.

  • To check how much free space you have left, you can use the command freespace 
    (base) [chania]s1873872: freespace
Quota Information for /afs/inf.ed.ac.uk/user/s18/s1873872

Total available:  30.00 GB
           Used:   1.80 GB (6%)
      Remaining:  28.20 GB (94%)
  • And to show you how slow the read/write speed to the AFS file system is 
    (base) [chania]s1873872: dd if=/dev/zero of=test conv=sync bs=384k count=1k ; rm test
1024+0 records in
1024+0 records out
402653184 bytes (403 MB, 384 MiB) copied, 29.1333 s, 13.8 MB/s
    it has a read/write speed of 13.8MB/s.
  • Now, you can use the command df -h to see the storage configuration on your computer 
    (base) [chania]s1873872: df -h
Filesystem                     Size  Used Avail Use% Mounted on
udev                            32G     0   32G   0% /dev
tmpfs                          6.3G  2.3M  6.3G   1% /run
/dev/mapper/vg0-root           137G   45G   85G  35% /
tmpfs                           32G     0   32G   0% /dev/shm
tmpfs                          5.0M  4.0K  5.0M   1% /run/lock
tmpfs                           32G     0   32G   0% /sys/fs/cgroup
/dev/nvme0n1p1                 511M  7.8M  504M   2% /boot/efi
/dev/mapper/vg0-tmp            9.2G   37M  8.6G   1% /tmp
/dev/mapper/vg0-var_cache_afs  7.3G  2.1G  4.9G  31% /var/cache/afs
/dev/mapper/vg0-disk_scratch   782G   41G  710G   6% /disk/scratch
AFS                            2.0T     0  2.0T   0% /afs
tmpfs                          6.3G   60K  6.3G   1% /run/user/997
tmpfs                          6.3G   36K  6.3G   1% /run/user/2067095
    This is the output on my allocated desktop, and you can see I have a 782GB storage disk at /disk/scratch, which is actually the local SSD of our desktop.
  • Let see what is the read/write speed of this drive is 
    (base) [chania]s1873872: cd /disk/scratch/
(base) [chania]s1873872: dd if=/dev/zero of=test conv=sync bs=384k count=1k ; rm test
1024+0 records in
1024+0 records out
402653184 bytes (403 MB, 384 MiB) copied, 0.121981 s, 3.3 GB/s
    3.3GB/s read/write. Imagine reading a large dataset on this disk vs on the AFS file system.
  • Each servers have their own local storage space, which are usually much quicker than the AFS file system, use the local storage when possible. Be aware with the following warning, though I don't think it applies to our allocated desktops. 
    +---------------------------------------------------------------+
|                          WARNING !!!                          |
|                                                               |
| Data in this 'scratch' directory is not backed up! This       |
| directory should only be used to house data which is either   |
| transient, or which is easily reproducible.                   |
|                                                               |
| On a server accessible by multiple users, files and/or        |
| directories in this 'scratch' directory which have not been   |
| accessed for a period of time may be deleted without warning, |
| if lack of space in the directory is causing a problem for    |
| other active users.                                           |
|                                                               |
| On teaching servers, files in this 'scratch' directory which  |
| are owned by students will be deleted after those students    |
| have completed the associated course.                         |
+---------------------------------------------------------------+
  • This will introduce other issues, such as having individual conda installation and environments on individual computers, which I will try to address later in this.

Install conda

Conda is a package management system and environment management system that simplifies software development workflow significantly, I believe is used in most courses and I believe it is a good practice to have a separate environment for each project. In this section I will describe how I personally setup miniconda on DICE computer.

  • Connect to our allocated system (see #ssh-config-file) 
    ssh chania
  • Download miniconda3 for Linux from docs.conda.io to your home directory 
    wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
  • Install miniconda 
    sh Miniconda3-latest-Linux-x86_64.sh
  • Agree to the terms and conditions
  • The installation script will ask you where do you want to install miniconda 
    Miniconda3 will now be installed into this location:
/afs/inf.ed.ac.uk/user/s18/s1873872/miniconda3

  - Press ENTER to confirm the location
  - Press CTRL-C to abort the installation
  - Or specify a different location below

[/afs/inf.ed.ac.uk/user/s18/s1873872/miniconda3] >>>
    By default, it would install in our home directory ~/miniconda3. However, I would advice against installing anything on the AFS system, because of the reasons specified in #understanding-the-dice-file-system, but also you will get Permission Error when the token between your allocated desktop and AFS timed out. I recommend install it in the local storage, hence I would provide the path /disk/scratch/miniconda3. 
    Miniconda3 will now be installed into this location:
/afs/inf.ed.ac.uk/user/s18/s1873872/miniconda3

  - Press ENTER to confirm the location
  - Press CTRL-C to abort the installation
  - Or specify a different location below

[/afs/inf.ed.ac.uk/user/s18/s1873872/miniconda3] >>> /disk/scratch/miniconda3
  • After installation, the script should automatically append the following lines to ~/.bashrc, which basically tells your system where miniconda3 is installed. 
    (base) [chania]s1873872: cat ~/.bashrc

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/disk/scratch/miniconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/disk/scratch/miniconda3/etc/profile.d/conda.sh" ]; then
        . "/disk/scratch/miniconda3/etc/profile.d/conda.sh"
    else
        export PATH="/disk/scratch/miniconda3/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<
    However, DICE machines don't read .bashrc on start (see computing.help.inf.ed.ac.uk/dice-bash), hence our DICE machine wouldn't know where conda is being installed. To fix this, we can create/edit ~/.profile, which would be read at each start up.
  • Copy and paste the content between # >>> conda initialize >>> and # <<< conda initialize <<< from ~/.bashrc to ~/.profile. You can use vim or nano to edit ~/.profile, for instance
    • nano ~/.profile
    • paste the content
    • press Control + X to exit
    • press Y to save
    • to verify you have it correctly, you can print .profile by cat ~/.profile 
      (base) [chania]s1873872: cat ~/.profile
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/disk/scratch/miniconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
  if [ $? -eq 0 ]; then
    eval "$__conda_setup"
  else
    if [ -f "/disk/scratch/miniconda3/etc/profile.d/conda.sh" ]; then
      . "/disk/scratch/miniconda3/etc/profile.d/conda.sh"
    else
      export PATH="/disk/scratch/miniconda3/bin:$PATH"
    fi
  fi
unset __conda_setup
# <<< conda initialize <<<
  • Open a new terminal and ssh into your desktop, run which conda to see if you have conda installed correctly 
    (base) [chania]s1873872: which conda
/disk/scratch/miniconda3/bin/conda
    You should be able to see if the path is correct.

sshfs: mount system as network drive

In this section, I will talk about how to mount our allocated desktop to our local computer as a network drive using sshfs, so that you can directly modify the content on your server directly from your local computer. For instance, you can use PyCharm/Sublime Text to edit text on the server directly, and inspect images/graphs on my local desktop.

  • Install sshfs on your local computer:
    • Ubuntu/Debian: apt install sshfs
    • macOS: download and install both FUSE and SSHFS from FUSE for macOS
    • Windows: install sshfs-win (p.s. I have never used sshfs-win so I don't know how it works exactly.)
  • If I want to mount the dedicated disk /disk/scratch/ from my allocated desktop to the folder ~/chania/ on my local system (which you have to create the folder ~/chania manually on your local computer), I can use the following command 
    sshfs -o allow_other,defer_permissions s1873872@chania.inf.ed.ac.uk:/disk/scratch ~/chania
    When I check ~/chania on my laptop, I can see the home directory of my AFS system. Note: use the command pwd -P to get the absolute path of the folder on your system that you want to mount.
  • To umount the network drive from my laptop, I use the command 
    umount ~/chania
  • To make my life a bit easier, I wrote a script to mount and unmount particular folders from my allocated desktop to my local laptop, feel free to modify and use this script connect-chania.sh 
    #!/bin/bash

FILE=/Users/bryanlimy/sshfs/chania/miniconda3
if [ -d "$FILE" ]; then
  echo "unmount ~/chania"
  umount -f ~/chania
else
  echo "mount ~/sshfs/chania"
  sshfs -o allow_other,defer_permissions s1873872@chania.inf.ed.ac.uk:/disk/scratch/ ~/chania
fi
    When I run sh connect-chania.sh, it will automatically check if the folder FILE=/Users/bryanlimy/sshfs/chania/miniconda3 exists on my local system; if it exists, meaning that the network drive is already mounted, then the script would unmount the drive; if the folder doesn't exists on my local system, then the script would mount /disk/scratch (the local SSD on our allocated system) to ~/chania on my local laptop.

Copy large files and folders

sshfs is useful when you want to modify small files (e.g. .py or .txt files) with a text editor, however, I would not recommand transfering large files or folders via sshfs, as it is slow and unstable. In this section, I will show you how to copy files and folders using scp and rsync. Note: I personally prefer rsync.

  • Copying files from local computer to allocated desktop. e.g. copy folder ~/data/passive_ds/ on my local laptop to my allocated desktop /disk/scratch/dataset/passive_ds. Note: this assumes /disk/scratch/dataset/ exists on your allocated desktop and that chania exists in your ~/.ssh/config (see SSH Config File).
    • with scp
      • you can use the following command 
        scp -r ~/data/passive_ds chania:/disk/scratch/dataset/passive_ds
      • then you should see the following as the files are being uploaded 
        (base) ➜ scp -r ~/data/passive_ds chania:/disk/scratch/dataset/passive_ds
Password:
.DS_Store                                                            100% 8196    44.1KB/s   00:00
.DS_Store                                                            100% 6148    27.4KB/s   00:00
preparation.npy                                                       10%   51MB   1.4MB/s   04:59 ETA
    • with rsync
      • you can use the following command 
        rsync -Pavzh ~/data/passive_ds 'chania:/disk/scratch/dataset/passive_ds'
      • then you should see the following as the files are being uploaded 
        (base) ➜ rsync -Pavzh ~/data/passive_ds 'chania:/disk/scratch/dataset/passive_ds'
Password:
building file list ...
2484 files to consider
passive_ds/
passive_ds/.DS_Store
      8.20K 100%    0.00kB/s    0:00:00 (xfer#1, to-check=2482/2484)
passive_ds/20200723_Thy171_alldata.mat
      70.88M 100%    1.38MB/s    0:00:49 (xfer#2, to-check=2481/2484)
passive_ds/20200723_Thy171_darkdata.mat
      2.75M  31%  869.90kB/s    0:00:06
  • To copy files and fodlers from your allocated desktop back to your local computer, you can use the same commands. For instance, to copy /disk/scratch/dataset/passive_ds to your local computer ~/passive_ds, then you can use the following command with scp: 
    scp -r 'chania:/disk/scratch/dataset/passive_ds' ~/passive_ds
    or with rsync: 
    rsync -Pavzh 'chania:/disk/scratch/dataset/passive_ds' ~/passive_ds
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment