Allocate 3 nodes with 1 task of 24 cpus each:
salloc -N 3 -n 3 -c 24Within the allocation, make sure you have dask etc (e.g. by activating a virtual or conda env).
Start a scheduler (on the first node) and three workers (one per node):
$ srun -n1 -N1 -r0 dask-scheduler --scheduler-file scheduler.json &>> scheduler.log &
| #!/bin/bash | |
| #Enables Playing on a moonlight server via an ssh tunnel. This is useful on restricted networks, as only the ssh port is needed. | |
| #Needs the complimentary script, redirectudp, running on the ssh server to function. | |
| #needs a private key for the ssh server | |
| #sudo apt install ssh socat snap | |
| #snap install moonlight |
Podman is such a cool project! However, there is no easy way to setup the nvidia-container-runtime and podman so that we can run unprivileged container in a gpu host. This is specially interesting in environments with multiple people accessing the same host (strong isolation between containers!!!).
Ubuntu: add-apt-repository -y ppa:projectatomic/ppa && apt install podman buildah skopeo
Log into your server with ssh, something like
ssh -i "my_secret.pem" ubuntu@12.123.12.123If it's a new server, you'll need to install a few things.
Install conda with
dhclient is the Dynamic Host Configuration Protocol (DHCP) Client one would use to allow a client to connect to a DHCP server.
$ sudo nano /etc/rc.local
#!/bin/bash
dhclient
exit 0
| # Instructions for 4.14 and cuda 9.1 | |
| # If upgrading from 4.13 and cuda 9.0 | |
| $ sudo apt-get purge --auto-remove libcud* | |
| $ sudo apt-get purge --auto-remove cuda* | |
| $ sudo apt-get purge --auto-remove nvidia* | |
| # also remove the container directory direcotory at /usr/local/cuda-9.0/ | |
| # Important libs required with 4.14.x with Cuda 9.X | |
| $ sudo apt install libelf1 libelf-dev |
| #!/bin/sh | |
| echo -e "-- Removing exited containers --\n" | |
| docker ps --all --quiet --filter="status=exited" | xargs --no-run-if-empty docker rm --volumes | |
| echo -e "\n\n-- Removing untagged images --\n" | |
| docker rmi --force $(docker images | awk '/^<none>/ { print $3 }') | |
| echo -e "\n\n-- Removing volume directories --\n" | |
| docker volume rm $(docker volume ls --quiet --filter="dangling=true") |
In this article, I will share some of my experience on installing NVIDIA driver and CUDA on Linux OS. Here I mainly use Ubuntu as example. Comments for CentOS/Fedora are also provided as much as I can.
Assuming your Anaconda Python is at the front of your path (type 'which ipython' to verify that it is using the Anaconda version). Here, Anaconda is installed in ~/anaconda
System Commands are preceded with ">"
Python Commands are preceded with ">>"
If you download the Pyrosetta Binaries, unzip them, source the SetPyrosettaEnvironment.sh and run ipython while using Ananconda's version of Ipython, you may see the following error:
| openssl rsa -in ~/.ssh/id_rsa -outform pem > id_rsa.pem | |
| chmod 600 id_rsa.pem |