In February 2017, Google announced the availability GPU-based VMs. I spun up a few of these instances, and ran some benchmarks. Along the way, I wrote down the steps taken to provision these VM instances, and install relevant drivers.
Update April 2019: Updated instructions to use instances with the Tesla T4 GPUs.
Some details:
- Each VM supports up to four Tesla K80 boards, which, if I recall correctly, comprises two GK210 GPUs.
- GPUs are billed per minute (10 min. minimum).
My only use case for GPU-based VMs is cracking (password) hashes, hence the test run with Hashcat.
Some remarks:
- I should warn you beforehand that the K80s are known to be sub-optimal for cracking (password) hashes (using Hashcat). Low cost per hour per GPU doesn't necessarily mean it's the best choice in terms of performance/cost ratio.
- For example, take a look at @jmgosney's GTX 1080 Hashcat benchmarks. If you compare these results to any K80 Hashcat benchmark, you will see that a single GTX 1080 is several times faster than a Tesla K80.
- Also, at this time, you can't attach GPUs to low-cost preemptible instances (similar to Amazon EC2 spot instances).
- Amazon, on the other hand, does support low-cost GPU-based spot instances, which makes cracking in the cloud a whole lot more interesting. See spot pricing.
- A Google Cloud Platform account.
- You need to have the command-line tool
gcloud
installed. - If you already have
gcloud
installed, verify that the current version (gcloud version
) is greater than (or equal) to144.0.0
. - You need to authorise (
gcloud auth login
) your local machine to access your Google Cloud account. - Make sure that you have enough GPUs available in both your project, and the desired
region/zone. You can check the current limit using:
gcloud compute regions describe europe-west4 | grep NVIDIA.*GPUS -B1
(replaceeurope-west4
with your desired region, and zone).
$ gcloud compute regions describe europe-west4 | grep NVIDIA.*GPU -B1
- limit: 8.0
metric: NVIDIA_T4_GPUS
--
- limit: 8.0
metric: PREEMPTIBLE_NVIDIA_T4_GPUS
- If the metric 'NVIDIA_T4_GPUS' isn't listed, or the limit is set to
0.0
, you need to ask your account manager to increase the limit, or submit an increase request. - Increase requests are usually granted in a matter of hours. However, you might be asked to make a prepayment of a few tens of dollars (Billing > select your billing account > 'Payment overview' > 'Make a payment').
Creating the virtual machine instance, and attaching GPUs, can be done in a single command:
$ gcloud compute instances create koenrh-gpu-instance-2 \
--machine-type n1-standard-2 \
--zone europe-west4-b \
--accelerator type=nvidia-tesla-t4,count=4 \
--image-family ubuntu-1804-lts \
--image-project ubuntu-os-cloud \
--maintenance-policy TERMINATE
Created [https://www.googleapis.com/compute/beta/projects/koenrh-test-drive/zones/europe-west4-b/instances/koenrh-gpu-instance-2].
NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS
koenrh-gpu-instance-2 europe-west4-b n1-standard-2 10.132.0.2 35.xxx.xx.xxx RUNNING
Besides the instance name, there are a few other arguments you might want to change:
--zone [zone]
: Make sure that GPUs are actually supported in the specified zone.--accelereator
:type
specifies the GPU model that you want to use.count
specifies the number of processors (maximum is8
). You can attach up to four K80 boards, which means there's a maximum of8
processors (K80 board houses two GK210 GPUs).
First, you need to authorise your SSH key by adding your SSH public key the just-provisioned
machine. Use the following command, but use your own instance name, instance region,
and path to your public key (after ssh-keys=
).
# NOTE: Update the path to your SSH public key
$ gcloud compute instances add-metadata koenrh-gpu-instance-2 \
--zone europe-west4-b \
--metadata-from-file ssh-keys=/Users/koen/.ssh/id_ed25519.pub
Updated [https://www.googleapis.com/compute/v1/projects/koenrh-test-drive/zones/europe-west4-b/instances/koenrh-gpu-instance-2].
- Get the NVIDIA signing key that is used to sign packages.
curl -o cudatools.asc http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
- Verify the integrity of the downloaded key. It should return with
cudatools.asc: OK
.
$ CUDA_KEY_CHECKSUM=47217c49dcb9e47a8728b354450f694c9898cd4a126173044a69b1e9ac0fba96
$ echo "$CUDA_KEY_CHECKSUM *cudatools.asc" | sha256sum --check --strict
cudatools.asc: OK
- Or verify the fingerprint of the downloaded key. It should match
AE09 FE4B BD22 3A84 B2CC FCE3 F60F 4B3D 7FA2 AF80
.
$ gpg --with-fingerprint --with-colons cudatools.asc | grep "^fpr:" | cut -d: -f 10
AE09FE4BBD223A84B2CCFCE3F60F4B3D7FA2AF80
- Import the CUDA signing key:
$ apt-key add cudatools.asc
OK
- Download the CUDA installer, again, verify its integrity, and add the repository
to the system using
dpkg
:
$ curl -sO http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-repo-ubuntu1804_10.1.105-1_amd64.deb
$ echo "2711e92b362972d6ba16dd1e1410f56ecc808831c3ae42b2b9fa98cea086f146 *cuda-repo-ubuntu1804_10.1.105-1_amd64.deb" | sha256sum --check --strict
cuda-repo-ubuntu1804_10.1.105-1_amd64.deb: OK
$ sudo dpkg -i cuda-repo-ubuntu1804_10.1.105-1_amd64.deb
...
OK
- Finally, install CUDA, which includes the NVIDIA drivers:
$ sudo apt-get update && sudo apt-get install -y cuda
...
This could take a while.
- Reboot the machine:
sudo reboot
When the installation has finished, verify that the driver is installed, and initialised
properly using the nvidia-smi
tool.
$ nvidia-smi
...
A little performance tuning. Before you can change the application clocks you need to put the GPUs in 'persistence mode'.
- Optionally, allow non-admin/root users to change the application clocks using the following command.
sudo nvidia-smi --applications-clocks-permission=UNRESTRICTED
- Enable 'persistence mode' with the following command.
sudo nvidia-smi --persistence-mode=ENABLED
- You must set both the memory and graphics clock, because the clock rates are
tied to a specific memory clock rate. Set the clock rates using the following
command (you can query the supported application clocks using
nvidia-smi -q -i 0 -d SUPPORTED_CLOCKS
).
sudo nvidia-smi --applications-clocks=2505,875
- First, we need to get hold of Hashcat's signing key. I hate to use key servers, but Hashcat doesn't seem to have published the (ASCII) signing key anywhere else, which means we have to fetch the key from a key server.
$ gpg --recv-keys "A70833229D040B4199CC00523C17DA8B8A16544F"
gpg: key 8A16544F: "Hashcat signing key <signing@hashcat.net>" not changed
gpg: Total number processed: 1
gpg: unchanged: 1
- Just to be sure, verify that you have imported the correct key. The fingerprint
should match
A708 3322 9D04 0B41 99CC 0052 3C17 DA8B 8A16 544F
.
$ gpg --list-keys --with-fingerprint A70833229D040B4199CC00523C17DA8B8A16544F
pub rsa2048/8A16544F 2015-12-19 [SC]
Key fingerprint = A708 3322 9D04 0B41 99CC 0052 3C17 DA8B 8A16 544F
uid [ unknown] Hashcat signing key <signing@hashcat.net>
sub rsa2048/911D7BDE 2015-12-19 [E]
- Download the archive containing the pre-compiled binaries, and signature from Hashcat's website.
curl -OO https://hashcat.net/files/hashcat-5.1.0.7z{,.asc}
- Verify the integrity of the downloaded Hashcat archive using the command depicted below. It should indicate that the signature is valid ("Good signature"). The warning indicates that the key that was used to sign this package doesn't have a trusted signature, therefore you should verify that the fingerprint of the key ('Primary key fingerprint' at the bottom) matches the fingerprint of the key that we have imported (and verified) in the previous steps.
$ gpg --verify hashcat-5.1.0.7z.asc hashcat-5.1.0.7z
gpg: Signature made Sun 02 Dec 2018 11:03:59 AM UTC
gpg: using RSA key 3C17DA8B8A16544F
gpg: checking the trustdb
gpg: no ultimately trusted keys found
gpg: Good signature from "Hashcat signing key <signing@hashcat.net>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg: There is no indication that the signature belongs to the owner.
Primary key fingerprint: A708 3322 9D04 0B41 99CC 0052 3C17 DA8B 8A16 544F
- Unpack the archive with Hashcat binaries.
7z x hashcat-5.1.0.7z
...
Everything is Ok
Folders: 39
Files: 601
Size: 57418237
Compressed: 2658676
- Run a full Hashcat benchmark.
cd ./hashcat-5.1.0/
./hashcat64.bin -b | tee hashcat-benchmark.txt
You can find the full results here (2x Tesla K80, 4x GPU).
- Automate provisioning of GPU VMs (whether AWS EC2 or GCP VM) using shell scripts or, better, Ansible.
- Figure out how to run
hashcat
in a distributed way, i.e. how to distribute password cracking jobs across servers or even clusters of servers.- Look into hashtopus.
- Look into management APIs or GUIs for remote management.
This is awesome, thanks for sharing! I hope to try this soon!