koenrh/gcp-gpu-vm-hashcat.md

## gcp-gpu-vm-hashcat.md

      
    Raw
  

              gcp-gpu-vm-hashcat.md
            
          
    Running Hashcat on Google Cloud's GPU-based VMs

In February 2017, Google announced
the availability GPU-based VMs. I spun up a few of these instances, and ran some
benchmarks. Along the way, I wrote down the steps taken to provision these VM
instances, and install relevant drivers.
Update April 2019: Updated instructions to use instances with the Tesla T4
GPUs.
Some details:

Each VM supports up to four Tesla K80 boards, which, if I recall correctly,
comprises two GK210 GPUs.
GPUs are billed per minute (10 min. minimum).

My only use case for GPU-based VMs is cracking (password) hashes, hence the test
run with Hashcat.
Some remarks:

I should warn you beforehand that the K80s are known to be sub-optimal for cracking
(password) hashes (using Hashcat). Low cost per hour per GPU doesn't necessarily
mean it's the best choice in terms of performance/cost ratio.
For example, take a look at @jmgosney's GTX
1080 Hashcat benchmarks. If you compare these results to any K80 Hashcat benchmark,
you will see that a single GTX 1080 is several times faster than a Tesla K80.
Also, at this time, you can't attach GPUs to low-cost preemptible instances
(similar to Amazon EC2 spot instances).
Amazon, on the other hand, does support low-cost GPU-based spot instances, which
makes cracking in the cloud a whole lot more interesting. See spot pricing.

Prerequisites


A Google Cloud Platform account.
You need to have the command-line tool gcloud installed.
If you already have gcloud installed, verify that the current version
(gcloud version) is greater than (or equal) to 144.0.0.
You need to authorise (gcloud auth login) your local machine to access your
Google Cloud account.
Make sure that you have enough GPUs available in both your project, and the desired
region/zone. You can check the current limit using:
gcloud compute regions describe europe-west4 | grep NVIDIA.*GPUS -B1 (replace
europe-west4 with your desired region, and zone).

$ gcloud compute regions describe europe-west4 | grep NVIDIA.*GPU -B1

- limit: 8.0
  metric: NVIDIA_T4_GPUS
--
- limit: 8.0
  metric: PREEMPTIBLE_NVIDIA_T4_GPUS

If the metric 'NVIDIA_T4_GPUS' isn't listed, or the limit is set to 0.0, you
need to ask your account manager to increase the limit, or
submit an increase request.
Increase requests are usually granted in a matter of hours. However, you might
be asked to make a prepayment of a few tens of dollars (Billing
> select your billing account > 'Payment overview' > 'Make a payment').

Creating VM instance (and attaching GPUs)

Creating the virtual machine instance, and attaching GPUs, can be done in a single
command:
$ gcloud compute instances create koenrh-gpu-instance-2 \
  --machine-type n1-standard-2 \
  --zone europe-west4-b \
  --accelerator type=nvidia-tesla-t4,count=4 \
  --image-family ubuntu-1804-lts \
  --image-project ubuntu-os-cloud \
  --maintenance-policy TERMINATE

Created [https://www.googleapis.com/compute/beta/projects/koenrh-test-drive/zones/europe-west4-b/instances/koenrh-gpu-instance-2].
NAME                   ZONE            MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP    STATUS
koenrh-gpu-instance-2  europe-west4-b  n1-standard-2               10.132.0.2   35.xxx.xx.xxx  RUNNING
Besides the instance name, there are a few other arguments you might want to change:

--zone [zone]: Make sure that GPUs are actually supported in the specified zone.
--accelereator: type specifies the GPU model that you want to use. count
specifies the number of processors (maximum is 8). You can attach up to four
K80 boards, which means there's a maximum of 8 processors (K80 board houses
two GK210 GPUs).

Authorising your SSH key

First, you need to authorise your SSH key by adding your SSH public key the just-provisioned
machine. Use the following command, but use your own instance name, instance region,
and path to your public key (after ssh-keys=).
# NOTE: Update the path to your SSH public key
$ gcloud compute instances add-metadata koenrh-gpu-instance-2 \
  --zone europe-west4-b \
  --metadata-from-file ssh-keys=/Users/koen/.ssh/id_ed25519.pub

Updated [https://www.googleapis.com/compute/v1/projects/koenrh-test-drive/zones/europe-west4-b/instances/koenrh-gpu-instance-2].
Installing drivers


Get the NVIDIA signing key that is used to sign packages.

curl -o cudatools.asc http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub


Verify the integrity of the downloaded key. It should return with
cudatools.asc: OK.

$ CUDA_KEY_CHECKSUM=47217c49dcb9e47a8728b354450f694c9898cd4a126173044a69b1e9ac0fba96
$ echo "$CUDA_KEY_CHECKSUM *cudatools.asc" | sha256sum --check --strict

cudatools.asc: OK


Or verify the fingerprint of the downloaded key. It should match
AE09 FE4B BD22 3A84 B2CC  FCE3 F60F 4B3D 7FA2 AF80.

$ gpg --with-fingerprint --with-colons cudatools.asc | grep "^fpr:" | cut -d: -f 10

AE09FE4BBD223A84B2CCFCE3F60F4B3D7FA2AF80


Import the CUDA signing key:

$ apt-key add cudatools.asc
OK

Download the CUDA installer, again, verify its integrity, and add the repository
to the system using dpkg:

$ curl -sO http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-repo-ubuntu1804_10.1.105-1_amd64.deb

$ echo "2711e92b362972d6ba16dd1e1410f56ecc808831c3ae42b2b9fa98cea086f146 *cuda-repo-ubuntu1804_10.1.105-1_amd64.deb" | sha256sum --check --strict

cuda-repo-ubuntu1804_10.1.105-1_amd64.deb: OK

$ sudo dpkg -i cuda-repo-ubuntu1804_10.1.105-1_amd64.deb

...
OK

Finally, install CUDA, which includes the NVIDIA drivers:

$ sudo apt-get update && sudo apt-get install -y cuda

...
This could take a while.

Reboot the machine: sudo reboot

When the installation has finished, verify that the driver is installed, and initialised
properly using the nvidia-smi tool.
$ nvidia-smi

...
Tuning the GPUs

A little performance tuning. Before you can change the application clocks you need
to put the GPUs in 'persistence mode'.

Optionally, allow non-admin/root users to change the application clocks using
the following command.

sudo nvidia-smi --applications-clocks-permission=UNRESTRICTED


Enable 'persistence mode' with the following command.

sudo nvidia-smi --persistence-mode=ENABLED

You must set both the memory and graphics clock, because the clock rates are
tied to a specific memory clock rate. Set the clock rates using the following
command (you can query the supported application clocks using
nvidia-smi  -q -i 0 -d SUPPORTED_CLOCKS).

sudo nvidia-smi --applications-clocks=2505,875

Running Hashcat


First, we need to get hold of Hashcat's signing key. I hate to use key servers,
but Hashcat doesn't seem to have published the (ASCII) signing key anywhere else,
which means we have to fetch the key from a key server.

$ gpg --recv-keys "A70833229D040B4199CC00523C17DA8B8A16544F"

gpg: key 8A16544F: "Hashcat signing key <signing@hashcat.net>" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1


Just to be sure, verify that you have imported the correct key. The fingerprint
should match A708 3322 9D04 0B41 99CC  0052 3C17 DA8B 8A16 544F.

$ gpg --list-keys --with-fingerprint A70833229D040B4199CC00523C17DA8B8A16544F

pub   rsa2048/8A16544F 2015-12-19 [SC]
      Key fingerprint = A708 3322 9D04 0B41 99CC  0052 3C17 DA8B 8A16 544F
uid         [ unknown] Hashcat signing key <signing@hashcat.net>
sub   rsa2048/911D7BDE 2015-12-19 [E]

Download the archive containing the pre-compiled binaries, and signature from
Hashcat's website.

curl -OO https://hashcat.net/files/hashcat-5.1.0.7z{,.asc}

Verify the integrity of the downloaded Hashcat archive using the command depicted
below. It should indicate that the signature is valid ("Good signature"). The
warning indicates that the key that was used to sign this package doesn't have
a trusted signature, therefore you should verify that the fingerprint of the
key ('Primary key fingerprint' at the bottom) matches the fingerprint of the
key that we have imported (and verified) in the previous steps.

$ gpg --verify hashcat-5.1.0.7z.asc hashcat-5.1.0.7z

gpg: Signature made Sun 02 Dec 2018 11:03:59 AM UTC
gpg:                using RSA key 3C17DA8B8A16544F
gpg: checking the trustdb
gpg: no ultimately trusted keys found
gpg: Good signature from "Hashcat signing key <signing@hashcat.net>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: A708 3322 9D04 0B41 99CC  0052 3C17 DA8B 8A16 544F

Unpack the archive with Hashcat binaries.

7z x hashcat-5.1.0.7z

...

Everything is Ok

Folders: 39
Files: 601
Size:       57418237
Compressed: 2658676

Run a full Hashcat benchmark.

cd ./hashcat-5.1.0/

./hashcat64.bin -b | tee hashcat-benchmark.txt
You can find the full results here
(2x Tesla K80, 4x GPU).
Notes to self


Automate provisioning of GPU VMs (whether AWS EC2 or GCP VM) using shell scripts
or, better, Ansible.
Figure out how to run hashcat in a distributed way, i.e. how to distribute
password cracking jobs across servers or even clusters of servers.

Look into hashtopus.


Look into management APIs or GUIs for remote management.

For example: hashview (site)