Skip to content

Instantly share code, notes, and snippets.

@anthonyclarka2
Forked from gpittarelli/run-batch-job.sh
Created March 7, 2019 18:35
Show Gist options
  • Save anthonyclarka2/bed7dbb97324c2e3c7a89adc8b0ffc28 to your computer and use it in GitHub Desktop.
Save anthonyclarka2/bed7dbb97324c2e3c7a89adc8b0ffc28 to your computer and use it in GitHub Desktop.
GNU Parallel autoscaling workload example
#!/usr/bin/env bash
(
while ((1)); do
gcloud --format=json compute instances list --filter="name~'nlp-workers-.*'" | \
jq -rc '.[]|select(.status=="RUNNING").name' > workers.hosts.txt
sleep 30s
done
) & # (or, just run this watcher in a separate script)
# Set an autoscaling instance group using cheaper preemptible vms
gcloud compute instance-groups managed resize nlp-workers --size=4
find /data/sentence-jobs/ -type f | parallel --eta --slf workers.hosts.txt --joblog job.log 'openie {} > {}.out'
gcloud compute instance-groups managed resize nlp-workers --size=0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment