Skip to content

Instantly share code, notes, and snippets.

@gpittarelli
Created March 7, 2019 18:23
Show Gist options
  • Save gpittarelli/5e14fb772ce0230a3c40ffad2c2262be to your computer and use it in GitHub Desktop.
Save gpittarelli/5e14fb772ce0230a3c40ffad2c2262be to your computer and use it in GitHub Desktop.
GNU Parallel autoscaling workload example
#!/usr/bin/env bash
(
while ((1)); do
gcloud --format=json compute instances list --filter="name~'nlp-workers-.*'" | \
jq -rc '.[]|select(.status=="RUNNING").name' > workers.hosts.txt
sleep 30s
done
) & # (or, just run this watcher in a separate script)
# Set an autoscaling instance group using cheaper preemptible vms
gcloud compute instance-groups managed resize nlp-workers --size=4
find /data/sentence-jobs/ -type f | parallel --eta --slf workers.hosts.txt --joblog job.log 'openie {} > {}.out'
gcloud compute instance-groups managed resize nlp-workers --size=0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment