Skip to content

Instantly share code, notes, and snippets.

@christippett
Last active May 8, 2019 14:31
Show Gist options
  • Save christippett/17f814ba683a7d632fc3ba6fe7d64b2d to your computer and use it in GitHub Desktop.
Save christippett/17f814ba683a7d632fc3ba6fe7d64b2d to your computer and use it in GitHub Desktop.
Shell script to poll successful completion of Dataflow job
#!/bin/sh
JOB_ID=$1
SLEEP_TIME=10
echo "Polling status for Dataflow job: ${JOB_ID}"
check_job_status() {
echo "${JOB_STATUS}" | grep "JOB_STATE_DONE\|JOB_STATE_CANCELLED\|JOB_STATE_FAILED\|JOB_STATE_DRAINED\|^$" > /dev/null
GREP_RETURN_CODE=$?
if [ ${GREP_RETURN_CODE} -eq 0 ]
then
printf "Dataflow job complete!\\r\\n"
exit 1
fi
}
while [ true ]
do
JOB_STATUS=$(gcloud --format json dataflow jobs describe $JOB_ID | jq -r '.currentState')
check_job_status
printf "\\r$JOB_STATUS"
for i in $(seq 1 $SLEEP_TIME); do printf "."; sleep 1; done
done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment