Skip to content

Instantly share code, notes, and snippets.

@JoshBroomberg
Last active October 15, 2020 19:47
Show Gist options
  • Save JoshBroomberg/9bd3196a8778b53031fa6be9abb76793 to your computer and use it in GitHub Desktop.
Save JoshBroomberg/9bd3196a8778b53031fa6be9abb76793 to your computer and use it in GitHub Desktop.
if [ "$SPARK_MODE" == "worker" ]; then
DRIVER_HOST="$(hostname | sed -E 's/spark-([a-z0-9]+)-(worker|master)-[0-9]+/spark-driver-\1.domino-compute.svc.cluster.local/g')"
echo "Starting ray worker targeting head at \"${DRIVER_HOST}\" after head startup delay"
for i in {1..10}; do ray start --address=${DRIVER_HOST}:6379 --redis-password=pass --num-gpus=1 && break || sleep 60; done
else
echo "taking no action on master"
# DRIVER_HOST="$(hostname | sed -E 's/spark-([a-z0-9]+)-(worker|master)-[0-9]+/spark-driver-\1.domino-compute.svc.cluster.local/g')"
# PROXY_PATH="$(hostname | sed -E 's/spark-([a-z0-9]+)-(worker|master)-[0-9]+/kevin_flansburg\/Ray\/workspaces\/\1\/spark\/ui/g')"
# echo "Starting ray ui proxying \"/${PROXY_PATH}\" to head at \"${DRIVER_HOST}\""
# touch /usr/sbin/proxy.txt
# configurable-http-proxy --ip 0.0.0.0 --port 8080 --log-level=debug --default-target=http://$DRIVER_HOST:4040 --no-include-prefix >> /usr/sbin/proxy.txt &
# sleep 5
# curl --header "Content-Type: application/json" \
# --request POST \
# --data "{\"target\":\"http://$DRIVER_HOST:4040\"}" \
# http://localhost:8081/api/routes/$PROXY_PATH
fi
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment