Skip to content

Instantly share code, notes, and snippets.

@mrchristine

mrchristine/get_spark_ui.py

Created February 12, 2020 23:54
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save mrchristine/92f5014795457358adbd6a8e0b5418bb to your computer and use it in GitHub Desktop.
Save mrchristine/92f5014795457358adbd6a8e0b5418bb to your computer and use it in GitHub Desktop.
Download ZIP
Script to get the Spark UI dynamically
Raw
get_spark_ui.py
ui_port = spark.sql("set spark.ui.port").collect()[0].value
env = "myenvironment.cloud.databricks.com"
cluster_id = dbutils.notebook.entry_point.getDbutils().notebook().getContext().clusterId().getOrElse(None)
url = "https://{0}/driver-proxy-api/o/0/{1}/{2}/api/v1/".format(env, cluster_id, ui_port)
import requests
token = "TOKEN"
token_header = {'Authorization': 'Bearer {0}'.format(token)}
api_endpoint = url + "applications"
print(api_endpoint)
resp = requests.get(api_endpoint, headers=token_header)
print(resp.text)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment