Skip to content

Instantly share code, notes, and snippets.

@mehd-io

mehd-io/list_hadoop_conf_pyspark.py

Created October 16, 2019 08:57
Show Gist options
  • Save mehd-io/f957c1e8c135171f8924f13534ed6b68 to your computer and use it in GitHub Desktop.
Save mehd-io/f957c1e8c135171f8924f13534ed6b68 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
list_hadoop_conf_pyspark.py
# List hadoop conf while running a pyspark job
hadoopConf = {}
iterator = sc._jsc.hadoopConfiguration().iterator()
while iterator.hasNext():
prop = iterator.next()
hadoopConf[prop.getKey()] = prop.getValue()
for item in sorted(hadoopConf.items()):
print(item)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment