Set up Apache Spark 1.5+ with Hadoop 2.6+ s3a
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
|# For a local environment|
|# Install hadoop and apache-spark via homebrew|
|# Apache Spark conf file|
|# Make the AWS jars available to Spark|
|# Add file|
Mar 1, 2016
Thanks for this - I struggled with the same thing for over a week until I found this gist. I've tried to leave an even more concise version of what you need on my own gist here: https://gist.github.com/chicagobuss/6557dbf1ad97e5a09709
Sep 7, 2016
Thanks for this.
pyspark --packages com.amazonaws:aws-java-sdk-pom:1.10.34,org.apache.hadoop:hadoop-aws:2.6.0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
A couple of important things:
aws-java-sdk-1.7.4is required as of Dec 19, 2015. Even though its a 2014 built JAR, the API changed.
spark.propertiesthe HDFS configurations can be set:
Or they can be set like this, where
scis an instance of
If you encounter an NPE while trying to write to an S3A URL:
then additional configuration is needed for the temporary scratch space. If your files are small enough you can instead just:
as this bypasses the temporary local disk scratch space.