Skip to content

Instantly share code, notes, and snippets.

@berlinbrown
Created January 1, 2024 14:30
Show Gist options
  • Save berlinbrown/e3f3d741b3a1eb722411fbc0d9a25efe to your computer and use it in GitHub Desktop.
Save berlinbrown/e3f3d741b3a1eb722411fbc0d9a25efe to your computer and use it in GitHub Desktop.
Example Setup Spark
====
When I start a spark-shell from machine B to spark master on B I get this:
scala> sc.master
res3: String = spark://machinB:7077
scala> sc.textFile("/tmp/data/myfile.csv").count()
res4: Long = 976
Act II - The Conflict
But when I start a spark-shell from machine A, pointing to spark master on B I get this:
scala> sc.master
res2: String = spark://machineB:7077
scala> sc.textFile("/tmp/data/myfile.csv").count()
org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file:/
https://stackoverflow.com/questions/32464894/where-does-spark-look-for-text-files
https://spark.apache.org/examples.html
'****'
C:\Users\berli\Downloads\spark-3.5.0-bin-hadoop3-scala2.13\spark-3.5.0-bin-hadoop3-scala2.13
****
https://spark.apache.org/examples.html
https://spark.apache.org/
https://en.wikipedia.org/wiki/Apache_Spark
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment