Michael mkmojo

## gist:bf2dfce4767ce50fb13e

      
        
          
            
              
              1 file
            
          
          
            
              
              0 forks
            
          
          
            
              
              0 comments
            
          
          
            
              
              0 stars
            
          
        
        
          
              
          
          
            
                mkmojo
                / gist:bf2dfce4767ce50fb13e
            
            
              Last active
              August 29, 2015 14:23
                — forked from laserson/gist:1d1185b412b41057810b
            
          
        
      
        
  
      
    Building Spark for PySpark use on top of YARN

Build Spark on local machine (only if using PySpark; otherwise, remote machine works)
(http://spark.apache.org/docs/latest/building-with-maven.html)
export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"
mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package

Copy the assembly/target/scala-2.10/...jar to the corresponding directory on
the cluster node and also into a location in HDFS.