eclipse-intelliJ.md

#Eclipse Scala:

1. Installed Scala IDE 4.4.1 (http://scala-ide.org/download/sdk.html)

2. File > New Project > Maven Project

3. Tick "Skip Archetype Selection" (simple project)

   

4. Add maven project info

5. Change to scala nature (Right click on proj > configure > add scala nature)

6. Add spark core dependency.

   <!-- https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.10 -->
   <dependency>
       <groupId>org.apache.spark</groupId>
       <artifactId>spark-core_2.10</artifactId>
       <version>1.5.2</version>
   </dependency>

7. Save pom.xml to trigger maven central download.

8. Update proj. for Scala (Project > Properties > Scala Compiler > Change Scala install to Fixed Scala 2.10.x (built in))

9. Add spark code (create new Scala Object, add spark imports, create SparkContext etc.)

   import org.apache.spark.SparkContext
   import org.apache.spark.SparkConf
   
   object AuctApp {
       def main(args: Array[String]) {
           val conf = new SparkConf().setAppName("AuctionsApp")
           val sc = new SparkContext(conf)
           println("Just a parallelize: " + sc.parallelize(List(1,2,3,4)).count())
       }
   }

10. Update pom.xml to include scala and fatjar plugins

    <build>
    	<plugins>
    		<!-- mixed scala/java compile -->
    		<plugin>
    			<groupId>org.scala-tools</groupId>
    			<artifactId>maven-scala-plugin</artifactId>
    			<executions>
    				<execution>
    					<id>compile</id>
    					<goals>
    						<goal>compile</goal>
    					</goals>
    					<phase>compile</phase>
    				</execution>
    				<execution>
    					<id>test-compile</id>
    					<goals>
    						<goal>testCompile</goal>
    					</goals>
    					<phase>test-compile</phase>
    				</execution>
    				<execution>
    					<phase>process-resources</phase>
    					<goals>
    						<goal>compile</goal>
    					</goals>
    				</execution>
    			</executions>
    		</plugin>
    		<plugin>
    			<artifactId>maven-compiler-plugin</artifactId>
    			<configuration>
    				<source>1.7</source>
    				<target>1.7</target>
    			</configuration>
    		</plugin>
    		<!-- for fatjar -->
    		<plugin>
    			<groupId>org.apache.maven.plugins</groupId>
    			<artifactId>maven-assembly-plugin</artifactId>
    			<version>2.4</version>
    			<configuration>
    				<descriptorRefs>
    					<descriptorRef>jar-with-dependencies</descriptorRef>
    				</descriptorRefs>
    			</configuration>
    			<executions>
    				<execution>
    					<id>assemble-all</id>
    					<phase>package</phase>
    					<goals>
    						<goal>single</goal>
    					</goals>
    				</execution>
    			</executions>
    		</plugin>
    		<plugin>
    			<groupId>org.apache.maven.plugins</groupId>
    			<artifactId>maven-jar-plugin</artifactId>
    			<configuration>
    				<archive>
    					<manifest>
    						<addClasspath>true</addClasspath>
    						<mainClass>fully.qualified.MainClass</mainClass>
    					</manifest>
    				</archive>
    			</configuration>
    		</plugin>
    	</plugins>
    </build>

11. Proj Rightclick > Maven > Update Project

12. Run As > Maven Build > package (as goal)

#IntelliJ Scala:

1. Install Scala, SBT and SBT Executor plugins. (Configure > Plugins > Browse Repositories)

   • Search for and find "Scala" under Languages
   • Find "SBT" under Build
   • Find "SBT Executor" under Build

2. Restart IDEA

3. Create New Project > Scala > SBT > Next > Select "Scala Version 2.10.4" > Finish

4. Add sbt dependency for spark core to build.sbt

   // https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.10
   libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.5.2"

5. Refresh Maven Project

6. Update plugins.sbt to contain the assemly plugin for fat jars

   addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.12.0")

7. Add dependency confling resolution to build.sbt for the assembly plugin

   assemblyMergeStrategy in assembly := {
      case PathList("META-INF", xs @ _*) => MergeStrategy.discard
      case x => MergeStrategy.first
   }

8. Add spark code (create new Scala Object, add spark imports, create SparkContext etc.)

   import org.apache.spark.SparkContext
   import org.apache.spark.SparkConf
   
   object AuctApp {
       def main(args: Array[String]) {
           val conf = new SparkConf().setAppName("AuctionsApp")
           val sc = new SparkContext(conf)
           println("Just a parallelize: " + sc.parallelize(List(1,2,3,4)).count())
       }
   }

9. Start the sbt console and run it, input "assembly" as the target