holacode
holacode
/ gist:551eca4a958f2f057aa8
Last active
November 1, 2015 10:31
Reading file from s3 using spark , jars and config for sparkshell
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Adding multiple jar to spark classpath | |
| #make comma seperated jar list and give input to --jar flag | |
| ./spark-shell --jars $(echo ~/lib/*.jar | tr ' ' ',') | |
| sc.hadoopConfiguration.set("fs.s3.impl", "org.apache.hadoop.fs.s3native.NativeS3FileSystem") | |
| sc.hadoopConfiguration.set("fs.s3.awsAccessKeyId","yourID") | |
| sc.hadoopConfiguration.set("fs.s3.awsSecretAccessKey","yourAccesskey") | |
| val input = sc.textFile("s3:/pathtoyourcsv") | |
| //some custom processing |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Adding Java Melody to Grails App | |
| Plugin Url : https://grails.org/plugin/grails-melody | |
| Installation step provided does not work with grails 2.3.4 or above. | |
| This method to install plugin has been deprecated in new version of grails. | |
| grails install-plugin grails-melody |