This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ec2-authorize default -p 2003 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
################################################################################################# | |
## This scripts install elastic search. Please save and run this script from directory ## | |
## newly created directory. ## | |
################################################################################################# | |
#Download and elastic search RPM | |
wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.3.noarch.rpm |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/bin/bash | |
# from here: http://www.codingsteps.com/install-redis-2-6-on-amazon-ec2-linux-ami-or-centos/ | |
# and here: https://raw.github.com/gist/257849/9f1e627e0b7dbe68882fa2b7bdb1b2b263522004/redis-server | |
############################################### | |
# To use: | |
# wget https://gist.github.com/aniketbhatnagar/6576857/raw/b8618f01dc9221c05101a391e95f2291fe662364/install-redis.sh | |
# chmod 777 install-redis.sh | |
# ./install-redis.sh | |
############################################### | |
echo "*****************************************" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
object EnumeratorFun extends App { | |
var counter: Int = 0 | |
val enumerator = Enumerator.generateM[Int]({ | |
val ret: Option[Int] = if (counter < 5) { | |
counter += 1 | |
Some(counter) | |
} else None | |
Future(ret) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
D:\scala\spark\fork\spark>mvn install dependency:tree -Phadoop-2.4 -Phive -Pyarn | |
-Dyarn.version=2.4.0 -Pkinesis-asl -DskipTests=true | |
[INFO] Scanning for projects... | |
[INFO] ------------------------------------------------------------------------ | |
[INFO] Reactor Build Order: | |
[INFO] | |
[INFO] Spark Project Parent POM | |
[INFO] Spark Project Core | |
[INFO] Spark Project Bagel |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
If any of the libraryDependencies is null, you will get a following exception. This can also happen if you have used lazy vals and have used dependendencies out of order. For example: | |
val combinedDeps = Seq(d1, d2) | |
val d1 = "..." %% ".." % "." | |
val d2 = "..." %% ".." % "." | |
Will always give the following NPE because d1 is referred before it is declared. Solution to this is to always use lazy vals. | |
java.lang.NullPointerException | |
at sbt.CacheIvy$L3$$anonfun$moduleToHL$1.apply(CacheIvy.scala:156) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
If you attempt to proxy pass CDH 5 Manager using apache, you may run into following error: | |
2014-06-29 08:43:02,977 WARN [600353515@scm-web-6:csrf.CsrfRefererInterceptor@68] Rejecting request originating from x.x.x.x for http://x.x.x.x:7180/cmf/parcel/validate with referrer blah | |
To resolve the issue, just set ProxyPreserveHost to On. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import play.api.libs.iteratee.Enumerator | |
import play.api.libs.iteratee.Enumeratee | |
import play.api.libs.iteratee.Iteratee | |
def grouped[T](enumerator: Enumerator[T], groupSize: Int): Enumerator[List[T]] = { | |
val groupingIteratee = Enumeratee.take[T](groupSize) &>> Iteratee.getChunks | |
val groupingEnumeratee = Enumeratee.grouped(groupingIteratee) | |
enumerator &> groupingEnumeratee | |
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Caused by: javax.ws.rs.ProcessingException: Error closing message content input stream. | |
at org.glassfish.jersey.message.internal.EntityInputStream.close(EntityInputStream.java:159) | |
at org.glassfish.jersey.message.internal.InboundMessageContext$EntityContent.close(InboundMessageContext.java:151) | |
at org.glassfish.jersey.message.internal.InboundMessageContext.close(InboundMessageContext.java:909) | |
at com.github.dockerjava.jaxrs.connector.ApacheConnectorClientResponse.close(ApacheConnectorClientResponse.java:45) | |
at org.glassfish.jersey.client.InboundJaxrsResponse.close(InboundJaxrsResponse.java:163) | |
at com.github.dockerjava.jaxrs.util.WrappedResponseInputStream.close(WrappedResponseInputStream.java:57) | |
at com.github.dockerjava.core.async.ResultCallbackTemplate.close(ResultCallbackTemplate.java:80) | |
at com.getintent.aerospike.embedded.AerospikeContainerMonitor.run(AerospikeContainerMonitor.java:41) | |
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import org.apache.spark.SparkContext | |
import org.apache.spark.sql.{Dataset, SQLContext, SparkSession} | |
object DataSetOptBug { | |
def main(args: Array[String]): Unit = { | |
if (System.getProperty("spark.master") == null) System.setProperty("spark.master", "local[*]") | |
val sparkSession: SparkSession = SparkSession.builder.appName("AudScale Context").getOrCreate | |
val sqlContext: SQLContext = sparkSession.sqlContext | |
val sparkContext: SparkContext = sparkSession.sparkContext |
OlderNewer