Skip to content

Instantly share code, notes, and snippets.

View csbond007's full-sized avatar

Kaustav Saha csbond007

2 followers · 0 following
View GitHub Profile
@csbond007
csbond007 / subject109_heart_beat_model_params.py
Created November 28, 2016 19:58
MODEL_PARAMS = \
{ 'aggregationInfo': { 'days': 0,
'fields': [],
'hours': 0,
'microseconds': 0,
'milliseconds': 0,
'minutes': 0,
'months': 0,
'seconds': 0,
'weeks': 0,
@csbond007
csbond007 / Numenta on CentOS 7
Last active November 9, 2016 02:57
///////////////////////////////////////////////////////////////////////
sudo yum group install "Development Tools"
sudo yum install python-devel.x86_64
wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
sudo rpm -ivh mysql-community-release-el7-5.noarch.rpm
sudo yum update
sudo yum install mysql-server
sudo systemctl start mysqld
@csbond007
csbond007 / Numenta_Installation_Attempt
Created November 4, 2016 22:17
Youtube : https://www.youtube.com/watch?v=rN-57iBvcT4
Resource Link : https://github.com/numenta/nupic
python --version
sudo apt-get install python2.7-dev
curl https://bootstrap.pypa.io/get-pip.py | sudo python
@csbond007
csbond007 / Spark_Shuffle_Fetch_Error
Created October 31, 2016 22:51
org.apache.spark.shuffle.FetchFailedException: Failed to connect to mesos102.itp.objectfrontier.com:45636
collectAsMap at DecisionTree.scala:1043
org.apache.spark.shuffle.FetchFailedException: Failed to connect to mesos102.itp.objectfrontier.com:45636
at org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:323)
at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:300)
at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:51)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
@csbond007
csbond007 / spark_working_sbt
Created October 25, 2016 18:44
name := "SampleApp"
version := "1.0"
scalaVersion := "2.10.6"
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.2" % "provided"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.6.2" % "provided"
@csbond007
csbond007 / spark_error_logs
Created October 25, 2016 01:10
[ksaha@mesos101 SampleApp]$ spark-submit --class "SampleApp" --master mesos://zk://10.10.40.138:2181/mesos --jars lib/spark-cassandra-connector-1.6.1-s_2.10.jar,lib/cassandra-driver-core-3.1.1.jar, target/scala-2.10/sampleapp_2.10-1.0.jar
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
16/10/24 21:07:33 INFO SparkContext: Running Spark version 1.6.2
16/10/24 21:07:33 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/10/24 21:07:33 INFO SecurityManager: Changing view acls to: ksaha
16/10/24 21:07:33 INFO SecurityManager: Changing modify acls to: ksaha
16/10/24 21:07:33 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(ksaha); users with modify permissions: Set(ksaha)
16/10/24 21:07:34 INFO Utils: Successfully started service 'sparkDriver' on port 46646.
16/10/24 21:07:34 INFO Slf4jLogger: Slf4jLogger started
@csbond007
csbond007 / spark_initial_working_code
Last active October 24, 2016 23:49
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import com.datastax.spark.connector._
object SampleApp {
def main(args: Array[String]) {
@csbond007
csbond007 / spark_working_logs
Created October 24, 2016 23:37
[ksaha@mesos101 SampleApp]$ spark-submit --class "SampleApp" --master mesos://zk://10.10.40.138:2181/mesos --jars lib/spark-cassandra-connector-1.6.1-s_2.10.jar,lib/cassandra-driver-core-3.1.1.jar, target/scala-2.10/sampleapp_2.10-1.0.jar
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
16/10/24 19:34:25 INFO SparkContext: Running Spark version 1.6.2
16/10/24 19:34:25 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/10/24 19:34:25 INFO SecurityManager: Changing view acls to: ksaha
16/10/24 19:34:25 INFO SecurityManager: Changing modify acls to: ksaha
16/10/24 19:34:25 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(ksaha); users with modify permissions: Set(ksaha)
16/10/24 19:34:25 INFO Utils: Successfully started service 'sparkDriver' on port 34208.
16/10/24 19:34:26 INFO Slf4jLogger: Slf4jLogger started
@csbond007
csbond007 / Spark_Scala_SBT_Eclipse_Cassandra_Connectivity
Last active October 24, 2016 18:21
Reference : http://www.nodalpoint.com/development-and-deployment-of-spark-applications-with-scala-eclipse-and-sbt-part-1-installation-configuration/
Make a Directory spark_sbt_eclipse_cassandra
SBT Installation : http://www.scala-sbt.org/0.13/docs/Installing-sbt-on-Linux.html
curl https://bintray.com/sbt/rpm/rpm | sudo tee /etc/yum.repos.d/bintray-sbt-rpm.repo
sudo yum install sbt
///////////////////////////////////////////////////////////////////////////////////////
@csbond007
csbond007 / Spark_Cassandra_Contectivity
Last active October 17, 2016 20:20
bin/spark-shell --conf spark.cassandra.connection.host=127.0.0.1 --packages datastax:spark-cassandra-connector:2.0.0-M2-s_2.11
import com.datastax.spark.connector._
///////////////////////////////////////////////////
//// Cassandra Table creation
CREATE KEYSPACE test WITH REPLICATION = {'class': 'SimpleStrategy', 'replication_factor': 1 };
CREATE TABLE test.words (word text PRIMARY KEY, count int);
INSERT INTO test.words (word, count) VALUES ('foo', 20);