Skip to content

Instantly share code, notes, and snippets.

View csbond007's full-sized avatar

Kaustav Saha csbond007

2 followers · 0 following
View GitHub Profile
@csbond007
csbond007 / spark_initial_working_code
Last active October 24, 2016 23:49
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import com.datastax.spark.connector._
object SampleApp {
def main(args: Array[String]) {
@csbond007
csbond007 / spark_error_logs
Created October 25, 2016 01:10
[ksaha@mesos101 SampleApp]$ spark-submit --class "SampleApp" --master mesos://zk://10.10.40.138:2181/mesos --jars lib/spark-cassandra-connector-1.6.1-s_2.10.jar,lib/cassandra-driver-core-3.1.1.jar, target/scala-2.10/sampleapp_2.10-1.0.jar
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
16/10/24 21:07:33 INFO SparkContext: Running Spark version 1.6.2
16/10/24 21:07:33 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/10/24 21:07:33 INFO SecurityManager: Changing view acls to: ksaha
16/10/24 21:07:33 INFO SecurityManager: Changing modify acls to: ksaha
16/10/24 21:07:33 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(ksaha); users with modify permissions: Set(ksaha)
16/10/24 21:07:34 INFO Utils: Successfully started service 'sparkDriver' on port 46646.
16/10/24 21:07:34 INFO Slf4jLogger: Slf4jLogger started
@csbond007
csbond007 / spark_working_sbt
Created October 25, 2016 18:44
name := "SampleApp"
version := "1.0"
scalaVersion := "2.10.6"
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.6.2" % "provided"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.6.2" % "provided"
@csbond007
csbond007 / Spark_Shuffle_Fetch_Error
Created October 31, 2016 22:51
org.apache.spark.shuffle.FetchFailedException: Failed to connect to mesos102.itp.objectfrontier.com:45636
collectAsMap at DecisionTree.scala:1043
org.apache.spark.shuffle.FetchFailedException: Failed to connect to mesos102.itp.objectfrontier.com:45636
at org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:323)
at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:300)
at org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:51)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371)
@csbond007
csbond007 / Numenta_Installation_Attempt
Created November 4, 2016 22:17
Youtube : https://www.youtube.com/watch?v=rN-57iBvcT4
Resource Link : https://github.com/numenta/nupic
python --version
sudo apt-get install python2.7-dev
curl https://bootstrap.pypa.io/get-pip.py | sudo python
@csbond007
csbond007 / Numenta on CentOS 7
Last active November 9, 2016 02:57
///////////////////////////////////////////////////////////////////////
sudo yum group install "Development Tools"
sudo yum install python-devel.x86_64
wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
sudo rpm -ivh mysql-community-release-el7-5.noarch.rpm
sudo yum update
sudo yum install mysql-server
sudo systemctl start mysqld
@csbond007
csbond007 / subject109_heart_beat_model_params.py
Created November 28, 2016 19:58
MODEL_PARAMS = \
{ 'aggregationInfo': { 'days': 0,
'fields': [],
'hours': 0,
'microseconds': 0,
'milliseconds': 0,
'minutes': 0,
'months': 0,
'seconds': 0,
'weeks': 0,
@csbond007
csbond007 / subject109_swarm_description.py
Created November 28, 2016 19:59
# ----------------------------------------------------------------------
# Numenta Platform for Intelligent Computing (NuPIC)
# Copyright (C) 2013, Numenta, Inc. Unless you have an agreement
# with Numenta, Inc., for a separate license for this software code, the
# following terms and conditions apply:
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU Affero Public License version 3 as
# published by the Free Software Foundation.
#
@csbond007
csbond007 / subject109_heart_beat.csv
Created November 28, 2016 20:22
We can't make this file beautiful and searchable because it's too large.
timestamp,heartbeat,V3,V4,V5,V6,V7,V8,V9,V10,V11,V12,V13,V14,V15,V16,V17,V18,V19,V20,V21,V22,V23,V24,V25,V26,V27,V28,V29,V30,V31,V32,V33,V34,V35,V36,V37,V38,V39,V40,V41,V42,V43,V44,V45,V46,V47,V48,V49,V50,V51,V52,V53,V54
datetime,int,int,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float,float
T,,C,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
2016-11-24 19:59:40.099,141,0,24.75,-9.98278,2.61759,2.46038,-10.2341,2.69055,2.45977,0.0429727,-0.042494,0.0261664,51.6354,3.95346,-11.2478,0.64893,-0.503093,-0.354745,0.447151,31.875,-0.268179,9.90328,-3.36347,-0.36191,9.84957,-3.20796,-0.0852794,0.0122102,-0.0184075,16.0398,-30.1135,32.846,0.512364,-0.741783,-0.360678,-0.239067,31.1875,10.7309,2.48629,-1.34938,10.6778,1.72467,-1.26455,0.714237,-0.235885,-0.341782,-44.
@csbond007
csbond007 / subject109_anomaly_data.csv
Created November 28, 2016 20:24
We can't make this file beautiful and searchable because it's too large.
2016-11-28 00:24:40.529,139,0,24.875,-7.12789,6.17922,0.181889,-7.11752,6.24527,0.274283,0.295609,0.639278,1.07341,34.8407,-27.1816,-32.6782,0.348372,-0.648166,0.206114,0.645008,32,2.13175,9.40628,-3.31838,2.13377,9.34543,-2.89201,0.440743,-0.348574,0.341198,10.7758,-31.9836,33.666,0.499629,-0.718336,-0.375219,-0.305901,31.3125,8.81372,3.57853,-2.8284,6.64808,5.7409,-3.82489,0.932368,-0.269509,-0.456748,-46.1014,-20.5801,11.8833,0.482755,-0.567856,0.643669,-0.328577
2016-11-28 00:24:40.629,139,0,24.875,-7.02708,6.09944,-0.162718,-7.04404,6.31996,-0.0279713,0.273202,0.670052,1.05155,35.2116,-27.4491,-32.0212,0.350709,-0.65005,0.208887,0.640941,32,1.83683,9.29661,-2.97488,2.01226,9.31575,-3.0576,0.238608,-0.286719,0.342594,10.0079,-31.8258,33.2367,0.498378,-0.72058,-0.37287,-0.305537,31.3125,11.0833,0.375515,1.71223,9.56186,11.0896,-7.78449,-0.247399,-0.0978458,0.833164,-45.4666,19.0838,3.66791,0.507424,-0.186061,0.71779,-0.439703
2016-11-28 00:24:40.728,139,0,24.875,-7.06327,6.13801,-0.124903,-7.01579,6.28937,