Skip to content

Instantly share code, notes, and snippets.

View VinACE's full-sized avatar
💭
I may be slow to respond.

VinACE

💭
I may be slow to respond.
6 followers · 24 following
View GitHub Profile
@VinACE
VinACE / 1.txt
Created September 21, 2014 13:48
DASI_PROJECT_PROPOSAL
Empty_test_file
@VinACE
VinACE / Teardown a framework in mesos,shutdown
Created November 11, 2016 09:37
curl -d@/tmp/post.txt -X POST http://your_mesos:5050/master/shutdown
vi /tmp/post.txt
frameworkId=23423-23423-234234-234234
@VinACE
VinACE / pyspark cassandra
Last active December 6, 2016 07:25
git clone https://github.com/TargetHolding/pyspark-cassandra.git
http://www.scala-sbt.org/release/docs/Installing-sbt-on-Linux.html
https://medium.com/@amirziai/running-pyspark-with-cassandra-in-jupyter-2bf5e95c319#.xhvqljmy6
./bin/spark-submit --packages com.datastax.spark:spark-cassandra-connector_2.11:1.5.0-M2 ../Main/Code/myPysparkFile.py
@VinACE
VinACE / python_streaming
Created November 25, 2016 06:48
https://community.cloudera.com/t5/Hadoop-101-Training-Quickstart/Python-Streaming-ImportError-No-module-named-streaming/td-p/42804
@VinACE
VinACE / pyspark_interactive_with_datastax_connector.py
Created November 25, 2016 07:10 — forked from spaceshipoperator/pyspark_interactive_with_datastax_connector.py
after successfully doing this http://planetcassandra.org/blog/kindling-an-introduction-to-spark-with-cassandra/ with spark-shell, one might reasonably think that the following would work....though I may be missing something trivial.
## invoking pyspark as follows: ##
# /path/to/spark-1.2.0-bin-hadoop2.4/bin/pyspark --jars /path/to/spark-1.2.0-bin-hadoop2.4/jars/spark-cassandra-connector-assembly-1.2.0-SNAPSHOT.jar
# first, stop the spark context launched by pyspark to avoid the conflict
sc.stop()
from py4j.java_gateway import java_import
from pyspark import SparkConf
conf = (SparkConf()
.setMaster("local")
@VinACE
VinACE / python_dataframe
Created November 25, 2016 14:39
http://takwatanabe.me/snippets/cs-pyspark.html#sql-in-pyspark
@VinACE
VinACE / df2json.py
Created November 28, 2016 07:27 — forked from mikedewar/df2json.py
A little script to convert a pandas data frame to a JSON object. Is there a better way?
"""
tiny script to convert a pandas data frame into a JSON object
"""
import ujson as json
import pandas
import numpy as np
df = pandas.DataFrame({
"time" : [1,2,3,4,5],
@VinACE
VinACE / gist:f13164c33d16735579572bb955067bf8
Last active December 8, 2016 09:52
Kafka test streaming with python
/home/centos/spark-2.0.2-bin-hadoop2.7/bin/spark-submit --packages org.apache.spark:spark-streaming-kafka_2.10:1.6.3 --conf spark.cassandra.connection.host=10.0.0.12 --master mesos://10.0.0.11:5050 testKafkastream.py 10.0.0.11:2181 ihealth
/home/centos/spark-2.0.2-bin-hadoop2.7/bin/spark-submit --packages com.datastax.spark:spark-cassandra-connector_2.11:2.0.0-M3,TargetHolding/pyspark-cassandra:0.1.5,org.apache.spark:spark-streaming-kafka-0-8_2.11:2.0.1 --conf spark.cassandra.connection.host=10.0.0.12 v_example.py ihealth
@VinACE
VinACE / jupyter_pyspark
Last active December 16, 2016 09:23
Jupyter_pyspark Links
@VinACE
VinACE / spark_frame_work tear_down
Last active December 19, 2016 07:18
sudo curl -d@/tmp/post.txt -X POST http://your_mesos:5050/master/shutdown
/tmp/post.txt is a file with the follow content:
frameworkId=23423-23423-234234-234234