Skip to content

Instantly share code, notes, and snippets.

Playing around with Big Data!

Maziyar Panahi maziyarpanahi

Playing around with Big Data!
View GitHub Profile
ruanbekker /
Last active Oct 26, 2021
Install a 3 Node Kubernetes Cluster on Ubuntu 16

Master: Dependencies

apt update && apt upgrade -y
curl | apt-key add -

cat <<EOF > /etc/apt/sources.list.d/kubernetes.list
deb kubernetes-xenial main
ytjohn /
Created Nov 16, 2016
quick notes on doing a kvm image from ubuntu cloud image
andypiper /
Last active Mar 27, 2021
Twitter API recipes for twurl +jq, and other useful Twitter code snippets and tools
nepsilon /
Last active Oct 11, 2021
Importing and Exporting CSV files with PostgreSQL — First published in issue #19

Importing and exporting CSV files with PostgreSQL

Let’s see how to use PostgreSQL to import and export CSV files painlessly with the COPY command.

Import CSV into table t_words:

COPY t_words FROM '/path/to/file.csv' DELIMITER ',' CSV;

You can tell quote char with QUOTE and change delimiter with DELIMITER.

View Spark_OnlineLDA_wikipedia_example.scala
import{CountVectorizer, RegexTokenizer, StopWordsRemover}
import org.apache.spark.mllib.clustering.{LDA, OnlineLDAOptimizer}
import org.apache.spark.mllib.linalg.Vector
import sqlContext.implicits._
val numTopics: Int = 100
val maxIterations: Int = 100
val vocabSize: Int = 10000
baraldilorenzo /
Last active Nov 16, 2021
VGG-16 pre-trained model for Keras

##VGG16 model for Keras

This is the Keras model of the 16-layer network used by the VGG team in the ILSVRC-2014 competition.

It has been obtained by directly converting the Caffe model provived by the authors.

Details about the network architecture can be found in the following arXiv paper:

Very Deep Convolutional Networks for Large-Scale Image Recognition
K. Simonyan, A. Zisserman
jkbradley / LDA_SparkDocs
Created Mar 24, 2015
LDA Example: Modeling topics in the Spark documentation
View LDA_SparkDocs
This example uses Scala. Please see the MLlib documentation for a Java example.
Try running this code in the Spark shell. It may produce different topics each time (since LDA includes some randomization), but it should give topics similar to those listed above.
This example is paired with a blog post on LDA in Spark:
import scala.collection.mutable
alexbowe /
Created Mar 21, 2011
Demonstration of extracting key phrases with NLTK in Python
import nltk
text = """The Buddha, the Godhead, resides quite as comfortably in the circuits of a digital
computer or the gears of a cycle transmission as he does at the top of a mountain
or in the petals of a flower. To think otherwise is to demean the Buddha...which is
to demean oneself."""
# Used when tokenizing words
sentence_re = r'''(?x) # set flag to allow verbose regexps
([A-Z])(\.[A-Z])+\.? # abbreviations, e.g. U.S.A.