- Probabilistic Data Structures for Web Analytics and Data Mining : A great overview of the space of probabilistic data structures and how they are used in approximation algorithm implementation.
- Models and Issues in Data Stream Systems
- Philippe Flajolet’s contribution to streaming algorithms : A presentation by Jérémie Lumbroso that visits some of the hostorical perspectives and how it all began with Flajolet
- Approximate Frequency Counts over Data Streams by Gurmeet Singh Manku & Rajeev Motwani : One of the early papers on the subject.
- [Methods for Finding Frequent Items in Data Streams](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.187.9800&rep=rep1&t
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
package forma; | |
import forma.WholeFileInputFormat; | |
import cascading.scheme.Scheme; | |
import cascading.tap.Tap; | |
import cascading.tuple.Fields; | |
import cascading.tuple.Tuple; | |
import cascading.tuple.TupleEntry; | |
import java.io.IOException; | |
import org.apache.hadoop.mapred.JobConf; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/bin/bash | |
condition="" | |
fs="\t" | |
while getopts c:F: OPT; do | |
case $OPT in | |
c ) condition=$OPTARG;; | |
F ) fs=$OPTARG;; | |
esac |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
package com.ebay.scoobitest | |
import edu.berkeley.cs.avro.marker._ | |
import edu.berkeley.cs.avro.runtime._ | |
import com.nicta.scoobi.Scoobi._ | |
case class LongRec(var f1: Long) extends AvroRecord | |
case class Cluster(var firstSearchTime: Long) extends AvroRecord |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/bin/sh | |
# one way (older scala version will be installed) | |
# sudo apt-get install scala | |
#2nd way | |
sudo apt-get remove scala-library scala | |
wget http://www.scala-lang.org/files/archive/scala-2.11.4.deb | |
sudo dpkg -i scala-2.11.4.deb | |
sudo apt-get update |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Check java version | |
JAVA_VER=$(java -version 2>&1 | sed 's/java version "\(.*\)\.\(.*\)\..*"/\1\2/; 1q') | |
if [ "$JAVA_VER" -lt 18 ] | |
then | |
# Download jdk 8 | |
echo "Downloading and installing jdk 8" | |
wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8-b132/jdk-8-linux-x64.rpm" | |
# Silent install |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#! /bin/bash | |
# Batch Convert Script by StevenTrux | |
# The Purpose of this Script is to batch convert any video file to mp4 or mkv format for chromecast compatibility | |
# this script only convert necessary tracks if the video is already | |
# in H.264 format it won't convert it saving your time! | |
# Put all video files need to be converted in a folder! | |
# the name of files must not have " " Space! | |
# Rename the File if contain space |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/bin/bash | |
# You must accept the Oracle JDK License Update | |
# https://www.oracle.com/java/technologies/javase-downloads.html | |
# usage: get_oracle_jdk_x64.sh <jdk_version> <platform> <ext> | |
# jdk_version: 14 | |
# platform: linux or osx or windows | |
# ext: rpm or dmg or tar.gz or exec | |
jdk_version=${1:-14} |
Adapted from here
- Install all packages
sudo apt-get update
sudo apt-get install php5 php5-pgsql php5-fpm php-apc php5-curl php5-cli postgresql nginx git
- Configure PostgresSQL
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
from pyspark.sql.types import StringType | |
from pyspark.sql.functions import udf | |
maturity_udf = udf(lambda age: "adult" if age >=18 else "child", StringType()) | |
df = spark.createDataFrame([{'name': 'Alice', 'age': 1}]) | |
df.withColumn("maturity", maturity_udf(df.age)) | |
df.show() |
OlderNewer