Skip to content

Instantly share code, notes, and snippets.

View bigsnarfdude's full-sized avatar

BigsnarfDude bigsnarfdude

285 followers · 266 following
View GitHub Profile
@bigsnarfdude
bigsnarfdude / gist:2d81595460859966d0d3
Created August 25, 2015 22:22
custom java scala buffer
// this section rewrites the each file byte by byte back to new zip file
while (e.hasMoreElements()) {
val entryIn = e.nextElement().asInstanceOf[ZipEntry]
if (!entryIn.getName.equalsIgnoreCase("abc.txt")) {
zos.putNextEntry(entryIn)
val is = zipFile.getInputStream(entryIn)
val buf = Array.ofDim[Byte](10)
var len: Int = 0
@bigsnarfdude
bigsnarfdude / gist:0c29a729d0dce6b0caae
Created May 13, 2014 17:38
Scala Algebird for IP Address count attack traffic
case class OutsideServer(val ipAddress: String, val LoginsPerMinute: Int) extends Ordered[OutsideServer] {
def compare(that: OutsideServer): Int = {
val count = this.LoginsPerMinute - that.LoginsPerMinute
if (count == 0) this.ipAddress.compareTo(that.ipAddress) else count
}
}
case class InsideServer(val ipAddress: String, val LoginsPerMinute: Int) extends Ordered[InsideServer] {
def compare(that: InsideServer): Int = {
val count = this.LoginsPerMinute - that.LoginsPerMinute
@bigsnarfdude
bigsnarfdude / gist:bd3f7b59e49d4c555bb0
Created June 6, 2014 03:07
HashTag Twitter Streamer
from tweepy import Stream
from tweepy import OAuthHandler
from tweepy.streaming import StreamListener
consumer_key = "HsYoJa0sdf980as8fbFh0w"
consumer_secret = "k698asdf9870sdf09a7s8df908a7sdf6A7m4"
access_token = "364asdf987asdfuiytsdaf765sdfasdfoiyLgS"
access_token_secret = "SFasdf9807asd09f87asf9087asdfbik5D"
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
@bigsnarfdude
bigsnarfdude / gist:992070c464c021644494
Created June 12, 2014 19:20
Dems alotta jars in Apache Spark
[info] Including: scala-library-2.10.4.jar
[info] Including: netty-all-4.0.17.Final.jar
[info] Including: jetty-server-8.1.14.v20131031.jar
[info] Including: javax.servlet-3.0.0.v201112011016.jar
[info] Including: jetty-continuation-8.1.14.v20131031.jar
[info] Including: jetty-http-8.1.14.v20131031.jar
[info] Including: jetty-io-8.1.14.v20131031.jar
[info] Including: jetty-util-8.1.14.v20131031.jar
[info] Including: jetty-plus-8.1.14.v20131031.jar
[info] Including: javax.transaction-1.1.1.v201105210645.jar
@bigsnarfdude
bigsnarfdude / gist:03c5b513fdcc4aed9233
Created June 13, 2014 17:19
spark jar list
'[info] Including: JavaEWAH-0.6.6.jar\n',
'[info] Including: ST4-4.0.4.jar\n',
'[info] Including: activation-1.1.jar\n',
'[info] Including: akka-actor_2.10-2.2.3-shaded-protobuf.jar\n',
'[info] Including: akka-remote_2.10-2.2.3-shaded-protobuf.jar\n',
'[info] Including: akka-slf4j_2.10-2.2.3-shaded-protobuf.jar\n',
'[info] Including: akka-zeromq_2.10-2.2.3-shaded-protobuf.jar\n',
'[info] Including: algebird-core_2.10-0.1.11.jar\n',
'[info] Including: ant-1.9.0.jar\n',
'[info] Including: ant-launcher-1.9.0.jar\n',
@bigsnarfdude
bigsnarfdude / SparkSQLexperiments.ipynb
Created July 1, 2014 05:03
Load Spark SQL from File, JSON file, or arrays
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@bigsnarfdude
bigsnarfdude / gist:45bd47e17b7daae60bed
Created July 20, 2014 23:13
docker
FROM ubuntu:14.04
MAINTAINER test@test.com
# need to run to make sure the apt-get installs work
RUN apt-get update
RUN apt-get install -y wget
WORKDIR /tmp
@bigsnarfdude
bigsnarfdude / gist:9ae01a7df8ffa10b5016
Created August 1, 2014 19:16
weather.scala video recreation "scala-in-action" talk
import org.apache.http.client.methods.HttpGet
object Weather {
private val locations = List(
"Berlin,Germany",
"Cologne,Germany",
"Hamburg,Germany",
"Munich,Germany",
"Nuremberg,Germany",
@bigsnarfdude
bigsnarfdude / gist:55b3b6c394f923bcbe66
Created August 3, 2014 02:54
problem with algebird 0.7.0 jar from maven repo
antigensAir:bin antigen$ ./scala -cp /Users/antigen/Downloads/algebird-core_2.10-0.7.0.jar
Welcome to Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_60).
Type in expressions to have them evaluated.
Type :help for more information.
scala> import com.twitter.algebird._
import com.twitter.algebird._
scala> import HyperLogLog._
import HyperLogLog._
@bigsnarfdude
bigsnarfdude / gist:c48dff7c79228b6180f3
Created August 3, 2014 02:56
algebird 0.7.0 REPL script
import com.twitter.algebird._
import HyperLogLog._
import com.twitter.algebird.Monoid
import com.twitter.algebird.DecayedValue
import com.twitter.algebird.Operators._
val hll = new HyperLogLogMonoid(4)