This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
package com.dvidr | |
import scala.concurrent.ExecutionContext.Implicits.global | |
import akka.actor.ActorSystem | |
import akka.stream.ActorMaterializer | |
import akka.stream.scaladsl.{Sink, Source} | |
import scala.concurrent.Future | |
import scala.util.{Success, Failure} | |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
package com.dvidr | |
import scala.concurrent.ExecutionContext.Implicits.global | |
import akka.actor.ActorSystem | |
import akka.stream.ActorMaterializer | |
import akka.stream.scaladsl.{RunnableGraph, Sink, Source, Keep} | |
import scala.concurrent.Future | |
import scala.util.{Success, Failure} | |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
package com.dvidr.storm.bolt | |
import com.twitter.algebird.Monoid | |
class MyMonoid[K, V <: Numeric] extends Monoid[Map[K, V]] { | |
override def zero = Map[K, V ]() | |
override def plus(x: Map[K, V], y: Map[K, V]): Map[K, V] = { | |
val list = x.toList ++ y.toList | |
return list.groupBy ( _._1) .map { case (k,v) => k -> v.map(_._2).sum } | |
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
package com.dvidr | |
import com.twitter.algebird.{Moments, Aggregator} | |
import scala.util.Random | |
/* | |
Please refer to AliceInAggregatorLand first: | |
https://gist.github.com/johnynek/814fc1e77aad1d295bb7 | |
This is an adaption where "Alice In Wonderland" is turned |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
package com.dvidr | |
import org.apache.spark.graphx.{VertexRDD, Edge, Graph} | |
import org.apache.spark.sql._ | |
import org.apache.spark.{SparkConf, SparkContext} | |
import scala.util.Random | |
case class Chat(id: Int, name: String, talk: String) | |
case class ChatGraph(id: Int, dst: Int, replyIn: Int, name: String, talk: String) | |
case class TopChat(id: Int, name: String, talk: String, inDeg: Int, outDeg: Int) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import networkx as nx | |
import matplotlib.pyplot as plt | |
import matplotlib | |
# -------------- Get Data/Graphs ------------- | |
with open('graph-data/g1.txt', 'r') as g: | |
g1 = map(lambda x: eval(x), g.readlines()) | |
with open('graph-data/g2.txt', 'r') as g: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
from __future__ import print_function | |
""" | |
Using OpenDNS domain query activity, we retrieve 5 days | |
of queries/hour to a domain for 240+ domains (stored | |
in dns.json). We predict the number of queries in | |
the next hour using a LSTM recurrent neural network. | |
An ad hoc anomaly detection is outlined in the final | |
for loop. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/usr/bin/python | |
import random | |
shuf = random.sample(range(100000), 1000) | |
# Keep bag of k=10 elements | |
# Prefill the bag with first shuf items | |
bag = shuf[:10] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
/** | |
* gradle clean | |
* gradle build | |
* | |
* hadoop jar build/libs/pivot-table-1.0-SNAPSHOT.jar com.dvidr.PivotTable src/main/resources/pivotdata.txt src/main/resources/output | |
* | |
*/ | |
package com.dvidr; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
/** | |
* gradle clean | |
* gradle build | |
* <p> | |
* hadoop jar build/libs/mapper-lmdb-1.0-SNAPSHOT.jar com.dvidr.MapperLMDB src/main/resources/keys.txt src/main/resources/output | |
*/ | |
package com.dvidr; | |
import org.apache.hadoop.conf.Configuration; |
OlderNewer