Skip to content

Instantly share code, notes, and snippets.

View quintona's full-sized avatar

Quinton Anderson quintona

  • Sydney Australia
View GitHub Profile
@quintona
quintona / gist:4752613
Created February 11, 2013 04:27
A test Gist, playing with a blog entry.
@SuppressWarnings({ "rawtypes", "unchecked" })
public static Stream getSentenceStream(TridentTopology topology, ITridentSpout spout){
Stream sentenceStream = null;
if(spout == null){
FixedBatchSpout fixedSpout = new FixedBatchSpout(new Fields("sentence"), 3,
new Values("the cow jumped over the moon"),
new Values("the man went to the store and bought some candy"),
new Values("four score and seven years ago"),
new Values("how many apples can you eat"),
new Values("to be or not to be the person"));
@quintona
quintona / gist:5385839
Last active December 16, 2015 05:39
Issue getting started with graphhopper
@Test
public void test() {
Date start = new Date();
//GraphHopper gh = new GraphHopper().forServer(); //<- this works, line below doesn't
GraphHopper gh = new GraphHopper().forServer().contractionHierarchies(true);
gh.load("./london.osm");
Date end = new Date();
System.out.println("Load Time: " + (end.getTime() - start.getTime()));
GHRequest request = new GHRequest(51.430896,0.115356,
51.564266,-0.219727);
@quintona
quintona / OuterJoinReducer
Created May 11, 2013 03:25
A way of doing an outer join of 2 separate streams, in storm trident. Use an outer join reducer. Here is the code for the reducer and associated state. Simply use topology.multiReduce(s1, s2, function, outputFields).
package storm.cookbook.tfidf.functions;
import java.util.Map;
import storm.trident.operation.MultiReducer;
import storm.trident.operation.TridentCollector;
import storm.trident.operation.TridentMultiReducerContext;
import storm.trident.tuple.TridentTuple;
import backtype.storm.tuple.Values;
@quintona
quintona / Instructions
Last active December 19, 2015 01:29
Follow the instructions to add NAT functionality to the OpenVPN for AWS VPC
In order to
* paste the text from the file in the gist: sudo vim /usr/local/sbin/configure-pat.sh
* Make it executable: sudo chmod u+x /usr/local/sbin/configure-pat.sh
* Edit /etc/rc.local: "sudo vim /etc/rc.local" and add the following line: "/usr/local/sbin/configure-pat.sh"
* Reboot the instance
@quintona
quintona / Aggregate Composition
Created November 25, 2013 10:37
Implementation of the aggregation test from first principles using a basic Monoid
trait M[A] {
def op( a1: A, a2: A): A
def zero: A
}
val sumIntMonoid = new M[Int]{
val zero = 0
def op(a1:Int, a2:Int) = a1 + a2
}
@quintona
quintona / gist:7670373
Created November 27, 2013 03:42
Simple example calculating the gini in R
calc.gini <- function(y)
{
num.pos <- sum(y==1)
num.neg <- sum(y==0)
tpr <- cumsum(y==1)/num.pos
fpr <- cumsum(y==0)/num.neg
area <- (tpr[-1] + tpr[1:(length(tpr)-1)]) %*% (fpr[-1] - fpr[1:(length(fpr)-1)])
1 - as.numeric(area)
}
@quintona
quintona / gist:8879309
Created February 8, 2014 09:10
The work around to get qibuild working on my Mac
There is a pathing issue with qiBuild. when I run qibuild configure I always get the following error:
Current build worktree: xxxx
Using toolchain: mytoolchain
Build type: Debug
[ERROR]: Could not find qibuild cmake framework path
Please file a bug report with the details of your installation
When I run the command in debug mode I see some paths that don't match my system:
Build type: Debug
@quintona
quintona / Build.scala
Created March 3, 2014 11:39
Build.scala
import sbt._
import Keys._
import sbtassembly.Plugin._, AssemblyKeys._
object build extends Build {
type Sett = Project.Setting[_]
lazy val standardSettings: Seq[Sett] =
Defaults.defaultSettings ++ Seq[Sett](
@quintona
quintona / build.scala
Created March 19, 2014 22:20
Very nice little hack that Laurence put together to change the way one can package dependencies with assembly. This is potential useful for only hadoop based deployments
val IncludeInLib = Configurations.config("include-lib")
def addSpecifiedDependenciesToLibInAssembly = inConfig(IncludeInLib)(Defaults.configSettings) ++ Seq(
(libraryDependencies in IncludeInLib) := Nil, // needed otherwise scala library gets included.
(assembledMappings in assembly) <<= (libraryDependencies in IncludeInLib, externalDependencyClasspath in IncludeInLib, assembledMappings in assembly) map ((specified, managed, mappings) => {
@quintona
quintona / SomeMain.scala
Created March 20, 2014 12:20
Simpler Tool for scalding that doesn't use reflection
object OpsMain {
def main(args : Array[String]) {
hadoop.util.ToolRunner.run(new hadoop.mapred.JobConf, new Tool[XXX]({args => new XXX(args)}), args)
}
}