Skip to content

Instantly share code, notes, and snippets.

View travishegner's full-sized avatar

Travis Hegner travishegner

View GitHub Profile
@travishegner
travishegner / VectorSum.scala
Created April 22, 2019 19:16
User Defined Aggregate Function: Vector Sum
package com.trilliumstaffing.hadoop.tools.udaf
import org.apache.spark.sql.expressions.{MutableAggregationBuffer, UserDefinedAggregateFunction}
import org.apache.spark.sql.types.{DataType, StructField, StructType}
import org.apache.spark.ml.linalg.SQLDataTypes.VectorType
import org.apache.spark.ml.linalg.{Vector, Vectors}
import breeze.linalg.{Vector => BV}
import org.apache.spark.sql.Row
class VectorSum extends UserDefinedAggregateFunction {
@travishegner
travishegner / HBase.scala
Created June 15, 2018 13:22
HBase scala object for writing and deleting from spark
package com.trilliumstaffing.hadoop.tools
import java.nio.ByteBuffer
import org.apache.hadoop.hbase.{HBaseConfiguration, TableName}
import org.apache.hadoop.hbase.client._
import org.apache.spark.sql.Dataset
import org.apache.spark.ml.linalg.{Vector, Vectors}
import org.apache.hadoop.hbase.protobuf.ProtobufUtil
import org.apache.hadoop.hbase.util.{Base64, Bytes}
@travishegner
travishegner / app.json
Created February 5, 2016 13:35
Sample app.json
{
"id":"/test",
"instances":1,
"cpus":1,
"mem":1024,
"container":{
"type":"DOCKER",
"docker":{
"network":"BRIDGE",
"image":"ubuntu:14.04",
@travishegner
travishegner / stderr
Created February 3, 2016 13:44
mesos-docker-executor logs -- successful launch
I0202 08:18:27.692706 7254 docker.cpp:614] Running docker -H unix:///var/run/docker.sock run --cpu-shares 102 --memory 536870912 -e MARATHON_APP_VERSION=2016-01-29T16:42:17.101Z -e HOST=node3.example.com -e MARATHON_APP_RESOURCE_CPUS=0.1 -e MARATHON_APP_DOCKER_IMAGE=registry.example.com/irc:e536abc322f21cf946cb9b9d0b86c1cfb2068dac -e PORT_10001=31847 -e MESOS_TASK_ID=irc.72170899-c9af-11e5-b589-02427fa617b6 -e PORT=31847 -e MARATHON_APP_RESOURCE_MEM=512.0 -e PORTS=31847 -e MARATHON_APP_RESOURCE_DISK=0.0 -e MARATHON_APP_LABELS= -e MARATHON_APP_ID=/irc -e PORT0=31847 -e MESOS_SANDBOX=/mnt/mesos/sandbox -e MESOS_CONTAINER_NAME=mesos-853e5e6e-d937-41c6-913d-48cc0a0270e3-S0.1eddc1e5-f53b-40dd-9355-e9c79c5a4044 -v /tmp/mesos/slaves/853e5e6e-d937-41c6-913d-48cc0a0270e3-S0/frameworks/3b552a3b-c098-40d6-af33-ed59ec2cd54f-0000/executors/irc.72170899-c9af-11e5-b589-02427fa617b6/runs/1eddc1e5-f53b-40dd-9355-e9c79c5a4044:/mnt/mesos/sandbox --net bridge --net=vx_0x10009 --name mesos-853e5e6e-d937-41c6-913d-48cc0a0270e3-S0
@travishegner
travishegner / haproxy2apache
Last active May 6, 2022 15:44
A quick bash one liner to convert haproxy logs to something like apache logs so that logstalgia can read them
tail -f log | stdbuf -o0 awk '{print $9" "$6" - - "$7" "$18" "$19" "$20" "$11" "$12}' | sed -u "s/:[0-9]\{4,5\}//g" | sed -u "s/\(:[0-6][0-9]\)\.[0-9]\{3\}/\\1 -400/g" | logstalgia --sync --paddle-mode vhost