Skip to content

Instantly share code, notes, and snippets.

View kerinin's full-sized avatar

Ryan Michael kerinin

View GitHub Profile
@kerinin
kerinin / stat.ino
Created December 6, 2012 19:48
Parsing Graphite Response
#include <Arduino.h>
String resp;
void setup() {
Serial.begin(9600);
Serial.write("\r\n");
Serial.println("-----------------------");
Serial.println("Arduino Context.IO demo");
### Keybase proof
I hereby claim:
* I am kerinin on github.
* I am kerinin (https://keybase.io/kerinin) on keybase.
* I have a public key ASCG5umtk3kRCyOUp2VwMM_doaZca1MkGktyOJrhm5Xhfwo
To claim this, I am signing this object:
// Source data, generated by some preprocessing pipeline or read from Kafka
var examples: DataStream[(Input,Target)] = null
// Algorithms to train against. An algorithm defines most of the values passed to SageMaker
// when creating training jobs and models. Algorithms can be defined statically or read from
// a live stream (ie Kafka). Algorithms have an associate "id" that can be used to train multiple
// algorithms against a single dataset.
var algorithms: DataStream[AlgorithmEvent] = null
// The `split_examples` uses a `Splitter` to partition an input stream into Training & Test datasets.
object Main {
def main(args: Array[String]) {
val env = StreamExecutionEnvironment.getExecutionEnvironment
clicks = env.readFile("myclicks.txt")
result = clicks
.map(new ClickTransformer)
.keyBy(0)
.window(EventTimeSessionWindows.withGap(Time.minutes(30L)))
var parsedEvent = JsonParser.Default.Parse<ParsedDataEventWebhook>(json);
//verify the signature to validate the webhook came from RP
var key = "CIO_SECRET"; //this should be your CIO auth secret
var encoding = BinaryStringEncoding.Utf8;
var algorithmProvider = MacAlgorithmProvider.OpenAlgorithm("HMAC_SHA256");
var contentBuffer =
CryptographicBuffer.ConvertStringToBinary(parsedEvent.checksum,
encoding);
var keyBuffer = CryptographicBuffer.ConvertStringToBinary(key, encoding);

Ryan Michael

My work emphasizes horizontal scalability, simple well-defined organizational boundaries and pervasive introspection through logging, metrics & alerting. My tools of choice are usually streams of immutable data, distributed data stores, canonical & unambiguous interface description languages and containerized runtimes

tl;dr

  • Enjoys: building performant, scalable infrastructure focused on robustness and maintainability
  • Speaks: Go, Ruby, Rust, Clojure, Python
  • Uses: Kafka, DynamoDB, Hadoop, Storm, Consul/Serf
// Code generated by protoc-gen-go.
// source: src/demo/demo.proto
// DO NOT EDIT!
/*
Package demo is a generated protocol buffer package.
It is generated from these files:
src/demo/demo.proto
package main
import (
"crypto/tls"
"crypto/x509"
"flag"
"fmt"
"log"
"time"
@kerinin
kerinin / Accounts_Kinesis.md
Created March 22, 2016 20:28
Accounts Kinesis

Accounts Kinesis

We currently need to solve a couple of problems in the mailservice:

  1. We need to react to account deletions (ie terminating ongoing processes, and preventing retries of failed processes)
  2. We need to increase processing capacity as the number of accounts being handled increases
  3. To reduce the probability of k8s scheduling conflicts, we need to minimize the resources requested by mailservice pods

The current approach to event notification is to write SQS events to signal state changes, for instance when a new account is created, the process creating the account writes an event to one of 64 "created" queues, which is consumed by the sync process. We could create a partitioned set of "deleted" queues and write deletion events into them. This approach feels brittle, as it relies on SQS events being created anywhere accounts are modified, and we're already doing this in at least two places (the API and the CIO kafka topic watcher).

@kerinin
kerinin / ResponseObject.rb
Last active December 25, 2015 23:19
Shower-thoughts on response objects
class ResponseObject
def initialize
@handlers = Hash.new { [] }
yield self
end
def do(state, *args)
@state = state
@args = args
end