Skip to content

Instantly share code, notes, and snippets.

View kppullin's full-sized avatar

Kevin Pullin kppullin

View GitHub Profile
@kppullin
kppullin / consumer-offset-parser.py
Last active January 7, 2022 00:27
Python Kafka __consumer_offset binary parser/formatter
# Port of `kafka/core/src/main/scala/kafka/coordinator/group/GroupMetadataManager.scala -> OffsetsMessageFormatter`
# `key` and `value` are the raw byte arrays from the `__consumer_offsets` topic.
import io
import struct
key_reader = io.BytesIO(key)
value_reader = io.BytesIO(value)
key_version = struct.unpack('>h', key_reader.read(2))[0]
@kppullin
kppullin / Cargo.toml
Created January 11, 2021 23:16
actix app data - avoiding double arc
[package]
name = "actix-data--no-double-arc"
version = "0.1.0"
edition = "2018"
[dependencies]
actix-rt = "1.1.1"
actix-web = { version = "3.3.2" }
futures = "0.3.8"
tokio = { version = "0.2.24", features = ["macros"] }
@kppullin
kppullin / config.fish
Created May 16, 2020 03:51
Fish shell + WSL2 + gnome-keyring / secret-tool
#
# This fish config sets up a working `gnome-keyring` on WSL2.
# I imagine it will work with WSL1 as well, perhaps after adjusting the `DISPLAY` value.
#
# Based off this bash script: https://askubuntu.com/questions/815327/running-gnome-keyring-on-wsl-windows-subsystem-for-linux
# Tested and working with `aws-vault` and `jetbrains-toolbox`.
#
# Be sure your x server is running!!!
set -x DISPLAY (cat /etc/resolv.conf | grep nameserver | awk '{print $2}'):0
@kppullin
kppullin / kafka-dump.sh
Last active November 15, 2023 13:48
Simple script to dump all kafka topics to json files
#!/usr/bin/env bash
# Overview:
# 1) Get list of all topic names
# 2) Iterate list, dumping each one to json
# Pre-reqs:
# 1) kafkacat
# 2) jq
@kppullin
kppullin / airflow-k8s-executor-minikube-helm.md
Last active September 12, 2022 19:47
Airflow w/ kubernetes executor + minikube + helm

Overview

The steps below bootstrap an instance of airflow, configured to use the kubernetes airflow executor, working within a minikube cluster.

This guide works with the airflow 1.10 release, however will likely break or have unnecessary extra steps in future releases (based on recent changes to the k8s related files in the airflow source).

Prerequisites

  • Docker installed
  • Minikube installed and started

Keybase proof

I hereby claim:

  • I am kppullin on github.
  • I am kppullin (https://keybase.io/kppullin) on keybase.
  • I have a public key whose fingerprint is 79FB 133B E75F 0522 9015 E180 26C4 ED7F 265F 0846

To claim this, I am signing this object:

@kppullin
kppullin / multiline-java-string-formatter.cs
Last active August 29, 2015 14:13
Create multiline Java string using C# + Linqpad
var s = @"SELECT *
FROM TABLE
WHERE ID = @ID
";
var sb = new StringBuilder();
var lines = Regex.Split(s, "\n");
for (int i = 0; i < lines.Length; ++i)
{
@kppullin
kppullin / ec2-ephemeral-swapon.sh
Created July 8, 2013 05:33
EC2 Ephemeral Swap Helper The intent here is to initialize and active a swap partition from the local ephemeral disk, if any. Generally using an ephemeral disk is faster than EBS and you don't incur EBS IOPS overhead (read: cost) for swappy systems.
#!/bin/bash
#
# Create a swap partition equal to the system RAM
# on the first ephemeral disk.
#
function activate_swap_on_device {
swap_on=`swapon -s | grep 'dev' | wc -l`
if [[ ${swap_on} -gt 0 ]]; then
echo "At least one swap partition is already active. Doing nothing (we assume zero or one only)"
@kppullin
kppullin / gist:5234495
Created March 25, 2013 02:09
A sample of using dynamic .NET ExpandoObjects with edge.js (https://github.com/tjanczuk/edge)
var edge = require('edge');
var testDynamic = edge.func(function() {/*
//#r "System.Core.dll"
//#r "Microsoft.CSharp.dll"
using System.Dynamic;
using System.Threading.Tasks;
public class Startup
@kppullin
kppullin / JsonNetSerializer.cs
Created November 5, 2011 22:31
Json.NET Serializer for Nancy
public class JsonNetSerializer : ISerializer
{
private Nancy.Responses.DefaultJsonSerializer _defaultSerializer = new Nancy.Responses.DefaultJsonSerializer();
public bool CanSerialize(string contentType)
{
return _defaultSerializer.CanSerialize(contentType);
}
public void Serialize<TModel>(string contentType, TModel model, Stream outputStream)