Skip to content

Instantly share code, notes, and snippets.

View Richard-Mathie's full-sized avatar

Richard Mathie Richard-Mathie

View GitHub Profile
@Richard-Mathie
Richard-Mathie / partition_iterator.py
Created November 8, 2018 17:33
Python Iterator Partitioner
import itertools
def partition(items, predicate=int, n=2):
""" Partition an iterable of `items` into `n` streams by `predicate`
"""
def filterer(i, tee):
return (item for pred, item in tee if pred == i)
pred_items = ((predicate(item), item) for item in items)
teed_items = enumerate(itertools.tee(pred_items, n))
return (filterer(x, t) for x, t in teed_items)
@Richard-Mathie
Richard-Mathie / kafka_worker.py
Created October 31, 2018 16:28
Kafka Consumer for Long Running Process
from threading import Thread, Event
from queue import Queue, Empty
from kafka import KafkaConsumer, KafkaProducer
from kafka.structs import TopicPartition, OffsetAndMetadata
from kafka.consumer.fetcher import ConsumerRecord
from kafka.errors import CommitFailedError
class KafkaWorker(object):
def __init__(self, topics, hosts, poll_timeout=1):
@Richard-Mathie
Richard-Mathie / config.yml
Last active October 15, 2018 16:55
CircleCI Docker Caching
version: 2.1
default_working_dir: &working_container
working_directory: /app
docker:
- image: docker:18.06.1-ce-git
default_steps:
steps:
- setup_remote_docker
@Richard-Mathie
Richard-Mathie / .gitignore
Last active July 12, 2018 11:08
Flask FTP Binary Stream
credentials.py
venv/
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@Richard-Mathie
Richard-Mathie / README.md
Last active January 26, 2018 11:02
Real Time Joins

Real Time Sorts and Joins

oh dir is there anything important in here?

@Richard-Mathie
Richard-Mathie / Redis Cluster Flask Cache.md
Last active June 12, 2017 14:10
Redis Cluster plugin for Flask Cache

Flask Cache Redis Cluster Backend

This impliments a redis cluster backend for Flask-Cache as the standard redis cache cannot handle a redis cluster.

Usage

class Config(object):
@Richard-Mathie
Richard-Mathie / Cassandra Concurrent writer.md
Last active March 29, 2017 10:43
Cassandra Concurrent writer

Cassandra Concurrent Writer

A class which constructs a concurrent writer object so that you can write async to cassandra, but limit the number of concurrent writes to some number, and block when we hit that limit. This can result in improved write throuput withough having to construct your query with genorators, as the default cassandra concurrent method requires.

I would sugest adding callbacks to handle write time outs, as you may want to retry the write.

Usage

@Richard-Mathie
Richard-Mathie / Redis Cluster Setup with Docker Swarm.md
Last active May 11, 2022 12:58
Redis cluster setup with docker swarm

Redis Cluster Setup with Docker Swarm

Setup

./redis.sh

Test

test the redis cluster

@Richard-Mathie
Richard-Mathie / Changing Labels on aws docker engine.md
Last active April 15, 2016 09:46
Changing Labels on aws docker engine launched by docker machine

First ssh into the node you want to add lable to

docker-machine ssh aws-node

Depending on wheather the machine is using upstart or systemd you will need to modify a config file:

systemd:

sudo vim /etc/systemd/system/docker.service

edit the line starting ExecStart to add label