Skip to content

Instantly share code, notes, and snippets.

View hiimivantang's full-sized avatar

hiimivantang

1 follower · 7 following
View GitHub Profile
@hiimivantang
hiimivantang / streamConsumer.py
Last active January 14, 2016 08:03
twitter stream consumer
import tweepy
from tweepy import auth
CONSUMER_KEY="" #FIXME
CONSUMER_SECRET="" #FIXME
TOKEN_KEY="" #FIXME
TOKEN_SECRET="" #FIXME
#override tweepy.StreamListener to add logic to on_status
@hiimivantang
hiimivantang / gdax_producer.py
Created November 6, 2017 17:30
kafka producer for gdax api
import gdax
class OrderBookProducer(gdax.WebsocketClient):
def on_open(self):
self.products = ["ETH-BTC"]
self.channels = ["level2"]
def on_message(self, msg):
if msg['type'] == 'snapshot':
print(msg)
def on_close(self):
print("-- Goodbye! --")
@hiimivantang
hiimivantang / container_linux_config.yaml
Last active November 20, 2017 08:54
passwd:
users:
- name: core
ssh_authorized_keys:
- ssh-rsa xxxxx ivantang@ngpdev
networkd:
units:
- name: internet.network
contents: |
@hiimivantang
hiimivantang / keybase.md
Created June 22, 2018 08:10

Keybase proof

I hereby claim:

  • I am hiimivantang on github.
  • I am hiimivantang (https://keybase.io/hiimivantang) on keybase.
  • I have a public key ASBGFt6G5OMl1nucZmt972ZAg7IZz1No_IAxVmI8dAUUrQo

To claim this, I am signing this object:

@hiimivantang
hiimivantang / async_kafka_to_containers
Created May 15, 2020 10:50
from aiokafka import AIOKafkaConsumer
import asyncio
import random
import json
import sys
async def no_sleep(previous_attempt_number, delay_since_first_attempt_ms):
"""Don't sleep at all before retrying."""
return 0
@hiimivantang
hiimivantang / hosts_rbac_ldap_no_ssl.yml
Created November 5, 2020 08:33
You want to deploy CP 6.0 with RBAC and LDAP integration. See `<<UPDATE: ..>` in yaml file.
---
all:
vars:
#secrets_protection_enabled: true
#secrets_protection_masterkey: ""
#secrets_protection_security_file: "/usr/secrets/security.properties"
ansible_connection: ssh
ansible_user: <<UPDATE: user with ssh access>>
ansible_become: true
ansible_ssh_private_key_file: <<UPDATE: private key location>>
@hiimivantang
hiimivantang / tc_playbook.yml
Last active December 15, 2020 12:44
Ansible playbook for introducing latency
---
- name: Simulating latency
hosts: all
vars:
latency_ms: 50
packet_loss_percent: 1
network_interface: eth0
remove_qdisc: false
tasks:
- name: Install tc
@hiimivantang
hiimivantang / librdkafka_clients_sidecar.py
Created January 10, 2021 11:58
Sidecar to expose librdkafka statistics in Prometheus exposition format.
from prometheus_client import start_http_server, Metric, REGISTRY
import json
import requests
import sys
import time
class JsonCollector(object):
def collect(self):
@hiimivantang
hiimivantang / get_message_count.sh
Created February 5, 2021 16:44
Sample bash script to get total message count of a Kafka topic.
#!/bin/bash
brokers="localhost:9092"
topic=<YOUR TOPIC>
sum_1=$(/<YOUR_KAFKA_BIN_DIRECTORY>/kafka-run-class kafka.tools.GetOffsetShell --broker-list $brokers --topic $topic --time -1 | grep -e ':[[:digit:]]*:' | awk -F ":" '{sum += $3} END {print sum}')
sum_2=$(/<YOUR_KAFKA_BIN_DIRECTORY>/kafka-run-class kafka.tools.GetOffsetShell --broker-list $brokers --topic $topic --time -2 | grep -e ':[[:digit:]]*:' | awk -F ":" '{sum += $3} END {print sum}')
echo "Number of records in topic ${topic}: "$((sum_1 - sum_2))
@hiimivantang
hiimivantang / useful-kafka-commands.sh
Last active April 25, 2021 23:59
Useful kafka-* commands
############################
# kafka-topics
############################
#Prints topics with offline partitions
kafka-topics --zookeeper <HOSTNAME>:<PORT> --describe --unavailable-partitions |\
sed -r 's/^\s+Topic:\s+(.*)\s+Partition:.*$/\1/'
#Prints offline partitions
kafka-topics --zookeeper <HOSTNAME>:<PORT> --describe --unavailable-partitions |\