Skip to content

Instantly share code, notes, and snippets.

View ns-mkusper's full-sized avatar
👋

Mark Kusper ns-mkusper

👋
  • Chicago
View GitHub Profile
@ns-mkusper
ns-mkusper / .lsp-docker
Created December 13, 2023 23:45 — forked from yosisa/.lsp-docker
Enable lsp-docker per project basis.
docker/image:name
--
-- PostgreSQL database dump
--
-- Dumped from database version 15.2
-- Dumped by pg_dump version 15.3 (Homebrew)
SET statement_timeout = 0;
SET lock_timeout = 0;
SET idle_in_transaction_session_timeout = 0;
@ns-mkusper
ns-mkusper / kafka_python_sasl_scram.py
Created January 19, 2023 17:19 — forked from alexlopes/kafka_python_sasl_scram.py
Kafka Python with SASL/SCRAM Authentication Example
import os
from kafka import KafkaProducer, KafkaConsumer
BOOTSTRAP_SERVERS=os.gentenv("KAFKA_BOOTSTRAP_SERVERS").split(",")
TOPIC_NAME="the-topic"
SASL_USERNAME=os.gentenv("KAFKA_SASL_USERNAME")
SASL_PASSWORD=os.gentenv("KAFKA_SASL_PASSWORD")
def consume():
consumer = KafkaConsumer(TOPIC_NAME, security_protocol="SASL_SSL", sasl_mechanism="SCRAM-SHA-512", sasl_plain_username=SASL_USERNAME, sasl_plain_password=SASL_PASSWORD, bootstrap_servers=BOOTSTRAP_SERVERS)
@ns-mkusper
ns-mkusper / txt2mp3
Last active November 25, 2022 19:47
#!/bin/bash
# requires flite, sox and ffmpeg
INPUT_TXT=$1
OUTPUT_WAV=${INPUT_TXT/txt/wav}
OUTPUT_MP3=${INPUT_TXT/txt/mp3}
OUTPUT_WAV_FAST=${INPUT_TXT/.txt/_fast.wav}
OUTPUT_MP3_FAST=${INPUT_TXT/.txt/_fast.mp3}
flite -f "${INPUT_TXT}" -o "$OUTPUT_WAV" 2>&1 > /dev/null
@ns-mkusper
ns-mkusper / remove_older_hdfs_files.sh
Last active June 6, 2022 21:24
script for removing older files in an hdfs directory
#!/bin/bash
usage="Usage: ./remove_older_hdfs_files.sh [path] [days]"
# use if working with incredibly large directories
# export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Xmx5g"
if [ ! "$1" ]
then
echo $usage;
exit 1;
@ns-mkusper
ns-mkusper / gethivejars.sh
Created January 31, 2020 16:40
Get current hive jars after HDP install
#!/bin/bash
cd /usr/hdp/current
# Get the current version of Hive
selectLine=$( hdp-select status hive-server2 )
parts=( $selectLine )
echo "SelectLine: ${parts[2]}"
version="${parts[2]}"
echo "Version: ${version}"
@ns-mkusper
ns-mkusper / http_to_https_redirect_for_albs.sh
Last active January 28, 2020 16:20
Add HTTP->HTTPS redirect for ALB's
for alb in $(aws elbv2 describe-load-balancers --query 'LoadBalancers[].LoadBalancerArn[]');do
DEFAULT_PORT=$(aws elbv2 describe-listeners --load-balancer-arn $alb --query 'Listeners[].{Port:Port}')
if [ "$DEFAULT_PORT" == "443" ];then
echo "ALB $alb is listening on port $DEFAULT_PORT"
echo "Adding listener on $alb to redirect $NEW_PORT to $DEFAULT_PORT"
aws elbv2 create-listener --load-balancer-arn $alb --protocol HTTP --port 80 --default-actions '[
{
"RedirectConfig": {
"Protocol": "HTTPS",
"Host": "#{host}",
@ns-mkusper
ns-mkusper / gist:cdbfce8e6baaddfb654dd950ee9dc686
Last active July 11, 2019 05:25
Get Sliding 60-second Owner Count Kinesis Stream Analytics
-- ** Aggregate (COUNT, AVG, etc.) + Sliding time window **
-- Performs function on the aggregate rows over a 10 second sliding window for a specified column.
-- .----------. .----------. .----------.
-- | SOURCE | | INSERT | | DESTIN. |
-- Source-->| STREAM |-->| & SELECT |-->| STREAM |-->Destination
-- | | | (PUMP) | | |
-- '----------' '----------' '----------'
-- STREAM (in-application): a continuously updated entity that you can SELECT from and INSERT into like a TABLE
-- PUMP: an entity used to continuously 'SELECT ... FROM' a source STREAM, and INSERT SQL results into an output STREAM
-- Create output stream, which can be used to send to a destination
@ns-mkusper
ns-mkusper / es_result_scanning.py
Created May 23, 2019 15:46
Minimal working example of large ElasticSearch query result grabbing in python
from elasticsearch import Elasticsearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth
import elasticsearch.helpers
import boto3
import datetime
import json
host = ''
region = ''
service = 'es'
import boto3
from datetime import datetime, timedelta
regions = [
"ap-northeast-1", "ap-northeast-2", "ap-south-1", "ap-southeast-1",
"eu-central-1", "eu-west-1", "eu-west-2", "us-east-1", "us-east-2",
"us-west-1", "us-west-2"
]
for region in regions: