This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
docker/image:name |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
-- | |
-- PostgreSQL database dump | |
-- | |
-- Dumped from database version 15.2 | |
-- Dumped by pg_dump version 15.3 (Homebrew) | |
SET statement_timeout = 0; | |
SET lock_timeout = 0; | |
SET idle_in_transaction_session_timeout = 0; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import os | |
from kafka import KafkaProducer, KafkaConsumer | |
BOOTSTRAP_SERVERS=os.gentenv("KAFKA_BOOTSTRAP_SERVERS").split(",") | |
TOPIC_NAME="the-topic" | |
SASL_USERNAME=os.gentenv("KAFKA_SASL_USERNAME") | |
SASL_PASSWORD=os.gentenv("KAFKA_SASL_PASSWORD") | |
def consume(): | |
consumer = KafkaConsumer(TOPIC_NAME, security_protocol="SASL_SSL", sasl_mechanism="SCRAM-SHA-512", sasl_plain_username=SASL_USERNAME, sasl_plain_password=SASL_PASSWORD, bootstrap_servers=BOOTSTRAP_SERVERS) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/bin/bash | |
# requires flite, sox and ffmpeg | |
INPUT_TXT=$1 | |
OUTPUT_WAV=${INPUT_TXT/txt/wav} | |
OUTPUT_MP3=${INPUT_TXT/txt/mp3} | |
OUTPUT_WAV_FAST=${INPUT_TXT/.txt/_fast.wav} | |
OUTPUT_MP3_FAST=${INPUT_TXT/.txt/_fast.mp3} | |
flite -f "${INPUT_TXT}" -o "$OUTPUT_WAV" 2>&1 > /dev/null |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/bin/bash | |
usage="Usage: ./remove_older_hdfs_files.sh [path] [days]" | |
# use if working with incredibly large directories | |
# export HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Xmx5g" | |
if [ ! "$1" ] | |
then | |
echo $usage; | |
exit 1; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/bin/bash | |
cd /usr/hdp/current | |
# Get the current version of Hive | |
selectLine=$( hdp-select status hive-server2 ) | |
parts=( $selectLine ) | |
echo "SelectLine: ${parts[2]}" | |
version="${parts[2]}" | |
echo "Version: ${version}" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
for alb in $(aws elbv2 describe-load-balancers --query 'LoadBalancers[].LoadBalancerArn[]');do | |
DEFAULT_PORT=$(aws elbv2 describe-listeners --load-balancer-arn $alb --query 'Listeners[].{Port:Port}') | |
if [ "$DEFAULT_PORT" == "443" ];then | |
echo "ALB $alb is listening on port $DEFAULT_PORT" | |
echo "Adding listener on $alb to redirect $NEW_PORT to $DEFAULT_PORT" | |
aws elbv2 create-listener --load-balancer-arn $alb --protocol HTTP --port 80 --default-actions '[ | |
{ | |
"RedirectConfig": { | |
"Protocol": "HTTPS", | |
"Host": "#{host}", |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
-- ** Aggregate (COUNT, AVG, etc.) + Sliding time window ** | |
-- Performs function on the aggregate rows over a 10 second sliding window for a specified column. | |
-- .----------. .----------. .----------. | |
-- | SOURCE | | INSERT | | DESTIN. | | |
-- Source-->| STREAM |-->| & SELECT |-->| STREAM |-->Destination | |
-- | | | (PUMP) | | | | |
-- '----------' '----------' '----------' | |
-- STREAM (in-application): a continuously updated entity that you can SELECT from and INSERT into like a TABLE | |
-- PUMP: an entity used to continuously 'SELECT ... FROM' a source STREAM, and INSERT SQL results into an output STREAM | |
-- Create output stream, which can be used to send to a destination |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
from elasticsearch import Elasticsearch, RequestsHttpConnection | |
from requests_aws4auth import AWS4Auth | |
import elasticsearch.helpers | |
import boto3 | |
import datetime | |
import json | |
host = '' | |
region = '' | |
service = 'es' |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import boto3 | |
from datetime import datetime, timedelta | |
regions = [ | |
"ap-northeast-1", "ap-northeast-2", "ap-south-1", "ap-southeast-1", | |
"eu-central-1", "eu-west-1", "eu-west-2", "us-east-1", "us-east-2", | |
"us-west-1", "us-west-2" | |
] | |
for region in regions: |
NewerOlder