Skip to content

Instantly share code, notes, and snippets.

View bdnf's full-sized avatar
Somewhere in the Clouds

Oleh Bodunov bdnf

Somewhere in the Clouds
  • TU Dresden
View GitHub Profile
@bdnf
bdnf / wc-stream.py
Created November 30, 2020 16:26
Word count from socket example using Spark in Python
from pyspark.sql.functions import *
spark = (SparkSession
.builder
.appName("PythonMnMCount")
.getOrCreate())
lines = (spark
.readStream.format("socket")
.option("host", "localhost")
@bdnf
bdnf / spark-wc-stream.scala
Last active November 30, 2020 16:27
Word count from socket using Spark in Scala
import org.apache.spark.sql.functions._
import org.apache.spark.sql.streaming._
val spark = SparkSession
.builder
.appName("MnMCount")
.getOrCreate()
val lines = spark
.readStream.format("socket")
Fetch configuration from SSM parameter store:
# /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c ssm:configuration-parameter-store-name -s
Fetch config from local path:
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m onPremise -s -c /etc/cwagentconfig/metrics.json
For check the status, run:
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a status
@bdnf
bdnf / user-data.sh
Last active October 31, 2020 16:14
Install Docker and Docker-Compose on EC2 (with User-Data script)
#! /bin/sh
yum update -y
amazon-linux-extras install docker
service docker start
usermod -a -G docker ec2-user
chkconfig docker on
sudo curl -L https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
@bdnf
bdnf / clickstream_gen.py
Created June 30, 2020 22:23
Generate clickstream events
import requests
import random
import sys
import argparse
def getClicked(rate):
if random.random() <= rate:
return True
else:
return False
@bdnf
bdnf / heartrate.sh
Created June 27, 2020 10:56
Simulate heart rate data from sensor and stream it to AWS Kinesis Stream
#!/bin/sh
while true
do
deviceID=$(( ( RANDOM % 10 ) + 1 ))
heartRate=$(jot -r 1 60 140)
echo "$deviceID,$heartRate"
aws kinesis put-record --stream-name <your_stream_name> --data "$deviceID,$heartRate"$'\n' --partition-key $deviceID --region us-east-1
done
@bdnf
bdnf / doctstrings-example.py
Created June 11, 2020 11:25
An example of proper docstrings for Python
class Pants:
"""The Pants class represents an article of clothing sold in a store
"""
def __init__(self, color, waist_size, length, price):
"""Method for initializing a Pants object
Args:
color (str)
waist_size (int)
@bdnf
bdnf / Cargo.toml
Created March 12, 2020 17:46
Steps on how to organize a library in Rust
[package]
name = "graph"
version = "0.1.0"
authors = ["me"]
edition = "2020"
[lib]
name = "service"
path = "src/lib.rs"
@bdnf
bdnf / airflow-s3-hook.py
Last active January 29, 2022 23:10
Creating an S3 hook in Apache Airflow
import datetime
import logging
from airflow import DAG
from airflow.models import Variable
from airflow.operators.python_operator import PythonOperator
from airflow.hooks.S3_hook import S3Hook
def list_keys():
@bdnf
bdnf / cloudwatch_reporter.js
Created January 30, 2020 09:38
A starter code that enables AWS Lambda to send data to Cloud Watch
const AWS = require('aws-sdk')
const axios = require('axios')
// Name of a service, any string
const serviceName = process.env.SERVICE_NAME
// URL of a service to test
const url = process.env.URL
// CloudWatch client