Skip to content

Instantly share code, notes, and snippets.

View asolera's full-sized avatar

Andrew Solera asolera

  • Secretaria Municipal de Direitos Humanos e Cidadania
  • São Paulo
View GitHub Profile
@asolera
asolera / msteams_notification.py
Last active January 14, 2021 15:26
Airflow (2.0.0+) DAG example that sends notifications to MS Teams using Webhooks
# Example of sending MS Teams Notifications on DAG failure (Airflow 2.0.0+)
import requests
from airflow.decorators import dag, task
from airflow.operators.python import task, get_current_context
from airflow.utils.dates import days_ago
from airflow.models import DagRun
# from airflow.models import Variable
@asolera
asolera / async.py
Created January 26, 2021 18:14
Airflow 2.0.0 Async HTTP Request with aiohttp DAG example
import aiohttp
import asyncio
from airflow.decorators import dag, task
from airflow.operators.python import task
from datetime import date, datetime, timedelta
async def async_main():
async with aiohttp.ClientSession() as session:
async with session.get('http://httpbin.org/get') as resp:
@asolera
asolera / concurrency.py
Last active January 29, 2021 20:11
Parallel (async) HTTP requests with configurable concurrency using Python/aiohttp/asyncio
import aiohttp
import nest_asyncio
nest_asyncio.apply()
from random import randint
class HttpClient:
def __init__(self, session, concurrency):
self.__session = session
self.concurrency = asyncio.Semaphore(concurrency)
@asolera
asolera / docker-compose.yml
Created February 16, 2021 20:33
Airflow 2.0.1 with Docker Compose and performance fixes
version: '3'
x-airflow-common:
&airflow-common
image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.0.1-python3.8}
environment:
&airflow-common-env
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow
@asolera
asolera / docker-compose.yml
Created February 16, 2021 20:55
PostgreSQL 13 with all logs enabled in docker compose
version: '3'
services:
postgres:
image: postgres:13.2
ports:
- 5432:5432
environment:
POSTGRES_USER: example
POSTGRES_PASSWORD: example
@asolera
asolera / auth.py
Last active February 17, 2021 18:09
Generating Cloudtrax API authorization/signature header with Python (client-side)
from time import time
import hmac
import hashlib
import uuid
def get_authorization_header(key: str) -> str:
nonce = 'YOUR_NONCE_PREFIX' + str(uuid.uuid4())
authorization = "key=" + key + ",timestamp=" + str(int(time())) + ",nonce=" + nonce
return authorization
@asolera
asolera / url_encode.sh
Created March 19, 2021 20:23
URL Encode with Shell Script
#!/usr/bin/env bash
#
# Credits to Meleu @ https://meleu.sh/urlencode/
# Usage: urlencode "#" => will output "%23"
urlencode() {
local LC_ALL=C
local string="$*"
local length="${#string}"
local char
@asolera
asolera / lambda_function.py
Created March 23, 2021 18:13
Start AWS EC2 Instances with Lambda
import boto3
import time
# Enter the region your instances are in, e.g. 'us-east-1'
region = 'us-east-1'
# Enter your instances here: ex. ['X-XXXXXXXX', 'X-XXXXXXXX']
instances = ['i-0ab7xdx71x8xeff33']
def lambda_handler(event, context):
ec2 = boto3.client('ec2', region_name=region)
ec2.start_instances(InstanceIds=instances)
@asolera
asolera / lambda_function.py
Created March 23, 2021 18:14
Stop AWS EC2 Instances with Lambda
import boto3
import time
# Enter the region your instances are in, e.g. 'us-east-1'
region = 'us-east-1'
# Enter your instances here: ex. ['X-XXXXXXXX', 'X-XXXXXXXX']
instances = ['i-03c24cxbd9x10ex51']
def lambda_handler(event, context):
ec2 = boto3.client('ec2', region_name=region)
ec2.stop_instances(InstanceIds=instances)
@asolera
asolera / tag.sh
Created March 25, 2021 14:47
Tagging AWS Cloudwatch Alarms via AWS CLI
#!/bin/bash
aws cloudwatch tag-resource --resource-arn arn:aws:cloudwatch:us-east-1:012345678901:alarm:Alarm-Name --tags Key=Example,Value=OK Key=Another,Value=Okay