Skip to content

Instantly share code, notes, and snippets.

Avatar

Andrew Solera asolera

  • Secretaria Municipal de Inovação e Tecnologia de São Paulo
  • São Paulo
View GitHub Profile
@asolera
asolera / docker-ami.md
Created Mar 22, 2021
Docker install on Amazon Linux 2 (AMI)
View docker-ami.md

I've created this document just in case the original one (https://gist.github.com/npearce/6f3c7826c7499587f00957fee62f8ee9) get deleted.

sudo yum update -y
sudo amazon-linux-extras install docker -y
sudo service docker start
sudo usermod -a -G docker ec2-user
sudo chkconfig docker on
sudo yum install -y git
sudo reboot
@asolera
asolera / url_encode.sh
Created Mar 19, 2021
URL Encode with Shell Script
View url_encode.sh
#!/usr/bin/env bash
#
# Credits to Meleu @ https://meleu.sh/urlencode/
# Usage: urlencode "#" => will output "%23"
urlencode() {
local LC_ALL=C
local string="$*"
local length="${#string}"
local char
@asolera
asolera / auth.py
Last active Feb 17, 2021
Generating Cloudtrax API authorization/signature header with Python (client-side)
View auth.py
from time import time
import hmac
import hashlib
import uuid
def get_authorization_header(key: str) -> str:
nonce = 'YOUR_NONCE_PREFIX' + str(uuid.uuid4())
authorization = "key=" + key + ",timestamp=" + str(int(time())) + ",nonce=" + nonce
return authorization
@asolera
asolera / docker-compose.yml
Created Feb 16, 2021
PostgreSQL 13 with all logs enabled in docker compose
View docker-compose.yml
version: '3'
services:
postgres:
image: postgres:13.2
ports:
- 5432:5432
environment:
POSTGRES_USER: example
POSTGRES_PASSWORD: example
@asolera
asolera / docker-compose.yml
Created Feb 16, 2021
Airflow 2.0.1 with Docker Compose and performance fixes
View docker-compose.yml
version: '3'
x-airflow-common:
&airflow-common
image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.0.1-python3.8}
environment:
&airflow-common-env
AIRFLOW__CORE__EXECUTOR: CeleryExecutor
AIRFLOW__CORE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
AIRFLOW__CELERY__RESULT_BACKEND: db+postgresql://airflow:airflow@postgres/airflow
@asolera
asolera / concurrency.py
Last active Jan 29, 2021
Parallel (async) HTTP requests with configurable concurrency using Python/aiohttp/asyncio
View concurrency.py
import aiohttp
import nest_asyncio
nest_asyncio.apply()
from random import randint
class HttpClient:
def __init__(self, session, concurrency):
self.__session = session
self.concurrency = asyncio.Semaphore(concurrency)
@asolera
asolera / async.py
Created Jan 26, 2021
Airflow 2.0.0 Async HTTP Request with aiohttp DAG example
View async.py
import aiohttp
import asyncio
from airflow.decorators import dag, task
from airflow.operators.python import task
from datetime import date, datetime, timedelta
async def async_main():
async with aiohttp.ClientSession() as session:
async with session.get('http://httpbin.org/get') as resp:
@asolera
asolera / msteams_notification.py
Last active Jan 14, 2021
Airflow (2.0.0+) DAG example that sends notifications to MS Teams using Webhooks
View msteams_notification.py
# Example of sending MS Teams Notifications on DAG failure (Airflow 2.0.0+)
import requests
from airflow.decorators import dag, task
from airflow.operators.python import task, get_current_context
from airflow.utils.dates import days_ago
from airflow.models import DagRun
# from airflow.models import Variable
@asolera
asolera / Dockerfile
Created Jan 11, 2021
Install Java (JDK8) inside Dockerfile from another image
View Dockerfile
FROM another/image:example
# Credits to https://stackoverflow.com/a/64062083
# some dockerfile commands...
ENV JAVA_FOLDER java-se-8u41-ri
ENV JVM_ROOT /usr/lib/jvm
ENV JAVA_PKG_NAME openjdk-8u41-b04-linux-x64-14_jan_2020.tar.gz
ENV JAVA_TAR_GZ_URL https://download.java.net/openjdk/jdk8u41/ri/$JAVA_PKG_NAME
@asolera
asolera / auth.xml
Created Dec 21, 2020
WSO2 Mediation Policy for External Authentication (before sending request) using Cookies
View auth.xml
<sequence name="ExternalAuth" statistics="enable" trace="enable" xmlns="http://ws.apache.org/ns/synapse">
<property expression="$axis2:REST_URL_POSTFIX" name="originalResource" scope="default" type="STRING"/>
<property action="remove" name="REST_URL_POSTFIX" scope="axis2"/>
<property name="BLOCKING_SENDER_PRESERVE_REQ_HEADERS" value="false"/>
<property name="HTTP_METHOD" scope="axis2" type="STRING" value="POST"/>
<callout initAxis2ClientOptions="false" serviceURL="https://api.external.example/auth?token=abcdefg">
<source type="envelope"/>
<target key="response"/>
</callout>
<property expression="$trp:Set-Cookie" name="setCookieHeader" scope="default" type="STRING"/>