Skip to content

Instantly share code, notes, and snippets.

View diogoaurelio's full-sized avatar

daurelio diogoaurelio

27 followers · 43 following
View GitHub Profile
@diogoaurelio
diogoaurelio / pyspark_fixtures.py
Created October 21, 2018 15:13
"""
Utilities common to all tests using spark
"""
import pytest
from pyspark.sql import SparkSession
from pyspark import SparkContext, SparkConf
import logging
@diogoaurelio
diogoaurelio / db_utils_pg8000.py
Created October 24, 2018 07:50
Sample utility functions for using pg8000 to connect with Postgres
"""
Utils to interact with DB using pg8000 library
Note: assumes py3.5+
[Example usage]
# OPTIONALLY retrieve DB password from AWS SSM
import boto3
@diogoaurelio
diogoaurelio / kube_config.yaml
Created November 12, 2018 17:09
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: <certificateAuthority.data from describe-cluster>
server: <endpoint from describe-cluster>
name: <cluster-name>
contexts:
- context:
cluster: <cluster-name>
user: aws
@diogoaurelio
diogoaurelio / min-char-rnn.py
Created November 21, 2018 21:45 — forked from karpathy/min-char-rnn.py
Minimal character-level language model with a Vanilla Recurrent Neural Network, in Python/numpy
"""
Minimal character-level Vanilla RNN model. Written by Andrej Karpathy (@karpathy)
BSD License
"""
import numpy as np
# data I/O
data = open('input.txt', 'r').read() # should be simple plain text file
chars = list(set(data))
data_size, vocab_size = len(data), len(chars)
@diogoaurelio
diogoaurelio / install_aws_nuke.sh
Created February 8, 2019 10:41
Install AWS-nuke utility script for Ubuntu
export AWS_NUKE_VERSION=2.7.0
sudo apt-get install -y wget
wget https://github.com/rebuy-de/aws-nuke/releases/download/v$AWS_NUKE_VERSION/aws-nuke-v$AWS_NUKE_VERSION-linux-amd64.tar.gz --no-check-certificate
tar xvf aws-nuke-v$AWS_NUKE_VERSION-linux-amd64.tar.gz
chmod +x aws-nuke-v$AWS_NUKE_VERSION-linux-amd64
sudo mv aws-nuke-v$AWS_NUKE_VERSION-linux-amd64 /usr/local/bin/aws-nuke
# test it
aws-nuke --help
@diogoaurelio
diogoaurelio / aws_nuke_sample_config_nuke_everything_except.yaml
Created February 8, 2019 16:47
AWS nuke sample config nuke everything except a given IAM resource
regions:
- eu-west-1
account-blacklist:
- "999999999999" # production
resource-types:
@diogoaurelio
diogoaurelio / aws_nuke_sample_config_nuke_only_target.yaml
Last active February 8, 2019 17:17
AWS nuke sample config nuke some specific targets
regions:
- eu-west-1
account-blacklist:
- "999999999999" # production
resource-types:
# only nuke these three resources
targets:
@diogoaurelio
diogoaurelio / jupyter_notebook_cloudera_setup.sh
Last active January 15, 2020 11:21
Bash script to setup python jupyter notebook with Cloudera Spark2
#!/usr/bin/env bash
set -x -e
JUPYTER_PASSWORD=${1:-myJupyterPassword}
PYTHON_VERSION=${2:-3.5}
test -d ~/venv_notebooks || python -m virtualenv venv_notebooks --python=python$PYTHON_VERSION
. venv_notebooks/bin/activate
@diogoaurelio
diogoaurelio / terraform_validate_wrong_resources.tf
Last active March 21, 2021 16:25
terraform_validate_resources
# terraform validate will catch typo in resource reference
resource "aws_s3_bukcet" "wrong_resource" {
name = "my-bucket"
}
# terraform validate will catch wrong CIDR
resource "aws_vpc" "default" {
cidr_block = "0.0.0.0/0"
}
@diogoaurelio
diogoaurelio / terratest_aws_example_skelleton.go
Created March 22, 2021 15:51
terratest basic skelleton
package tests
import (
"github.com/gruntwork-io/terratest/modules/aws"
"github.com/gruntwork-io/terratest/modules/terraform"
"github.com/stretchr/testify/assert"
"testing"
)
func TestTerraformAwsEnvironment(t *testing.T) {