Skip to content

Instantly share code, notes, and snippets.

@kangks
kangks / ec2TagCheck.py
Created May 19, 2017 21:01
Lambda function triggered by CloudTrail events to check for tagging, and notify the owner through SES if tagging not found
import boto3
required_keys = [ "key01", "key02", "key03", "key04" ]
ses_source = 'source@maildomain.com'
ses_destination = ['somebody@domain.com']
def lambda_handler(event, context):
if 'detail' in event and 'instance-id' in event['detail']:
ec2_instance_id = event['detail']['instance-id']
@kangks
kangks / postgres96Ec2.json
Last active January 17, 2019 09:25
AWS Cloudformation for Ec2 with Postgres 96
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "Create Linux EC2 with postgresql96",
"Metadata": {
"Comment": "Postgres 9.6 on a r3.8xlarge + 20G EBS in a privateSubnet, and pgbench in publicSubnet"
},
"Parameters": {
"Ec2KeyNameParam": {
"Description": "Name of an existing EC2 KeyPair to enable SSH access to the instance",
"Type": "AWS::EC2::KeyPair::KeyName",
@kangks
kangks / aws_emr.template.json
Created May 24, 2017 23:27
AWS CloudFormation template to create EMR-5.1.0 cluster
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "Create AMR Cluster",
"Parameters": {
"ProductParam": {
"Description": "The EMR Product tag",
"Type": "String",
"Default": "aProductParam",
"MinLength": "3",
"MaxLength": "50",
@kangks
kangks / S3-Lambda-Kinesis.yaml
Created June 9, 2017 04:22
Stack for S3 event -> Lambda transform from csv to json -> kinesis stream -> downstream Lambda for each row processing. These are the purpose of the demo: - Retry of downstream Lambda when failed. The record in Kinesis Stream will not be removed until success Runtime - With large CSV, the Lambda will parallel according to number of shards of Kin…
AWSTemplateFormatVersion: '2010-09-09'
Metadata:
Comment: "Stack for S3 event -> Lambda transform from csv to json -> kinesis stream -> downstream Lambda for each row processing.
These are the purpose of the demo:
- Retry of downstream Lambda when failed. The record in Kinesis Stream will not be removed until success Runtime
- With large CSV, the Lambda will parallel according to number of shards of Kinesis Stream"
Resources:
IAMRole:
Type: "AWS::IAM::Role"
Properties:
import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
import json
import requests
from pyspark import sql
from uuid import UUID
@kangks
kangks / python37_installation.ipynb
Created January 5, 2020 05:48
Python 3.7 installation Jupyter Notebook
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@kangks
kangks / persistent_client.py
Last active May 29, 2020 01:41
AWS IoT Persistent client test
from datetime import datetime
import time
import threading
import json
import logging
import paho.mqtt.client as mqtt
import paho.mqtt.publish as mqtt_publish
import boto3
subscriber_keepalive_sec = 40 * 60 # 40 mins
@kangks
kangks / create_aws_iot_ota.py
Created June 29, 2020 03:47
Simple Python 3 script to create an AWS IoT OTA job
import sys, argparse
import boto3
import pathlib
from pathlib import Path
import random
parser = argparse.ArgumentParser(description='Script to start OTA update')
parser.add_argument("--aws_profile", help="Profile name created using aws configure", default="default", required=False)
parser.add_argument("--s3bucket", help="S3 bucket to store firmware updates", required=True)
parser.add_argument("--s3prefix", help="S3 bucket prefix to store firmware updates", required=False)
!pip install --upgrade mxnet-cu$(ls /usr/local/cuda/lib64/libcudart.so.*.? | sed -e 's/.*\.\([0-9]*\.[0-9]*\)/\$1/g' -e 's/\.//g')
@kangks
kangks / gist:49aaf9c273ddefce68fba75ade80231a
Created October 5, 2020 00:47
AWS Lambda getting public IP address
import json
import urllib3
def lambda_handler(event, context):
return {
'statusCode': 200,
'body': get_public_ip()
}
def get_public_ip():