Skip to content

Instantly share code, notes, and snippets.

View mnoumanshahzad's full-sized avatar

Muhammad Nouman Shahzad mnoumanshahzad

View GitHub Profile
function _cdk_completer {
STACK_CMDS="list synthesize bootstrap deploy destroy diff metadata init context docs doctor"
if [ "$3" == "cdk" ]; then
COMPREPLY=($(compgen -W "$STACK_CMDS" $2))
elif [[ -d "cdk.out" ]] && ! [[ "$2" == "-"* ]]; then
TEMPLATES=$(ls -1 cdk.out/*.template.json | awk '{split($0,t,/\/|\./); print t[3]}')
COMPREPLY=($(compgen -W "$TEMPLATES" $2))
else
COMPREPLY=()
@brettswift
brettswift / CfnInitMetadataBuilder.ts
Last active October 2, 2022 20:36
CDK 1.10.1 - ASG Cloudformation Init example
import autoscaling = require("@aws-cdk/aws-autoscaling")
import scriptAssets = require("./CfnInitScriptAsset")
import iam = require('@aws-cdk/aws-iam')
import cdk = require('@aws-cdk/core')
/**
* Helpful context into what was built.
* Use these to get logical ID's when constructing your userdata.
*/
@ian-whitestone
ian-whitestone / notes.md
Last active March 1, 2023 01:45
Best practices for presto sql

Presto Specific

  • Don’t SELECT *, Specify explicit column names (columnar store)
  • Avoid large JOINs (filter each table first)
    • In PRESTO tables are joined in the order they are listed!!
    • Join small tables earlier in the plan and leave larger fact tables to the end
    • Avoid cross joins or 1 to many joins as these can degrade performance
  • Order by and group by take time
    • only use order by in subqueries if it is really necessary
  • When using GROUP BY, order the columns by the highest cardinality (that is, most number of unique values) to the lowest.
@mohanpedala
mohanpedala / bash_strict_mode.md
Last active July 4, 2024 12:40
set -e, -u, -o, -x pipefail explanation
@shortjared
shortjared / list.txt
Last active June 27, 2024 13:53
List of AWS Service Principals
a4b.amazonaws.com
access-analyzer.amazonaws.com
account.amazonaws.com
acm-pca.amazonaws.com
acm.amazonaws.com
airflow-env.amazonaws.com
airflow.amazonaws.com
alexa-appkit.amazon.com
alexa-connectedhome.amazon.com
amazonmq.amazonaws.com

Custom credential provider for AWS EMRFS and Spark applications

Background

Frequently, our EMR applications need to perform cross-account read/write, i.e., the cluster is created under one AWS billing account, but the data lives under another (let's call it "guest account"). Because security concerns, we cannot grant blank S3 access to the guest account. Instead, we should rely on assume-role function of AWS STS to provide ephemeral authentication for read/write transactions. The basic logic for calling STS service is not difficult, but there are some pitfalls when you want to integrate the assume-role authentication with EMRFS.

Custom credential provider

For hadoop/Spark, the authentication process is handled within the file system itself, so the application code can write to a S3 file without worrying about the underlying nitty-gritty details. EMRFS is an implementation o

FWIW: I (@rondy) am not the creator of the content shared here, which is an excerpt from Edmond Lau's book. I simply copied and pasted it from another location and saved it as a personal note, before it gained popularity on news.ycombinator.com. Unfortunately, I cannot recall the exact origin of the original source, nor was I able to find the author's name, so I am can't provide the appropriate credits.


Effective Engineer - Notes

What's an Effective Engineer?

@cahna
cahna / parse_ps_aux.py
Last active September 27, 2023 22:39
Parse the output of `ps aux` into a list of dictionaries representing the parsed process information from each row of the output.
#!/usr/bin/python
# -*- coding: utf-8 -*-
import pprint
import subprocess
def get_processes():
"""
Parse the output of `ps aux` into a list of dictionaries representing the parsed
@motleytech
motleytech / casper.md
Created June 21, 2016 01:00
Deal with casper

Unofficial Guide to JAMF Casper

This page documents the external/internal behavior of Casper.

Files

Casper installed itself into the following directories:

  • /Library/Application Support/JAMF
    • ManagementFrameworkScripts - a set of scripts that runs on certain events
      • StartupScript.sh - Activated when jamf is started
      • loginhook.sh - Activated when user login
  • logouthook.sh - Activated when user logout