akoumjian

// The Kafka documentation makes it very confusing to set up plain text SASL authentication while also using TLS / SSL.
// MAKE SURE THE KEYSTORE YOU ARE USING ON THE KAFKA CLUSTER IS BUILT WITH RSA ALGO, OTHERWISE GO CAN'T TALK TO JAVA OVER TLS / SSL
package main

import (
	"crypto/tls"
	"fmt"
	"github.com/Shopify/sarama"
)

akoumjian

"""
This utility lets you instrument a function with New Relic even when it's in a third party library and called as a subprocess.

This is necessary because New Relic refuses to send data when the agent is initialized in a parent PID.
If the subprocess function is directly part of your code, you can simply wrap your code in something that initializes
and shuts down the agent.

Usage:

from instrument import subprocess_wrapper

akoumjian

"""
Google makes it difficult to verify if local files match the ones on their cloud storage.
This snippet takes a file path to run the CRC32C hashing and then base64 encode it so it matches the results
from the google-cloud-storage python blob api

You will need pip install:
  - google-cloud-storage
 
"""
import google_crc32c

akoumjian

sudo su -

# stuff we need to build from source
apt-get install libpcre3-dev build-essential libssl-dev

# get the nginx source
cd /opt/
wget http://nginx.org/download/nginx-0.8.54.tar.gz
tar -zxvf nginx*
# we'll put the source for nginx modules in here

akoumjian

akoumjian

import boto3


def s3_object_needs_restore(bucket, key):
    s3_client = boto3.client("s3")
    head_response = s3_client.head_object(Bucket=bucket, Key=key)
    """
    The Head Response will include an `x-amz-archive-status` value if the object
    is in INTELLIGENT_TIERING and is currently archived
    """

akoumjian

Boostrapping Salt on Linux EC2 with Cloud-Init

Salt is a great tool for remote execution and configuration management, however you will still need to bootstrap the daemon when spinning up a new node. One option is to create and save a custom AMI, but this creates another resource to maintain and document.

A better method for Linux machines uses Canonical's CloudInit to run a bootstrap script during an EC2 Instance initialization. Cloud-init takes the user_data string passed into a new AWS instance and runs it in a manner similar to rc.local. The bootstrap script needs to:

1. Install Salt with dependencies
2. Point the minion to the master

akoumjian

AWS Glue surpriginsly does not support reading NDJSON out of the box. You have to create a custom classifier, but it is incredibly simple.

Use the pattern $[*] with a custom JSON classifier and it should read the NDJSON schema more or less correctly.

NOTE: Editing classifiers doesn't seem to work well, or they are denormalized in the crawler. Instead, delete and create new classifiers when making adjustments.

akoumjian

Salt Masterless Quickstart

Running a masterless salt-minion lets you use salt's configuration management for a single machine. It is also useful for testing out state trees before

akoumjian

def testing_ghost_stuff(foo):
  return foo**2