Hammond95

preBootstrapCommands: &preboot
  - "export DOCKER_SECRET='my-docker-secret'"
  - "export DISK_WITH_RAID=''"
  - "export DISK_MOUNT=''"
  - |
    #!/bin/bash -x

    # Writing Docker Secret
    if [[ ! -z "${DOCKER_SECRET}" ]]; then
      secret=$(aws --region $AWS_REGION secretsmanager get-secret-value --secret-id ${DOCKER_SECRET} --query SecretString --output text)

Hammond95

How to install 'logstash-input-ganglia' on an EMR cluster

Since I have wasted a lot of time trying to make this work, I just wanted to post this gist to help anyone having issues to setup the plugin.

What you need

• An AWS account
• A running cluster on EMR with Ganglia installed.
• An elasticsaerch cluster running.

Steps

Hammond95

#! /bin/bash
# Note: run from /var/lib/ganglia directory, no need for subdir
# Run with: find /var/lib/ganglia/rrds/ -type d -exec sh -c 'cd "{}" ; /var/lib/ganglia/rrd_exporter.sh ;' \;
mkdir -p /tmp/ganglia-export/xml/

RUNDATE="$(date --date='now' +%F-%R)"
BACKUPFILE="/tmp/ganglia-export/rrds-${RUNDATE}.tar"
EXPORTFILE="/tmp/ganglia-export/xml-export-${RUNDATE}.tar"

# backup current rrd files -- tar all and place in /tmp

Hammond95

#!/bin/bash
commit_hash=$(git rev-parse HEAD | head -c8)
text="*[DEPLOY] - ($commit_hash) I have made a new deploy for Project! :bomb:*"
token="YOUR_TOKEN"
slackhost="YOUR_SLACK_HOST"
channel="CHANNEL_NAME"
escapedText=$(echo "$text" | sed 's/"/\"/g' | sed "s/'/\'/g" )
json="{\"channel\": \"#$channel\", \"text\": \"$escapedText\",\"icon_emoji\":\":apple:\",\"username\":\"My Bot\"}"

# Example of complex data

Hammond95

#!/bin/bash

SCRIPTDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

COMMITS_VERSIONS=$(git log --follow -- "${SCRIPTDIR}/../VERSION" | grep -E '^commit' | sed 's#commit ##g')
NEXT=$(echo -e "$COMMITS_VERSIONS" | head -1)
COMMITS_VERSIONS=$(echo -e "$COMMITS_VERSIONS" | tail -n+2)

for commit in $COMMITS_VERSIONS; do
    VERSION=$(git show ${NEXT}:../pycuebiq/VERSION)

Hammond95

def explode_from_fields(df, target_columns: list, separator: str):
    """ df = dataframe to split,
        target_columns = list of the columns containing the values to split,
                         if the elements returned from the split are not the
                         same for each columns, the shorter ones are extended
                         to the longest.
        separator = the symbol used to perform the split
        returns: A dataframe with each entry for the target column separated, 
                 with each element moved into a new row.
                 The values in the other columns are duplicated across the newly divided rows."""