Skip to content

Instantly share code, notes, and snippets.

View meconlin's full-sized avatar

Mark Conlin meconlin

12 followers · 13 following
View GitHub Profile
@meconlin
meconlin / es cluster yaml
Last active May 23, 2016 15:26
---
- name: provision 3 instances for ES
hosts: localhost
connection: local
gather_facts: false
strategy: free
vars:
instance_name: "bcd2-elasticsearch"
aws_region: "{{ gg_aws_region }}"
tasks:
@meconlin
meconlin / standalone python s3 log script for zabbix
Created April 15, 2016 17:46
#!/usr/bin/env python
import os
import sys
from datetime import datetime, timedelta
import boto
import yaml
@meconlin
meconlin / s3 log check for zabbix shell
Last active April 19, 2016 17:42
#!/bin/bash
#
# USAGE:
# <script>.sh <bucketname> <level> <appname>
# ./logs.sh mybucket stage myapp
#
# Use DEBUG flag to run locally and see debug print lines
# ~/.s3cfg should be set correctly for this to work in non DEBUG mode
# date -d doesnt work on mac osx so watch out for that
#
@meconlin
meconlin / gist:f2925a7a35f9bfdbac19
Created January 19, 2016 13:58
opsworks : django-setup : logs from successful run
```
[2016-01-18T18:32:40+00:00] INFO: AWS OpsWorks instance 4d017ba9-378f-46db-886d-fb6e8cbcf300, Agent version 3431-20151222151215
[2016-01-18T18:32:40+00:00] INFO: Starting chef-zero on port 8889 with repository at repository at /opt/aws/opsworks/current
One version per cookbook
data_bags at /var/lib/aws/opsworks/data/data_bags
nodes at /var/lib/aws/opsworks/data/nodes
[2016-01-18T18:32:40+00:00] INFO: Forking chef instance to converge...
[2016-01-18T18:32:41+00:00] INFO: *** Chef 11.10.4 ***
[2016-01-18T18:32:41+00:00] INFO: Chef-client pid: 23428
@meconlin
meconlin / gist:d2d3780a6f1917094120
Created October 16, 2015 15:06
AWS SWF : kill stuck workflow executions
require 'rubygems'
require 'aws-sdk'
require 'pry'
class SwfManager
def client
credentials = Aws::SharedCredentials.new(profile_name: 'default')
swf = Aws::SWF::Client.new(
region: 'us-east-1',
@meconlin
meconlin / gist:eb0fc77c9cdedd760a7e
Created September 21, 2015 15:25
aws datapipeline create, upload, init cli example
aws datapipeline create-pipeline --name vin-reference-unzip --unique-id vin-reference-unzip
# returned : df-0757905EU1V52FPMH4U
aws datapipeline put-pipeline-definition --pipeline-id df-0757905EU1V52FPMH4U --pipeline-definition file:///Users/mark/workspace/buycheck_etl/aws-datapipeline/dataone-unzip-vin-reference.json
aws datapipeline activate-pipeline --pipeline-id df-0757905EU1V52FPMH4U
aws datapipeline list-pipelines
@meconlin
meconlin / unzip-csv-awk-aws-datapipeline.json
Created September 21, 2015 15:24
aws data pipeline unzip from s3 -> to s3 filtering csv via awk
{
"objects": [
{
"directoryPath": "#{myS3OutputLoc}/",
"name": "S3OutputLocation",
"id": "S3OutputLocation",
"type": "S3DataNode"
},
{
"period": "1 day",
@meconlin
meconlin / gist:aec0405da73cac6a436a
Created August 21, 2015 01:35
AWS SWF : node launch of a simple workflow
// -- ------------------------------------------
// SWF - sending work to a workflow
//
var swf = new AWS.SWF();
var params = {
domain: 'FTPToS3Domain', /* required */
workflowId: 'STRING_VALUE', /* required */
workflowType: { /* required */
name: 'FTPToS3WorkFlow.run', /* required */
version: '1.7' /* required */
@meconlin
meconlin / gist:5a66c5532578aaf9a73e
Last active August 29, 2015 14:27
AWS SNS node sdk make it happen
var AWS = require('aws-sdk')
var sns = new AWS.SNS();
var inner_message = { name:"mymessage", other:"this is other" };
inner_message = JSON.stringify(inner_message);
var payload =
{
"default": inner_message,
};
payload = JSON.stringify(payload);
@meconlin
meconlin / email template example
Created August 4, 2015 15:40
Mallorys email template example
```
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<title>Confirm Your CarLingo Account</title>
<meta name="viewport" content="width=device-width, initial-scale=1.0"/>