rupakg

ETL job processing with Serverless, Lambda, and AWS Redshift

Build an ETL job service by fetching data from a public API endpoint and dumping it into an AWS Redshift database.

Published on Feb 20th, 2018 at https://serverless.com/blog/etl-job-processing-with-serverless-lambda-and-redshift/

One of the big use cases of using serverless is ETL job processing: dumping data into a database, and possibily visualizing the data.

In this post, I'll go over the process step by step. We'll build a serverless ETL job service that will fetch data from a public API endpoint and dump it into an AWS Redshift database. The service will be scheduled to run every hour, and we'll visualize the data using Chart.io.

rupakg

AWSTemplateFormatVersion: 2010-09-09
Transform: AWS::Serverless-2016-10-31
Description: Video processing app with AWS Fargate using Stackery
Resources:
  objectStore6A12D98C:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Sub ${AWS::StackName}-objectstore6a12d98c
    Metadata:
      StackeryName: rg-stackery-video-files

rupakg

AWSTemplateFormatVersion: "2010-09-09"
Outputs:
  DeploymentHistoryId:
    Description: "Stackery Deployment History ID"
    Value: "4148"
Parameters:
  StackeryEnvironmentTagName:
    Default: development
    Type: String
    Description: "Environment Name (injected by Stackery at deployment time)"

rupakg

// define
const delay = ms => new Promise(res => setTimeout(res, ms))

// call
await delay(5000)

rupakg

Writing an Event-driven Serverless Application with Full Local Development Experience

Learn how to write an event-driven serverless application with full local development experience using the Serverless Application Platform.

Published on Sept. 12th, 2017 at https://serverless.com/blog/event-driven-serverless-app-local-dev-exp/

In my previous post on Anatomy of a Serverless Application, I lay the foundation for building a very simple application with an email service using the Serverless Framework, deployed to AWS Lambda.

In this post, we will build mailman, an event-driven serverless application. The application has a simple frontend using curl that calls into a couple of backend services: a users service and an email service. The post will highlight event-driven application development with focus on full local development experience. We will

rupakg

# You don't need Fog in Ruby or some other library to upload to S3 -- shell works perfectly fine
# This is how I upload my new Sol Trader builds (http://soltrader.net)
# Based on a modified script from here: http://tmont.com/blargh/2014/1/uploading-to-s3-in-bash

S3KEY="my aws key"
S3SECRET="my aws secret" # pass these in

function putS3
{
  path=$1

rupakg

Go Software and Tools

A curated list of awesome Go storage projects and libraries

Web and Middleware

1. Negroni: Web middleware in Go.
2. Martini: Package for quickly writing modular web applications/services in Golang. Sinatra for Go.
3. Gorilla Mux: URL router and dispatcher for Go.

rupakg

# This marathon.json deploys cadvisor to each node and exposes it on port 3002.

cadvisor/marathon.json
{
  "id": "cadvisor",
  "cpus": 0.1,
  "mem": 100,
  "disk": 0,
  "instances": 8,                // equal to number of agent nodes you have
  "constraints": [["hostname", "UNIQUE"]],

rupakg

#!/bin/bash 
# A simple script to backup an organization's GitHub repositories.

# NOTE: if you have more than 100 repositories, you'll need to step thru the list of repos 
# returned by GitHub one page at a time, as described at https://gist.github.com/darktim/5582423

GHBU_BACKUP_DIR=${GHBU_BACKUP_DIR-"github-backups"}                  # where to place the backup files
GHBU_ORG=${GHBU_ORG-"<CHANGE-ME>"}                                   # the GitHub organization whose repos will be backed up
                                                                     # (if you're backing up a user's repos instead, this should be your GitHub username)
GHBU_UNAME=${GHBU_UNAME-"<CHANGE-ME>"}                               # the username of a GitHub account (to use with the GitHub API)

rupakg

#start mongodb
 mongod run --config /usr/local/Cellar/mongodb/1.4.3-x86_64/mongod.conf

#launch mongodb automatically on logon
launchctl load -w ~/Library/LaunchAgents/org.mongodb.mongod.plist