bigorn0

bash-4.3# curl -4 \
               --insecure \ 
               -sSf \
               -w '\nLookup time:\t%{time_namelookup}\nConnect time:\t%{time_connect}\nAppCon time:\t%{time_appconnect}\nRedirect time:\t%{time_redirect}\nPreXfer time:\t%{time_pretransfer}\nStartXfer time:\t%{time_starttransfer}\n\nTotal time:\t%{time_total}\n' \
               -o /dev/null 
               -XPUT https://my-endpoint

bigorn0

Tracking tools and components

Clair

Coreos project : security analysis for docker images https://github.com/coreos/clair

Traefik

Moder dynamic routing written in go https://github.com/containous/traefik

bigorn0

PRO TIP: Transfer git repo without `.git` folder using tar.gz | ssh `git archive HEAD | gzip -c | ssh <host> 'tar -zxvf -'

bigorn0

Start to learn go

https://golang.org/doc/code.html

Glide Dependency management

https://github.com/Masterminds/glide

Toml config

https://github.com/BurntSushi/toml

Go by example

bigorn0

How to split/redirect part of the logs based on level

apache-spark-stderr-and-stdout

Try this in log4j.properties passed to Spark (or modify default configuration under Spark/conf)

# Log to stdout and stderr
log4j.rootLogger=INFO, stdout, stderr

bigorn0

AppArmor

https://medium.com/@lucjuggery/docker-apparmor-30-000-foot-view-60c5a5deb7b#.nj5v0if6h

bigorn0

Running spark jogs from scala code

http://henningpetersen.com/post/22/running-apache-spark-jobs-from-applications

Example out of spark test classes

// NOTE: This is an expensive operation in terms of time (10 seconds+). Use sparingly.
  private def runSparkSubmit(args: Seq[String]): Unit = {
    val sparkHome = sys.props.getOrElse("spark.test.home", fail("spark.test.home is not set!"))
    val process = Utils.executeCommand(
 Seq("./bin/spark-submit") ++ args,

bigorn0

#!/bin/sh

str_in_list() {
    str="$1"
    shift
    list="$@"
    if test "${list#*$str}" != "$list"
    then
        return 0    # $str is in $list
    else

bigorn0

import java.sql._
dataframe.coalesce("NUMBER OF WORKERS").mapPartitions((d) => Iterator(d)).foreach { batch =>
val dbc: Connection = DriverManager.getConnection("JDBCURL")
val st: PreparedStatement = dbc.prepareStatement("YOUR PREPARED STATEMENT")

batch.grouped("# Of Rows you want per batch").foreach { session =>
  session.foreach { x =>
    st.setDouble(1, x.getDouble(1)) 
    st.addBatch()
  }

bigorn0

// Restify Server CheatSheet.
// More about the API: http://mcavage.me/node-restify/#server-api
// Install restify with npm install restify


// 1.1. Creating a Server.
// http://mcavage.me/node-restify/#Creating-a-Server


var restify = require('restify');