samklr

We subscribe to the Git Featrue Branch workflow, briefly described in that link.

In practice, it works as follows:

FEATURE DEVELOPMENT

Steps to Follow:

1. Start with an updated local development branch -- by checking out the dev branch and pulling changes:
   git checkout development
git pull origin development

samklr

CREATE OR REPLACE VIEW admin.v_space_by_schema
AS
WITH CAPACITY AS
(
  SELECT SUM(capacity) FROM stv_partitions
),
USAGE AS
(
  SELECT TRIM(pgdb.datname) AS DATABASE,
         TRIM(pgn.nspname) AS SCHEMA,

samklr

Standalone Spark 2.0.0 with s3

###Tested with:

• Spark 2.0.0 pre-built for Hadoop 2.7
• Mac OS X 10.11
• Python 3.5.2

Goal

Use s3 within pyspark with minimal hassle.

samklr

SELECT TRIM(pgdb.datname) AS DATABASE,
         TRIM(pgn.nspname) AS SCHEMA,
         TRIM(a.name) AS TABLE,
         b.mbytes,
         a.rows
  FROM (SELECT db_id,
               id,
               name,
               SUM(ROWS) AS ROWS
        FROM stv_tbl_perm a

samklr

#!/usr/bin/env bash
set -x -e

JUPYTER_PASSWORD=${1:-"myJupyterPassword"}
NOTEBOOK_DIR=${2:-"s3://myS3Bucket/notebooks/"}

# home backup
if [ ! -d /mnt/home_backup ]; then
  sudo mkdir /mnt/home_backup
  sudo cp -a /home/* /mnt/home_backup

samklr

/* Terraform setup to evaluate kafka performances on various aws instances types and ebs sizes */

provider "aws" {
  region = "eu-west-1"
}

variable "ssh_key_name" {
  default = "ben@ici"
}

samklr

package no.norsk_tipping.poc.streams.util;

import org.apache.kafka.streams.TopologyDescription;

import java.io.StringWriter;
import java.util.stream.Stream;

/**
 *
 */

samklr

## Consumer Throughput: Single consumer thread, no compression
## Consumer Throughput: 3 consumer thread, no compression

bin/kafka-consumer-perf-test.sh --topic benchmark-3-3-none \
--zookeeper kafka-zk-1:2181,kafka-zk-2:2181,kafka-zk-3:2181 \
--messages 15000000 \
--threads 1

samklr

PSQL

Magic words:

psql -U postgres

Some interesting flags (to see all, use -h):

• -E: will describe the underlaying queries of the \ commands (cool for learning!)
• -l: psql will list all databases and then exit (useful if the user you connect with doesn't has a default database, like at AWS RDS)

samklr

Installing Postgres via Brew

Pre-Reqs

Brew Package Manager

In your command-line run the following commands:

1. brew doctor
2. brew update