samklr

Enforcing invariants in Scala datatypes

Scala provides many tools to help us build programs with less runtime errors. Instead of relying on nulls, the recommended practice is to use the Option type. Instead of throwing exceptions, Try and Either types are used for representing potential error scenarios. What’s common with these features is that they’re used for capturing runtime features in the type system, thus lifting the runtime scenario handling to the compilation phase: your program doesn’t compile until you’ve explicitly handled nulls, exceptions, and other runtime features in your code.

In his “Strategic Scala Style” blog post series,

samklr

Kafka Cheat Sheet

Display Topic Information

$ kafka-topics.sh --describe --zookeeper localhost:2181 --topic beacon
Topic:beacon	PartitionCount:6	ReplicationFactor:1	Configs:
	Topic: beacon	Partition: 0	Leader: 1	Replicas: 1	Isr: 1
	Topic: beacon	Partition: 1	Leader: 1	Replicas: 1	Isr: 1

samklr

version: '3'
services:

  master:
    image: gettyimages/spark:2.2.1-hadoop-2.7
    command: bin/spark-class org.apache.spark.deploy.master.Master
    ports:
      - 4040:4040
      - 6066:6066
      - 7077:7077

samklr

from __future__ import print_function

import json
import boto3
from kafka import KafkaProducer
import urllib
import ssl
import logging

root = logging.getLogger()

samklr

TODO

implement security measures
git config
config files
full sublimetext config
set up openvpn
rdesktop and network drive to terra
set up evolution
RStudio

samklr

sudo apt-get install -y python-software-properties debconf-utils
sudo add-apt-repository -y ppa:webupd8team/java
sudo apt-get update
echo "oracle-java8-installer shared/accepted-oracle-license-v1-1 select true" | sudo debconf-set-selections
sudo apt-get install -y oracle-java8-installer

samklr

apt-get install -y software-properties-common
apt-add-repository -y ppa:webupd8team/java
apt-get update
echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | /usr/bin/debconf-set-selections
apt-get install -y oracle-java8-installer

samklr

We subscribe to the Git Featrue Branch workflow, briefly described in that link.

In practice, it works as follows:

FEATURE DEVELOPMENT

Steps to Follow:

1. Start with an updated local development branch -- by checking out the dev branch and pulling changes:
   git checkout development
git pull origin development

samklr

ZK="ZK_SERVER1:2181,ZK_SERVER2:2181,ZK_SERVER3:2181"
sum=0
for i in $(kafka-topics --list --zookeeper $zookeeper ); do 
  count=$(kafka-topics.sh  --describe --zookeeper $zookeeper  --topic $i |grep Leader | wc -l);
 
 sum=`expr $sum + $count` ; 
 
 echo 'total partitions is ' $sum; done

samklr

mysqldump -f --max_allowed_packet=2M --skip-extended-insert \
          -u jobteaser -p main_prod -h debezium.cqvkb5fch8kh.eu-west-1.rds.amazonaws.com  >  dbz_dump.sql