donigian

Building a Production Data Warehouse

So I hear business is going well...customers are flocking to your site/app...people just can't get enough? Congratulations, this is a huge step most companies don't reach.

What now? Well, business analysts, modelers, engineers, customer support and marketing want to access an authoritative, consistent, timely access to your organization's data. Perhaps we can build some beautiful dashboards to visualize KPIs for executive management.

You're in need of a Data Warehouse? Maybe a Data Lake? Where do you start? What sort of considerations & alternatives should you think about.

Turns out there are a variety of things you must take into consideration depending on your requirements. A good place to start is to interview & establish a relationship with team leads to better understand their needs. As you're going through the discovery process, here's an algorithm of the types of things you'll need to consider in your discovery phase.

What is a Data Warehouse? Data Lak

donigian

import tensorflow as tf

with tf.device('/cpu:0'):
    a_c = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a-cpu')
    b_c = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b-cpu')
    c_c = tf.matmul(a_c, b_c, name='c-cpu')

with tf.device('/gpu:0'):
    a_g = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a-gpu')
    b_g = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b-gpu')

donigian

github setup

cd project_dir
git init
git add .
git commit -am 'initial commit'
git log

git remote add origin https://github.com/donigian/Hello-World.git
# Creates a remote named "origin" pointing at your GitHub repo

donigian

Hadoop File System Commands

Syntax Overview

The basic syntax of HDFS commands is as follows:

$ hadoop fs -command [extra arguments]
For example:

$ hadoop fs -ls

donigian

Set up the submodule for the first time:
[~]$  cd ~/main/
[main]$  git submodule add git://github.com/my/submodule.git ./subm
[main]$  git submodule update --init
[main]$  git commit ./submodule -m "Added submodule as ./subm"
Fetch submodules after cloning a repository:
[~]$  git clone git://github.com/my/main.git ~/main
[~]$  cd ~/main/
[main]$  git submodule update --init
Pull upstream main repo changes and update submodule contents:

donigian

#!/bin/bash
echo "Checking for CUDA and installing."
# Check for CUDA and try to install.
if ! dpkg-query -W cuda; then
  # The 16.04 installer works with 16.10.
  curl -O http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_8.0.61-1_amd64.deb
  dpkg -i ./cuda-repo-ubuntu1604_8.0.61-1_amd64.deb
  apt-get update
  apt-get install cuda -y
fi

donigian

sudo service postgresql restart 
sudo service postgresql status
sudo service postgresql stop 
sudo service postgresql status
sudo crontab -l

tail -f  /var/log/postgresql/postgresql-9.1-main.log

/etc/services

donigian

Vagrant Cheat Sheet

Essential Commands

vagrant up - Starts your virtual machine.
vagrant status - Tells you if your virtual machine is running.
vagrant suspend - Saves the state of your virtual machine. (Similar to putting it to sleep.)
vagrant resume - Restores a suspended virtual machine. (Similar to waking it up from sleep.)
After your first vagrant up, a suspend/resume operation only takes a few seconds.
Destroying your virtual machine means that you'll need to wait through the ~20 minute boostrap process the next time that you vagrant up.

donigian

configure the python2.7 environment:

conda create -n py27 python=2.7
source activate py27
conda install notebook ipykernel
ipython kernel install


configure the python3.5 environment:

donigian

To set up a new Chef workstation:
+ download vagrant
+ download virtualbox
+ vagrant box add precise64
+ vagrant plugin install vagrant-berkshelf
+ vagrant plugin install vagrant-omnibus
+ git clone git@bitbucket.org:gpn-dev/chef-repo.git
+ vagrant up

vagrant up