joeljacobson

Data and Innovation Summit hands-on Workshop

Lab 1

CREATE SCHEMA demo_datalake_catalog.transactions;

Lab 2

CREATE TABLE demo_datalake_catalog.transactions."orders" (custkey ,orderkey , totalprice , orderdate ,orderstatus, orderyear , ordermonth) WITH (partitioned_by = ARRAY[ 'orderyear', 'ordermonth' ] , format= 'ORC') AS SELECT custkey, orderkey, totalprice,orderdate,orderstatus , extract( year from orderdate) , extract (month from orderdate) FROM tpch.tiny.orders;

joeljacobson

Setup

These instructions are for DC/OS 1.11.0 and Kubernetes package 1.0.0-1.9.3.

Service Accounts

Install the DC/OS Enterprise CLI, then create a keypair and a service account for Kubernetes. Securely store the private key in the DC/OS secrets store.

joeljacobson

Latency Comparison Numbers
--------------------------
L1 cache reference                           0.5 ns
Branch mispredict                            5   ns
L2 cache reference                           7   ns                      14x L1 cache
Mutex lock/unlock                           25   ns
Main memory reference                      100   ns                      20x L2 cache, 200x L1 cache
Compress 1K bytes with Zippy             3,000   ns        3 us
Send 1K bytes over 1 Gbps network       10,000   ns       10 us
Read 4K randomly from SSD*             150,000   ns      150 us          ~1GB/sec SSD

joeljacobson

#Use Case

A bank wants to help locate and tag all their expenses/transactions in their bank account to help them categorise their spending. The users will be able to tag any expense/transaction to allow for efficient retrieval and reporting. There will be 10 millions customers with on average 500 transactions a year. Some business customers may have up to 10,000 transactions a year. The client wants the tagged items to show up in searches in less than a second to give users a seamless experience between devices.

The client would like

1. to understand how this can be done with DSE
2. some latency examples of how they can search all of a users data
3. the user to be able to filter the queries using time as well as tags e.g. show me all shopping vs luxury in the last 3 months.
4. some assurances that peak traffic of over 10,000 writes, 4,000 reads and 1000 searches per second can be accommodated by the solution using DSE.

joeljacobson

#!/bin/bash
# Maintainer Joel Jacobson

VERSION="5.0.0"

       # to start the services you must be root
       if [ "$EUID" -ne 0 ]; then
       echo "Please run as root"
       exit 1
       fi

joeljacobson

docker rmi -f $(docker images -f "dangling=true" -q)

joeljacobson

### DML ###

# Keyspace Name
keyspace: stresscql

# The CQL for creating a keyspace (optional if it already exists)
keyspace_definition: |
  CREATE KEYSPACE stresscql WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1};

# Table name

joeljacobson

1  cd Downloads/
    2  chmod +x NVIDIA-Linux-x86_64-352.30.run 
    3  sudo ./NVIDIA-Linux-x86_64-352.30.run 
    4  cd
    5  cat /var/log/nvidia-installer.log 
    6  sudo add-apt-repository ppa:ubuntu-x-swat/x-updates
    7  sudo apt-get update
    8  sudo apt-get install nvidia-current
    9  cd Downloads/
   10  sudo ./NVIDIA-Linux-x86_64-352.30.run --no-x-check

joeljacobson

I ran into this at work this week (also on RHEL6.5). I believe the lib-device-mapper that you have isn't exporting a symbol ("Base") that Docker needs. I solved this by upgrading lib-device-mapper to version 1.02.90.

You may have to enable the public_ol6_latest repo in order to get this package.

sudo yum-config-manager --enable public_ol6_latest

And then install the package:

sudo yum install device-mapper-event-libs

joeljacobson

lang en_US
langsupport en_US
keyboard us
timezone Etc/UTC
text
install
skipx
halt

# Ridiculous URL... I know...