nxbdi

LAB

This lab is part of a 'Sql on Hadoop' webinar. The recording and slides can be found here

Purpose

How/when to use Hive vs Phoenix vs SparkSQL

nxbdi

Hbase indexing to solr in HDP 2.3

• Background:

The HBase Indexer provides the ability to stream events from HBase to Solr for near real time searching. The HBase indexer is included with HDPSearch as an additional service. The indexer works by acting as an HBase replication sink. As updates are written to HBase, the events are asynchronously replicated to the HBase Indexer processes, which in turn creates Solr documents and pushes them to Solr.

• References:
  • http://doc.lucidworks.com/hdpsearch23/JobJarGuide.html#_hbase_indexer
  • https://github.com/NGDATA/hbase-indexer/wiki/Tutorial

nxbdi

Phoenix/Spark demo

• These demo is part of a webinar on 'HBase for Mission Critical Applications'
• The webinar recording /slides are available at http://hortonworks.com/partners/learn/#hbase2
• The Ambari service for OpenTSDB is available at https://github.com/abajwa-hw/opentsdb-service
• The full list of available workshops is available at http://hortonworks.com/partners/learn

Option 1: prebuilt VM

There is a prebuilt Centos 6.5 VM with the below components installed:

• HDP 2.3.0.0-1754
• Spark 1.3.1

nxbdi

<!-- place this in an %angular paragraph -->

<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/leaflet/0.7.5/leaflet.css" />
<div id="map" style="height: 800px; width: 100%"></div>

<script type="text/javascript">
function initMap() {
    var map = L.map('map').setView([30.00, -30.00], 3);

    L.tileLayer('http://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png', {

nxbdi

End-to-end Latency

0.0543 ms ms (median)
0.003125 ms (99th percentile)
5  ms (99.9th percentile)

Producer and consumer

Producer - 1431170.2 records/sec (136.49 MB/sec) 
Consumer - 3276754.7021 records/sec (312.4957 MB/sec)

nxbdi

HDF workshop

Setup already done

• Download and import latest HDP 2.3 sandbox VM from http://hortonworks.com/sandbox and import into VMware Fusion
• Deployed and install VNC ambari service (to be able to 'remote desktop' in and use eclipse)

  sudo git clone https://github.com/hortonworks-gallery/ambari-vnc-service.git /var/lib/ambari-server/resources/stacks/HDP/2.3/services/VNCSERVER
  
  

nxbdi

REPLACEME: TITLE OF YOUR GRAPHGIST

Introduction

nxbdi

Latency Comparison Numbers
--------------------------
L1 cache reference                           0.5 ns
Branch mispredict                            5   ns
L2 cache reference                           7   ns                      14x L1 cache
Mutex lock/unlock                           25   ns
Main memory reference                      100   ns                      20x L2 cache, 200x L1 cache
Compress 1K bytes with Zippy             3,000   ns        3 us
Send 1K bytes over 1 Gbps network       10,000   ns       10 us
Read 4K randomly from SSD*             150,000   ns      150 us          ~1GB/sec SSD

nxbdi

import javax.net.ssl.SSLSocket;
import javax.net.ssl.SSLSocketFactory;
import java.io.*;

/** Establish a SSL connection to a host and port, writes a byte and
 * prints the response. See
 * http://confluence.atlassian.com/display/JIRA/Connecting+to+SSL+services
 */
public class SSLPoke {
    public static void main(String[] args) {

nxbdi

# using four part files to construct "minitweet"
cat rawtweets/part-0000[1-3] > minitweets

# change log4j properties to WARN to reduce noise during demo
mv conf/log4j.properties.template conf/log4j.properties
vim conf/log4j.properties # Change to WARN

# launch Spark shell REPL
./bin/spark-shell