abajwa-hw

#Sets up single node HDP+Nifi+credit card demo on CentOS 6.6
#Run script below and then use ~/CreditCardTransactionMonitor/startDemoServices.sh to start demo

export ambari_password=${ambari_password:-admin} 
export host_count=${host_count:-1}   #choose number of nodes
export ambari_services=${ambari_services:-HDFS HIVE PIG SPARK MAPREDUCE2 TEZ YARN ZOOKEEPER ZEPPELIN NIFI HBASE PHOENIX STORM KAFKA AMBARI_INFRA ATLAS}  #AMBARI_METRICS can be added post-install
export hdp_ver=${hdp_ver:-2.6}
export hdf_mpack="http://public-repo-1.hortonworks.com/HDF/centos6/3.x/updates/3.0.1.1/tars/hdf_ambari_mp/hdf-ambari-mpack-3.0.1.1-5.tar.gz"
export vm_name=sandbox
export ambari_version=2.5.1.0 

abajwa-hw

#!/usr/bin/env bash
# Launch Centos 7 Vm on IBM Power with at least 8 cores / 32Gb mem / 20Gb disk
# Then run:
# curl -sSL https://gist.github.com/abajwa-hw/31b656ee03419f23bbcea66a635661b6/raw | sudo -E sh  


export ambari_version=2.6.0.5
export ambari_repo="http://public-repo-1.hortonworks.com/ambari/centos7-ppc/2.x/updates/2.6.0.5-1/ambari.repo"
export hdp_repo="http://public-repo-1.hortonworks.com/HDP/centos7-ppc/2.x/updates/2.6.3.0/hdp.repo"
export hdf_repo="http://public-repo-1.hortonworks.com/HDF/centos7-ppc/3.x/updates/3.0.3.0/hdf.repo"

abajwa-hw

# Run this script on Ranger host to 
# 1. create users in format $host.openstacklocal@apache.nifi e.g. abajwa-hdf-qe-bp-1.openstacklocal@apache.nifi
# 2. create Ranger policies for above Nifi users:
#  a) read policy for /flow 
#  b) read/write policies for /proxy


export admin=${admin:-nifiadmin}
export cluster=${cluster:-HDF}
export hosts=${hosts:-myhost1 myhost2 myhost3}

abajwa-hw

Automated deployment of fresh cluster via blueprints (using ambari-bootstrap)

• Note: You will now need to set the following env vars with your Cloudera paywall credentials: username paywall_password

• Bring up 4 VMs imaged with RHEL/CentOS 6.x (e.g. node1-4 in this case)

• On non-ambari nodes (e.g. nodes 2-4), install ambari-agents and point them to ambari node (e.g. node1 in this case)

export ambari_server=node1
export ambari_version=2.7.5.0
export username=myuser

abajwa-hw

Hbase indexing to solr in HDP 2.3

• Background:

The HBase Indexer provides the ability to stream events from HBase to Solr for near real time searching. The HBase indexer is included with HDPSearch as an additional service. The indexer works by acting as an HBase replication sink. As updates are written to HBase, the events are asynchronously replicated to the HBase Indexer processes, which in turn creates Solr documents and pushes them to Solr.

• References:
  • http://doc.lucidworks.com/hdpsearch23/JobJarGuide.html#_hbase_indexer
  • https://github.com/NGDATA/hbase-indexer/wiki/Tutorial

abajwa-hw

Phoenix/Spark demo

• These demo is part of a webinar on 'HBase for Mission Critical Applications'
• The webinar recording /slides are available at http://hortonworks.com/partners/learn/#hbase2
• The Ambari service for OpenTSDB is available at https://github.com/abajwa-hw/opentsdb-service
• The full list of available workshops is available at http://hortonworks.com/partners/learn

Option 1: prebuilt VM

There is a prebuilt Centos 6.5 VM with the below components installed:

• HDP 2.3.0.0-1754
• Spark 1.3.1

abajwa-hw

#!/usr/bin/env bash
# Launch Centos/RHEL 7 VM with at least 8 vcpu / 32Gb+ memory / 100Gb disk
# Then run:
# curl -sSL https://gist.github.com/abajwa-hw/9bf11bcaadaa7f94f9075432ce5b3d0b/raw | sudo -E sh  

export create_image=${create_image:-true}
export ambari_version=2.7.3.0
#export mpack_url="http://public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.4.0.0/tars/hdf_ambari_mp/hdf-ambari-mpack-3.4.0.0-155.tar.gz"  
#export mpack_url="http://public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.4.1.0/tars/hdf_ambari_mp/hdf-ambari-mpack-3.4.1.0-5.tar.gz"  
export mpack_url="http://public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.4.1.1/tars/hdf_ambari_mp/hdf-ambari-mpack-3.4.1.1-4.tar.gz"

abajwa-hw

#!/usr/bin/env bash
# Launch Centos/RHEL 7 Vm  with at least 4 cores / 16Gb mem / 60Gb disk
# Then run:
# curl -sSL https://gist.github.com/abajwa-hw/c37d0e847054cf519813066401c33388/raw | sudo -E sh  

export ambari_password=${ambari_password:-StrongPassword}
export db_password=${db_password:-StrongPassword}
export nifi_password=${nifi_password:-StrongPassword}
export ambari_services="ZOOKEEPER STREAMLINE NIFI KAFKA STORM REGISTRY NIFI_REGISTRY KNOX AMBARI_METRICS"
export cluster_name=${cluster_name:-hdf}

abajwa-hw

# curl -sSL https://gist.github.com/abajwa-hw/bca3d23fe146c3ebd59a9b5fd19480a3/raw | sudo -E sh 
export host=$(hostname -f)
export realm=${realm:-CLOUDERA.COM}
export domain=${domain:-cloudera.com}
export kdcpassword=${kdcpassword:-BadPass#1}

set -e

sudo yum -y install krb5-server krb5-libs krb5-auth-dialog krb5-workstation

abajwa-hw

#!/usr/bin/env bash
export cluster_name=$1
export host=$(hostname -f)
export ambari_pass=$2
export knox_ldap_pass=$3
hostname=$(hostname -f)
current_dir=$(pwd)
cd /tmp
#Copy config-update to /tmp
cp /tmp/masterclass/ranger-atlas/HortoniaMunichSetup/config_update.py .