Cluster install

Cluster Install info

——————————————————————————————————————
*** Initial setup before Ambari setup ***
——————————————————————————————————————


OS checks:



yum install wget

wget -qO- --no-check-certificate https://github.com/hortonworks/HDP-Public-Utilities/raw/master/Installation/install_tools.sh | bash

cd hdp and create Hostdetail.txt with all the host entries ( use FQDN )



handy command, in case we are using hosts file

cat /etc/hosts | awk '{ print $2'}   ( note: It may be $2 or $3) 



Once Hostdetail.txt is under /root/hdp , test the scripts


ssh-keygen and follow the prompt
chmod 700 .ssh
cd .ssh
cat id_rsa.pub >> authorized_keys
chmod 600 authorized_keys
scp authorized_keys to all the nodes under .ssh 
or try  ssh-copy-id -i ~/.ssh/id_rsa.pub remote-host


/etc/hosts - All hosts in your system must be configured for DNS and Reverse DNS.
If DNS can’t be setup then set entries in /etc/hosts file


./run_command.sh 'hostname'

./copy_file.sh source_filename destination_filename ( for example: ./copy_file.sh /etc/hosts /etc/hosts) 


*** IPtables  ***

 ./run_command.sh 'chkconfig iptables off'   
./run_command.sh '/etc/init.d/iptables stop'


*** Disable SELinux and PackageKit and check the UMASK Value ***

./run_command.sh 'setenforce 0'
./run_command.sh 'umask 022'


*** Disable THP ***

In ambari host vi /etc/rc.local  and add following lines

if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi

./copy_file.sh /etc/rc.local /etc/rc.local 
./run_command.sh '/etc/rc.local' 

verify: look for always madvise [never] 

./run_command.sh 'cat /sys/kernel/mm/transparent_hugepage/enabled'

./run_command.sh 'cat /sys/kernel/mm/transparent_hugepage/defrag'


***  ntp  ***

./run_command.sh '/etc/init.d/ntpd status '

if not installed then 

./run_command.sh  ' yum  -y install ntp  '

./run_command.sh '/etc/init.d/ntpd start '
./run_command.sh ' chkconfig ntpd on'

***  swap  ***
cat /proc/sys/vm/swappiness
sysctl -w vm.swappiness=0
./run_command.sh ' sysctl -w vm.swappiness=0'

*** Check the Maximum Open File Descriptors  *****

recommended maximum number of open file descriptors is 10000, 

 ./run_command.sh 'sysctl fs.file-max'
./run_command.sh 'sysctl -w fs.file-max=300000'
./run_command.sh 'cat /proc/sys/fs/file-max'

 echo fs.file-max=300000 >>  /etc/sysctl.con
grep fs.file-max /etc/sysctl.conf 

——————————————————————————————————————
*** Ambari Installation ***
——————————————————————————————————————
http://doc.hortonworks.com


Typical Ambari install




——————————————————————————————————————
*** Tune the cluster ***

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.1/bk_installing_manually_book/content/rpm-chap1-11.html

——————————————————————————————————————
*** HDFS and NN Testing ***
——————————————————————————————————————

TestDFSIO
	su - hdfs
	export YARN_EXAMPLES=$YARN_HOME/usr/hdp/current/hadoop-mapreduce-client/
	yarn jar $YARN_EXAMPLES/hadoop-mapreduce-client-jobclient-tests.jar TestDFSIO -write -nrFiles 100 -fileSize 100
	yarn jar $YARN_EXAMPLES/hadoop-mapreduce-client-jobclient-tests.jar TestDFSIO -read -nrFiles 100 -fileSize 100
	yarn jar $YARN_EXAMPLES/hadoop-mapreduce-client-jobclient-tests.jar TestDFSIO -clean

Terasort
	su - hdfs
	export YARN_EXAMPLES=$YARN_HOME/usr/hdp/current/hadoop-mapreduce-client/
	yarn jar $YARN_EXAMPLES/hadoop-mapreduce-examples.jar teragen 1000000 /benchmarks/terasort/terasort-input
	yarn jar $YARN_EXAMPLES/hadoop-mapreduce-examples.jar terasort /benchmarks/terasort/terasort-input /benchmarks/terasort/terasort-output
	yarn jar $YARN_EXAMPLES/hadoop-mapreduce-examples.jar teravalidate /benchmarks/terasort/terasort-output /benchmarks/terasort/teravalidate-output



Read/Write Data

*** TeraGen ***

hdfs dfs -mkdir /benchmarks

hdfs dfs -mkdir /benchmarks/terasort

# This will generate 1,000,000 100 byte records as input for Terasort
# use find  / -name hadoop-mapreduce-examples.jar  to exact location 
#user -Dmapred.reduce.tasks=#ofmappers
 hadoop jar /usr/hdp/current/hadoop-mapreduce/hadoop-mapreduce-examples.jar teragen -Dmapred.map.tasks=30 1000000 /benchmarks/terasort/terasort-input


*** TeraSort ***

# Sort the 1,000,000 records generated by TeraGen
hadoop jar /usr/hdp/2.2.3.0-2610/hadoop-mapreduce/hadoop-mapreduce-examples.jar terasort -Dmapred.reduce.tasks=30 /benchmarks/terasort/terasort-input /benchmarks/terasort/terasort-output

*** TeraValidate ***

# Validate that the sort was successful and correct
hadoop jar /usr/hdp/2.2.3.0-2610/hadoop-mapreduce/hadoop-mapreduce-examples.jar teravalidate /benchmarks/terasort/terasort-output /benchmarks/terasort/teravalidate-output

 
*** NameNode Exercise ***

login as hdfs user

*** TestDFSIO Write Test ***

# -fileSize argument is, by default, in units of MB.  This should write 10 GB of files

hadoop jar /usr/hdp/2.2.3.0-2610/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.6.0.2.2.3.0-2610-tests.jar TestDFSIO -write -nrFiles 100 -fileSize 100


TestDFSIO Read Test

hadoop jar /usr/hdp/2.2.3.0-2610/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.6.0.2.2.3.0-2610-tests.jar TestDFSIO -read -nrFiles 100 -fileSize 100

TestDFSIO Cleanup

hadoop jar /usr/hdp/2.2.3.0-2610/hadoop-mapreduce/hadoop-mapreduce-client-jobclient-2.6.0.2.2.3.0-2610-tests.jar TestDFSIO -clean

——————————————————————————————————————
*** HA - Enable and Test ***
——————————————————————————————————————

NN test:
Login to ambari and check for Active Name Node http://<hostname>:8080/#/main/services/HDFS/summary

ssh into Active NN
ps -ef | grep namenode
kill -9 <pid>

Login to Ambari and you see Active Node  changing 

Terasort test


# Sort the 1,000,000 records generated by TeraGen
hadoop jar /usr/hdp/2.2.3.0-2610/hadoop-mapreduce/hadoop-mapreduce-examples.jar terasort /benchmarks/terasort/terasort-input /benchmarks/terasort/terasort-output


- look for application id or application url 

- 


——————————————————————————————————————
*** Initial Environment Setup for POC testing ***
——————————————————————————————————————

./run_command.sh 'groupadd hadoopusers'
./run_command.sh 'groupadd analyst'
./run_command.sh 'groupadd shipment_analyst'

./run_command.sh 'grep hadoopusers\|analyst /etc/group'


 ./run_command.sh 'useradd -g hadoopusers platfora'
./run_command.sh 'useradd -g hadoopusers 353559'
./run_command.sh 'useradd -g hadoopusers 687671'

./run_command.sh 'grep platfora\|353559\|687671 /etc/passwd'


 HDFS setup

hdfs dfs -mkdir /user/platfora
hdfs dfs -chown -R platfora:hdfs /user/platfora

hdfs dfs -mkdir /user/353559
hdfs dfs -chown -R 353559:hdfs /user/platfora

hdfs dfs -mkdir /user/687671
hdfs dfs -chown -R 687671:hdfs /user/platfora

hdfs dfs -mkdir /tmp/hadoop-platfora
hdfs dfs -mkdir /tmp/353559
hdfs dfs -mkdir /tmp/687671

hdfs dfs -chown -R platfora:hdfs /tmp/hadoop-platfora
hdfs dfs -chown -R 353559:hdfs /tmp/353559
hdfs dfs -chown -R 687671:hdfs /tmp/687671

hdfs dfsadmin -refreshUserToGroupsMappings


hdfs dfs -ls /tmp/
hdfs dfs -ls /user/


——————————————————————————————————————
 *** HDFS ACLs ***
——————————————————————————————————————

dfs.namenode.acls.enabled  = true in hdfs-site.xml


hdfs dfs -mkdir /data
hdfs dfs -mkdir /data/shipment
hdfs dfs -mkdir /data/revenue


hdfs dfs -setfacl -R -m group:shipment_analyst:r-- /data/shipment
hdfs dfs -setfacl -R -m group:revenue_analyst:r-- /data/revenue

hdfs dfs -getfacl /data
hdfs dfs -getfacl /data/shipment
hdfs dfs -getfacl /data/revenue


Hive Database: login as hdfs user

CREATE DATABASE IF NOT EXISTS shipment_schema COMMENT 'Shipment schema' Location '/data/shipment/shipment_schema';
CREATE DATABASE IF NOT EXISTS shipment_schema COMMENT 'Revenue schema' Location '/data/shipment/revenue_schema';

Tables: tracking_info, invoice needs to be created. Followup with the customer. 


——————————————————————————————————————
*** Ranger setup ***

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.2.0/Ranger_U_Guide_v22/Ranger_U_Guide_v22.pdf
——————————————————————————————————————

HDFS

create HDFS policy 

Hive

create HIVE policy

HBASE

create HBASE policy

Storm

Knox

——————————————————————————————————————
*** Views ***

——————————————————————————————————————


Installation

cd /var/lib/ambari-server/resources/views/

wget http://public-repo-1.hortonworks.com/HDP-LABS/Projects/Views/tp1/files-0.1.0-tp1.jar
wget http://public-repo-1.hortonworks.com/HDP-LABS/Projects/Views/tp1/hive-0.2.0-tp1.jar
wget http://public-repo-1.hortonworks.com/HDP-LABS/Projects/Views/tp1/pig-0.1.0-tp1.jar
wget http://public-repo-1.hortonworks.com/HDP-LABS/Projects/Views/tp1/capacity-scheduler-0.3.0-tp1.jar


core-site.xml

hadoop.proxyuser.root.groups=*
hadoop.proxyuser.root.hosts=*

hadoop.proxyuser.hcat.groups=*
hadoop.proxyuser.hcat.hosts=*

webhcat-site.xml

webhcat.proxyuser.root.groups=*
webhcat.proxyuser.root.hosts=*

hdfs dfs -mkdir -p /user/admin/
 hdfs dfs -chown admin:hdfs /user/admin


Job


yarn.ats.url*   - get this from ambari  ( Yarn —> yarn.timeline-service.webapp.address,  http://<host>:8188) 

yarn.resourcemanager.url*  ( RM address http://<rm_host>:8088 )



Tez

YARN Timeline Server URL*   ( Yarn —> yarn.timeline-service.webapp.address,  http://<host>:8188) 
yarn.resourcemanager.url*  ( RM address http://<rm_host>:8088 )


Slider

ambari.server.url*    http://ambari_server:8080/api/v1/clusters/cluster_name

ambari username and password


Hive

webhdfs.url*     webhdfs://active_namenode:50070

webhcat.url*  http://webhcat_host:50111/templeton/v1
scripts.dir* /user/${username}/hive/scripts
jobs.dir*  /user/${username}/hive/jobs
hive.host* Hiveserver2
hive.port* 10000
hive.auth  auth=NONE;user=${username}


Capacity scheudler

http://ambari_server:8080/api/v1/clusters/cluster_name

File

WebHDFS FileSystem URI*  hdfs-site.xml  —> dfs.namenode.http-address  for example:  webhdfs://master2:50070 ( HA consideration - Active NN  )