Skip to content

Instantly share code, notes, and snippets.

@sturadnidge
Last active December 26, 2015 00:09
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save sturadnidge/7062575 to your computer and use it in GitHub Desktop.
Save sturadnidge/7062575 to your computer and use it in GitHub Desktop.
Build out an HttpFS gateway. Assumes you have a Sun JDK installed and JAVA_HOME set for root.
# Add Cloudera RPM-GPG-KEY and repo
rpm --import http://archive.cloudera.com/cdh4/redhat/6/x86_64/cdh/RPM-GPG-KEY-cloudera
rpm -ivh http://archive.cloudera.com/cdh4/one-click-install/redhat/6/x86_64/cloudera-cdh-4-0.x86_64.rpm
# note: if you want to install a specific version,
# modify /etc/yum.repos.d/cloudera-cdh4.repo accordingly.
# For example, if you want to install 4.2.1, use the following baseurl:
# baseurl=http://archive.cloudera.com/cdh4/redhat/6/x86_64/cdh/4.2.1/
# Install CDH4 httpfs Base
yum -y install hadoop-httpfs
# create a file /etc/hadoop/conf/hadoop-env.sh and export JAVA_HOME in there
# ensure DNS is working, or that you have an entry in /etc/hosts for both
# <hostname.fqdn> and <namenode.fqdn>
# on each node of the hadoop cluster, add the following in core-site.xml
<!-- HTTPFS proxy user setting -->
<property>
<name>hadoop.proxyuser.httpfs.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.httpfs.groups</name>
<value>*</value>
</property>
# on the httpfs gateway, add the follwing in /etc/hadoop/conf/core-site.xml
<property>
<name>fs.default.name</name>
<value>hdfs://<namenode.fqdn>:8020</value>
</property>
# export JAVA_HOME in /etc/hadoop-httpfs/conf/httpfs-env.sh
# Start httpfs - you should see some fairly typical Tomcat startup messages
service hadoop-httpfs start
# Check everything worked, run '$JAVA_HOME/bin/jps' and look for the following processes
Bootstrap
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment