Skip to content

Instantly share code, notes, and snippets.

@JoeyAndres
JoeyAndres / mapred-site.xml
Created October 30, 2017 21:08
mapred-site.xml for cluster config.
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
@JoeyAndres
JoeyAndres / yarn-site.xml
Created October 30, 2017 21:07
yarn-site.xml cluster configuration.
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>linux-01</value>
<description>The hostname of the RM.</description>
@JoeyAndres
JoeyAndres / hdfs-site.xml
Created October 30, 2017 21:01
hdfs-site.xml for cluster with 2 nodes.
<configuration>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/opt/hadoop/namenode</value>
</property>
<property>
@JoeyAndres
JoeyAndres / core-site.xml
Last active October 30, 2017 20:58
core-site.xml for cluster configuration
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://linux-01</value>
</property>
</configuration>
@JoeyAndres
JoeyAndres / hosts
Created October 30, 2017 15:28
Ubuntu modified /etc/hosts, based on the Hadoop's requirement.
127.0.0.1 localhost
192.168.1.1 linux-01
192.168.1.2 linux-02
# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
@JoeyAndres
JoeyAndres / hosts
Last active October 30, 2017 15:25
Ubuntu default /etc/hosts for linux-01 hostname
127.0.0.1 localhost
127.0.1.1 linux-01
# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
@JoeyAndres
JoeyAndres / core-site.xml
Created October 30, 2017 14:30
Multi-node core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:9000</value>
</property>
</configuration>
@JoeyAndres
JoeyAndres / hdfs-site.xml
Created October 30, 2017 14:08
Pseudo-distributed hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
@JoeyAndres
JoeyAndres / core-site.xml
Created October 30, 2017 14:07
Pseudo-distributed core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
@JoeyAndres
JoeyAndres / interfaces
Last active October 27, 2017 20:06
/etc/network/interfaces static ip setup
auto eth0
iface eth0 inet static
address 192.168.1.1
netmask 255.255.0.0
gateway 192.168.0.1
dns-nameservers 8.8.8.8 8.8.4.4