Skip to content

Instantly share code, notes, and snippets.

@bradkarels
Created January 21, 2015 18:41
Show Gist options
  • Save bradkarels/e3cc535c65d25d684cd2 to your computer and use it in GitHub Desktop.
Save bradkarels/e3cc535c65d25d684cd2 to your computer and use it in GitHub Desktop.
A brief history setting up Hadoop (HDFS) locally (psuedo distributed)
890 cd ~/Downloads/
891 wget http://mirror.cc.columbia.edu/pub/software/apache/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz
915 cd ~
917 tar xzf Downloads/hadoop-2.6.0.tar.gz
920 mv hadoop-2.6.0/ hadoop/
923 cd hadoop/
924 vim ~/.bashrc
Set HADOOP_HOME and add HADOOP_HOME/bin to PATH
925 . ~/.bashrc
926 ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
927 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
928 ssh localhost
986 sudo less /var/log/secure
987 cd .ssh/
989 chmod 600 authorized_keys
992 sudo tail /var/log/secure
993 cd ..
994 chmod 700 .ssh
998 chmod 600 .ssh/authorized_keys
999 sudo service sshd restart
1022 ssh-add
1023 ssh localhost
1025 cd hadoop
1027 vim etc/hadoop/hadoop-env.sh
1030 vim etc/hadoop/core-site.xml
1031 vim etc/hadoop/hdfs-site.xml
1032 hdfs namenode -format
1033 sbin/start-dfs.sh
1034 hdfs dfs -mkdir /user
1035 hdfs dfs -mkdir /user/bkarels
1042 hdfs dfs -put ~/Downloads/kafka_2.10-0.8.1.1.tgz /user/bkarels
1044 hdfs dfs -ls /user/bkarels
1047 hdfs dfs -ls -h /user/bkarels
1048 hdfs dfs -rm /user/bkarels/kafka_2.10-0.8.1.1.tgz
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment