Skip to content

Instantly share code, notes, and snippets.

Ayush Gupta ayushgp

View GitHub Profile
View number_and_format.py
"""
Reads a filegiven as CLI arg, puts a number as prefix on each line
and adds the seperator between the lines.
"""
import sys
file_name = sys.argv[1]
f = open(file_name, 'r', encoding="utf8")
contents = f.readlines()
View run-map-reduce-job.sh
# Format the namenode
bin/hadoop namenode -format
# Start hadoop related processes
sbin/start-dfs.sh
sbin/start-yarn.sh
# Create input directory and put relavent text file in it
echo "Creating input directory $1 and putting textfile.txt on it."
bin/hadoop fs -mkdir $2
View jps.sh
$ jps
3923 Jps
3188 NodeManager
3061 ResourceManager
3531 NameNode
2894 SecondaryNameNode
2687 DataNode
View yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
View mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
View passphraseless-ssh.sh
$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
$ chmod 0600 ~/.ssh/authorized_keys
View hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
View hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
View core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
You can’t perform that action at this time.