Skip to content

Instantly share code, notes, and snippets.

Ayush Gupta ayushgp

Block or report user

Report or block ayushgp

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
View number_and_format.py
"""
Reads a filegiven as CLI arg, puts a number as prefix on each line
and adds the seperator between the lines.
"""
import sys
file_name = sys.argv[1]
f = open(file_name, 'r', encoding="utf8")
contents = f.readlines()
View run-map-reduce-job.sh
# Format the namenode
bin/hadoop namenode -format
# Start hadoop related processes
sbin/start-dfs.sh
sbin/start-yarn.sh
# Create input directory and put relavent text file in it
echo "Creating input directory $1 and putting textfile.txt on it."
bin/hadoop fs -mkdir $2
View jps.sh
$ jps
3923 Jps
3188 NodeManager
3061 ResourceManager
3531 NameNode
2894 SecondaryNameNode
2687 DataNode
View yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
View mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
View passphraseless-ssh.sh
$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
$ chmod 0600 ~/.ssh/authorized_keys
View hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
View hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
View core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
You can’t perform that action at this time.