Issues in installation and configuration Hadoop

Issues in installation and configuration Hadoop

1. connect to host localhost port 22: Connection refused
Stopping namenodes on [localhost]
localhost: ssh: connect to host localhost port 22: Connection refused
localhost: ssh: connect to host localhost port 22: Connection refused
Stopping secondary namenodes [0.0.0.0]
0.0.0.0: ssh: connect to host 0.0.0.0 port 22: Connection refused

Fix:
thanooj@ubuntu:/$ sudo apt-get install openssh-server
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  ncurses-term openssh-sftp-server ssh-import-id
Suggested packages:
  ssh-askpass rssh molly-guard monkeysphere
The following NEW packages will be installed:
  ncurses-term openssh-server openssh-sftp-server ssh-import-id
0 upgraded, 4 newly installed, 0 to remove and 322 not upgraded.
Need to get 636 kB of archives.
After this operation, 5,145 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 ncurses-term all 6.0+20160213-1ubuntu1 [249 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 openssh-sftp-server amd64 1:7.2p2-4ubuntu2.1 [38.8 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 openssh-server amd64 1:7.2p2-4ubuntu2.1 [338 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 ssh-import-id all 5.5-0ubuntu1 [10.2 kB]
Fetched 636 kB in 4s (144 kB/s)     
Preconfiguring packages ...
Selecting previously unselected package ncurses-term.
(Reading database ... 204538 files and directories currently installed.)
Preparing to unpack .../ncurses-term_6.0+20160213-1ubuntu1_all.deb ...
Unpacking ncurses-term (6.0+20160213-1ubuntu1) ...
Selecting previously unselected package openssh-sftp-server.
Preparing to unpack .../openssh-sftp-server_1%3a7.2p2-4ubuntu2.1_amd64.deb ...
Unpacking openssh-sftp-server (1:7.2p2-4ubuntu2.1) ...
Selecting previously unselected package openssh-server.
Preparing to unpack .../openssh-server_1%3a7.2p2-4ubuntu2.1_amd64.deb ...
Unpacking openssh-server (1:7.2p2-4ubuntu2.1) ...
Selecting previously unselected package ssh-import-id.
Preparing to unpack .../ssh-import-id_5.5-0ubuntu1_all.deb ...
Unpacking ssh-import-id (5.5-0ubuntu1) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for systemd (229-4ubuntu10) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for ufw (0.35-0ubuntu2) ...
Setting up ncurses-term (6.0+20160213-1ubuntu1) ...
Setting up openssh-sftp-server (1:7.2p2-4ubuntu2.1) ...
Setting up openssh-server (1:7.2p2-4ubuntu2.1) ...
Creating SSH2 RSA key; this may take some time ...
2048 SHA256:Z0Ua5wQ/Ys2VkMXY6EHO3piE8wR2eDVWKfJ8QjmBf+s root@ubuntu (RSA)
Creating SSH2 DSA key; this may take some time ...
1024 SHA256:QVf/9MFGymmWAt6GPeRdQPtfL0XJiq9jzFtPyax5iZo root@ubuntu (DSA)
Creating SSH2 ECDSA key; this may take some time ...
256 SHA256:FO49jCgn6J1yciS0OxC3KnRgzLGIty0mlBXXEcIYerA root@ubuntu (ECDSA)
Creating SSH2 ED25519 key; this may take some time ...
256 SHA256:NRTS+9qFjP1GdOvAFN1iXfQslb5scIGLJg52foOc+Nw root@ubuntu (ED25519)
Setting up ssh-import-id (5.5-0ubuntu1) ...
Processing triggers for systemd (229-4ubuntu10) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for ufw (0.35-0ubuntu2) ...
thanooj@ubuntu:/$
thanooj@ubuntu:/$ sudo /etc/init.d/ssh restart
[ ok ] Restarting ssh (via systemctl): ssh.service.
thanooj@ubuntu:/$
thanooj@ubuntu:/$ netstat -tulpn | grep 22
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -               
tcp6       0      0 :::22                   :::*                    LISTEN      -               
thanooj@ubuntu:/$
thanooj@ubuntu:/$ start-dfs.sh
16/11/12 12:05:20 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
The authenticity of host 'localhost (127.0.0.1)' can't be established.
ECDSA key fingerprint is SHA256:FO49jCgn6J1yciS0OxC3KnRgzLGIty0mlBXXEcIYerA.
Are you sure you want to continue connecting (yes/no)? yes
localhost: Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
thanooj@localhost's password: 
localhost: starting namenode, logging to /home/thanooj/hadoop-2.7.3/logs/hadoop-thanooj-namenode-ubuntu.out
thanooj@localhost's password: 
localhost: starting datanode, logging to /home/thanooj/hadoop-2.7.3/logs/hadoop-thanooj-datanode-ubuntu.out
Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
ECDSA key fingerprint is SHA256:FO49jCgn6J1yciS0OxC3KnRgzLGIty0mlBXXEcIYerA.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts.
thanooj@0.0.0.0's password: 
0.0.0.0: starting secondarynamenode, logging to /home/thanooj/hadoop-2.7.3/logs/hadoop-thanooj-secondarynamenode-ubuntu.out
16/11/12 12:06:34 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
thanooj@ubuntu:/$ 
thanooj@ubuntu:/$ jps
11254 SecondaryNameNode
11078 DataNode
10936 NameNode
11418 Jps
thanooj@ubuntu:/$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /home/thanooj/hadoop-2.7.3/logs/yarn-thanooj-resourcemanager-ubuntu.out
thanooj@localhost's password: 
localhost: starting nodemanager, logging to /home/thanooj/hadoop-2.7.3/logs/yarn-thanooj-nodemanager-ubuntu.out
thanooj@ubuntu:/$ jps
11254 SecondaryNameNode
11078 DataNode
10936 NameNode
11690 Jps
11596 NodeManager
11471 ResourceManager
thanooj@ubuntu:/$ 




Note: An alternative to start hadoop without ssh is by using :
hadoop-daemon.sh start [daemon-name]




2. thanooj@ubuntu:/$ start-dfs.sh
16/11/12 12:23:33 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Fix:

thanooj@ubuntu:/$ gedit ~/.bashrc
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"

save it and close gedit, then, source
thanooj@ubuntu:/$ source ~/.bashrc





3. Passwordless ssh login

thanooj@ubuntu:/$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/thanooj/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/thanooj/.ssh/id_rsa.
Your public key has been saved in /home/thanooj/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:2kIaoMf+FvOeCUQqOYwdhwrZ13cY9vMRKiEeG+XTIUg thanooj@ubuntu
The key's randomart image is:
+---[RSA 2048]----+
|     .Eo* . .    |
| o . o.B B o .   |
|o = + + * * .    |
|+* B   . + o .   |
|B.= o . S   .    |
| = .o+ o         |
|  . o+o .        |
|   ....+         |
|   ...+          |
+----[SHA256]-----+
thanooj@ubuntu:/$ ssh-copy-id thanooj@127.0.0.1
The authenticity of host '127.0.0.1 (127.0.0.1)' can't be established.
ECDSA key fingerprint is SHA256:FO49jCgn6J1yciS0OxC3KnRgzLGIty0mlBXXEcIYerA.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
thanooj@127.0.0.1's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'thanooj@127.0.0.1'"
and check to make sure that only the key(s) you wanted were added.

thanooj@ubuntu:/$ ssh thanooj@127.0.0.1
thanooj@127.0.0.1's password: 
Welcome to Ubuntu 16.04 LTS (GNU/Linux 4.4.0-45-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

319 packages can be updated.
30 updates are security updates.

thanooj@ubuntu:~$ ssh 'thanooj@localhost'
thanooj@localhost's password: 
Welcome to Ubuntu 16.04 LTS (GNU/Linux 4.4.0-45-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

319 packages can be updated.
30 updates are security updates.

Last login: Sat Nov 12 12:38:51 2016 from 127.0.0.1
thanooj@ubuntu:~$ ssh-copy-id thanooj@0.0.0.0
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/thanooj/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
thanooj@0.0.0.0's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'thanooj@0.0.0.0'"
and check to make sure that only the key(s) you wanted were added.

thanooj@ubuntu:~$ ssh 'thanooj@0.0.0.0'
thanooj@0.0.0.0's password: 
Welcome to Ubuntu 16.04 LTS (GNU/Linux 4.4.0-45-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

319 packages can be updated.
30 updates are security updates.

Last login: Sat Nov 12 12:45:57 2016 from 127.0.0.1
thanooj@ubuntu:~$ 

thanooj@ubuntu:~$ start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [localhost]
thanooj@localhost's password: 
localhost: starting namenode, logging to /home/thanooj/hadoop-2.7.3/logs/hadoop-thanooj-namenode-ubuntu.out
thanooj@localhost's password: 
localhost: starting datanode, logging to /home/thanooj/hadoop-2.7.3/logs/hadoop-thanooj-datanode-ubuntu.out
Starting secondary namenodes [0.0.0.0]
thanooj@0.0.0.0's password: 
0.0.0.0: starting secondarynamenode, logging to /home/thanooj/hadoop-2.7.3/logs/hadoop-thanooj-secondarynamenode-ubuntu.out
starting yarn daemons
starting resourcemanager, logging to /home/thanooj/hadoop-2.7.3/logs/yarn-thanooj-resourcemanager-ubuntu.out
thanooj@localhost's password: 
localhost: starting nodemanager, logging to /home/thanooj/hadoop-2.7.3/logs/yarn-thanooj-nodemanager-ubuntu.out
thanooj@ubuntu:~$ jps
17107 DataNode
17446 ResourceManager
17879 Jps
17756 NodeManager
17293 SecondaryNameNode
16957 NameNode
thanooj@ubuntu:~$