Hadoop

hadoop config

Things to care about
 
OS configs :
 
ulimit : unlimited
increase the max_file handle : cat /proc/sys/fs/file-max
remove reserve block count (currently - 1%)
noatime
 
[root@ip-192-10-16-125 ~]# cat /proc/sys/net/core/somaxconn
128
 
[root@ip-192-10-16-125 ~]# cat /proc/sys/fs/file-max
740846
[root@ip-192-10-16-125 ~]# cat /proc/sys/net/ipv4/tcp_max_syn_backlog
2048
 
selective_ack (dsack,sack,ecn) are enabled.
 
Should we increase the transmit queue lenght. currenlty (txqueuelen : 1000)
 
We are using CFQ as IO scheduler.
 
fs.epoll.max_user_instances = 4096