Operating system level tuning for Hadoop

1)

THC should be disabled

echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
Add the above to /etc/rc.local

2)

The noatime mount for disks to speed up reads

3)

The open file limit ulimit to high number 32000

4)

Turn off caching on controller

5)

net.core.somaxconn=1024
Socket listen size

6)

Disable SE Linux

SELINUX=disabled


7)

umask 022

8)

sysctl.conf

vm.swappiness=0
vm.overcommit_memory = 1
vm.overcommit_ratio = 100

9)

Disable ip6

10)

Host DNS is properly set

References

http://www.slideshare.net/vgogate/hadoop-configuration-performance-tuning