Steps to build 64bit native libs for Hadoop 0.23.1

gistfile1.md

Hadoop 0.23.1 only comes with 32bit libs. Here is how to rebuild them for your platform.

Compile them

1. Install gcc, autoconf, automake, libtool, zlib-devel packages

2. Fix pom.xml to avoid classpath error when running javah:

   1. in hadoop-common-project/hadoop-common/pom.xml, find the dependency on hadoop-annotations
   2. change the scope of the dependency from provided to compile

3. Build the native libs:

    cd hadoop-common-project
    mvn package -Pnative -DskipTests
    # make a cup of tea while Maven downloads the internet
   

Libs should be created in hadoop-common-project/hadoop-common/target/native/target/usr/local/lib

Use them

To use the libs, add them to the Java library path:

java -Djava.library.path=$HADOOP_SRC/hadoop-common-project/hadoop-common/target/native/target/usr/local/lib

Checking it worked

In the log4j logs, you should get a message like this:

util.NativeCodeLoader (NativeCodeLoader.java:50) - Loaded the native-hadoop library