Skip to content

Instantly share code, notes, and snippets.

@alxn

alxn/Hadoop-Native-OSX.md

Created October 8, 2015 18:11
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save alxn/d3917760581aaad46d23 to your computer and use it in GitHub Desktop.
Save alxn/d3917760581aaad46d23 to your computer and use it in GitHub Desktop.
Download ZIP
Information about building the hadoop native tools for OS-X
Raw
Hadoop-Native-OSX.md

Best source of information.

I wanted 2.5.2, so grab the -src tarball.

Assuming you are using brew:

brew install protobuf25
brew install cmake

Check that you are using Java 1.7, you may want to set your .mavenrc:

export JAVA_HOME=$(/usr/libexec/java_home -v 1.7)

Be sure to look over hadoop-2.5.2-src/BUILDING.txt, this section especially:

Building on OS/X

A one-time manual step is required to enable building Hadoop OS X with Java 7 every time the JDK is updated. see: https://issues.apache.org/jira/browse/HADOOP-9350

$ sudo mkdir `/usr/libexec/java_home`/Classes
$ sudo ln -s `/usr/libexec/java_home`/lib/tools.jar `/usr/libexec/java_home`/Classes/classes.jar

Finally pick your target from the BUILDING.txt:

$ mvn package -Pdist,native,docs -DskipTests -Dtar

Grab the result from hadoop-dist/target/hadoop-2.5.2.tar.gz.

See if it works:

HADOOP_CLASSPATH=share/hadoop/yarn/test/hadoop-yarn-server-tests-2.5.2-tests.jar \
 bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.5.2-tests.jar \
 minicluster -rmport 8000 -jhsport 9000 -nodemanagers 2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment