This worked for me for the versions of Hadoop and Spark that I needed to use. Probably will work for other versions as well, until macOS takes the next step further from the light...
-
Install dependencies with homebrew:
- snappy
- zlib
- zstd
- bzip2
-
Install SDKMAN.
-
sdk install 8.0.292.hs-adpt
and make default -
sdk install spark 3.1.2
and make default -
sdk install hadoop 3.2.2
and make default -
sdk install maven
-
Clone hadoop project (
git@github.com:apache/hadoop.git
) and checkoutbranch-3.2.2
-
cd
intohadoop/hadoop-common-project/hadoop-common
-
Add
<ZLIB_LIBRARY>${zlib.lib}</ZLIB_LIBRARY>
under cmake-compile profile vars inpom.xml
. Fixes cmake problem. -
Build with:
mvn clean package -Pdist,native -DskipTests -Dmaven.javadoc.skip -Dzlib.lib=/usr/local/opt/zlib/lib/libz.dylib -Drequire.bzip2=false -Drequire.snappy=true -Drequire.zstd=true -Dsnappy.prefix=/usr/local/opt/snappy/ -Dsnappy.lib=/usr/local/opt/snappy/lib/libsnappy.1.dylib -Dsnappy.include=/usr/local/opt/snappy/include/ -Dzstd.prefix=/usr/local/opt/zstd/ -Dzstd.lib=/usr/local/opt/zstd/lib/libzstd.1.dylib -Dzstd.include=/usr/local/opt/zstd/include/
NOTE: This means bzip2 isn't native and openssl isn't included. If those are problems, you're probably stuck figuring out how to do this without System Integrity Protection because you're going to need to mess with library paths.
-
Copy
target/native/target/usr/local/lib/libhadoop.1.0.0.dylib
to~/.sdkman/candidates/hadoop/3.2.2/lib/native
and symlinklibhadoop.dylib
to it in the same folder -
Go to
~/.sdkman/candidates/java/current/bin/
and linklibsnappy.1.dylib
andlibzstd.1.dylib
from/usr/local/lib/
into it. You'll have to do this for every java version you use. -
run
hadoop checknative -a
and make sure there are no errors and that it says true for the codecs. -
I've set these env vars. Definitely only some of them do anything!
export HADOOP_HOME=$HOME/.sdkman/candidates/hadoop/current
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib/native"
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export JAVA_LIBRARY_PATH=$HADOOP_HOME/lib/native:$JAVA_LIBRARY_PATH
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native