Building native libraries of Hadoop on macOS Catalina

building_HADOOP_on_macOS.md

Building native libraries of Hadoop-3.1.x on macOS Catalina

start

mvn -Pdist,native -DskipTests -Dtar package

error no.1

A different version of protobuf is already installed on macOS but Hadoop needs protobuf@2.5.0

[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.1.3:protoc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecutionException: protoc version is 'libprotoc 3.12.4', expected version is '2.5.0' -> [Help 1]

solution (link)

wget https://github.com/google/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.bz2
tar xvf protobuf-2.5.0.tar.bz2
cd protobuf-2.5.0
./configure CC=clang CXX=clang++ CXXFLAGS='-std=c++11 -stdlib=libc++ -O3 -g' LDFLAGS='-stdlib=libc++' LIBS="-lc++ -lc++abi"
make -j 4 
sudo make install
protoc --version

note

If you

• already have installed protobuf > 2.5.0 for some reason (e.g. mysql),
• or don't want to do make install,

proceed up to make -j 4 and export PATH=<where_you_untarred_protobuf-2.5.0.tar>/src:$PATH and mvn -Pdist,native -DskipTests -Dtar package -rf :hadoop-common.

error no.2

If you are suffering from building hadoop-yarn-server-nodemanager, follow the instruction on link

error no.3

The following error message occurs while hadoop-pipes is being built:

ld: cannot link directly with dylib/framework, your binary is not an allowed client of /usr/lib/libcrypto.dylib for architecture x86_64

solution

vi hadoop-tools/hadoop-pipes/pom.xml 
# find <openssl.prefix> </openssl.pefix> and make it <openssl.prefix>/usr/local/opt/openssl</openssl.prefix>
rm -rf hadoop-tools/hadoop-pipes/target # note that the target directory contains stale instructions including wrong openssl-related paths
mvn -Pdist,native -DskipTests -Dtar package -rf :hadoop-pipes

note

Exporting OPENSSL_ROOT_DIR is not enough because of the following profile:

    <profile>
      <id>native</id>
      <activation>
        <activeByDefault>false</activeByDefault>
      </activation>
      <properties>
        <openssl.prefix></openssl.prefix>
      </properties>
      <build>
        <plugins>
          <plugin>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-maven-plugins</artifactId>
            <executions>
              <execution>
                <id>cmake-compile</id>
                <phase>compile</phase>
                <goals><goal>cmake-compile</goal></goals>
                <configuration>
                  <source>${basedir}/src</source>
                  <vars>
                    <JVM_ARCH_DATA_MODEL>${sun.arch.data.model}</JVM_ARCH_DATA_MODEL>
                    <OPENSSL_ROOT_DIR>${openssl.prefix} </OPENSSL_ROOT_DIR>
                  </vars>
                </configuration>
              </execution>
            </executions>
          </plugin>
        </plugins>
      </build>
    </profile>

$openssl.prefix is going to override OPENSSL_ROOT_DIR