Skip to content

Instantly share code, notes, and snippets.

@evansde77
Created July 5, 2017 21:15
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save evansde77/b7b629431bc7a4d7e65d6d7658fe240b to your computer and use it in GitHub Desktop.
Save evansde77/b7b629431bc7a4d7e65d6d7658fe240b to your computer and use it in GitHub Desktop.
Installation steps to install hdfs3 and underlying libhdfs3 on a Mac
#
# required downloading source of cmakem protobuf, gsasl as well as hdfs3
#
mv ~/Downloads/cmake-3.9.0-rc5.tar.gz .
tar -zxf cmake-3.9.0-rc5.tar.gz
cd cmake-3.9.0-rc5
./bootstrap && make
sudo make install
mv ~/Downloads/protobuf-3.3.0.tar.gz .
tar -zxf protobuf-3.3.0.tar.gz
cd protobuf-3.3.0
bash autogen.sh
./configure && make
sudo make install
mv ~/Downloads/gsasl-1.8.0.tar.gz .
tar -zxf gsasl-1.8.0.tar.gz
cd gsasl-1.8.0
./configure && make
sudo make install
mv ~/Downloads/pivotalrd-libhdfs3-2.2.31.tar.gz .
tar -zxf pivotalrd-libhdfs3-2.2.31.tar.gz
cd pivotalrd-libhdfs3-2.2.31
mkdir build
cd build
bash ../bootstrap
make
sudo make install # doesnt seem to work
cd ..
sudo cp -r ./dist/include/hdfs /usr/local/include
sudo cp ./dist/lib/* /usr/local/lib
sudo cp ./dist/lib/pkgconfig/libhdfs3.pc /usr/local/lib/pkgconfig/
cd ..
virtualenv venv
. venv/bin/activate
pip install hdfs3
vi venv/lib/python2.7/site-packages/hdfs3/lib.py # edit libhdfs3.so => libhdfs3.dylib
python
>>> import hdfs3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment