abishekmuthian/cudf_ARM64_jetson.md

## cudf_ARM64_jetson.md

      
    Raw
  

              cudf_ARM64_jetson.md
            
          
    My setup

I'm using Nvidia Jetson nano.
Quad-core ARM® Cortex®-A57 MPCore processor
NVIDIA Maxwell™ architecture with 128 NVIDIA CUDA® cores
4 GB 64-bit LPDDR4 1600MHz - 25.6 GB/s
Ubuntu 18.04 LTS
Python 3.6.9
Environment variables for CUDA

export CUDACXX=/usr/local/cuda/bin/nvcc
export CUDA_HOME=/usr/local/cuda
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64
export PATH=$PATH:$CUDA_HOME/bin

Pip Dependencies

$ pip install cmake-setuptools

Dependencies

apache arrow

Follow my earlier apache arrow installation guide for ARMv8 - https://gist.github.com/heavyinfo/04e1326bb9bed9cecb19c2d603c8d521.
DLPack

$ git clone https://github.com/dmlc/dlpack.git
$ cd dlpack
$ mkdir build && cd build
$ cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local/lib/dlpack
$ make -j4
$ make install

RMM

$ git clone --recurse-submodules https://github.com/rapidsai/rmm.git
$ cd rmm
$ mkdir build && cd build
$ cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local/lib/dlpack
$ make -j4
$ sudo -E make install

Installing rmm python library.

cd python
sudo -E python3 setup.py build_ext --inplace
sudo -E python3 setup.py install

Export RMM path to LD_LIBRARY_PATH

export LD_LIBRARY_PATH=/usr/local/lib:/usr/local/lib/rmm/lib:$LD_LIBRARY_PATH
sudo ldconfig

Download cudf

I'm checking out the commit to reflect the HEAD I used when building cudf.
You can use later versions if there aren't any deviations from this install procedure.
git clone https://github.com/rapidsai/cudf.git
git checkout d6b4794b4a3ed7e6577042596a32732bd62fd074

Building and installing libcudf

Set the paths according to your own locations.
-DGPU_ARCHS="" was needed for jetson nano as the cuda compute capability is 5.3 and cudf requires 6.0+. If you are building for a device is GPU with cuda compute 6.0+, this flag can be ommited.
export CUDF_HOME=/home/abishek/Downloads/cudf
export DLPACK_ROOT=/usr/local/lib/dlpack/
export RMM_ROOT=/usr/local/lib/rmm/
cd $CUDF_HOME/cpp/build/
cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local/lib/cudf/ -DCMAKE_CXX11_ABI=ON -DRMM_INCLUDE=/usr/local/lib/rmm/include/ -DDLPACK_INCLUDE=/usr/local/lib/dlpack/include/ -DGPU_ARCHS=""
make #will take a day to build, more process will crash the build due to low memory.
sudo -E make install

Building and installing nvstrings python module

cd $CUDF_HOME/python/nvstrings
export NVSTRINGS_ROOT=/usr/local/lib/cudf/
export NVSTRINGS_LIBRARY=/usr/local/lib/cudf/lib/
export NVTEXT_LIBRARY=/usr/local/lib/cudf/lib/
export NVCATEGORY_LIBRARY=/usr/local/lib/cudf/lib/
sudo -E python3 setup.py install

Export CUDF path to LD_LIBRARY_PATH

export LD_LIBRARY_PATH=/usr/local/lib:/usr/local/lib/cudf/lib:/usr/local/lib/rmm/lib:$LD_LIBRARY_PATH
sudo ldconfig

Fixing cudf python module setup

cd $CUDF_HOME/python/cudf
nano setup.py

To fix this issue - rapidsai/cudf#4121,
Add the following include paths to setup.py at this location - https://github.com/rapidsai/cudf/blob/d6b4794b4a3ed7e6577042596a32732bd62fd074/python/cudf/setup.py#L41-L50
Change the path to your appropirate location if you haven't followed the earlier paths for installation of these libraries.
"/usr/local/lib/",
"/usr/local/lib/cudf",
"/usr/local/lib/include",
"/usr/local/lib/rmm/include/",
"/usr/local/lib/dlpack/include/",

Building cudf python module

cd $CUDF_HOME/python/cudf
sudo -E python3 setup.py build_ext --inplace
sudo -E python3 setup.py install

Notes


You could probably do away with sudo while installing python libraries if you don't face any permission errors. If you use sudo for python library installations, understand the risks involved in it.

References

https://github.com/rapidsai/cudf/blob/branch-0.13/CONTRIBUTING.md#setting-up-your-build-environment
rapidsai/cudf#2770 (comment)
rapidsai/cudf#2505 (comment)