glennklockwood/cudf_ARM64_jetson.md

## cudf_ARM64_jetson.md

      
    Raw
  

              cudf_ARM64_jetson.md
            
          
    Installing cuDF on Jetson Nano

These instructions move in the direction of building a cuDF docker image that
works on Jetson Nano.
My setup

I'm using Nvidia Jetson nano.

Quad-core ARM® Cortex®-A57 MPCore processor
NVIDIA Maxwell™ architecture with 128 NVIDIA CUDA® cores
4 GB 64-bit LPDDR4 1600MHz - 25.6 GB/s
Ubuntu 18.04 LTS
Python 3.6.9

Dependencies

Start from the L4T-ml image in NGC.  Use a docker-compose.yml like this:
---
version: "3"
services:
  l4t-ml-bash:
    image: "nvcr.io/nvidia/l4t-ml:r32.4.4-py3"
    devices:
      - /dev/video0
    stdin_open: true
    tty: true
    entrypoint: /bin/bash
Then
$ docker-compose run --rm l4t-ml-bash

See my Getting started with the NVIDIA Jetson Nano page for how to get docker-compose running.
Miniforge for ARM64

Install Miniforge:
$ cd /
$ wget https://github.com/conda-forge/miniforge/releases/download/4.9.2-5/Miniforge3-4.9.2-5-Linux-aarch64.sh
$ sh ./Miniforge3-4.9.2-5-Linux-aarch64.sh -b -p /conda
$ /conda/bin/conda update -y -n base conda

$ export PATH=${PATH}:/conda/bin
$ conda init --all
$ /bin/bash # to enable `conda activate`

Create a cudf conda environment in which we will install our dependencies:
$ conda create -y -n cudf cmake boost pyarrow
$ conda activate cudf

DLPack

$ cd /
$ wget https://github.com/dmlc/dlpack/archive/v0.3.tar.gz
$ tar zxf v0.3.tar.gz
$ mkdir -p dlpack-0.3/build
$ cd dlpack-0.3/build
$ cmake .. -DCMAKE_INSTALL_PREFIX=$CONDA_PREFIX
$ make -j4
$ make install

RMM

$ cd /
$ git clone -b v0.17.0 --recurse-submodules https://github.com/rapidsai/rmm.git
$ mkdir -p /rmm/build
$ cd /rmm/build
$ cmake .. -DCMAKE_INSTALL_PREFIX=${CONDA_PREFIX} -DBUILD_TESTS=OFF
$ make
$ make install

Are these other RMM steps necesssary?
Installing rmm python library.

cd python
sudo -E python3 setup.py build_ext --inplace
sudo -E python3 setup.py install

Apache Arrow

You can't get away with just installing Arrow from conda because you need to build Arrow with CUDA support.  So,
cd /
conda install -y rapidjson snappy brotli gflags libthrift protobuf re2 libutf8proc grpc-cpp orc
apt update && apt install -y llvm-10 clang
wget https://github.com/apache/arrow/archive/apache-arrow-3.0.0.tar.gz
tar zxf apache-arrow-3.0.0.tar.gz
cd arrow-apache-arrow-3.0.0/cpp
mkdir build
cd build
export ARROW_HOME=$CONDA_PREFIX/lib
cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
      -DCMAKE_INSTALL_LIBDIR=lib  \
      -DARROW_FLIGHT=ON \
      -DARROW_GANDIVA=ON  \
      -DARROW_ORC=ON  \
      -DARROW_WITH_BZ2=ON \
      -DARROW_WITH_ZLIB=ON  \
      -DARROW_WITH_ZSTD=ON  \
      -DARROW_WITH_LZ4=ON \
      -DARROW_WITH_SNAPPY=ON  \
      -DARROW_WITH_BROTLI=ON  \
      -DARROW_PARQUET=ON  \
      -DARROW_PYTHON=ON \
      -DARROW_PLASMA=ON \
      -DARROW_CUDA=ON \
      -DARROW_BUILD_TESTS=OFF \
      -DPYTHON_EXECUTABLE=$(which python3) \
      -DBUILD_TESTS=OFF \
      ..
make -j4


This will take a while (30-60 minutes)
cudf

I'm checking out a tagged commit to reflect the HEAD I used when building cudf.
You can use later versions if there aren't any deviations from this install procedure.
wget https://github.com/rapidsai/cudf/archive/v0.17.0.tar.gz
tar zxf v0.17.0.tar.gz
ln -s cudf-0.17.0 cudf
mkdir -p /cudf/cpp/build
cd /cudf/cpp/build
cmake .. -DCMAKE_INSTALL_PREFIX=${CONDA_PREFIX} -DBUILD_TESTS=OFF -Wno-dev
make

This step takes forever and it consumes a huge amount of memory with the version of cuDF above.
According to a memory profile I ran, it needs ~12 GB.  I got around this by adding a swap SSD.
This is where I stopped.  Everything below here is untested so far.  Original
build instructions included in case mine are insufficient.

-DGPU_ARCHS="" was needed for jetson nano as the cuda compute capability is 5.3 and cudf requires 6.0+. If you are building for a device is GPU with cuda compute 6.0+, this flag can be ommited.
cmake .. -DCMAKE_INSTALL_PREFIX=/usr/local/lib/cudf/ -DCMAKE_CXX11_ABI=ON -DRMM_INCLUDE=/usr/local/lib/rmm/include/ -DDLPACK_INCLUDE=/usr/local/lib/dlpack/include/ -DGPU_ARCHS=""
make #will take a day to build, more process will crash the build due to low memory.
sudo -E make install

Building and installing nvstrings python module

cd $CUDF_HOME/python/nvstrings
export NVSTRINGS_ROOT=/usr/local/lib/cudf/
export NVSTRINGS_LIBRARY=/usr/local/lib/cudf/lib/
export NVTEXT_LIBRARY=/usr/local/lib/cudf/lib/
export NVCATEGORY_LIBRARY=/usr/local/lib/cudf/lib/
sudo -E python3 setup.py install

Export CUDF path to LD_LIBRARY_PATH

export LD_LIBRARY_PATH=/usr/local/lib:/usr/local/lib/cudf/lib:/usr/local/lib/rmm/lib:$LD_LIBRARY_PATH
sudo ldconfig

Fixing cudf python module setup

cd $CUDF_HOME/python/cudf
nano setup.py

To fix this issue - rapidsai/cudf#4121,
Add the following include paths to setup.py at this location - https://github.com/rapidsai/cudf/blob/d6b4794b4a3ed7e6577042596a32732bd62fd074/python/cudf/setup.py#L41-L50
Change the path to your appropirate location if you haven't followed the earlier paths for installation of these libraries.
"/usr/local/lib/",
"/usr/local/lib/cudf",
"/usr/local/lib/include",
"/usr/local/lib/rmm/include/",
"/usr/local/lib/dlpack/include/",

Building cudf python module

cd $CUDF_HOME/python/cudf
sudo -E python3 setup.py build_ext --inplace
sudo -E python3 setup.py install

Notes


You could probably do away with sudo while installing python libraries if you don't face any permission errors. If you use sudo for python library installations, understand the risks involved in it.

References

https://github.com/rapidsai/cudf/blob/branch-0.13/CONTRIBUTING.md#setting-up-your-build-environment
rapidsai/cudf#2770 (comment)
rapidsai/cudf#2505 (comment)