yunqu/build-ray.md

## build-ray.md

      
    Raw
  

              build-ray.md
            
          
    Building Ray for aarch64

There are multiple ways to build ray on aarch64 processors.
We will use the PYNQ image v2.4 with Ubuntu 18.04 OS as an example.
In this document, we will introduce 2 approaches to build ray; we will
always prefer the second approach (building on Amazon A1)
since it is faster, cleaner, and easy to reproduce.
Building from Source on the Target

The easiest way is to build ray from source on the board. Since the
bootstrapping is going to take a lot of disk space, it is highly recommended
to use a 32GB SD card for your PYNQ image v2.4. Again, we are only targetting
64-bit ARM architectures, so please only go through the following steps
on Zynq Ultrascale boards.
Step 1. Install dependencies

As instructed in the offical document,
we will need to install a few dependent libraries first.
sudo apt-get update
sudo apt-get install -y build-essential curl unzip psmisc

Since we already have a Python 3.6 environment, we will just stick with
our Python 3 commands. Now install the following:
pip3 install cython==0.29.0 setproctitle psutil

Step 2. Prepare bazel

You need to first install a Java environment.
sudo apt-get install -y openjdk-11-jdk-headless

After that, we have a pre-compiled bazel binary for aarch64.
You can copy it over to /usr/local/bin and change its permissions.
sudo chmod 777 /usr/local/bin/bazel

Verify the command is working:
bazel

It should show a help message.
Step 3. Build

You can clone the ray repository, checkout a version, and build from it.
If you do not want to build from that tag, another commit that you can try is
931e6a2.
cd /opt/builds
git clone https://github.com/ray-project/ray.git
cd ray
git checkout -b ray-0.6.3 tags/ray-0.6.3
cd python
pip3 install -e .

This will take a long time. But after this you are done!
Building Wheels on AWS A1 Platform

Now we will build wheels on AWS A1, copy the wheels over to your board,
and install them! This takes the shortcut since all the Zynq Ultrascale
processors share the aarch64 so we do not have to rebuild the packages for
different boards.
Let us assume you are on an A1 platform from now on.
Step 1. Prepare bazel

You need to first install a Java environment.
sudo apt-get update
sudo apt-get install -y openjdk-11-jdk-headless

After that, since we have a pre-compiled bazel binary for aarch64,
you can copy it over to /usr/local/bin and change its permissions.
For building bazel on Amazon A1, please see my other document.
sudo chmod 777 /usr/local/bin/bazel

Verify the command is working:
bazel

Step 2. Install arrow

We need to prepare this package first since ray depends on it.
We will follow the steps for installation.
For simplicity, the steps are summarized here.
sudo apt-get install -y libjemalloc-dev libboost-dev \
					libboost-filesystem-dev \
                    libboost-system-dev \
                    libboost-regex-dev \
                    python3-dev \
                    autoconf \
                    flex bison
pip3 install six numpy pandas cython pytest psutil

I used /opt/builds as my working directory to host various repositories.
For a few additional packages if you have not installed them yet:
sudo apt-get install -y libssl-dev curl cmake

Now we need to set up a few environment variables.
cd /opt/builds
mkdir arrow-dist
export ARROW_HOME=$(pwd)/arrow-dist
export LD_LIBRARY_PATH=$(pwd)/arrow-dist/lib:$LD_LIBRARY_PATH

Eventually the directory arrow-dist will contain all the *.so files that
you need to copy over onto the board.
Now clone the repository and build arrow from source.
cd /opt/builds
git clone https://github.com/apache/arrow/
cd arrow
git checkout -b apache-arrow-0.12.0 tags/apache-arrow-0.12.0
cd cpp
mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX=$ARROW_HOME \
	  -DCMAKE_INSTALL_LIBDIR=lib \
      -DARROW_PYTHON=ON \
      -DARROW_PLASMA=ON \
      -DARROW_BUILD_TESTS=ON \
	  -DPYTHON_EXECUTABLE=/usr/bin/python3 \
      ..
make
sudo make install
cd /opt/builds/arrow/python
PYARROW_WITH_PLASMA=1 python3 setup.py bdist_wheel

After this you should be able to see file
pyarrow-0.12.0-cp36-cp36m-linux_aarch64.whl under
/opt/builds/arrow/python/dist. Copy this onto the board and install.
sudo pip3 install pyarrow-0.12.0-cp36-cp36m-linux_aarch64.whl

Also, copy the arrow-dist folder onto the board, and run
cp -rf arrow-dist/* /usr/

This delivers all the *.so files as well as the header files.
Step 3. Build Wheel

We will build the wheel for ray. One thing to notice, is that with bazel,
we are able to install new versions (not constrained to ray-0.6.3 anymore).
cd /opt/builds
git clone https://github.com/ray-project/ray.git
cd ray
git checkout -b ray-0.6.5 tags/ray-0.6.5
cd python

Patch the following file:
vi /opt/builds/ray/build.sh

Comment the following section around line 98. This makes sure we don't install
arrow from pip since we already have it built locally.
$PYTHON_EXECUTABLE -m pip install \
    --target=$ROOT_DIR/python/ray/pyarrow_files pyarrow==0.12.0.RAY \
    --find-links https://s3-us-west-2.amazonaws.com/arrow-wheels/ca1fa51f0901f5a4298f0e4faea00f24e5dd7bb7/index.html

Try to see if you can pip install it.
python3.6 -m pip install -e . --verbose

Finally build the wheel.
python3 setup.py bdist_wheel

You now should be able to see file ray-0.6.5-cp36-cp36m-linux_aarch64.whl
under /opt/builds/ray/python/dist. Copy this onto the board and install.
sudo pip3 install ray-0.6.5-cp36-cp36m-linux_aarch64.whl

If you want to rebuild from a clean repository, run in the git repository:
git clean -f -f -x -d