Skip to content

Instantly share code, notes, and snippets.

@yunqu

yunqu/build-bazel-qemu-arm.md

Last active February 19, 2022 01:49
Show Gist options
  • Star 1 You must be signed in to star a gist
  • Fork 1 You must be signed in to fork a gist
  • Save yunqu/87568921ba2758b87dd0ec57f301efaf to your computer and use it in GitHub Desktop.
Save yunqu/87568921ba2758b87dd0ec57f301efaf to your computer and use it in GitHub Desktop.
Download ZIP
Build Bazel on QEMU for Pynq-Z1/Pynq-Z2
Raw
build-bazel-qemu-arm.md

Building Bazel on armv7l QEMU

Here you'll learn how to build Bazel for Pynq-Z1 (image v2.4). A similar flow can be run directly on the board (native build). However, this guide is based on QEMU flow, which is running faster and cleaner than a native build on the board.

The same flow can also target aarch64 (ZCU104). You just have to work with the correct image (instead of Pynq-Z1-2.4.img, work with ZCU104-2.4.img).

Requirement

  • A Ubuntu machine as the building environment for PYNQ images.

Steps

1. Prepare the Building Environment

QEMU

You need to prepare a QEMU environment. Usually, if you have a PYNQ building environment, you will already have a QEMU 2.8.0 installed in your Ubuntu system under /opt. However, it is known that the Java is broken on QEMU-2.8.

The first step is to make sure docker can be run without problems. If you have problems with broken packages, please make sure the /etc/apt/sources.list has the default contents.

sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io

It's likely that you will have permission problems with dockers. You need to request docker access in your IDM. If you cannot, please do the following (every time you restarted the machine; not recommended).

sudo chmod 666 /var/run/docker.sock

Test:

docker run hello-world

Let us upgrade it to QEMU 3.1.0. We can install it as shown below.

cd /opt
rm -rf qemu/
wget http://wiki.qemu-project.org/download/qemu-3.1.0.tar.bz2
tar -xf qemu-3.1.0.tar.bz2
cd qemu-3.1.0
./configure --target-list=arm-linux-user,aarch64-linux-user --prefix=/opt/qemu --static
make
sudo make install
cd /opt/qemu/bin
sudo rm -rf qemu-arm-static qemu-aarch64-static
sudo ln -s qemu-arm qemu-arm-static
sudo ln -s qemu-aarch64 qemu-aarch64-static

After a successful installation, you should be able to check:

qemu-arm-static -version

to see QEMU 3.1.0 has been enabled.

Mount the image

The new image can be mounted so you have a QEMU environment of your Pynq-Z1. For example, to mount it in my working directory /opt/builds/PYNQ_20180606:

./scripts/mount_image.sh /opt/builds/PYNQ_20180606/sdbuild/output/Pynq-Z1-2.4.img \
	/opt/builds/PYNQ_20180606/sdbuild/build/bionic.Pynq-Z1
sudo mount -o bind /proc /opt/builds/PYNQ_20180606/sdbuild/build/bionic.Pynq-Z1/proc
sudo mount -o bind /dev /opt/builds/PYNQ_20180606/sdbuild/build/bionic.Pynq-Z1/dev
sudo mount -o bind /run /opt/builds/PYNQ_20180606/sdbuild/build/bionic.Pynq-Z1/run

The default image does not have enough space for building bazel. To solve this issue, we will mount a temporary build folder into out QEMU environment.

sudo mkdir -p /opt/builds/PYNQ_20180606/sdbuild/build/bionic.Pynq-Z1/local/tmp
sudo mount -o bind /tmp /opt/builds/PYNQ_20180606/sdbuild/build/bionic.Pynq-Z1/local/tmp

We will later make sure we indeed use this temporary folder.

Chroot environment

Run

sudo -E chroot /opt/builds/PYNQ_20180606/sdbuild/build/bionic.Pynq-Z1 bash

Now you have an environment just like you are on the Pynq-Z1 board. From now on, we assume you are on that chroot environment.

Once you are on that chroot environment, you can double-check that you have enough disk space for your /local/tmp folder using df -h. For me, this is 100+ GB.

2. Install Basic Dependencies

First, update apt-get to make sure it knows where to download everything. You can skip this step most likely if you are on a fresh image. Then install some packages that we will need.

apt-get update
apt-get install -y pkg-config zip g++ zlib1g-dev unzip 
apt-get install -y autoconf automake libtool openjdk-11-jdk
apt-get install -y python3-pip python3-numpy swig python3-dev
pip3 install wheel

You may want to change gcc/g++ version:

update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-7 10 --slave /usr/bin/g++ g++ /usr/bin/g++-7
update-alternatives --config gcc

And you should be able to check the gcc/g++ version has been updated:

gcc -v
g++ -v

We need to configure the default Java environment now. Put the following script as /usr/lib/jvm/change-java.sh. Change permissions if necessary (chmod 777 change-java.sh).

#!/bin/bash

for file in /usr/lib/jvm/java-1.11.0-openjdk-armhf/bin/*
do
   if [ -x $file ]
   then
      filename=`basename $file`
      update-alternatives --install /usr/bin/$filename $filename $file 20000
      update-alternatives --set $filename $file
      echo $filename
   fi
done

Then run this script.

cd /usr/lib/jvm
./change-java.sh

It will update all the Java paths. After that, check:

java -version

Also, you need to make sure you can compile a Java program. Create a file /local/tmp/HelloWorld.java as follows:

public class HelloWorld {
    public static void main(String[] args) {
        // Prints "Hello, World" to the terminal window.
        System.out.println("Hello, World");
    }
}

Compile it and run it to make sure nothing is broken.

cd /local/tmp
javac HelloWorld.java
java HelloWorld

Finally, for cleanliness, make a directory that will hold the Bazel repository.

cd /local/tmp
mkdir work
cd work

3. Make Patches

To build Bazel, we're going to need to download a zip file containing a distribution archive. Let's do that now and extract it into a new directory called bazel (Releases later than 0.5.3 failed to build):

wget https://github.com/bazelbuild/bazel/releases/download/0.5.3/bazel-0.5.3-dist.zip
unzip -d bazel bazel-0.5.3-dist.zip
cd bazel

Now we need to change the permissions of every files in the bazel project with:

chmod u+w ./* -R

Run the following to set environment:

export JAVA_HOME=/usr/lib/jvm/java-1.11.0-openjdk-armhf
export TMPDIR=/local/tmp

Patch 1

Before building Bazel, we need to set the javac maximum heap size for this job, or else we'll get an OutOfMemoryError. To do this, we need to make a small addition to bazel/scripts/bootstrap/compile.sh.

vi scripts/bootstrap/compile.sh

Move down to line around 120, where you'll see the following block of code:

run "${JAVAC}" -classpath "${classpath}" -sourcepath "${sourcepath}" \
      -d "${output}/classes" -source "$JAVA_VERSION" -target "$JAVA_VERSION" \
      -encoding UTF-8 "@${paramfile}"

At the end of this block, add in the -J-Xmx2048M flag, which sets the maximum size of the Java heap to 2048 MB:

run "${JAVAC}" -classpath "${classpath}" -sourcepath "${sourcepath}" \
      -d "${output}/classes" -source "$JAVA_VERSION" -target "$JAVA_VERSION" \
      -encoding UTF-8 "@${paramfile}" -J-Xmx2048M

Patch 2

Go to file (depending on your version, the file can also be mapped_file_posix.inc):

vi src/tools/singlejar/mapped_file.h

Comment out the line:

#error This code for 64 bit Unix.

Patch 3

Go to file

vi tools/cpp/CROSSTOOL

Add the following lines when you locate linker_flag:

linker_flag: "-Wl,-R/usr/lib/arm-linux-gnueabihf"
cxx_builtin_include_directory: "/usr/lib/gcc/arm-linux-gnueabihf/7/include"
cxx_builtin_include_directory: "/usr/lib/gcc/arm-linux-gnueabihf/7/include-fixed"

Patch 4

The following patch is based on git commit cc8e716. Depending on the bazel version, you may not need to do this.

vi src/java_tools/buildjar/java/com/google/devtools/build/buildjar/javac/plugins/errorprone/ErrorPronePlugin.java

Add in line 21:

import com.google.errorprone.BaseErrorProneJavaCompiler;

Remove around line 33:

import com.sun.tools.javac.util.JavacMessages;

Modify the line around line 69 from

JavacMessages.instance(context).add("com.google.errorprone.errors");

into

BaseErrorProneJavaCompiler.setupMessageBundle(context);

4. Build Bazel

Now we can build Bazel!

./compile.sh

This takes a long time (at least 30 mins). When the build finishes, you end up with a new binary output/bazel. Copy that to your /usr/local/bin directory.

cp output/bazel /usr/local/bin/bazel

To make sure it's working properly, run bazel on the command line and verify it prints help text.

cd /local/tmp
bazel

Usage: bazel <command> <options> ...

Available commands:
  analyze-profile     Analyzes build profile data.
  build               Builds the specified targets.
  canonicalize-flags  Canonicalizes a list of bazel options.
  clean               Removes output files and optionally stops the server.
  dump                Dumps the internal state of the bazel server process.
  fetch               Fetches external repositories that are prerequisites to the targets.
  help                Prints help for commands, or the index.
  info                Displays runtime info about the bazel server.
  mobile-install      Installs targets to mobile devices.
  query               Executes a dependency graph query.
  run                 Runs the specified target.
  shutdown            Stops the bazel server.
  test                Builds and runs the specified test targets.
  version             Prints version information for bazel.

Getting more help:
  bazel help <command>
                   Prints help and options for <command>.
  bazel help startup_options
                   Options for the JVM hosting bazel.
  bazel help target-syntax
                   Explains the syntax for specifying targets.
  bazel help info-keys
                   Displays a list of keys used by the info command.

Now you can copy that binary around onto any Pynq-Z1 board with v2.4 image! Congratulations!

5. Cleaning Up

You can exit the chroot environment now.

exit

Check the image you have mounted:

sudo losetup -a

If you have an image mounted, you will get a return something like:

/dev/loop0: [2049]:3671625 (/opt/builds/PYNQ_20180606/sdbuild/output/Pynq-Z1-2.4.img)

You need to unmount all the images:

sudo umount -l /opt/builds/PYNQ_20180606/sdbuild/build/bionic.Pynq-Z1/proc
sudo umount -l /opt/builds/PYNQ_20180606/sdbuild/build/bionic.Pynq-Z1/dev
sudo umount -l /opt/builds/PYNQ_20180606/sdbuild/build/bionic.Pynq-Z1/run
sudo umount -l /opt/builds/PYNQ_20180606/sdbuild/build/bionic.Pynq-Z1/local/tmp
sudo umount /opt/builds/PYNQ_20180606/sdbuild/build/bionic.Pynq-Z1/boot
sudo umount /opt/builds/PYNQ_20180606/sdbuild/build/bionic.Pynq-Z1
sudo kpartx -d /opt/builds/PYNQ_20180606/sdbuild/output/Pynq-Z1-2.4.img

Finally check you can check whether you have a clean environment again. The following command should return nothing.

sudo losetup

References

  1. Building TensorFlow for Jetson TK1 (an update to this post is available here)
  2. Turning a USB Drive into swap
  3. Safely removing USB swap space
  4. Original guide
  5. Tensorflow for Go on rpi3
  6. Compiling the C++ interface
  7. Compiling for Tensorflow C++ API
  8. Mfpu neon issue thread
  9. Building TensorFlow 1.3.0-rc1 for Raspberry Pi/Ubuntu 16.04: a Step-By-Step Guide
  10. Install newer GCC versions in Ubuntu
  11. How to install java 1.5 JDK in Ubuntu
  12. Instructions on installing JAVA
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment