Using the `distcc` distributed compiler with Bela
Hints on getting faster compile times without a cross-compiler
Compiling programs on the BeagleBone Black, depending on what you are developing and how frequently, can be slow.
Distributed compilation allows you to use a more powerful processor (a host in this article, likely your personal laptop/desktop) to do the grunt work for you.
This article explains how to use the distributed compiler
distcc to compile programs on Bela, including Bela programs themselves (although the latter workflow is currently "imperfect").
This article is broken down into these sections:
- Setting up
- A Bela, preferably with image version ≥
v0.3.1. The version is displayed when you
ssh email@example.com Bela.
- A host machine (e.g. your laptop) connected to Bela by Ethernet over USB
- A recent release of
- Other dependencies (macOS):
brew install autoconf)
- Clang 3.9. This can be installed via MacPorts (try
sudo port install clang @3.9or
sudo port install clang-3.9) (apparently this fails on newer versions of
macOS), or download the pre-built binaries from LLVM.org, (Direct download), and add them to your
- Optional: if you may want to cross-compile without
distcclater, then install
sshfs("FUSE for OS X" on
For Ubuntu 18.04:
sudo apt update sudo apt install autoconf clang-3.9
Installing on Bela
Since Bela image version
distcc is installed by default.
If you are running an older version of the Bela image, you may want to consider upgrading rather than installing
See this article for more on flashing a Bela image: https://github.com/BelaPlatform/Bela/wiki/Manage-your-SD-card
Installing on the host
distcc on your host machine, first download the latest
distcc release from here: https://github.com/distcc/distcc/releases. Documentation for
distcc can be found here: https://distcc.github.io/.
Steps to install (tested on
cd /path/to/distcc autoconf configure.ac # if this fails, run the line below instead sudo ./autogen.sh # if `aclocal` not found: `brew install automake` ./configure --without-libiberty --disable-Werror sudo make && sudo make install
For information about the flags for
./configure, see: marksatt/DistCode#13
Setting up on Bela
NOTE 2022-01-05: As of Bela Image 0.3.8f (and possibly earlier, haven't tested), the following setup on Bela is already done by default.
There are two steps to setting up:
- Creating and making executable scripts that will run
clang-3.9on Bela with custom options as needed
- Creating additional executable scripts that will run the above script using
The goal of setting up
distcc on Bela is to have a simple syntax for running
distcc, without depending on edits to a profile such as
To achieve this, the environment variables for
distcc are going to be set in the scripts we create in the second step.
On Bela, create the following script in
#!/bin/bash clang-3.9 $@
Then, create the following script in
#!/bin/bash clang++-3.9 $@ -stdlib=libstdc++
Make the scripts executable:
chmod +x /usr/local/bin/clang-3.9-arm chmod +x /usr/local/bin/clang++-3.9-arm
... Or do the above in a single step if you are feeling fly:
printf '#!/bin/bash\nclang-3.9 $@' > /usr/local/bin/clang-3.9-arm printf '#!/bin/bash\nclang++-3.9 $@' > /usr/local/bin/clang++-3.9-arm chmod +x /usr/local/bin/clang-3.9-arm chmod +x /usr/local/bin/clang++-3.9-arm
On Bela, create the following script in
#!/bin/bash export DISTCC_HOSTS=192.168.7.1 export DISTCC_VERBOSE=0 export DISTCC_FALLBACK=0 # does not seem to apply to the version of distcc in use export DISTCC_BACKOFF_PERIOD=0 export TMPDIR=/tmp/distcc mkdir -p /tmp/distcc distcc clang-3.9-arm $@
Then, create the following script in
#!/bin/bash export DISTCC_HOSTS=192.168.7.1 export DISTCC_VERBOSE=0 export DISTCC_FALLBACK=0 # does not seem to apply to the version of distcc in use export DISTCC_BACKOFF_PERIOD=0 export TMPDIR=/tmp/distcc mkdir -p /tmp/distcc distcc clang++-3.9-arm $@
Make these scripts executable:
chmod +x /usr/local/bin/distcc-clang chmod +x /usr/local/bin/distcc-clang++
Setting up on the host
This section will differ depending on if you installed Clang 3.9 via MacPorts or from LLVM.org.
Namely, the MacPorts executables will be called
clang++-mp-3.9, etc, whereas the LLVM.org executables will be named
Alter the following instructions as necessary.
Only one set of scripts are needed on the host, which essentially create a shortcut to running
clang-3.9 with some specific command line options to make it generate code for ARM, our target platform.
Create the folder
~/arm or similar (if you choose something else, then modify the scripts that follow accordingly):
mkdir -p ~/arm
On the host, create the following script in
#!/bin/bash /path/to/your/clang-3.9 -target armv7l-unknown-linux-gnueabihf --sysroot ~/arm $@
Create the following script in
#!/bin/bash /path/to/your/clang++-3.9 -target armv7l-unknown-linux-gnueabihf --sysroot ~/arm $@
Make them executable:
sudo chmod +x /usr/local/bin/clang-3.9-arm sudo chmod +x /usr/local/bin/clang++-3.9-arm
Note on setting up 1: We created
clang-3.9-armon the host so that we can easily compile to ARM.
distccrequires us to have on Bela an executable file called
clang-3.9-arm, so that the local and remote compiler are "the same". While we could have on Bela used a
/usr/bin/clang, the method we have used allows us more flexibility to inject custom options as needed.
Note on setting up 2: We added
--sysroot ~/arm $@to the host scripts. These should not be necessary with
distcc, since all the
includesare resolved on the client (Bela), and so the host does not really need any paths. However if you wanted to cross-compile without
distccyou would need this path. To do this you will also need to have installed
sshfs firstname.lastname@example.org: ~/arm.
distcc requires two processes:
- Running the
distccddaemon on the host
- Compile programs from Bela
distcc daemon) on the host:
distccd --verbose --no-detach --daemon --allow 192.168.7.2 --log-level debug --log-file ~/distccd.log
--make-me-a-botnetmay also be required, otherwise
distccdwill search in the wrong paths and when monitoring the log
~/distccd.logyou will something like:
distccd (dcc_warn_masquerade_whitelist) CRITICAL! /usr/local/lib/distcc not found. It seems that
distccthinks that user installed software on a Mac should be in
/usr/local, but this changed a few macOS' ago.
On Bela, you should now be able to compile a test program:
Which will produce the build output
This will not try to compile remotely, you need to have
-c for something to be sent out to the server. The reason for this is that without
-c then there is a linking stage included, and the linking is always done locally:
distcc-clang -c ~/Bela/resources/network/udp-client.c
distcc by default on Bela
Make the following edit to the Bela Makefile at
DISTCC ?= 1 # set this to 1 to use distcc by default
distcc to compile Bela programs
While it is possible to compile Bela programs using
distcc, there is currently an issue with re-compiling dependencies:
"if files are compiled with distcc, then dependency files point to the wrong target" - BelaPlatform/Bela#354
What this means is that when a
.h file is modified,
make may fail to realise that all the
.cpp files that include it need to be recompiled.
To workaround this, if you are editing a
.h file, make sure that the
.cpp file is also touched (or the old
.o file is deleted), so that
make will know that it has to recompile it.
To see the logs, run on the host:
tail -f ~/distccd.log
To make sure the process is actually called on Bela, run on Bela:
watch -n 0.4 bash -c "ps aux | grep clang-3.9-arm"
watch -n 0.4 bash -c "ps aux | grep clang++-3.9-arm"
From Bela, you should also be able to see it in
htop and the
macOS app "Activity Monitor" on the host.
On Bela, you can also inspect the
stderr returned by the host
More usage notes
You may experience an issue where the
DISTCC_BACKOFF_PERIOD flag is not obeyed, possibly due to the version of
distcc being used on Bela.
You can compensate for this by running a program such as:
watch -n 0.4 rm -rf "~/.distcc/lock/*"
This will compensate for
DISTCC_BACKOFF_PERIOD=0 not working by removing periodically any "lock" files that
distcc uses internally to timeout the backoff.