Skip to content

Instantly share code, notes, and snippets.

@jarmitage jarmitage/.bashrc (bela)
Last active Nov 11, 2019

Embed
What would you like to do?
Using the `distcc` distributed compiler with Bela
#!/bin/bash
alias watchdistcc="watch -n 0.4 rm -rf \"~/.distcc/lock/*\""
#!/bin/bash
alias distccdbela="distccd --verbose --no-detach --daemon --allow 192.168.7.2 --log-level debug --log-file ~/.distcc/distccd.log --enable-tcp-insecure & tail -f ~/.distcc/distccd.log"
alias killdistccd="killall distccd"
layout title date categories description image author
post
Using the `distcc` distributed compiler with Bela
2018-12-04
Tutorials
Software
Hints on getting faster compile times without a cross-compiler
bela-and-supercollider-live-coding-sensors/alo-allik.jpg
giulio

Compiling programs on the BeagleBone Black, depending on what you are developing and how frequently, can be slow. Distributed compilation allows you to use a more powerful processor (a host in this article, likely your personal laptop/desktop) to do the grunt work for you. This article explains how to use the distributed compiler distcc to compile programs on Bela, including Bela programs themselves (although the latter workflow is currently "imperfect").

This article is broken down into these sections:

  • Requirements
  • Installing distcc
  • Setting up distcc
  • Using distcc

Requirements

  • A Bela, preferably with image version ≥v0.3.1. The version is displayed when you ssh root@192.168.7.2 into Bela.
  • A host machine (e.g. your laptop) connected to Bela by Ethernet over USB
  • A recent release of distcc: https://github.com/distcc/distcc/releases
  • Other dependencies (macOS):
    • autoconf (e.g. brew install autoconf)
    • Clang 3.9. This can be installed via MacPorts (try sudo port install clang @3.9 or sudo port install clang-3.9) (apparently this fails on newer versions of macOS), or download the pre-built binaries from LLVM.org, (Direct download), and add them to your PATH environment variable.
  • Optional: if you may want to cross-compile without distcc later, then install sshfs ("FUSE for OS X" on macOS: https://osxfuse.github.io/)

For Ubuntu 18.04:

sudo apt update
sudo apt install autoconf clang-3.9

Installing distcc

Installing on Bela

Since Bela image version v0.3.1, distcc is installed by default. If you are running an older version of the Bela image, you may want to consider upgrading rather than installing distcc manually. See this article for more on flashing a Bela image: https://github.com/BelaPlatform/Bela/wiki/Manage-your-SD-card

Installing on the host

To install distcc on your host machine, first download the latest distcc release from here: https://github.com/distcc/distcc/releases. Documentation for distcc can be found here: https://distcc.github.io/.

Steps to install (tested on macOS Mojave):

cd /path/to/distcc

autoconf configure.ac # if this fails, run the line below instead
sudo ./autogen.sh # if `aclocal` not found: `brew install automake`

./configure --without-libiberty --disable-Werror
sudo make && sudo make install

For information about the flags for ./configure, see: https://github.com/marksatt/DistCode/issues/13

Setting up distcc

Setting up on Bela

There are two steps to setting up:

  • Creating and making executable scripts that will run clang-3.9 on Bela with custom options as needed
  • Creating additional executable scripts that will run the above script using distcc

The goal of setting up distcc on Bela is to have a simple syntax for running distcc, without depending on edits to a profile such as ~/.bashrc. To achieve this, the environment variables for distcc are going to be set in the scripts we create in the second step.

The clang-3.9 scripts

On Bela, create the following script in /usr/local/bin/clang-3.9-arm:

#!/bin/bash
clang-3.9 $@

Then, create the following script in /usr/local/bin/clang++-3.9-arm:

#!/bin/bash
clang++-3.9 $@ -stdlib=libstdc++

Make the scripts executable:

chmod +x /usr/local/bin/clang-3.9-arm
chmod +x /usr/local/bin/clang++-3.9-arm

... Or do the above in a single step if you are feeling fly:

printf '#!/bin/bash\nclang-3.9 $@' > /usr/local/bin/clang-3.9-arm
printf '#!/bin/bash\nclang++-3.9 $@' > /usr/local/bin/clang++-3.9-arm
chmod +x /usr/local/bin/clang-3.9-arm
chmod +x /usr/local/bin/clang++-3.9-arm

The distcc scripts

On Bela, create the following script in /usr/local/bin/distcc-clang:

#!/bin/bash
export DISTCC_HOSTS=192.168.7.1
export DISTCC_VERBOSE=0
export DISTCC_FALLBACK=0 # does not seem to apply to the version of distcc in use
export DISTCC_BACKOFF_PERIOD=0
export TMPDIR=/tmp/distcc
mkdir -p /tmp/distcc

distcc clang-3.9-arm $@

Then, create the following script in /usr/local/bin/distcc-clang++:

#!/bin/bash
export DISTCC_HOSTS=192.168.7.1
export DISTCC_VERBOSE=0
export DISTCC_FALLBACK=0 # does not seem to apply to the version of distcc in use
export DISTCC_BACKOFF_PERIOD=0
export TMPDIR=/tmp/distcc
mkdir -p /tmp/distcc

distcc clang++-3.9-arm $@

Make these scripts executable:

chmod +x /usr/local/bin/distcc-clang
chmod +x /usr/local/bin/distcc-clang++

Setting up on the host

This section will differ depending on if you installed Clang 3.9 via MacPorts or from LLVM.org. Namely, the MacPorts executables will be called clang-mp-3.9, clang++-mp-3.9, etc, whereas the LLVM.org executables will be named clang-3.9, clang++, etc. Alter the following instructions as necessary.

Only one set of scripts are needed on the host, which essentially create a shortcut to running clang-3.9 with some specific command line options to make it generate code for ARM, our target platform.

Create the folder ~/arm or similar (if you choose something else, then modify the scripts that follow accordingly):

mkdir -p ~/arm

On the host, create the following script in /usr/local/bin/clang-3.9-arm:

#!/bin/bash
/path/to/your/clang-3.9 -target armv7l-unknown-linux-gnueabihf --sysroot ~/arm $@

Create the following script in /usr/local/bin/clang++-3.9-arm:

#!/bin/bash
/path/to/your/clang++-3.9 -target armv7l-unknown-linux-gnueabihf --sysroot ~/arm $@

Make them executable:

sudo chmod +x /usr/local/bin/clang-3.9-arm
sudo chmod +x /usr/local/bin/clang++-3.9-arm

Note on setting up 1: We created clang-3.9-arm on the host so that we can easily compile to ARM. distcc requires us to have on Bela an executable file called clang-3.9-arm, so that the local and remote compiler are "the same". While we could have on Bela used a symlink to /usr/bin/clang, the method we have used allows us more flexibility to inject custom options as needed.

Note on setting up 2: We added --sysroot ~/arm $@ to the host scripts. These should not be necessary with distcc, since all the includes are resolved on the client (Bela), and so the host does not really need any paths. However if you wanted to cross-compile without distcc you would need this path. To do this you will also need to have installed sshfs and run sshfs root@192.168.7.2: ~/arm.

Using distcc

Using distcc requires two processes:

  • Running the distccd daemon on the host
  • Compile programs from Bela

Start distccd (the distcc daemon) on the host:

distccd --verbose --no-detach --daemon --allow 192.168.7.2 --log-level debug --log-file ~/distccd.log

macOS note: --make-me-a-botnet may also be required, otherwise distccd will search in the wrong paths and when monitoring the log ~/distccd.log you will something like: distccd[53245] (dcc_warn_masquerade_whitelist) CRITICAL! /usr/local/lib/distcc not found. It seems that distcc thinks that user installed software on a Mac should be in /usr instead of /usr/local, but this changed a few macOS' ago.

On Bela, you should now be able to compile a test program:

distcc-clang ~/Bela/resources/network/udp-client.c

Which will produce the build output a.out.

This will not try to compile remotely, you need to have -c for something to be sent out to the server. The reason for this is that without -c then there is a linking stage included, and the linking is always done locally:

distcc-clang -c ~/Bela/resources/network/udp-client.c

Using distcc by default on Bela

Make the following edit to the Bela Makefile at /root/Bela/Makefile:

 endif

 ifeq ($(COMPILER), clang)
-  CLANG_PATH?=/usr/bin/clang
+ CLANG_PATH=distcc-clang
   CC=$(CLANG_PATH)
   CXX=$(CLANG_PATH)++
   DEFAULT_CPPFLAGS += -DNDEBUG # Maybe we should add back in -no-integrated-as?

Using distcc to compile Bela programs

While it is possible to compile Bela programs using distcc, there is currently an issue with re-compiling dependencies:

"if files are compiled with distcc, then dependency files point to the wrong target" - https://github.com/BelaPlatform/Bela/issues/354

What this means is that when a .h file is modified, make may fail to realise that all the .cpp files that include it need to be recompiled.

To workaround this, if you are editing a .h file, make sure that the .cpp file is also touched (or the old .o file is deleted), so that make will know that it has to recompile it.

Monitoring distcc

To see the logs, run on the host:

tail -f ~/distccd.log

To make sure the process is actually called on Bela, run on Bela:

watch -n 0.4 bash -c "ps aux | grep clang-3.9-arm"
watch -n 0.4 bash -c "ps aux | grep clang++-3.9-arm"

From Bela, you should also be able to see it in top or htop and the macOS app "Activity Monitor" on the host.

On Bela, you can also inspect the stderr returned by the host

cat /tmp/distcc/*txt

More usage notes

You may experience an issue where the DISTCC_BACKOFF_PERIOD flag is not obeyed, possibly due to the version of distcc being used on Bela. You can compensate for this by running a program such as:

watch -n 0.4 rm -rf "~/.distcc/lock/*"

This will compensate for DISTCC_BACKOFF_PERIOD=0 not working by removing periodically any "lock" files that distcc uses internally to timeout the backoff.

#!/bin/bash
clang++-3.9 $@ -stdlib=libstdc++
#!/bin/bash
/path/to/your/clang++-3.9 -target armv7l-unknown-linux-gnueabihf --sysroot ~/arm $@
#!/bin/bash
clang-3.9 $@
#!/bin/bash
/path/to/your/clang-3.9 -target armv7l-unknown-linux-gnueabihf --sysroot ~/arm $@
#!/bin/bash
export DISTCC_HOSTS=192.168.7.1
export DISTCC_VERBOSE=0
export DISTCC_FALLBACK=0 # does not seem to apply to the version of distcc in use
export DISTCC_BACKOFF_PERIOD=0
export TMPDIR=/tmp/distcc
mkdir -p /tmp/distcc
distcc clang-3.9-arm $@
#!/bin/bash
export DISTCC_HOSTS=192.168.7.1
export DISTCC_VERBOSE=0
export DISTCC_FALLBACK=0 # does not seem to apply to the version of distcc in use
export DISTCC_BACKOFF_PERIOD=0
export TMPDIR=/tmp/distcc
mkdir -p /tmp/distcc
distcc clang++-3.9-arm $@
@Digital-Larry

This comment has been minimized.

Copy link

Digital-Larry commented Jun 6, 2019

  1. To be more clear, in 2018-12-04_cross-compiling.md:
  • Make the following edit to the Bela Makefile: /root/Bela/Makefile
  1. Add instructions for Ubuntu 18.04:
sudo apt update
sudo apt install autoconf clang-3.9
@jarmitage

This comment has been minimized.

Copy link
Owner Author

jarmitage commented Jun 16, 2019

@Digital-Larry thanks, updated

@jarmitage

This comment has been minimized.

Copy link
Owner Author

jarmitage commented Nov 11, 2019

macOS Catalina breaks this workflow by disabling clang-3.9 binaries because they are not notarized

Screenshot 2019-11-11 at 15 31 54

Presumably, these binaries would need to be recompiled by the LLVM project with a fix for notarization

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.