Multiple Servers

.gitignore

.vagrant
ubuntu-xenial-16.04-cloudimg-console.log

MULTIPLE_SERVERS.md

MULTIPLE SERVERS

Distributed setup of:

• ccache
• memcached
• distcc/distccd

By default the servers will only listen to 127.0.0.1 (localhost), so you will need to change to 0.0.0.0 (wildcard) or suitable subset.

Edit the server configuration on each, in:

• /etc/default/distcc
• /etc/memcached.conf

Make sure to restart the services afterwards.

You will need to define your servers:

$ export DISTCC_HOSTS="<distccd>"
$ export MEMCACHED_SERVERS="<memcached>"

Configure your ccache with memcached support:

$ export CCACHE_PREFIX=distcc
$ export CCACHE_MEMCACHED_CONF=--SERVER=<memcached>:11211

And your compiler cache should now be distributed!

Update regarding the preprocessor

When using ccache "run_second_cpp" (which is now the default), you will need to run distcc in pump mode - for multiple files.

$ export DISTCC_HOSTS="<distccd>,lzo,cpp"
$ eval `distcc-pump --startup`
$ make -j80
$ distcc-pump --shutdown

Otherwise distcc will precompile your files, before sending them over the network to the actual compile server, regardless.

Running a local memcached proxy

It is possible to run a local moxi server, that will proxy the actual network requests to the actual memcached server(s).

Doing so means higher performance by caching in local memory, and also being less susceptible to transient network errors.

$ sudo service moxi-server start
$ export CCACHE_MEMCACHED_CONF=--SERVER=localhost:11211

---

For the impatient, here's a Vagrantfile to do it for you.

Vagrantfile

# -*- mode: ruby -*-
# vi: set ft=ruby :

# All Vagrant configuration is done below. The "2" in Vagrant.configure
# configures the configuration version (we support older styles for
# backwards compatibility). Please don't change it unless you know what
# you're doing.
Vagrant.configure("2") do |config|
  # The most common configuration options are documented and commented below.
  # For a complete reference, please see the online documentation at
  # https://docs.vagrantup.com.

  # Every Vagrant development environment requires a box. You can search for
  # boxes at https://vagrantcloud.com/search.
  config.vm.box = "ubuntu/xenial64"

  # Disable automatic box update checking. If you disable this, then
  # boxes will only be checked for updates when the user runs
  # `vagrant box outdated`. This is not recommended.
  # config.vm.box_check_update = false

  if Vagrant.has_plugin?("vagrant-cachier")
    config.cache.scope = :machine
  end

  # distccd
  config.vm.define "distccd" do |distccd|
    distccd.vm.hostname = 'distccd'
    distccd.vm.network "private_network", ip: "192.168.60.10"
    distccd.vm.provision :shell, :inline => <<-SHELL
      apt-get update
      apt-get install -y distcc gcc
      sed -e 's/^STARTDISTCC.*/STARTDISTCC="true"/' -i.orig /etc/default/distcc
      sed -e 's/^ALLOWEDNETS.*/ALLOWEDNETS="192.168.60.0\\/24"/' -i /etc/default/distcc
      sed -e 's/^LISTENER.*/LISTENER="0.0.0.0"/' -i /etc/default/distcc
      service distcc restart
    SHELL
  end

  # memcached
  config.vm.define "memcached" do |memcached|
    memcached.vm.hostname = 'memcached'
    memcached.vm.network "private_network", ip: "192.168.60.11"
    memcached.vm.provision :shell, :inline => <<-SHELL
      apt-get update
      apt-get install -y memcached
      # assume a default memory of 512M, increase from 64M
      sed -e 's/^-m.*/-m 256/' -i.orig /etc/memcached.conf
      sed -e 's/^-l.*/-l 0.0.0.0/' -i /etc/memcached.conf
      service memcached restart
    SHELL
  end

  # ccache
  config.vm.define "ccache" do |ccache|
    ccache.vm.hostname = 'ccache'
    ccache.vm.network "private_network", ip: "192.168.60.12"
    ccache.vm.provision :shell, :privileged => false, :inline => <<-SHELL
      sudo apt-get update
      sudo apt-get install -y build-essential git

      git clone https://github.com/afbjorklund/memccache.git
      cd memccache
      sudo apt-get install -y autoconf
      ./autogen.sh

      sudo apt-get install -y gperf libmemcached-dev
      ./configure
      make -j `nproc`
      make unittest
      sudo apt-get install -y --no-install-recommends asciidoc xsltproc docbook-xml docbook-xsl
      sudo make install

      sudo apt-get install -y distcc-pump libmemcached-tools
      export DISTCC_HOSTS=192.168.60.10,lzo,cpp
      export MEMCACHED_SERVERS=192.168.60.11
      memcping

      sudo apt-get install -y nutcracker
      sudo mkdir -p /etc/nutcracker
      sudo tee /etc/nutcracker/nutcracker.yml <<EOF
pool:
    listen: 127.0.0.1:11311
    preconnect: false
    redis: false
    servers:
        - 192.168.60.11:11211:1
EOF
      sudo /etc/init.d/nutcracker stop
      sudo /etc/init.d/nutcracker start

      export CCACHE=memccache
      export CCACHE_NOCPP2=1
      export CCACHE_COMPRESS=1
      export CCACHE_COMPRESSTYPE=lz4f
      export CCACHE_PREFIX=distcc
      export CCACHE_MEMCACHED_CONF=--SERVER=127.0.0.1:11311

      export DISTCC=distcc-pump
      export DISTCC_LOG=$PWD/distcc.log DISTCC_VERBOSE=1
      export CCACHE_LOGFILE=$PWD/ccache.log
      echo 'int main() {}' > test.c
      $DISTCC $CCACHE cc -c test.c
      cat ccache.log
      cat distcc.log

      export MEMCACHED_SERVERS=192.168.60.11:11211
      set -x
      memcping
      memcdump
      set +x

      $CCACHE -C
      $CCACHE cc -c test.c
      $CCACHE -s
      set -x
      memcstat
      set +x

      true
    SHELL
  end
end