ccache/memcached/distcc

.gitignore

.vagrant/
ubuntu-xenial-16.04-cloudimg-console.log

GETTING_STARTED.md

GETTING STARTED

How to get started with:

• ccache
• memcached
• distcc/distccd

Pull request:

ccache/ccache#58

These instructions are for Ubuntu 16.04 LTS.

distcc (server)

Install the distcc client/server component.

$ sudo apt-get install distcc

Setup the server configuration (127.0.0.1):

$ export DISTCC_HOSTS=127.0.0.1

Edit /etc/default/distcc to allow nets/hosts.

$ sudo service distcc restart

Run distccd with --stats, for statistics. [optional]

DAEMON_ARGS="$DAEMON_ARGS --stats"
$ telnet localhost 3633

memcached (server)

Install the memcached server component.

$ sudo apt-get install memcached

Setup the server configuration (default=64M):

$ export MEMCACHED_SERVERS=localhost

Edit /etc/memcached.conf to increase memory.

$ sudo service memcached restart

Also install memstat, for statistics. [optional]

$ sudo apt-get install libmemcached-tools
$ memcstat # ubuntu has renamed this tool

ccache (client)

Build ccache with memcached support (needs libmemcached):

$ git clone https://github.com/ccache/ccache.git
$ cd ccache
$ git checkout dev/memcached
$ sudo apt-get install autoconf
$ ./autogen.sh

$ sudo apt-get install libmemcached-dev
$ ./configure --enable-memcached
$ make -j4
$ make test
$ sudo apt-get install --no-install-recommends asciidoc xsltproc docbook-xml docbook-xsl
$ sudo make install

Upload the existing $CCACHE_DIR cache contents [optional]:

$ sudo apt-get install python-memcache
$ ./upload-memcached.py

Configure memcached servers to use for ccache (see above):

$ export CCACHE_MEMCACHED_CONF="--SERVER=localhost"

Configure ccache to use distcc for compilation (see above):

$ export CCACHE_PREFIX="distcc"

The compilation will be done by the distcc server (distccd), only the preprocessing will be done locally (from ccache).

Configure ccache to only run the preprocessor (cpp) once:

$ export CCACHE_NOCPP2=1

testing (logging)

Make a simple compilation, with verbose logging enabled:

$ echo 'int main() {}' > test.c
$ export CCACHE_LOGFILE=$PWD/ccache.log
$ export DISTCC_VERBOSE=1 DISTCC_LOG=$PWD/distcc.log
$ ccache cc -c test.c

This will create one manifest entry and one object entry:

664202a7dafbe30973103db470837ccd-143
fc5f78b19f525061889c52036a902cbd-253

Enabling us to get direct hits, even when not in local cache...

You can now continue by adding more (i.e. non-local) servers.

---

For the impatient, here's a Vagrantfile to do it for you.

Vagrantfile

# -*- mode: ruby -*-
# vi: set ft=ruby :

# All Vagrant configuration is done below. The "2" in Vagrant.configure
# configures the configuration version (we support older styles for
# backwards compatibility). Please don't change it unless you know what
# you're doing.
Vagrant.configure("2") do |config|
  # The most common configuration options are documented and commented below.
  # For a complete reference, please see the online documentation at
  # https://docs.vagrantup.com.

  # Every Vagrant development environment requires a box. You can search for
  # boxes at https://vagrantcloud.com/search.
  config.vm.box = "ubuntu/xenial64"

  # Disable automatic box update checking. If you disable this, then
  # boxes will only be checked for updates when the user runs
  # `vagrant box outdated`. This is not recommended.
  # config.vm.box_check_update = false

  if Vagrant.has_plugin?("vagrant-cachier")
    config.cache.scope = :machine
  end

  config.vm.provision :shell, :privileged => true, :inline => <<-SHELL
    apt-get update

    apt-get install -y distcc
    sed -e 's/^STARTDISTCC.*/STARTDISTCC="true"/' -i.orig /etc/default/distcc
    sed -e 's/^ALLOWEDNETS.*/ALLOWEDNETS="127.0.0.0\\/16"/' -i /etc/default/distcc
    service distcc restart

    apt-get install -y memcached
    # assume a default memory of 512M, increase from 64M
    sed -e 's/^-m.*/-m 256/' -i.orig /etc/memcached.conf
    service memcached restart
  SHELL

  config.vm.provision :shell, :privileged => false, :inline => <<-SHELL
    sudo apt-get install -y build-essential git

    git clone https://github.com/afbjorklund/memccache.git
    cd memccache
    sudo apt-get install -y autoconf
    ./autogen.sh

    sudo apt-get install -y gperf libmemcached-dev
    ./configure
    make -j `nproc`
    make unittest
    sudo apt-get install -y --no-install-recommends asciidoc xsltproc docbook-xml docbook-xsl
    sudo make install

    sudo apt-get install -y distcc libmemcached-tools
    export DISTCC_HOSTS=127.0.0.1
    export MEMCACHED_SERVERS=127.0.0.1
    memcping

    export CCACHE=memccache
    export CCACHE_NOCPP2=1
    export CCACHE_COMPRESS=1
    export CCACHE_COMPRESSTYPE=lz4f
    export CCACHE_PREFIX=distcc
    export CCACHE_MEMCACHED_CONF=--SERVER=127.0.0.1:11211

    export DISTCC_LOG=$PWD/distcc.log DISTCC_VERBOSE=1
    export CCACHE_LOGFILE=$PWD/ccache.log
    echo 'int main() {}' > test.c
    $CCACHE cc -c test.c
    cat ccache.log
    cat distcc.log

    sudo more /var/log/distccd.log
    export MEMCACHED_SERVERS=127.0.0.1:11211
    set -x
    memcping
    memcdump
    set +x

    $CCACHE -C
    $CCACHE cc -c test.c
    $CCACHE -s
    set -x
    memcstat
    set +x

    true
  SHELL
end

sudo make install for ccache assumes asciidoc is installed. Might want to add sudo apt-get install asciidoc before this line

https://gist.github.com/itensionanders/2bd0056027f8308d4cd8/revisions#diff-df17b1aeaf5e7d797809ea45cc1271acR62

Hmm, asciidoc installs the world as depends... Or at least recommends.

Few questions on the implementation before i try it out. This seems to be good fit for requirement.

1. In case ccache hit misses on localdisk and found it in memcache, will it copy results to ccache based on local disk too ? So that next time double search can be prevented?
2. Can i have memcached server connected to multiple distcc servers? Will it allow parallel writes to the memcached server from all the distcc servers?
3. What happens in case two distcc servers try to write same hash object at the same time to memcache server?
4. Can i have multiple memcached servers? If so, how do CCACHE search for available ccache objects among multiple servers?
5. Do ccache keep quiet in case memcached server is down and proceed to compilation? or exit with error?
6. Do we have a man page for any additional settings for this code?

On a side note:

Do you have any patch to read/write ccache results to multiple locations i.e., local disk and NFS to share the cached results?

@hkrpavan; totally missed this comment...

1. Yes. Unless you use CCACHE_MEMCACHED_ONLY, that is how it works. However, your local disk might be slower than memcached.
2. You can have multiple memcached servers, and the shards (divided into chunks of maximum 1M each) will be split between them.
3. The same hash always have similar content (possibly with new timestamps etc), and the second write will just overwrite the first
4. Yes, and it uses the "ketama" method (MEMCACHED_DISTRIBUTION_CONSISTENT) for choosing which server stores the key/value.
5. If the server is down, it will eventually timeout and compilation will continue.
6. Just the MANUAL.adoc document, as included in the source code (branch).

@afbjorklund the ccache checout dont work, the branch dev/memcached is missing. :(

root@item:/tmp# git clone https://github.com/ccache/ccache.git
Cloning into 'ccache'...
remote: Enumerating objects: 37, done.
remote: Counting objects: 100% (37/37), done.
remote: Compressing objects: 100% (25/25), done.
remote: Total 16332 (delta 14), reused 24 (delta 12), pack-reused 16295
Receiving objects: 100% (16332/16332), 7.43 MiB | 9.37 MiB/s, done.
Resolving deltas: 100% (11672/11672), done.
root@item:/tmp# cd ccache
root@item:/tmp/ccache# git checkout dev/memcached
error: pathspec 'dev/memcached' did not match any file(s) known to git
root@item:/tmp/ccache#

The code now lives at https://github.com/afbjorklund/memccache