freeman-lab/notes.md

## notes.md

      
    Raw
  

              notes.md
            
          
    Notes on compiling chromium using an EC2 cluster

This note lays out the machine specifications and configurations used to get a distributed Chome build working on cloud compute (in this case AWS EC2) using the icecc tool.
These links were a good starting point:

https://chromium.googlesource.com/chromium/src/+/master/docs/linux_faster_builds.md
https://github.com/icecc/icecream
http://mkollaro.github.io/2015/05/08/compiling-chromium-with-clang-and-icecc/

but I couldn't fine a full walk through, so hopefully we can make it easier for others.
(with @maxogden)
Machines we need to orchestrate

The idea behind iccee is that one machine acts as a scheduler, and one or more machines do the compilation via the icecc daemon. In a typical configuration, you'd setup one machine with the shceduler, and one or more machines with daemens. One thing I found confusing at first: it's ok to run a daemon on the scheduler! It's optional, but useful in testing, because it means you can get a full distributed build working with just two nodes. Finally, you trigger the build itself from a machine on the same network that can also be running a daemon. I think running a daemon on this machine is optional too, but againt this lets you test with a minimal cluster.
I was able to get it all working on cloud compute (at least EC2) so long as the neccessary ports were open on all the nodes. It was also very useful to run the monitoring tool icemon on one of the nodes (I ran it on the scheduler), but make sure you have -X forwarding as it's GUI-only.
I used the following set of machines:
SCHEDULER

runs the icecc scheduler
runs an icecc deamon (optional)

MASTER

runs ninja
runs an icecc daemon (optional)

WORKER

runs an icecc daemon

WORKER

runs an icecc daemon

etc
This should just require two Dockerfiles, one for the MASTER, and one for the SCHEDULER + WORKERs. See comment below.
Instance types

The following worked fine:
ubuntu aws image, m3.large, 100GB storage
Make sure to open the following ports:
80, 8765, 10245
Machine configuration

MASTER

sudo apt-get update
sudo apt-get install git
git clone https://chromium.googlesource.com/chromium/tools/depot_tools.git
echo 'export PATH=$PATH:/home/ubuntu/depot_tools' >> ~/.bash_profile
source ~/.bash_profile
fetch --nohooks --no-history chromium
echo 'deb http://us.archive.ubuntu.com/ubuntu/ trusty multiverse' | sudo tee -a /etc/apt/sources.list
echo 'deb-src http://us.archive.ubuntu.com/ubuntu/ trusty multiverse' | sudo tee -a /etc/apt/sources.list
echo 'deb http://us.archive.ubuntu.com/ubuntu/ trusty-updates multiverse' | sudo tee -a /etc/apt/sources.list
echo 'deb-src http://us.archive.ubuntu.com/ubuntu/ trusty-updates multiverse' | sudo tee -a /etc/apt/sources.list
cd src
sudo ./build/install-build-deps.sh
sudo apt-get install icecc
gclient runhooks
SCHEDULER

sudo apt-get update
sudo apt-get install icecc icecc-monitor
WORKER

(cna be same as SCHEDULER, though the monitor is optional)
Job running

To run on the shceduler:

icecc-scheduler -n chrome -u ubuntu -vv
sudo iceccd -u ubuntu -vv -s SCHEDULER_IP -n chrome
To run on each worker:

sudo iceccd -u ubuntu -vv -s SCHEDULER_IP -n chrome
To run on the master (once the rest is setup):

sudo iceccd -u ubuntu -vv -s SERVER_IP -n chrome
export GYP_DEFINES="$GYP_DEFINES clang=0 linux_use_debug_fission=0 linux_use_bundled_binutils=0"
ninja -j 10 -C out/Release chrome
Other notes:

In debugging how all this works, it was very useful to run the monitoring tool icemon (great name!)
AFAIK this tool is GUI only, so I just used -X forwarding and ran it on the scheduler with
icemon -n chrome