Installing a complete monitoring stack on Ubuntu 12.04

monitoring-stack-ubuntu-precise.md

Follow these steps to install Graphite on a fresh Ubuntu 12.04 instance. Make sure you read and understand the commands because by the time you read this, things could be outdated.

Graphite

Installing dependencies

# apt-get install libpq-dev
# apt-get install python-dev python-pip python-cairo python-psycopg2
# apt-get install python-django python-django-tagging
# apt-get install postgresql postgresql-client
# apt-get install build-essential

Setup Postgresql

Start the postgresql service:

# service postgresql start

Create a user and database for graphite:

# sudo -u postgres createuser graphite
# sudo -u postgres createdb -O graphite graphite

Change the password of the postgres and graphite users:

# sudo -u postgres psql -d template1
template1=# ALTER USER postgres WITH PASSWORD '<password>';
template1=# ALTER USER graphite WITH PASSWORD '<password>';

Install Graphite

# pip install carbon
# pip install whisper
# pip install graphite-web

Graphite configuration

1. Removing pyc's

For some reason, there are pyc files in the webapp. Let's delete those:

# cd /opt/graphite/webapp/graphite
# rm `find -name "*.pyc"`

2. Database

Here you will want to edit the settings file so that graphite can connect to postgesql. First you have to create a copy of the example settings file:

# cd /opt/graphite/webapp/graphite
# cp local_settings.py{.example,}

Here's what you probably want to modify in config:

DATABASE_ENGINE   = 'postgresql_psycopg2'
DATABASE_NAME     = 'graphite'
DATABASE_USER     = 'graphite'
DATABASE_PASSWORD = '<graphite postgresql user password>'
DATABASE_HOST     = ''
DATABASE_PORT     = ''

You may also want to change the timezone:

TIME_ZONE = 'America/Montreal'

After you are done, you have to tell Django to populate the database:

# python manage.py syncdb

3. Carbon configuration

# cd /opt/graphite/conf

Let's create copies of some conf files:

# cp carbon.conf{.example,}
# cp storage-schemas.conf{.example,}

Optionally, make carbon listen to localhost:

# cat carbon.conf | sed s/0.0.0.0/127.0.0.1/g > carbon.conf

Let's configure the storage schema. I will replace the default one by the following one, which is inspired by Etsy's setup:

[stats]
priority = 110
pattern = ^stats\..*
retentions = 10:2160,60:10080,600:262974

That translates to:

• 6 hours of 10 second data (what we consider "near-realtime")
• 1 week of 1 minute data
• 5 years of 10 minute data

We'll make a directory to store all the example configs, to clean up things:

# mkdir examples
# mv *.example examples/

Permissions

Let's create a graphite user:

# adduser graphite

Then, lets make him own the graphite install:

# chown -R graphite:graphite /opt/graphite

Starting up Graphite

Log in as the graphite user:

# su graphite

Starting carbon:

# python /opt/graphite/bin/carbon-cache.py start

Starting the Graphite server:

# python /opt/graphite/bin/run-graphite-devel-server.py /opt/graphite

Hope that it works!

Go to:

http://localhost:8080

You should see this if it works properly:

If you get a broken image, it most likely means that something is wrong py2cairo and cairo.

Check the debug output here:

http://localhost:8080/render

Statsd

Before proceeding, logout of the 'graphite' user.

Installing dependencies

# apt-get install git nodejs npm redis-server

Installing statsd

# cd /opt/
# git clone https://github.com/etsy/statsd.git

Then let's create a copy of the example config, and modify the config according to your graphite/backend settings:

# cd statsd
# cp exampleConfig.js localConfig.js

Starting statsd

node stats.js localConfig.js

Graphiti

# cd /opt/

Let's clone the graphiti repository:

# https://github.com/paperlesspost/graphiti.git

Installing dependencies

# apt-get install ttf-droid

RVM

First make sure you are logged in as the graphite user (that you created when you installed graphite):

# su graphite

Then, let's follow the rvm installation steps (https://rvm.io/rvm/install/):

# curl -L get.rvm.io | bash -s stable
# source /etc/profile.d/rvm.sh

Check what you need to do before installing ruby:

# rvm requirements

Here's what I had to do, it could be different for you. You'll have to be root for this step:

# apt-get install build-essential openssl libreadline6 libreadline6-dev curl git-core zlib1g zlib1g-dev libssl-dev libyaml-dev 
# apt-get install libsqlite3-dev sqlite3 libxml2-dev libxslt-dev autoconf libc6-dev ncurses-dev automake libtool bison subversion
# apt-get install libcurl4-gnutls-dev

Then login as the 'graphite' user:

# su graphite

Install and set ruby 1.8.7 and 1.9.3:

# rvm install 1.8.7
# rvm use 1.8.7
# rvm install 1.9.3
# rvm use 1.9.3

We need to install 'bundle':

# gem install bundle

I had some errors about 'lib', but I just ignored them because they didn't scare me.

After that, we do:

# bundle install

Configure Graphiti

# cd /opt/graphiti

Make sure you are logged in as the 'graphite' user:

# su graphite

Let's create copies of some conf files:

# cd config
# cp amazon_s3.yml{.example,}
# cp settings.yml{.example,}

Modify the settings.yml file, especially the part about the graphite/graphiti url:

graphiti_base_url: http://localhost:5000
graphite_base_url: http://localhost:8080

You can ignore the s3 settings file if you don't want to save screenshots on s3.

Also, modify the 'unicorn.rb' file:

listen 5000                                          
worker_processes 2                              
pid "/var/run/graphiti/unicorn.pid"
stderr_path "/var/log/graphiti/unicorn.stderr.log"
stdout_path "/var/log/graphiti/unicorn.stdout.log"

If that config is used, we have to create dirs:

# sudo su
# mkdir /var/{run,log}/graphiti
# chown graphite:graphite /var/{run,log}/graphiti

start-graphiti.sh

#!/bin/sh
# Just drop this in the root of your graphiti install (ex. /opt/graphiti)

bundle exec unicorn -c config/unicorn.rb -E production -D

stop-graphiti.sh

#!/bin/sh
# Just drop this in the root of your graphiti install (ex. /opt/graphiti)

PID='/var/run/graphiti/unicorn.pid'

if [ -e $PID ]
then
    kill `cat $PID`
    rm $PID
fi

Thanks mate, I could finally install Graphite with your instructions.