distributed logging discussion uds2010

Distributed logging for physical deployment (for UEC and more)


User story:
  As a UEC admin I can view all logs from every physical system include in my UEC deployment in one central place so that I can easily find out and debug why some components of my UEC deployment are failing.
  
 * flume:
   - java component
   - centrally configured (redundant via zookeeper) 
   - agents / forwarders / sinks / management server
   - not packaged -- low complexity to do so
   
 * central syslog: rsyslog, relp (MIR librelp), rsyslog output modules (sql Database, hdfs).
   - rsyslog
     * agent -> * can use [ udp / tcp / relp ] to forward
       * RELP Reliable Event Logging Protocol - http://www.rsyslog.com/doc/relp.html
         - supports reliable delivery end-to-end
     * "sinks" have output plugins
       * existing plugins: sql & files
     * centralized configs == config management, i.e. puppet
     * TLS enabled for TCP
     
 * Reconoiter
   - Created by OmniTI 
   - Log aggregation and introspection
   - Highly Polished
   - can be used for reporting and data presentation
   - experimental debian packaging. http://github.com/skorgu/reconnoiter-debian
   - Reasonable dependencies -- python, pcre, postgres, etc. (for the web ui)
   - Minimal dependencies for the agents.
   
   
 * real-time debugging
   - tail + grep on log "sink" -- for UEC, CLC
     - tail -f /var/log/uec/* | grep $INSTANCE_ID
     - highlighting tools with UEC knowledge -- swatch, etc.
     - Web interface for watching/querying logs
       - leverage text searching rather than grep
       - http://www.linuxjournal.com/content/centralized-logging-web-interface
       
 * scribe:
   - c++ binary
   - built on top of thrift
   
 * Swatch
 * 
 
ACTIONS:
 * Support relp in main (MIR librelp)
 * write puppet recipes to automatically configure rsyslog
 * Integrate in UEC:
   * configure central rsyslog on the ClC
   * configure aggegrator rsyslog on the CC
   * configure central logging via rsyslog on the NC, SC, Walrus
   * use syslog for all UEC components
   * write a script (grep++) to automatically track messages related to an InstanceId
 * package Reconoiter in ubuntu to use it for reporting and presentation