GaryRogers/RSyslogSetup.md

## RSyslogSetup.md

      
    Raw
  

              RSyslogSetup.md
            
          
    RSyslog Forwarding Setup

Overview

I'm looking to centralize logging for our dev team into Elasticsearch via Logstash. The wrinkle is that we aren't a Java shop, so installing java on our hosts just to ship logs back to a central Logstash indexer is something we'd like to avoid. So, I'm approaching things as a chance to understand RSyslog and its capabilities as a log shipper.
Procedure

Set up TCP listening on the log index host

Uncomment the following lines in /etc/rsyslog.conf. This will enable the rsyslog daemon to listen for incoming requests on TCP port 514. We're using TCP here so that we can have some confidence that the messages from the agent hosts reach the indexer. (More on this below)
# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514

Add a line to /etc/rsyslog.conf to actually put the received logs in a specific file.
local3.*                        /local/logs/httpd-error
local4.*                        /local/logs/httpd-access

Finally, restart the rsyslog process.
service rsyslog restart
Set up the agent host

On the agent host, the host that is running apache, add a file, /etc/rsyslog.d/apache.conf. This will be read at syslog start time. This file tells rsyslog to read /var/log/httpd/error_log (the default apache error log on CentOS) every 10 seconds and send its messages to the local3.info facility in syslog. (Expanded to also read Access logs and send those to local4.info)
$ModLoad imfile

# Default Apache Error Log
$InputFileName /var/log/httpd/error_log
$InputFileTag httpd-error-default:
$InputFileStateFile stat-httpd-error
$InputFileSeverity info
$InputFileFacility local3
$InputRunFileMonitor

# Default Apache Access Log
$InputFileName /var/log/httpd/access_log
$InputFileTag httpd-access-default:
$InputFileStateFile stat-httpd-access
$InputFileSeverity info
$InputFileFacility local4
$InputRunFileMonitor

$InputFilePollInterval 10


Next, modify /etc/rsyslog.conf uncomment or add the following lines to the end of the file. This tells rsyslog to set up a log queue and forward any local3 and local4 facility log messages to the TCP port of 192.168.10.11. @@ is rsyslog shorthand for the TCP syslog port. If you want to forward via UDP, use a single @ instead. However, it's probably not worth setting up the queue in that case, as rsyslog doesn't have a way to assure that UDP packets are received by the index host.
In this configuration the agent host will store any log messages that can't be sent to the index host. This is good for dealing with times that the index host is being rebooted, or otherwise unavailable.
$WorkDirectory /var/lib/rsyslog # where to place spool files
$ActionQueueFileName fwdRule1 # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList   # run asynchronously
$ActionResumeRetryCount -1    # infinite retries if host is down
local3.* @@192.168.10.11
local4.* @@192.168.10.11

Lastly, restart rsyslog on the agent host.
service rsyslog restart
Test the configuration

Now, if you restart the apache server (service httpd restart) you should see logs being generated in /local/logs/httpd-error on the index host. If not check to see if there are any firewall blocks between the hosts, and that your rsyslog config changes are being parsed properly. You can check your rsyslog configuration with this command: /sbin/rsyslogd -c5 -f /etc/rsyslog.conf -N1
Configure Logstash to consume the new files

This configuration uses a custom grok parser to pull the error level out of the message, as well as pull the tag we set in the rsyslog config into its own field as well. For help with custom grok filters check out http://grokdebug.herokuapp.com/
input {
  file {
    type => "httpd-error-log"
    path => ["/local/logs/httpd-error"]
    sincedb_path => "/opt/logstash/sincedb-access"
    discover_interval => 10
  }

  file {
    type => "httpd-access-log"
    path => ["/local/logs/httpd-access"]
    sincedb_path => "/opt/logstash/sincedb-access"
    discover_interval => 10
  }
}

filter {
  if [type] == "httpd-error-log" {
    grok {
      match       => [ "message", "\S+ \d+ \d+:\d+:\d+ %{HOSTNAME} %{NOTSPACE:tag}: \[%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}\] \[%{LOGLEVEL:level}\] %{GREEDYDATA}" ]
    }
    mutate {
      rename      => [ "program", "vhost" ]
    }
  }

  if [type] == "httpd-access-log" {
    grok {
      match       => [ "message", "\S+ \d+ \d+:\d+:\d+ %{HOSTNAME} %{NOTSPACE:tag}: %{COMBINEDAPACHELOG}" ]
      add_field		=> { "level", "info" }
    }
  }
}

output {
  elasticsearch {
    host => "localhost"
  }
}