Rotating Server Logs

rotate.md

#Rotating Server Logs with LogRotate

Logrotate is a UNIX utility that does following operations:

1. Rotates the log file when file size reaches a specific size or after a specified duration
2. Continues to write the log information to the newly created file after rotating the old log file
3. Compresses the rotated log files
4. Specifies compression option for the rotated log files
5. Rotates the old log files with the date in the filename
6. Executes custom shell scripts immediately after log rotation
7. Remove older rotated log files

##Configuring Logrotate

1. Open /etc/logrotate.conf
2. Goto the end of the file and add the configuration for the files you wish to rotate. The configuration will look similar to this.

/home/deploy/APPENV/APPNAME/current/log/*.log {
  weekly
  missingok
  rotate 7
  compress
  delaycompress
  notifempty
  copytruncate
}

Make sure to point to the log directory with the *.log bit on the end so that we rotate only the log files.

##How It Works

1. weekly – Rotate the log files weekly. You can also use daily or monthly here instead.
2. missingok – If the log file doesn’t exist, ignore it
3. rotate 7 – Only keep 7 rotates of logs around
4. compress – GZip the log file on rotation
5. delaycompress – Rotate the file one day, then compress it the next day so we can be sure that it won’t interfere with the Rails server
6. notifempty – Don’t rotate the file if the logs are empty
7. copytruncate – Copy the log file and then empties it. This makes sure that the log file Rails is writing to always exists so you won’t get problems because the file does not actually change. If you don’t use this, you would need to restart your application each time.

##Running Logrotate

To run logrotate manually, we just do:

sudo /usr/sbin/logrotate -f /etc/logrotate.conf

##Uploading the log files to s3

1. Install the s3cmd on your system from here.
2. Open the logrotate.conf file and append the postrotate script. The final configuration would look like this:

/home/deploy/APPENV/APPNAME/current/log/*.log {
  weekly
  missingok
  rotate 7
  compress
  delaycompress
  notifempty
  copytruncate
  postrotate
        /usr/bin/s3cmd sync /home/deploy/APPENV/APPNAME/current/log/*.gz s3://S3_BUCKET_NAME/APPENV/logs/
  endscript
}