Split nginx logs

split-nginx-logs.md

Suppose you have these logs:

access.log
api.access.log
api.error.log
frontend.access.log
frontend.error.log
services.access.log

And you want to split them like this:

05-11/
  access.log
  api.access.log
  api.error.log
  frontend.access.log
  frontend.error.log
  services.access.log
05-12/
  access.log
  api.access.log
  api.error.log
  frontend.access.log
  frontend.error.log
  services.access.log
05-13/
  access.log
  api.access.log
  api.error.log
  frontend.access.log
  frontend.error.log
  services.access.log

Split access logs by file name and then date:

for f in *access.log; do awk '
  NR==1{
    system("mkdir -p tmp/"FILENAME)
  }
  {
    MON=sprintf("%02d",(match("JanFebMarAprMayJunJulAugSepOctNovDec",substr($4,5,3))+2)/3);
    print > "tmp/"FILENAME"/"MON"-"substr($4,2,2)
  }' $f; done

Since error logs have different time format, use this to split them:

for f in *error.log; do awk '
  NR==1{
    system("mkdir -p tmp/"FILENAME)
  }
  {
    print > "tmp/"FILENAME"/"substr($1,6,2)"-"substr($1,9,2)
  }' $f; done

Enter tmp directory to see if you have directories of each log file:

$ cd tmp
$ ls
access.log
api.access.log
api.error.log
frontend.access.log
frontend.error.log
services.access.log

Run this command to move files by date then log type:

find * -type f | awk -F/ '{print "mkdir -p "$2"; mv "$1"/"$2" "$2"/"$1}' | bash
rmdir *.log

Now ls should list directories by date:

$ ls
05-11
05-12
05-13

And optionally if you want to compress them (replace pigz with gzip if you don't have pigz):

find . -type f -not -name '*.gz' -exec pigz {} \;