Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save IanVaughan/d669310820ae776850b64f10bc9c0a21 to your computer and use it in GitHub Desktop.
Save IanVaughan/d669310820ae776850b64f10bc9c0a21 to your computer and use it in GitHub Desktop.
Datadog grok logstash rules for custom sidekiq logs

Samples:

I, [2020-11-12T12:12:18.883446 #4]  INFO -- : sidekiq-stats {"processed":48469869,"failed":30091,"scheduled_size":0,"retry_size":0,"dead_size":0,"processes_size":1,"default_queue_latency":0,"workers_size":0,"enqueued":0}
I, [2020-11-12T12:03:11.112090 #4]  INFO -- : sidekiq-queues [[{"name":"default"},{"count":0},{"size":0},{"latency":0}],[{"name":"import"},{"count":0},{"size":0},{"latency":0}],[{"name":"low"},{"count":26},{"size":26},{"latency":183.94694018363953}]]

Rules:

autoFilledRule1 I,\s+\[%{date("yyyy-MM-dd'T'HH:mm:ss.SSSSSS"):date}\s+#4\]\s+%{word:level}\s+--\s+\:\s+sidekiq-stats\s+\{\"processed\"\:%{integer:processed},\"failed\"\:%{integer:failed},\"scheduled_size\"\:%{integer:scheduled_size},\"retry_size\"\:%{integer:retry_size},\"dead_size\"\:%{integer:dead_size},\"processes_size\"\:%{integer:processes_size},\"default_queue_latency\"\:%{number:default_queue_latency},\"workers_size\"\:%{integer:workers_size},\"enqueued\"\:%{integer:enqueued}\}
autoFilledRule2 I,\s+\[%{date("yyyy-MM-dd'T'HH:mm:ss.SSSSSS"):date}\s+#4\]\s+%{word:level}\s+--\s+\:\s+sidekiq-queues\s+\[\[\{\"name\"\:\"default\"\},\{\"count\"\:%{integer:default.count}\},\{\"size\"\:%{integer:default.size}\},\{\"latency\"\:%{number:default.latency}\}\],\[\{\"name\"\:\"import\"\},\{\"count\"\:%{integer:import.count}\},\{\"size\"\:%{integer:import.size}\},\{\"latency\"\:%{number:import.latency}\}\],\[\{\"name\"\:\"low\"\},\{\"count\"\:%{integer:low.count}\},\{\"size\"\:%{integer:low.size}\},\{\"latency\"\:%{number:low.latency}\}\]\]

match autoFilledRule1

{
  "processes_size": 1,
  "default_queue_latency": 0,
  "workers_size": 0,
  "date": 1605183138883,
  "level": "INFO",
  "processed": 48469869,
  "enqueued": 0,
  "failed": 30091,
  "scheduled_size": 0,
  "retry_size": 0,
  "dead_size": 0
}

match autoFilledRule2

{
  "import": {
    "latency": 0,
    "count": 0,
    "size": 0
  },
  "low": {
    "count": 26,
    "size": 26,
    "latency": 183.94694018363953
  },
  "date": 1605182591112,
  "level": "INFO",
  "default": {
    "count": 0,
    "size": 0,
    "latency": 0
  }
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment