Skip to content

Instantly share code, notes, and snippets.

@QaDeS QaDeS/aws
Last active Feb 7, 2019

Embed
What would you like to do?
Grok patterns to parse AWS access logs
IAMUSERID %{NUMBER:iam_user_id}
IAMUSERNAME [a-zA-Z0-9._-]+
IAMUSER arn:aws:iam::%{IAMUSERID}:user/%{IAMUSERNAME:iam_user_name}
S3TYPE (?:OBJECT)
S3OP [a-zA-z]+
S3SOAPOPERATION (?:SOAP\.%{S3OP})
S3RESTOPERATION (?:REST\.%{S3OP}\.%{S3TYPE})
S3WEBSITEOPERATION (?:WEBSITE\.%{S3OP}\.%{S3TYPE})
S3OPERATION (?:%{S3SOAPOPERATION}|%{S3RESTOPERATION}|%{S3WEBSITEOPERATION})
S3ERRORCODE [a-zA-z]+
S3KEY [a-zA-Z0-9\/_\.]+
S3ACCESSLOG1 %{BASE16NUM:s3_owner} (-|%{HOSTNAME:s3_bucket}) \[%{HTTPDATE:timestamp}\] %{IP:s3_remote_ip} (-|%{IAMUSER:s3_requester})
S3ACCESSLOG2 %{BASE16NUM:s3_request_id} %{S3OPERATION:s3_operation} (-|%{S3KEY:s3_key}) \"%{DATA:s3_request}\"
S3ACCESSLOG3 %{NUMBER:s3_status} (-|%{S3ERRORCODE:s3_error_code}) (-|%{NUMBER:s3_bytes_sent}) %{NUMBER:s3_object_size}
S3ACCESSLOG4 %{NUMBER:s3_request_time} %{NUMBER:s3_turnaround_time} \"(-|%{DATA:s3_referrer})\" \"%{DATA:s3_user_agent}\" (-|%{BASE16NUM:s3_version_id})
S3ACCESSLOG %{S3ACCESSLOG1} %{S3ACCESSLOG2} %{S3ACCESSLOG3} %{S3ACCESSLOG4}
@QaDeS

This comment has been minimized.

Copy link
Owner Author

QaDeS commented Feb 11, 2014

This is a work in progress and currently only applicable to S3 access logs. Please enhance if you can.

@lasconic

This comment has been minimized.

Copy link

lasconic commented Feb 19, 2014

Hi,
Thank you very much for this !
I improved it a bit. See https://gist.github.com/lasconic/9089182
I ran > 4 millions lines log and got no parse errors.

@adamlamar

This comment has been minimized.

Copy link

adamlamar commented Sep 21, 2017

@QaDeS @lasconic Just to clarify, what's the license on these patterns?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.