Skip to content

Instantly share code, notes, and snippets.

@vdv7
Last active March 19, 2017 03:31
Show Gist options
  • Save vdv7/1b5ebbc08d5083fd35ea7daa3df79d4a to your computer and use it in GitHub Desktop.
Save vdv7/1b5ebbc08d5083fd35ea7daa3df79d4a to your computer and use it in GitHub Desktop.
Extended Log File Format (ELF), version 1.1.
Extended Log File Format (ELF), version 1.1.
ELF version 1.0 is specified here: https://www.w3.org/TR/WD-logfile.html
In addition to ELF 1.0 fields, 1.1 allows the following fields:
Field names that require no prefix:
epochs, epochms, seconds and milliseconds since epoch
Field names that require sc-, cs-, s-, or c- prefix:
port, server port
protocol connection protocol (e.g. http, https, ws)
raw, raw, all data coming over the wire, including headers
head, head, entire string of headers
body, body, data coming over the wire, minus the headers
Fields date and time did not used to allow prefix, but now allow s- or c-:
s-date, s-time, local date/time on server
c-date, c-time, local date/time on client
In addition to ELF 1.0 time definition, 1.1 allows time-zone offset:
<time>=2<digit>":"2<digit>[":"2<digit>["."*<digit>]]["+"|"-"2<digit>[":"2<digit>]]');
In addition to ELF 1.0 value types, 1.1 allows *tight JSON strings.
*tight JSON: JSON without any optional whitespace
Example of ELF v1.1:
#Version: 1.1
#Date: 2017-03-17 13:20:58-04:00
#Fields: s-date s-time cs-protocol s-port uri sc-body
2017-03-17 13:21:05 http 80 / "<html><body>\nHello World!\n</body></html>"
2017-03-17 13:21:13 http 80 /somescript.py {"x":"Hello World!\nLook, double-quotes: \"","y":23,"z":true}
2017-03-17 13:21:15 http 80 /somescript.py {"x":"Goodbye World."}
To give a bit of context -- Imagine developing a server that will include the ability to log messages between server and client for web-apps (perhaps using ajax or jsonp or websockets). Obviously this requires the addition of cs-body and sc-body fields. I'm assuming other users will want to store variants on these fields, such as cs-raw, sc-raw, cs-head, and sc-head. Another issue here is that the values for these fields will include newlines and other characters that should be escaped. Since much of modern webapp traffic is in the form of JSON (which uses backslashes to escape special characters between quotes), it seems like allowing values to be valid tight (without whitespace between values) JSON strings would be a useful addition to ELF.
Being that message-exchange logs can get fairly large for dynamic web-apps, you may also like to store a timestamp as milliseconds since epoch (i.e. javascript getTime()), since it's a fairly condensed and precise date-time format, thus we need epochms. I'm assuming there'll be users who prefer unix time, so i included epochs, as well.
Third, UTC time is not human-readable, it does the one thing that logging isn't supposed to do -- it confuses the reader. Thus, I beg for the option to add s- and c- prefixes to date/time fields so as to allow date/time specification in local server time (and keep track of local client time, as well). This would require that #Date, #Start-Date, and #End-Date directives be specified with offset, thus the addition to the <time> field definition (I believe <date> <time> would still comply with ISO).
Finally, I am not sure why port and protocol were not in the original ELF, but these are definitely needed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment