kawanet/flutned_forward_protocol.md

## flutned_forward_protocol.md

      
    Raw
  

              flutned_forward_protocol.md
            
          
    Flutned Forward Protocol (Draft)

This is NOT an official protocol specification published by the fluentd maintainers.
Abstract

This specification describes an the fluentd forward protocol, which is used to forward event records.
Terminology

The keywords "MUST", "MUST NOT", "SHOULD", "SHOULD NOT" and "MAY" in this document are to be interpreted as described in RFC 2119. The following terms are also used:

client: an endpoint that sends event records. out_forward plugin is the reference client implementation shipped with the fluentd distribution.
server: an endpoint that receives event records. in_forward plugin is the reference server implementation shipped with the fluentd distribution.
connection: a TCP connection between two endpoints.
msgpack: a light weight binary representation of serialized objects.

Fluentd Foward Client


Client sends a msgpack array which contains one or more event records to the server through TCP connection.
Client MUST choose a carrier mode from three: Message, Forward and PackedForward described below.
Client MAY send a msgpack nil value for heartbeat health check usage without any event record payload.
Client MAY send multiple msgpack arrays on a single connection, continuaslly.

Fluentd Foward Server


Server MUST receive one or more msgpack arrays from the client through TCP connection.
Server MUST detect the carrier mode by inspecting the second element of the array.
Server SHOULD ignore any request value other than array format.
Note: In addition to the three carrier modes, in_forward plugin also accepts JSON representation of a single event record for convenience. It detect it by the first byte of the request.

Message Mode

It carries a single event record.

tag is a string separated with '.' (e.g. myapp.access) to categorize event records.
time is a number of seconds since Unix epoch per default. Since the fluentd v0.xx.xx, EventTime MAY send nanosecond precision of time with msgpack ext format of type 0 is used when time_as_integer is false. Server SHOULD accept both formats.
record is key-value pairs of the event record.
option is optional. See below.


name
Ruby type
msgpack fromat
content


tag
String
str
tag name


time
Integer | EventTime
int | ext
Unix time (second)


record
Object
map
event payload


option
Object
map
option (optional)


[
  "tag.name",
  1441588984,
  {"message": "bar"},
  {"option": "optional"}
]
Forward Mode

It carries a series of events as a msgpack array on a single request.


name
Ruby type
msgpack fromat
content


tag
String
str
tag name


entries
MultiEventStream
array
list of Entry


option
Object
map
option (optional)


[
  "tag.name",
  [
    [1441588984, {"message": "foo"}],
    [1441588985, {"message": "bar"}],
    [1441588986, {"message": "baz"}]
  ],
  {"option": "optional"}
]
PackedForward Mode

It carries a series of events as a msgpack binary on a single request.

entries is a binary chunk of MessagePackEventStream which contains multiple raw msgpack representations of Entry.
Client SHOULD send a MessagePackEventStream as msgpack bin format as it's a binary representation.
Client MAY send a MessagePackEventStream as msgpack str format by compatibility reasons.
Server MUST accept both formats of bin and str.
Server MAY decode individual event records on demand but not on a time of request. It means it MAY costs less, compared to Forward mode, when decoding is not needed by any plugins.
Note: out_forward plugin sends event records by the PackedForward mode. It encloses event records with msgpack str format instead of bin format for a backward compatibility reason.


name
Ruby type
msgpack format
content


tag
String
str
tag name


entries
MessagePackEventStream
str | bin
msgpack stream of Entry


option
Object
map
option (optional)


[
  "tag.name",
  "<<MessagePackEventStream>>",
  {"option": "optional"}
]
Entry

It's the event record format used in Forward and PackedForward mode.


name
Ruby type
msgpack fromat
content


time
Integer | EventTime
int | ext
Unix time (second)


record
Object
map
event payload


Option

It carries an optional meta data for the request.

Client MAY send key-value pairs of options.
Server MAY just ignore any options given.
chunk: Clients MAY send the chunk option to confirm the server receives event records. The value is a string of Base64 representation of 128 bits unique_id which is a ID of an event.

{"chunk": "p8n9gmxTQVC8/nh2wlKKeQ=="}
Response


Server SHOULD close the connection silently with no response when the chunk option is not sent.
ack: Server SHOULD respond ack when the chunk option is sent by client. The ack response value MUST be the same value given by chunk option from client. Client SHOULD retry to send event records later when the request has a chunk but the response has no ack.

{"ack": "p8n9gmxTQVC8/nh2wlKKeQ=="}
EventTime Ext Format

EventTime uses msgpack extension format of type 0 to carry nanosecond precision of time.

Client MAY send EventTime instead of plain integer representation of second since unix epoch.
Server SHOULD accept both formats of integer and EventTime.

+----+----+----+----+----+----+----+----+----+----+
|  1 |  2 |  3 |  4 |  5 |  6 |  7 |  8 |  9 | 10 |
+----+----+----+----+----+----+----+----+----+----+
| C7 | 00 | second from epoch |     nanosecond    |
+----+----+----+----+----+----+----+----+----+----+
|ext |type| 32bits integer BE | 32bits integer BE |
+----+----+----+----+----+----+----+----+----+----+
Grammar


Name? means that Name is optional.
Name* means that Name can occur zero or more times.
<<Name>> means binary msgpack representation of Name.
[ A, B, C ] means an array.
nil, string, integer and object means as it is.

Connection ::= <<Request>>*

Request ::= Message | Forward | PackedForward | nil

Message ::= [ Tag, Time, Record, Option? ]

Forward ::= [ Tag, MultiEventStream, Option? ]

MultiEventStream ::= [ Event* ]

PackedForward ::= [ Tag, MessagePackEventStream, Option? ]

MessagePackEventStream ::= <<Event>>*

Event ::= [ Time, Record ]

Tag ::= string

Time ::= integer | EventTime

Record ::= object

Option ::= object
name	Ruby type	msgpack fromat	content
tag	String	str	tag name
time	Integer \| EventTime	int \| ext	Unix time (second)
record	Object	map	event payload
option	Object	map	option (optional)
name	Ruby type	msgpack fromat	content
tag	String	str	tag name
entries	MultiEventStream	array	list of Entry
option	Object	map	option (optional)