Skip to content

Instantly share code, notes, and snippets.

@kawanet
Last active March 16, 2016 20:09
Show Gist options
  • Save kawanet/078e274952638fd53150 to your computer and use it in GitHub Desktop.
Save kawanet/078e274952638fd53150 to your computer and use it in GitHub Desktop.
Flutned Forward Protocol (Draft)

Flutned Forward Protocol (Draft)

This is NOT an official protocol specification published by the fluentd maintainers.

Abstract

This specification describes an the fluentd forward protocol, which is used to forward event records.

Terminology

The keywords "MUST", "MUST NOT", "SHOULD", "SHOULD NOT" and "MAY" in this document are to be interpreted as described in RFC 2119. The following terms are also used:

  • client: an endpoint that sends event records. out_forward plugin is the reference client implementation shipped with the fluentd distribution.
  • server: an endpoint that receives event records. in_forward plugin is the reference server implementation shipped with the fluentd distribution.
  • connection: a TCP connection between two endpoints.
  • msgpack: a light weight binary representation of serialized objects.

Fluentd Foward Client

  • Client sends a msgpack array which contains one or more event records to the server through TCP connection.
  • Client MUST choose a carrier mode from three: Message, Forward and PackedForward described below.
  • Client MAY send a msgpack nil value for heartbeat health check usage without any event record payload.
  • Client MAY send multiple msgpack arrays on a single connection, continuaslly.

Fluentd Foward Server

  • Server MUST receive one or more msgpack arrays from the client through TCP connection.
  • Server MUST detect the carrier mode by inspecting the second element of the array.
  • Server SHOULD ignore any request value other than array format.
  • Note: In addition to the three carrier modes, in_forward plugin also accepts JSON representation of a single event record for convenience. It detect it by the first byte of the request.

Message Mode

It carries a single event record.

  • tag is a string separated with '.' (e.g. myapp.access) to categorize event records.
  • time is a number of seconds since Unix epoch per default. Since the fluentd v0.xx.xx, EventTime MAY send nanosecond precision of time with msgpack ext format of type 0 is used when time_as_integer is false. Server SHOULD accept both formats.
  • record is key-value pairs of the event record.
  • option is optional. See below.
name Ruby type msgpack fromat content
tag String str tag name
time Integer | EventTime int | ext Unix time (second)
record Object map event payload
option Object map option (optional)
[
  "tag.name",
  1441588984,
  {"message": "bar"},
  {"option": "optional"}
]

Forward Mode

It carries a series of events as a msgpack array on a single request.

name Ruby type msgpack fromat content
tag String str tag name
entries MultiEventStream array list of Entry
option Object map option (optional)
[
  "tag.name",
  [
    [1441588984, {"message": "foo"}],
    [1441588985, {"message": "bar"}],
    [1441588986, {"message": "baz"}]
  ],
  {"option": "optional"}
]

PackedForward Mode

It carries a series of events as a msgpack binary on a single request.

  • entries is a binary chunk of MessagePackEventStream which contains multiple raw msgpack representations of Entry.
  • Client SHOULD send a MessagePackEventStream as msgpack bin format as it's a binary representation.
  • Client MAY send a MessagePackEventStream as msgpack str format by compatibility reasons.
  • Server MUST accept both formats of bin and str.
  • Server MAY decode individual event records on demand but not on a time of request. It means it MAY costs less, compared to Forward mode, when decoding is not needed by any plugins.
  • Note: out_forward plugin sends event records by the PackedForward mode. It encloses event records with msgpack str format instead of bin format for a backward compatibility reason.
name Ruby type msgpack format content
tag String str tag name
entries MessagePackEventStream str | bin msgpack stream of Entry
option Object map option (optional)
[
  "tag.name",
  "<<MessagePackEventStream>>",
  {"option": "optional"}
]

Entry

It's the event record format used in Forward and PackedForward mode.

name Ruby type msgpack fromat content
time Integer | EventTime int | ext Unix time (second)
record Object map event payload

Option

It carries an optional meta data for the request.

  • Client MAY send key-value pairs of options.
  • Server MAY just ignore any options given.
  • chunk: Clients MAY send the chunk option to confirm the server receives event records. The value is a string of Base64 representation of 128 bits unique_id which is a ID of an event.
{"chunk": "p8n9gmxTQVC8/nh2wlKKeQ=="}

Response

  • Server SHOULD close the connection silently with no response when the chunk option is not sent.
  • ack: Server SHOULD respond ack when the chunk option is sent by client. The ack response value MUST be the same value given by chunk option from client. Client SHOULD retry to send event records later when the request has a chunk but the response has no ack.
{"ack": "p8n9gmxTQVC8/nh2wlKKeQ=="}

EventTime Ext Format

EventTime uses msgpack extension format of type 0 to carry nanosecond precision of time.

  • Client MAY send EventTime instead of plain integer representation of second since unix epoch.
  • Server SHOULD accept both formats of integer and EventTime.
+----+----+----+----+----+----+----+----+----+----+
|  1 |  2 |  3 |  4 |  5 |  6 |  7 |  8 |  9 | 10 |
+----+----+----+----+----+----+----+----+----+----+
| C7 | 00 | second from epoch |     nanosecond    |
+----+----+----+----+----+----+----+----+----+----+
|ext |type| 32bits integer BE | 32bits integer BE |
+----+----+----+----+----+----+----+----+----+----+

Grammar

  • Name? means that Name is optional.
  • Name* means that Name can occur zero or more times.
  • <<Name>> means binary msgpack representation of Name.
  • [ A, B, C ] means an array.
  • nil, string, integer and object means as it is.
Connection ::= <<Request>>*

Request ::= Message | Forward | PackedForward | nil

Message ::= [ Tag, Time, Record, Option? ]

Forward ::= [ Tag, MultiEventStream, Option? ]

MultiEventStream ::= [ Event* ]

PackedForward ::= [ Tag, MessagePackEventStream, Option? ]

MessagePackEventStream ::= <<Event>>*

Event ::= [ Time, Record ]

Tag ::= string

Time ::= integer | EventTime

Record ::= object

Option ::= object
@tagomoris
Copy link

@kawanet We (Fluentd committers) want to add a wiki entry about Fluentd forward protocol details.
Can we use this entry for original version of it?

@tagomoris
Copy link

I'm sorry that I missed fluent/fluentd#671

@kawanet
Copy link
Author

kawanet commented Mar 16, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment