Skip to content

Instantly share code, notes, and snippets.

@arrawatia
Created February 6, 2015 07:27
Show Gist options
  • Save arrawatia/7ceaec87580d3090583d to your computer and use it in GitHub Desktop.
Save arrawatia/7ceaec87580d3090583d to your computer and use it in GitHub Desktop.
[~/code/src/java-scripting-experiments/src/main/resources (master *=) ]➔ kafkacat -P -X list -b fb-log-0.demo-us-east-1b.quantezza.com -t test
## Global configuration properties
Property | C/P | Default | Description
-----------------------------------------|-----|--------------:|--------------------------
client.id | * | rdkafka | Client identifier.
metadata.broker.list | * | | Initial list of brokers. The application may also use `rd_kafka_brokers_add()` to add brokers during runtime.
message.max.bytes | * | 4000000 | Maximum transmit message size.
receive.message.max.bytes | * | 100000000 | Maximum receive message size. This is a safety precaution to avoid memory exhaustion in case of protocol hickups. The value should be at least fetch.message.max.bytes * number of partitions consumed from + messaging overhead (e.g. 200000 bytes).
metadata.request.timeout.ms | * | 60000 | Non-topic request timeout in milliseconds. This is for metadata requests, etc.
topic.metadata.refresh.interval.ms | * | 10000 | Topic metadata refresh interval in milliseconds. The metadata is automatically refreshed on error and connect. Use -1 to disable the intervalled refresh.
topic.metadata.refresh.fast.cnt | * | 10 | When a topic looses its leader this number of metadata requests are sent with `topic.metadata.refresh.fast.interval.ms` interval disregarding the `topic.metadata.refresh.interval.ms` value. This is used to recover quickly from transitioning leader brokers.
topic.metadata.refresh.fast.interval.ms | * | 250 | See `topic.metadata.refresh.fast.cnt` description
topic.metadata.refresh.sparse | * | false | Sparse metadata requests (consumes less network bandwidth)
debug | * | | A comma-separated list of debug contexts to enable: all,generic,broker,topic,metadata,producer,queue,msg,protocol
socket.timeout.ms | * | 60000 | Timeout for network requests.
socket.send.buffer.bytes | * | 0 | Broker socket send buffer size. System default is used if 0.
socket.receive.buffer.bytes | * | 0 | Broker socket receive buffer size. System default is used if 0.
socket.keepalive.enable | * | false | Enable TCP keep-alives (SO_KEEPALIVE) on broker sockets
socket.max.fails | * | 3 | Disconnect from broker when this number of send failures (e.g., timed out requests) is reached. Disable with 0. NOTE: The connection is automatically re-established.
broker.address.ttl | * | 300000 | How long to cache the broker address resolving results.
broker.address.family | * | any | Allowed broker IP address families: any, v4, v6
statistics.interval.ms | * | 0 | librdkafka statistics emit interval. The application also needs to register a stats callback using `rd_kafka_conf_set_stats_cb()`. The granularity is 1000ms. A value of 0 disables statistics.
error_cb | * | | Error callback (set with rd_kafka_conf_set_error_cb())
stats_cb | * | | Statistics callback (set with rd_kafka_conf_set_stats_cb())
log_cb | * | | Log callback (set with rd_kafka_conf_set_log_cb())
log_level | * | 6 | Logging level (syslog(3) levels)
socket_cb | * | | Socket creation callback to provide race-free CLOEXEC
open_cb | * | | File open callback to provide race-free CLOEXEC
opaque | * | | Application opaque (set with rd_kafka_conf_set_opaque())
internal.termination.signal | * | 0 | Signal that librdkafka will use to quickly terminate on rd_kafka_destroy(). If this signal is not set then there will be a delay before rd_kafka_wait_destroyed() returns true as internal threads are timing out their system calls. If this signal is set however the delay will be minimal. The application should mask this signal as an internal signal handler is installed.
queued.min.messages | C | 100000 | Minimum number of messages per topic+partition in the local consumer queue.
queued.max.messages.kbytes | C | 1000000 | Maximum number of kilobytes per topic+partition in the local consumer queue. This value may be overshot by fetch.message.max.bytes.
fetch.wait.max.ms | C | 100 | Maximum time the broker may wait to fill the response with fetch.min.bytes.
fetch.message.max.bytes | C | 1048576 | Maximum number of bytes per topic+partition to request when fetching messages from the broker.
fetch.min.bytes | C | 1 | Minimum number of bytes the broker responds with. If fetch.wait.max.ms expires the accumulated data will be sent to the client regardless of this setting.
fetch.error.backoff.ms | C | 500 | How long to postpone the next fetch request for a topic+partition in case of a fetch error.
group.id | C | | Consumer group id string. All clients sharing the same group.id belong to the same consumer group.
queue.buffering.max.messages | P | 100000 | Maximum number of messages allowed on the producer queue.
queue.buffering.max.ms | P | 1000 | Maximum time, in milliseconds, for buffering data on the producer queue.
message.send.max.retries | P | 2 | How many times to retry sending a failing MessageSet. **Note:** retrying may cause reordering.
retry.backoff.ms | P | 100 | The backoff time in milliseconds before retrying a message send.
compression.codec | P | none | Compression codec to use for compressing message sets: none, gzip or snappy
batch.num.messages | P | 1000 | Maximum number of messages batched in one MessageSet.
dr_cb | P | | Delivery report callback (set with rd_kafka_conf_set_dr_cb())
dr_msg_cb | P | | Delivery report callback (set with rd_kafka_conf_set_dr_msg_cb())
## Topic configuration properties
Property | C/P | Default | Description
-----------------------------------------|-----|--------------:|--------------------------
request.required.acks | P | 1 | This field indicates how many acknowledgements the leader broker must receive from ISR brokers before responding to the request: *0*=broker does not send any response, *1*=broker will wait until the data is written to local log before sending a response, *-1*=broker will block until message is committed by all in sync replicas (ISRs) before sending response. *>1*=for any number > 1 the broker will block waiting for this number of acknowledgements to be received (but the broker will never wait for more acknowledgements than there are ISRs).
enforce.isr.cnt | P | 0 | Fail messages locally if the currently known ISR count for a partition is less than this value. **NOTE**: The ISR count is fetched from the broker at regular intervals (`topic.metadata.refresh.interval.ms`) and might thus be outdated.
request.timeout.ms | P | 5000 | The ack timeout of the producer request in milliseconds. This value is only enforced by the broker and relies on `request.required.acks` being > 0.
message.timeout.ms | P | 300000 | Local message timeout. This value is only enforced locally and limits the time a produced message waits for successful delivery. A time of 0 is infinite.
produce.offset.report | P | false | Report offset of produced message back to application. The application must be use the `dr_msg_cb` to retrieve the offset from `rd_kafka_message_t.offset`.
partitioner | P | | Partitioner callback (set with rd_kafka_topic_conf_set_partitioner_cb())
opaque | * | | Application opaque (set with rd_kafka_topic_conf_set_opaque())
auto.commit.enable | C | true | If true, periodically commit offset of the last message handed to the application. This commited offset will be used when the process restarts to pick up where it left off. If false, the application will have to call `rd_kafka_offset_store()` to store an offset (optional). **NOTE:** There is currently no zookeeper integration, offsets will be written to local file according to offset.store.path.
auto.commit.interval.ms | C | 60000 | The frequency in milliseconds that the consumer offsets are commited (written) to offset storage.
auto.offset.reset | C | largest | Action to take when there is no initial offset in offset store or the desired offset is out of range: 'smallest' - automatically reset the offset to the smallest offset, 'largest' - automatically reset the offset to the largest offset, 'error' - trigger an error which is retrieved by consuming messages and checking 'message->err'.
offset.store.path | C | . | Path to local file for storing offsets. If the path is a directory a filename will be automatically generated in that directory based on the topic and partition.
offset.store.sync.interval.ms | C | -1 | fsync() interval for the offset file, in milliseconds. Use -1 to disable syncing, and 0 for immediate sync after each write.
offset.store.method | C | file | Offset commit store method: 'file' - local file store (offset.store.path, et.al), 'broker' - broker commit store (requires Apache Kafka 0.8.1 or later on the broker).
### C/P legend: C = Consumer, P = Producer, * = both
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment