Skip to content

Instantly share code, notes, and snippets.

View MarkC0x's full-sized avatar

Mark Cox MarkC0x

  • San Francisco Bay Area
View GitHub Profile
@jkreps
jkreps / benchmark-commands.txt
Last active June 17, 2024 03:54
Kafka Benchmark Commands
Producer
Setup
bin/kafka-topics.sh --zookeeper esv4-hcl197.grid.linkedin.com:2181 --create --topic test-rep-one --partitions 6 --replication-factor 1
bin/kafka-topics.sh --zookeeper esv4-hcl197.grid.linkedin.com:2181 --create --topic test --partitions 6 --replication-factor 3
Single thread, no replication
bin/kafka-run-class.sh org.apache.kafka.clients.tools.ProducerPerformance test7 50000000 100 -1 acks=1 bootstrap.servers=esv4-hcl198.grid.linkedin.com:9092 buffer.memory=67108864 batch.size=8196
@rmoff
rmoff / gist:f32543f78d821b25502f6db49eee9259
Created August 29, 2017 13:08
Kafka Connect JDBC source with JSON converter
{
"name": "jdbc_source_mysql_foobar_01",
"config": {
"_comment": "The JDBC connector class. Don't change this if you want to use the JDBC Source.",
"connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
"_comment": "How to serialise the value of keys - here use the Json converter. We want to retain the schema in the message (which will generate a schema/payload JSON document) and so set schemas.enable=true",
"key.converter": "org.apache.kafka.connect.json.JsonConverter",
"key.converter.schemas.enable":"true",
@wheresalice
wheresalice / rest_proxy.yaml
Last active April 14, 2022 08:28
Swagger OpenAPI spec for Confluent REST Proxy
openapi: 3.0.1
info:
title: REST Proxy API
description: >-
The Confluent REST Proxy provides a RESTful interface to a Kafka cluster, making it easy to produce and consume messages, view the state of the cluster, and perform administrative actions without using the native Kafka protocol or clients.
<p>Some example use cases are</p>
<ul>
<li>Reporting data to Kafka from any frontend app built in any language not supported by official Confluent clients</li>
<li>Ingesting messages into a stream processing framework that doesn’t yet support Kafka</li>
<li>Scripting administrative actions</li>