Skip to content

Instantly share code, notes, and snippets.

View rmoff's full-sized avatar

Robin Moffatt rmoff

480 followers · 0 following
View GitHub Profile
@rmoff
rmoff / docker-compose.yml
Last active March 18, 2024 03:34
Multi-node Kafka cluster (three brokers)
---
version: '3.8'
services:
zookeeper-1:
image: confluentinc/cp-zookeeper:5.5.1
ports:
- '32181:32181'
environment:
ZOOKEEPER_CLIENT_PORT: 32181
ZOOKEEPER_TICK_TIME: 2000
@rmoff
rmoff / go_version.txt
Last active June 25, 2020 08:01
go version go1.13.7 darwin/amd64
@rmoff
rmoff / smt.md
Created January 24, 2020 11:14
Kafka Connect - IllegalArgumentException: Invalid decimal scale: 127

From Mayank Patel on http://cnfl.io/slack

Problem:

org.apache.kafka.connect.errors.ConnectException: Tolerance exceeded in error handler
	at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:178)
	at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:104)
	at org.apache.kafka.connect.runtime.WorkerSourceTask.convertTransformedRecord(WorkerSourceTask.java:284)
	at org.apache.kafka.connect.runtime.WorkerSourceTask.sendRecords(WorkerSourceTask.java:309)
@rmoff
rmoff / gist:cfa93f2d8b722f4e6bf96aedec5074ef
Created January 22, 2020 17:34
InfluxDB Sink connector example
kafkacat -b localhost:9092 -P -t testdata-json4 <<EOF
{ "schema": { "type": "struct", "fields": [ { "type": "map", "keys": { "type": "string", "optional": false }, "values": { "type": "string", "optional": false }, "optional": false, "field": "tags" }, { "field": "sn", "optional": false, "type": "string" }, { "field": "value", "optional": false, "type": "float" } ], "optional": false, "version": 1 }, "payload": { "tags": { "tagnum": "5" }, "sn": "FOO", "value": 500.0 } }
EOF
curl -i -X PUT -H "Accept:application/json" \
-H "Content-Type:application/json" http://localhost:8083/connectors/SINK_INFLUX_01/config \
-d '{
"connector.class" : "io.confluent.influxdb.InfluxDBSinkConnector",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter.schemas.enable": "true",
@rmoff
rmoff / foo.adoc
Created January 21, 2020 14:52

Asciidoc code formatting

Example 01

-- SQL Code block
SELECT SOME_FIELD FROM FOO;
@rmoff
rmoff / kafkacat.adoc
Last active January 5, 2024 19:59
Show last three messages from a Kafka topic with kafkacat 
kafkacat -b localhost:9092 \
         -t _kafka-connect-group-01-status \
         -C \
         -o-3 \
         -c3 \
         -f 'Topic %t / Partition %p / Offset: %o / Timestamp: %T\nHeaders: %h\nKey (%K bytes): %k\nPayload (%S bytes): %s\n--\n'
@rmoff
rmoff / docker-compose.yml
Last active December 2, 2019 17:01
ksqlDB with external Kafka Connect worker and installed connector plugin
---
version: '3'
services:
zookeeper:
image: confluentinc/cp-zookeeper:5.3.1
container_name: zookeeper
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
@rmoff
rmoff / docker-compose.yml
Last active December 2, 2019 17:01
ksqlDB with external Kafka Connect worker
---
version: '3'
services:
zookeeper:
image: confluentinc/cp-zookeeper:5.3.1
container_name: zookeeper
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
@rmoff
rmoff / ccloud_connect.adoc
Created September 26, 2019 11:18
Configuring Kafka Connect with Confluent Cloud

Configuring Kafka Connect with Confluent Cloud

So you want to connect Kafka Connect to Confluent Cloud? Here’s how.

Overview

  1. Configure the worker

  2. Pre-create any topics that your source connector will write to (since auto-topic creation is not enabled on Confluent Cloud)

@rmoff
rmoff / Single Message Transform to add topic, partition, offset to sink message
Created July 15, 2019 15:12
curl -i -X PUT -H "Content-Type:application/json" \
http://localhost:8083/connectors/sink-elastic-orders-00/config \
-d '{
"connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
"topics": "orders",
"connection.url": "http://elasticsearch:9200",
"type.name": "type.name=kafkaconnect",
"key.ignore": "true",
"schema.ignore": "false",
"errors.tolerance":"all",