Skip to content

Instantly share code, notes, and snippets.

Avatar

Robin Moffatt rmoff

View GitHub Profile
@rmoff
rmoff / docker-compose.yml
Last active Oct 12, 2020
Multi-node Kafka cluster (three brokers)
View docker-compose.yml
---
version: '3.8'
services:
zookeeper-1:
image: confluentinc/cp-zookeeper:5.5.1
ports:
- '32181:32181'
environment:
ZOOKEEPER_CLIENT_PORT: 32181
ZOOKEEPER_TICK_TIME: 2000
View go_version.txt
go version go1.13.7 darwin/amd64
@rmoff
rmoff / smt.md
Created Jan 24, 2020
Kafka Connect - IllegalArgumentException: Invalid decimal scale: 127
View smt.md

From Mayank Patel on http://cnfl.io/slack

Problem:

org.apache.kafka.connect.errors.ConnectException: Tolerance exceeded in error handler
	at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:178)
	at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:104)
	at org.apache.kafka.connect.runtime.WorkerSourceTask.convertTransformedRecord(WorkerSourceTask.java:284)
	at org.apache.kafka.connect.runtime.WorkerSourceTask.sendRecords(WorkerSourceTask.java:309)
@rmoff
rmoff / gist:cfa93f2d8b722f4e6bf96aedec5074ef
Created Jan 22, 2020
InfluxDB Sink connector example
View gist:cfa93f2d8b722f4e6bf96aedec5074ef
kafkacat -b localhost:9092 -P -t testdata-json4 <<EOF
{ "schema": { "type": "struct", "fields": [ { "type": "map", "keys": { "type": "string", "optional": false }, "values": { "type": "string", "optional": false }, "optional": false, "field": "tags" }, { "field": "sn", "optional": false, "type": "string" }, { "field": "value", "optional": false, "type": "float" } ], "optional": false, "version": 1 }, "payload": { "tags": { "tagnum": "5" }, "sn": "FOO", "value": 500.0 } }
EOF
curl -i -X PUT -H "Accept:application/json" \
-H "Content-Type:application/json" http://localhost:8083/connectors/SINK_INFLUX_01/config \
-d '{
"connector.class" : "io.confluent.influxdb.InfluxDBSinkConnector",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter.schemas.enable": "true",
View foo.adoc

Asciidoc code formatting

Example 01

-- SQL Code block
SELECT SOME_FIELD FROM FOO;
@rmoff
rmoff / kafkacat.adoc
Last active Jan 17, 2020
Show last three messages from a Kafka topic with kafkacat
View kafkacat.adoc
kafkacat -b localhost:9092 \
         -t _kafka-connect-group-01-status \
         -C \
         -o-3 \
         -c3 \
         -f 'Topic %t / Partition %p / Offset: %o / Timestamp: %T\nHeaders: %h\nKey (%K bytes): %k\nPayload (%S bytes): %s\n--\n'
@rmoff
rmoff / docker-compose.yml
Last active Dec 2, 2019
ksqlDB with external Kafka Connect worker and installed connector plugin
View docker-compose.yml
---
version: '3'
services:
zookeeper:
image: confluentinc/cp-zookeeper:5.3.1
container_name: zookeeper
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
@rmoff
rmoff / docker-compose.yml
Last active Dec 2, 2019
ksqlDB with external Kafka Connect worker
View docker-compose.yml
---
version: '3'
services:
zookeeper:
image: confluentinc/cp-zookeeper:5.3.1
container_name: zookeeper
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
@rmoff
rmoff / ccloud_connect.adoc
Created Sep 26, 2019
Configuring Kafka Connect with Confluent Cloud
View ccloud_connect.adoc

Configuring Kafka Connect with Confluent Cloud

So you want to connect Kafka Connect to Confluent Cloud? Here’s how.

Overview

  1. Configure the worker

  2. Pre-create any topics that your source connector will write to (since auto-topic creation is not enabled on Confluent Cloud)

View Single Message Transform to add topic, partition, offset to sink message
curl -i -X PUT -H "Content-Type:application/json" \
http://localhost:8083/connectors/sink-elastic-orders-00/config \
-d '{
"connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
"topics": "orders",
"connection.url": "http://elasticsearch:9200",
"type.name": "type.name=kafkaconnect",
"key.ignore": "true",
"schema.ignore": "false",
"errors.tolerance":"all",
You can’t perform that action at this time.