Skip to content

Instantly share code, notes, and snippets.

Avatar

Robin Moffatt rmoff

View GitHub Profile
@rmoff
rmoff / raycast.md
Created March 17, 2023 12:17
Raycast AI helping with regex
View raycast.md

rmoff: mac regex replace in file from CLI

Raycast AI: You can use the sed command on macOS to perform a regex replacement in a file from the command line. Here's the general syntax:

sed 's/regex/replacement/g' input_file > output_file
  • regex: The regular expression you want to search for.
  • replacement: The text you want to replace the matched regex with.
@rmoff
rmoff / docker-compose.yml
Last active January 1, 2023 09:06
Multi-node Kafka cluster (three brokers)
View docker-compose.yml
---
version: '3.8'
services:
zookeeper-1:
image: confluentinc/cp-zookeeper:5.5.1
ports:
- '32181:32181'
environment:
ZOOKEEPER_CLIENT_PORT: 32181
ZOOKEEPER_TICK_TIME: 2000
View go_version.txt
go version go1.13.7 darwin/amd64
@rmoff
rmoff / smt.md
Created January 24, 2020 11:14
Kafka Connect - IllegalArgumentException: Invalid decimal scale: 127
View smt.md

From Mayank Patel on http://cnfl.io/slack

Problem:

org.apache.kafka.connect.errors.ConnectException: Tolerance exceeded in error handler
	at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:178)
	at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:104)
	at org.apache.kafka.connect.runtime.WorkerSourceTask.convertTransformedRecord(WorkerSourceTask.java:284)
	at org.apache.kafka.connect.runtime.WorkerSourceTask.sendRecords(WorkerSourceTask.java:309)
@rmoff
rmoff / gist:cfa93f2d8b722f4e6bf96aedec5074ef
Created January 22, 2020 17:34
InfluxDB Sink connector example
View gist:cfa93f2d8b722f4e6bf96aedec5074ef
kafkacat -b localhost:9092 -P -t testdata-json4 <<EOF
{ "schema": { "type": "struct", "fields": [ { "type": "map", "keys": { "type": "string", "optional": false }, "values": { "type": "string", "optional": false }, "optional": false, "field": "tags" }, { "field": "sn", "optional": false, "type": "string" }, { "field": "value", "optional": false, "type": "float" } ], "optional": false, "version": 1 }, "payload": { "tags": { "tagnum": "5" }, "sn": "FOO", "value": 500.0 } }
EOF
curl -i -X PUT -H "Accept:application/json" \
-H "Content-Type:application/json" http://localhost:8083/connectors/SINK_INFLUX_01/config \
-d '{
"connector.class" : "io.confluent.influxdb.InfluxDBSinkConnector",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter.schemas.enable": "true",
View foo.adoc

Asciidoc code formatting

Example 01

-- SQL Code block
SELECT SOME_FIELD FROM FOO;
@rmoff
rmoff / kafkacat.adoc
Last active January 10, 2023 19:19
Show last three messages from a Kafka topic with kafkacat
View kafkacat.adoc
kafkacat -b localhost:9092 \
         -t _kafka-connect-group-01-status \
         -C \
         -o-3 \
         -c3 \
         -f 'Topic %t / Partition %p / Offset: %o / Timestamp: %T\nHeaders: %h\nKey (%K bytes): %k\nPayload (%S bytes): %s\n--\n'
@rmoff
rmoff / docker-compose.yml
Last active December 2, 2019 17:01
ksqlDB with external Kafka Connect worker and installed connector plugin
View docker-compose.yml
---
version: '3'
services:
zookeeper:
image: confluentinc/cp-zookeeper:5.3.1
container_name: zookeeper
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
@rmoff
rmoff / docker-compose.yml
Last active December 2, 2019 17:01
ksqlDB with external Kafka Connect worker
View docker-compose.yml
---
version: '3'
services:
zookeeper:
image: confluentinc/cp-zookeeper:5.3.1
container_name: zookeeper
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
@rmoff
rmoff / ccloud_connect.adoc
Created September 26, 2019 11:18
Configuring Kafka Connect with Confluent Cloud
View ccloud_connect.adoc

Configuring Kafka Connect with Confluent Cloud

So you want to connect Kafka Connect to Confluent Cloud? Here’s how.

Overview

  1. Configure the worker

  2. Pre-create any topics that your source connector will write to (since auto-topic creation is not enabled on Confluent Cloud)