Skip to content

Instantly share code, notes, and snippets.

View rmoff's full-sized avatar

Robin Moffatt rmoff

View GitHub Profile
@rmoff
rmoff / docker-compose.yml
Last active March 18, 2024 03:34
Multi-node Kafka cluster (three brokers)
---
version: '3.8'
services:
zookeeper-1:
image: confluentinc/cp-zookeeper:5.5.1
ports:
- '32181:32181'
environment:
ZOOKEEPER_CLIENT_PORT: 32181
ZOOKEEPER_TICK_TIME: 2000

This is a collection for Postman to access the Apache Flink SQL Gateway. It’s based on the yaml published for version 1.18.1.

I’ve added Tests to set Session and Operation handles, and amended the collection to use these variables.

You need to set a Global Variable for baseURL to your Flink SQL Gateway.

If you launch the gateway locally with:

This file has been truncated, but you can view the full file.
[0.004s][info][class,load] opened: /Users/rmoff/.sdkman/candidates/java/11.0.21-tem/lib/modules
[0.006s][info][class,load] java.lang.Object source: jrt:/java.base
[0.006s][info][class,load] java.io.Serializable source: jrt:/java.base
[0.006s][info][class,load] java.lang.Comparable source: jrt:/java.base
[0.006s][info][class,load] java.lang.CharSequence source: jrt:/java.base
[0.006s][info][class,load] java.lang.String source: jrt:/java.base
[0.006s][info][class,load] java.lang.reflect.AnnotatedElement source: jrt:/java.base
[0.006s][info][class,load] java.lang.reflect.GenericDeclaration source: jrt:/java.base
[0.006s][info][class,load] java.lang.reflect.Type source: jrt:/java.base
This file has been truncated, but you can view the full file.
[0.004s][info][class,load] opened: /Users/rmoff/.sdkman/candidates/java/11.0.21-tem/lib/modules
[0.006s][info][class,load] java.lang.Object source: jrt:/java.base
[0.006s][info][class,load] java.io.Serializable source: jrt:/java.base
[0.006s][info][class,load] java.lang.Comparable source: jrt:/java.base
[0.006s][info][class,load] java.lang.CharSequence source: jrt:/java.base
[0.006s][info][class,load] java.lang.String source: jrt:/java.base
[0.006s][info][class,load] java.lang.reflect.AnnotatedElement source: jrt:/java.base
[0.006s][info][class,load] java.lang.reflect.GenericDeclaration source: jrt:/java.base
[0.006s][info][class,load] java.lang.reflect.Type source: jrt:/java.base
@rmoff
rmoff / kafkacat.adoc
Last active January 5, 2024 19:59
Show last three messages from a Kafka topic with kafkacat
kafkacat -b localhost:9092 \
         -t _kafka-connect-group-01-status \
         -C \
         -o-3 \
         -c3 \
         -f 'Topic %t / Partition %p / Offset: %o / Timestamp: %T\nHeaders: %h\nKey (%K bytes): %k\nPayload (%S bytes): %s\n--\n'
@rmoff
rmoff / foo.md
Last active January 5, 2024 17:17
cx_Oracle install on MacOS
  1. Download Instant Client:
  • instantclient-basic-macos.x64-11.2.0.4.0.zip
  • instantclient-sdk-macos.x64-11.2.0.4.0.zip
  • instantclient-sqlplus-macos.x64-11.2.0.4.0.zip
  1. Unzip and move to /opt

  2. Create symlink

@rmoff
rmoff / contexts.ipynb
Created April 5, 2023 18:59
SparkContext & SparkSession
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
@rmoff
rmoff / Continuous Conversion of Kafka topics from JSON to Avro with KSQL.adoc
Last active August 18, 2023 21:11
Continuous Conversion of Kafka topics from JSON to Avro with KSQL

@rmoff / 04 Apr 2018

This is easy with KSQL :)

Here’s a dummy topic, in JSON:

$ kafka-console-consumer --bootstrap-server localhost:9092 --from-beginning --topic mysql_users
{"uid":1,"name":"Cliff","locale":"en_US","address_city":"St Louis","elite":"P"}
@rmoff
rmoff / 00_numeric.mapping_README.md
Last active August 11, 2023 02:58
Kafka Connect JDBC connector - numeric.mapping
@rmoff
rmoff / docker-compose.yml
Last active July 22, 2023 16:29
Docker-Compose for Kafka and Zookeeper with internal and external listeners
---
version: '2'
services:
zookeeper:
image: confluentinc/cp-zookeeper:latest
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
kafka: