Skip to content

Instantly share code, notes, and snippets.

View rmoff's full-sized avatar

Robin Moffatt rmoff

480 followers · 0 following
View GitHub Profile
@rmoff
rmoff / docker-compose.yml
Last active May 22, 2019 11:30
Example Docker Compose showing Apache Kafka listener configuration
[…]
kafka0:
image: "confluentinc/cp-enterprise-kafka:5.2.1"
ports:
- '9092:9092'
- '29094:29094'
depends_on:
- zookeeper
environment:
@rmoff
rmoff / connector-delete.sh
Created May 1, 2019 10:05
Selectively delete Kafka Connect connectors using peco
curl -s "http://localhost:8083/connectors"| \
jq '.[]'| \
peco | \
xargs -I{connector_name} curl -s -XDELETE "http://localhost:8083/connectors/"{connector_name}
@rmoff
rmoff / connect_status.sh
Created May 1, 2019 09:58
Show status of all Kafka Connect connectors
curl -s "http://localhost:8083/connectors"| \
jq '.[]'| \
xargs -I{connector_name} curl -s "http://localhost:8083/connectors/"{connector_name}"/status" | \
jq -c -M '[.name,.connector.state,.tasks[].state]|join(":|:")' | \
column -s : -t | \
sed 's/\"//g' | \
sort
@rmoff
rmoff / gist:c3db21aae4b9232b07699fbe07e10d68
Created March 13, 2019 16:43
Neo4j/twitter Kafka Connect demo
= Testing the Neo4j Connector with Twitter as a source of data
Caveat: I have never used Neo4j before. I'm familiar with the general concept of a property graph though.
Environment: https://github.com/rmoff/neo4j-streams/blob/master/kafka-connect-neo4j/docker/docker-compose.yml
== Start here
Create a Twitter source
@rmoff
rmoff / example.pcap
Created January 24, 2019 10:27
pcap sample
{
"timestamp": "1547735265961",
"layers": {
"frame": {
"frame_frame_interface_id": "0",
"frame_interface_id_frame_interface_name": "en0",
"frame_frame_encap_type": "1",
"frame_frame_time": "Jan 17, 2019 14:27:45.961581000 GMT",
"frame_frame_offset_shift": "0.000000000",
"frame_frame_time_epoch": "1547735265.961581000",
@rmoff
rmoff / connect-jdbc-driver.adoc
Last active January 15, 2019 12:50
JDBC Driver installation with Confluent CLI

JDBC Driver installation with Confluent CLI

Install Confluent Platform

  1. Go to https://www.confluent.io/download/ and download 5.1

  2. Move and unpack it:

    $ mv ~/Downloads/confluent-5.1.0-2.11.tar.gz ~/cp/
@rmoff
rmoff / 00_numeric.mapping_README.md
Last active August 11, 2023 02:58
Kafka Connect JDBC connector - numeric.mapping
@rmoff
rmoff / docker-compose.yml
Last active July 22, 2023 16:29
Docker-Compose for Kafka and Zookeeper with internal and external listeners
---
version: '2'
services:
zookeeper:
image: confluentinc/cp-zookeeper:latest
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
kafka:
@rmoff
rmoff / ksql null workaround.ksql
Created September 19, 2018 21:03
KSQL NULL workaround
# Set up some sample data:
docker run --interactive --rm --network ksql-troubleshooting_default \
confluentinc/cp-kafkacat \
kafkacat -b kafka:29092 \
-t topic_with_nulls \
-P <<EOF
{"col1":1,"col2":16000,"col3":"foo"}
{"col1":2,"col2":42000}
{"col1":3,"col2":94000,"col3":"bar"}
{"col1":4,"col2":12345}
@rmoff
rmoff / ksql-asciinema.adoc
Last active September 17, 2018 11:31
Recording Docker container sessions with asciinema 
asciinema rec \
    --idle-time-limit 1 \
    -c "docker run --network ksql-troubleshooting_default --interactive --tty --rm confluentinc/cp-ksql-cli:5.0.0 http://ksql-server:8088" \
    target.cast