Skip to content

Instantly share code, notes, and snippets.

Avatar

Robin Moffatt rmoff

View GitHub Profile
@rmoff
rmoff / docker-compose.yml
Last active May 22, 2019
Example Docker Compose showing Apache Kafka listener configuration
View docker-compose.yml
[…]
kafka0:
image: "confluentinc/cp-enterprise-kafka:5.2.1"
ports:
- '9092:9092'
- '29094:29094'
depends_on:
- zookeeper
environment:
@rmoff
rmoff / connector-delete.sh
Created May 1, 2019
Selectively delete Kafka Connect connectors using peco
View connector-delete.sh
curl -s "http://localhost:8083/connectors"| \
jq '.[]'| \
peco | \
xargs -I{connector_name} curl -s -XDELETE "http://localhost:8083/connectors/"{connector_name}
@rmoff
rmoff / connect_status.sh
Created May 1, 2019
Show status of all Kafka Connect connectors
View connect_status.sh
curl -s "http://localhost:8083/connectors"| \
jq '.[]'| \
xargs -I{connector_name} curl -s "http://localhost:8083/connectors/"{connector_name}"/status" | \
jq -c -M '[.name,.connector.state,.tasks[].state]|join(":|:")' | \
column -s : -t | \
sed 's/\"//g' | \
sort
@rmoff
rmoff / gist:c3db21aae4b9232b07699fbe07e10d68
Created Mar 13, 2019
Neo4j/twitter Kafka Connect demo
View gist:c3db21aae4b9232b07699fbe07e10d68
= Testing the Neo4j Connector with Twitter as a source of data
Caveat: I have never used Neo4j before. I'm familiar with the general concept of a property graph though.
Environment: https://github.com/rmoff/neo4j-streams/blob/master/kafka-connect-neo4j/docker/docker-compose.yml
== Start here
Create a Twitter source
View example.pcap
{
"timestamp": "1547735265961",
"layers": {
"frame": {
"frame_frame_interface_id": "0",
"frame_interface_id_frame_interface_name": "en0",
"frame_frame_encap_type": "1",
"frame_frame_time": "Jan 17, 2019 14:27:45.961581000 GMT",
"frame_frame_offset_shift": "0.000000000",
"frame_frame_time_epoch": "1547735265.961581000",
@rmoff
rmoff / connect-jdbc-driver.adoc
Last active Jan 15, 2019
JDBC Driver installation with Confluent CLI
View connect-jdbc-driver.adoc

JDBC Driver installation with Confluent CLI

Install Confluent Platform

  1. Go to https://www.confluent.io/download/ and download 5.1

  2. Move and unpack it:

    $ mv ~/Downloads/confluent-5.1.0-2.11.tar.gz ~/cp/
@rmoff
rmoff / 00_numeric.mapping_README.md
Last active May 15, 2020
Kafka Connect JDBC connector - numeric.mapping
View 00_numeric.mapping_README.md
@rmoff
rmoff / docker-compose.yml
Last active Apr 20, 2020
Docker-Compose for Kafka and Zookeeper with internal and external listeners
View docker-compose.yml
---
version: '2'
services:
zookeeper:
image: confluentinc/cp-zookeeper:latest
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
kafka:
View ksql null workaround.ksql
# Set up some sample data:
docker run --interactive --rm --network ksql-troubleshooting_default \
confluentinc/cp-kafkacat \
kafkacat -b kafka:29092 \
-t topic_with_nulls \
-P <<EOF
{"col1":1,"col2":16000,"col3":"foo"}
{"col1":2,"col2":42000}
{"col1":3,"col2":94000,"col3":"bar"}
{"col1":4,"col2":12345}
@rmoff
rmoff / ksql-asciinema.adoc
Last active Sep 17, 2018
Recording Docker container sessions with asciinema
View ksql-asciinema.adoc
asciinema rec \
    --idle-time-limit 1 \
    -c "docker run --network ksql-troubleshooting_default --interactive --tty --rm confluentinc/cp-ksql-cli:5.0.0 http://ksql-server:8088" \
    target.cast
You can’t perform that action at this time.