Skip to content

Instantly share code, notes, and snippets.

Robin Moffatt rmoff

Block or report user

Report or block rmoff

Hide content and notifications from this user.

Learn more about blocking users

Contact Support about this user’s behavior.

Learn more about reporting abuse

Report abuse
View GitHub Profile
@rmoff
rmoff / connector-delete.sh
Created May 1, 2019
Selectively delete Kafka Connect connectors using peco
View connector-delete.sh
curl -s "http://localhost:8083/connectors"| \
jq '.[]'| \
peco | \
xargs -I{connector_name} curl -s -XDELETE "http://localhost:8083/connectors/"{connector_name}
@rmoff
rmoff / connect_status.sh
Created May 1, 2019
Show status of all Kafka Connect connectors
View connect_status.sh
curl -s "http://localhost:8083/connectors"| \
jq '.[]'| \
xargs -I{connector_name} curl -s "http://localhost:8083/connectors/"{connector_name}"/status" | \
jq -c -M '[.name,.connector.state,.tasks[].state]|join(":|:")' | \
column -s : -t | \
sed 's/\"//g' | \
sort
@rmoff
rmoff / gist:c3db21aae4b9232b07699fbe07e10d68
Created Mar 13, 2019
Neo4j/twitter Kafka Connect demo
View gist:c3db21aae4b9232b07699fbe07e10d68
= Testing the Neo4j Connector with Twitter as a source of data
Caveat: I have never used Neo4j before. I'm familiar with the general concept of a property graph though.
Environment: https://github.com/rmoff/neo4j-streams/blob/master/kafka-connect-neo4j/docker/docker-compose.yml
== Start here
Create a Twitter source
View example.pcap
{
"timestamp": "1547735265961",
"layers": {
"frame": {
"frame_frame_interface_id": "0",
"frame_interface_id_frame_interface_name": "en0",
"frame_frame_encap_type": "1",
"frame_frame_time": "Jan 17, 2019 14:27:45.961581000 GMT",
"frame_frame_offset_shift": "0.000000000",
"frame_frame_time_epoch": "1547735265.961581000",
@rmoff
rmoff / connect-jdbc-driver.adoc
Last active Jan 15, 2019
JDBC Driver installation with Confluent CLI
View connect-jdbc-driver.adoc

JDBC Driver installation with Confluent CLI

Install Confluent Platform

  1. Go to https://www.confluent.io/download/ and download 5.1

  2. Move and unpack it:

    $ mv ~/Downloads/confluent-5.1.0-2.11.tar.gz ~/cp/
@rmoff
rmoff / 00_numeric.mapping_README.md
Last active Feb 5, 2019
Kafka Connect JDBC connector - numeric.mapping
View 00_numeric.mapping_README.md
@rmoff
rmoff / docker-compose.yml
Created Sep 21, 2018
Docker-Compose for Kafka and Zookeeper with internal and external listeners
View docker-compose.yml
---
version: '2'
services:
zookeeper:
image: confluentinc/cp-zookeeper:latest
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
kafka:
View ksql null workaround.ksql
# Set up some sample data:
docker run --interactive --rm --network ksql-troubleshooting_default \
confluentinc/cp-kafkacat \
kafkacat -b kafka:29092 \
-t topic_with_nulls \
-P <<EOF
{"col1":1,"col2":16000,"col3":"foo"}
{"col1":2,"col2":42000}
{"col1":3,"col2":94000,"col3":"bar"}
{"col1":4,"col2":12345}
@rmoff
rmoff / ksql-asciinema.adoc
Last active Sep 17, 2018
Recording Docker container sessions with asciinema
View ksql-asciinema.adoc
asciinema rec \
    --idle-time-limit 1 \
    -c "docker run --network ksql-troubleshooting_default --interactive --tty --rm confluentinc/cp-ksql-cli:5.0.0 http://ksql-server:8088" \
    target.cast
@rmoff
rmoff / decoding-json-with-jq.adoc
Created Jul 5, 2018
How to view a schema's JSON representation from the Confluent Schema Registry (Decoding encoded JSON with jq)
View decoding-json-with-jq.adoc

Confluent Schema Registry supports REST to pull down schema definitions:

$ curl -s localhost:8081/subjects/asgard.demo.CUSTOMERS-raw-value/versions/latest

{"subject":"asgard.demo.CUSTOMERS-raw-value","version":1,"id":6,"schema":"{\"type\":\"record\",\"name\":\"Envelope\",\"namespace\":\"asgard.demo.CUSTOMERS\",\"fields\":[{\"name\":\"before\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"Value\",\"fields\":[{\"name\":\"id\",\"type\":\"int\"},{\"name\":\"first_name\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"last_name\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"email\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"gender\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"club_status\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"comments\",\"type\":[\"null\",\"string\"],\"default\":null},{\"name\":\"create_ts\",\"type\":{\"type\":\"string\",\"connect.version\":1,\"connect.default\":\"1970-01-01T00:00:00
You can’t perform that action at this time.