Skip to content

Instantly share code, notes, and snippets.

Avatar

Robin Moffatt rmoff

View GitHub Profile
@rmoff
rmoff / gist:f32543f78d821b25502f6db49eee9259
Created Aug 29, 2017
Kafka Connect JDBC source with JSON converter
View gist:f32543f78d821b25502f6db49eee9259
{
"name": "jdbc_source_mysql_foobar_01",
"config": {
"_comment": "The JDBC connector class. Don't change this if you want to use the JDBC Source.",
"connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
"_comment": "How to serialise the value of keys - here use the Json converter. We want to retain the schema in the message (which will generate a schema/payload JSON document) and so set schemas.enable=true",
"key.converter": "org.apache.kafka.connect.json.JsonConverter",
"key.converter.schemas.enable":"true",
@rmoff
rmoff / docker-compose.yml
Last active Mar 3, 2021
Docker-Compose for Kafka and Zookeeper with internal and external listeners
View docker-compose.yml
---
version: '2'
services:
zookeeper:
image: confluentinc/cp-zookeeper:latest
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
kafka:
@rmoff
rmoff / 0_update.adoc
Last active Feb 26, 2021
Example of Using Kafka Single Message Transform TimestampConverter
View 0_update.adoc
@rmoff
rmoff / List all available Kafka Connect plugins.md
Created May 18, 2018
List all available Kafka Connect plugins
View List all available Kafka Connect plugins.md
$ curl -s -XGET http://localhost:8083/connector-plugins|jq '.[].class'

"io.confluent.connect.activemq.ActiveMQSourceConnector"
"io.confluent.connect.elasticsearch.ElasticsearchSinkConnector"
"io.confluent.connect.hdfs.HdfsSinkConnector"
"io.confluent.connect.hdfs.tools.SchemaSourceConnector"
"io.confluent.connect.ibm.mq.IbmMQSourceConnector"
"io.confluent.connect.jdbc.JdbcSinkConnector"
"io.confluent.connect.jdbc.JdbcSourceConnector"
"io.confluent.connect.jms.JmsSourceConnector"
@rmoff
rmoff / 00_numeric.mapping_README.md
Last active Feb 15, 2021
Kafka Connect JDBC connector - numeric.mapping
View 00_numeric.mapping_README.md
View OGG_Swingbench_Extract.md
@rmoff
rmoff / docker-compose.yml
Last active Jan 17, 2021
Multi-node Kafka cluster (three brokers)
View docker-compose.yml
---
version: '3.8'
services:
zookeeper-1:
image: confluentinc/cp-zookeeper:5.5.1
ports:
- '32181:32181'
environment:
ZOOKEEPER_CLIENT_PORT: 32181
ZOOKEEPER_TICK_TIME: 2000
@rmoff
rmoff / foo.md
Last active Dec 11, 2020
cx_Oracle install on MacOS
View foo.md
  1. Download Instant Client:
  • instantclient-basic-macos.x64-11.2.0.4.0.zip
  • instantclient-sdk-macos.x64-11.2.0.4.0.zip
  • instantclient-sqlplus-macos.x64-11.2.0.4.0.zip
  1. Unzip and move to /opt

  2. Create symlink

@rmoff
rmoff / 00.README.md
Last active Nov 6, 2020
RMarkdown - repeating child block with changing variable value
View 00.README.md

When you use knit_expand it appears that the inclusion of the Rmd is done on the first pass, and then the complete document evaluated. This means that a Rmd block referenced in loop with knit_expand will only evaluate changing variables at their last value.

This can be worked around by passing the literal value of the variable at the time of the knit_expand with {{var}} syntax.

This is documented in the knitr_expand docs, but less clear (to an R noob like me) for embedded documents rather than strings.

@rmoff
rmoff / jmeter_ibmmq.jmx
Last active Oct 16, 2020
IBM MQ JMeter test script
View jmeter_ibmmq.jmx
<?xml version="1.0" encoding="UTF-8"?>
<jmeterTestPlan version="1.2" properties="4.0" jmeter="4.0 r1823414">
<hashTree>
<TestPlan guiclass="TestPlanGui" testclass="TestPlan" testname="Test Plan" enabled="true">
<stringProp name="TestPlan.comments"></stringProp>
<boolProp name="TestPlan.functional_mode">false</boolProp>
<boolProp name="TestPlan.tearDown_on_shutdown">true</boolProp>
<boolProp name="TestPlan.serialize_threadgroups">false</boolProp>
<elementProp name="TestPlan.user_defined_variables" elementType="Arguments" guiclass="ArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true">
<collectionProp name="Arguments.arguments">