Skip to content

Instantly share code, notes, and snippets.

@AdamMcCormick

AdamMcCormick/aws-aurora-to-maxwell-kinesis-producer.md

Forked from abacaphiliac/aws-aurora-to-maxwell-kinesis-producer.md
Last active January 18, 2018 18:33
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save AdamMcCormick/7c5ba06218a3f7f6b7be0729c7d8b537 to your computer and use it in GitHub Desktop.
Save AdamMcCormick/7c5ba06218a3f7f6b7be0729c7d8b537 to your computer and use it in GitHub Desktop.
Download ZIP
AWS Aurora to Maxwell Kinesis Producer
Raw
aws-aurora-to-maxwell-kinesis-producer.md

Prerequisites

  1. AWS Aurora to Maxwell STDOUT Producer
  2. Create an AWS Kinesis Stream
  3. Configuring the AWS Command Line Interface

After running through the prerequisites, you will have:

  • a Maxwell image named osheroff/maxwell
  • an AWS Aurora instance
  • an AWS Kinesis Stream
  • AWS Credentials stored in ~/.aws/credentials

Start Maxwell with Kinesis Producer

docker run -it --rm --name maxwell -v `cd && pwd`/.aws:/root/.aws osheroff/maxwell sh -c "\
    cp /app/kinesis-producer-library.properties.example /app/kinesis-producer-library.properties && \
    echo "Region=YOUR_REGION" >> /app/kinesis-producer-library.properties && \
    bin/maxwell --user=AURORA_USER --password=AURORA_PASSWORD --host=AURORA_HOST --producer=kinesis --kinesis_stream=KINESIS_STREAM "

This command will:

  • share your host AWS credentials with the Maxwell container
  • copy the Kinesis properties file into place
  • append YOUR_REGION to the Kinesis properties file
  • connect Maxwell to your Aurora instance
  • connect Maxwell to your Kinesis Stream

output:

18:39:18,394 INFO  KinesisProducerConfiguration - Attempting to load config from file kinesis-producer-library.properties
18:39:18,425 WARN  KinesisProducerConfiguration - Property Port ignored as there is no corresponding set method in KinesisProducerConfiguration
18:39:18,436 INFO  KinesisProducer - Extracting binaries to /tmp/amazon-kinesis-producer-native-binaries
18:39:19,163 INFO  Daemon - Asking for trace
18:39:19,196 INFO  Maxwell - Maxwell v1.7.0 is booting (MaxwellKinesisProducer), starting at BinlogPosition[mysql-bin-changelog.000002:91000]
18:39:19,274 INFO  LogInputStreamReader - [2017-01-17 18:39:19.259719] [0x00000024][0x00007f2cdc4c0780] [info] [logging.cc:83] Set boost log level to info
18:39:19,274 INFO  LogInputStreamReader - [2017-01-17 18:39:19.259809] [0x00000024][0x00007f2cdc4c0780] [info] [logging.cc:170] Set AWS Log Level to WARN
18:39:19,274 INFO  LogInputStreamReader - [2017-01-17 18:39:19.262697] [0x00000024][0x00007f2cdc4c0780] [info] [main.cc:346] Setting CA path to /tmp/amazon-kinesis-producer-native-binaries
18:39:19,274 INFO  LogInputStreamReader - [2017-01-17 18:39:19.262716] [0x00000024][0x00007f2cdc4c0780] [info] [main.cc:382] Starting up main producer
18:39:19,274 INFO  LogInputStreamReader - [2017-01-17 18:39:19.262736] [0x00000024][0x00007f2cdc4c0780] [info] [kinesis_producer.cc:87] Using Region: us-west-2
18:39:19,275 INFO  LogInputStreamReader - [2017-01-17 18:39:19.262747] [0x00000024][0x00007f2cdc4c0780] [info] [kinesis_producer.cc:48] Using default Kinesis endpoint
18:39:19,275 INFO  LogInputStreamReader - [2017-01-17 18:39:19.262775] [0x00000024][0x00007f2cdc4c0780] [info] [kinesis_producer.cc:87] Using Region: us-west-2
18:39:19,275 INFO  LogInputStreamReader - [2017-01-17 18:39:19.262782] [0x00000024][0x00007f2cdc4c0780] [info] [kinesis_producer.cc:48] Using default CloudWatch endpoint
18:39:19,275 INFO  LogInputStreamReader - [2017-01-17 18:39:19.262849] [0x00000024][0x00007f2cdc4c0780] [info] [main.cc:393] Entering join
18:39:19,903 INFO  MysqlSavedSchema - Restoring schema id 1 (last modified at BinlogPosition[mysql-bin-changelog.000002:3521])
18:39:23,243 INFO  OpenReplicator - starting replication at mysql-bin-changelog.000002:91000
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment