AWS Aurora to Maxwell STDOUT Producer

aws-aurora-to-maxwell-stdout-producer.md

Prerequisites

1. docker pull osheroff/maxwell

After running through the prerequisites, you will have:

• a Maxwell image named osheroff/maxwell

AWS Aurora

1. Create an Aurora instance: Creating an Amazon Aurora DB Cluster
2. Create a DB Cluster Parameter Group: Creating a DB Parameter Group
3. Activate the binlog in your DB Cluster Parameter Group: Binary Logging Format
4. Apply the DB Cluster Parameter Group to the Aurora Instance
   1. Navigate to RDS Instances.
   2. Select the new Aurora instance.
   3. Select Modify from Instance Actions.
   4. Select the new DB Cluster Parameter Group.
   5. Continue to save.
   6. Select Reboot from Instance Actions.
5. Generate a schema.

Maxwell STDOUT Producer

Maxwell's Dockerfile uses the Kafka producer by default, but we're going to override the producer.

Run a self-destructing Maxwell container with the STDOUT producer:

docker run -it --rm --name maxwell osheroff/maxwell bin/maxwell --user=AURORA_USER --password=AURORA_PASSWORD --host=AURORA_HOST --producer=stdout

output:

19:17:12,414 INFO  Maxwell - Maxwell v1.7.0 is booting (StdoutProducer), starting at BinlogPosition[mysql-bin-changelog.000002:45695]
19:17:13,070 INFO  MysqlSavedSchema - Restoring schema id 1 (last modified at BinlogPosition[mysql-bin-changelog.000002:3521])
19:17:16,381 INFO  OpenReplicator - starting replication at mysql-bin-changelog.000002:45695
{"database":"AURORA_DATABASE","table":"AURORA_TABLE","type":"update","ts":1484594323,"xid":1579291,"commit":true,"data":{"id":4,"first_name":"Tim","last_name":"Younger"},"old":{"first_name":"Timothy"}}

Maxwell to Kafka Producer

Maxwell to Kafka Producer