Skip to content

Instantly share code, notes, and snippets.

@jcustenborder
jcustenborder / README.md
Last active Apr 2, 2018
Example docker-compose.yml for the Confluent Platform.
View README.md

Introduction

This is an example docker-compose.yml for the Confluent Platform.

Networking

In order to properly use this docker-compose make sure that your containers are resolvable by their container name. Take a look at the hosts file. Each container must be resolvable from the host machine if you are connecting from the host to the docker environment.

View kafka.service
[Unit]
Description = Apache Kafka by Confluent
Wants = basic.target
After = basic.target network.target
[Service]
EnvironmentFile = /etc/sysconfig/kafka
ExecStart = /usr/bin/kafka-server-start /etc/kafka/server.properties
LimitNOFILE = 131072
KillMode = process
@jcustenborder
jcustenborder / README.md
Last active Feb 13, 2019
kafka-connect-hdfs s3
View README.md

Download the AWS Java SDK Copy aws-java-sdk-1.10.11.jar to $CONFLUENT_HOME/share/java/kafka-connect-hdfs/ Download hadoop-aws-2.6.0.jar to $CONFLUENT_HOME/share/java/kafka-connect-hdfs/ Download

View HDFS
#http://docs.confluent.io/3.0.0/connect/connect-hdfs/docs/configuration_options.html
flush.size=50
hdfs.url=hdfs:///
hadoop.conf.dir=
hive.conf.dir=hive.home
hive.database=kafka
hive.home=hdfs:///user/hive/warehouse
hive.metastore.uris=
You can’t perform that action at this time.