Download the AWS Java SDK Copy aws-java-sdk-1.10.11.jar to $CONFLUENT_HOME/share/java/kafka-connect-hdfs/ Download hadoop-aws-2.6.0.jar to $CONFLUENT_HOME/share/java/kafka-connect-hdfs/ Download
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[Unit] | |
Description = Apache Kafka by Confluent | |
Wants = basic.target | |
After = basic.target network.target | |
[Service] | |
EnvironmentFile = /etc/sysconfig/kafka | |
ExecStart = /usr/bin/kafka-server-start /etc/kafka/server.properties | |
LimitNOFILE = 131072 | |
KillMode = process |
This is an example docker-compose.yml for the Confluent Platform.
In order to properly use this docker-compose make sure that your containers are resolvable by their container name. Take a look at the hosts file. Each container must be resolvable from the host machine if you are connecting from the host to the docker environment.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#http://docs.confluent.io/3.0.0/connect/connect-hdfs/docs/configuration_options.html | |
flush.size=50 | |
hdfs.url=hdfs:/// | |
hadoop.conf.dir= | |
hive.conf.dir=hive.home | |
hive.database=kafka | |
hive.home=hdfs:///user/hive/warehouse | |
hive.metastore.uris= |