Download the AWS Java SDK Copy aws-java-sdk-1.10.11.jar to $CONFLUENT_HOME/share/java/kafka-connect-hdfs/ Download hadoop-aws-2.6.0.jar to $CONFLUENT_HOME/share/java/kafka-connect-hdfs/ Download
Last active
February 13, 2019 16:35
-
-
Save jcustenborder/63d22be110109a8443621fb99c06de36 to your computer and use it in GitHub Desktop.
kafka-connect-hdfs s3
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
name=hdfs-sink | |
connector.class=io.confluent.connect.hdfs.HdfsSinkConnector | |
tasks.max=1 | |
topics=test_hdfs | |
hdfs.url=s3a://<awsAccessKeyId>:<awsSecretAccessKey>@s3-us-west-2.amazonaws.com/<bucket>/<path> | |
flush.size=3 |
Please note that AWS Java SDK link points to .zip file, not .jar. You need to extract the package first to get aws-java-sdk-1.10.11.jar
Sad programmer after 3h of debugging classpath...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Thank you, Jeremy, it was priceless to find these hints!