Skip to content

Instantly share code, notes, and snippets.

@jcustenborder

jcustenborder/README.md

Last active February 13, 2019 16:35
Show Gist options
  • Save jcustenborder/63d22be110109a8443621fb99c06de36 to your computer and use it in GitHub Desktop.
Save jcustenborder/63d22be110109a8443621fb99c06de36 to your computer and use it in GitHub Desktop.
Download ZIP
kafka-connect-hdfs s3
Raw
README.md

Download the AWS Java SDK Copy aws-java-sdk-1.10.11.jar to $CONFLUENT_HOME/share/java/kafka-connect-hdfs/ Download hadoop-aws-2.6.0.jar to $CONFLUENT_HOME/share/java/kafka-connect-hdfs/ Download

Raw
s3.properties
name=hdfs-sink
connector.class=io.confluent.connect.hdfs.HdfsSinkConnector
tasks.max=1
topics=test_hdfs
hdfs.url=s3a://<awsAccessKeyId>:<awsSecretAccessKey>@s3-us-west-2.amazonaws.com/<bucket>/<path>
flush.size=3
@vyrwu
Copy link

vyrwu commented Feb 13, 2019 •
edited
Loading

Please note that AWS Java SDK link points to .zip file, not .jar. You need to extract the package first to get aws-java-sdk-1.10.11.jar
Sad programmer after 3h of debugging classpath...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment