Skip to content

Instantly share code, notes, and snippets.

@ksingh7
Created June 14, 2020 14:49
Show Gist options
  • Save ksingh7/f87570b61f8144382673ef7eb04186e1 to your computer and use it in GitHub Desktop.
Save ksingh7/f87570b61f8144382673ef7eb04186e1 to your computer and use it in GitHub Desktop.
Kafka Connector YAML for AWS S3
apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaConnector
metadata:
name: s3-sink-connector
labels:
strimzi.io/cluster: my-connect-cluster
spec:
class: org.apache.camel.kafkaconnector.CamelSinkConnector
tasksMax: 1
config:
key.converter: org.apache.kafka.connect.storage.StringConverter
value.converter: org.apache.kafka.connect.storage.StringConverter
topics: sensor-data-ingest-topic
#camel.sink.url: aws-s3://k2s3?keyName=${date:now:yyyyMMdd-HHmmssSSS}-${exchangeId}
camel.sink.url: aws-s3://k2s3?keyName=${date:now:yyyyMMdd}/${exchangeId}
camel.sink.maxPollDuration: 10000
camel.component.aws-s3.configuration.autocloseBody: false
camel.component.aws-s3.accessKey: ${file:/opt/kafka/external-configuration/aws-credentials/aws-credentials.properties:aws_access_key_id}
camel.component.aws-s3.secretKey: ${file:/opt/kafka/external-configuration/aws-credentials/aws-credentials.properties:aws_secret_access_key}
camel.component.aws-s3.region: US_EAST_1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment