Created
July 15, 2019 15:12
-
-
Save rmoff/d2b2e3ae4039707036420f4a54eb6618 to your computer and use it in GitHub Desktop.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
curl -i -X PUT -H "Content-Type:application/json" \ | |
http://localhost:8083/connectors/sink-elastic-orders-00/config \ | |
-d '{ | |
"connector.class": "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector", | |
"topics": "orders", | |
"connection.url": "http://elasticsearch:9200", | |
"type.name": "type.name=kafkaconnect", | |
"key.ignore": "true", | |
"schema.ignore": "false", | |
"errors.tolerance":"all", | |
"transforms": "addTS,InsertTopic,InsertOffset,InsertPartition", | |
"transforms.addTS.type": "org.apache.kafka.connect.transforms.InsertField$Value", | |
"transforms.addTS.timestamp.field": "op_ts", | |
"transforms.InsertTopic.type":"org.apache.kafka.connect.transforms.InsertField$Value", | |
"transforms.InsertTopic.topic.field":"source_topic", | |
"transforms.InsertOffset.type":"org.apache.kafka.connect.transforms.InsertField$Value", | |
"transforms.InsertOffset.offset.field":"source_offset", | |
"transforms.InsertPartition.type":"org.apache.kafka.connect.transforms.InsertField$Value", | |
"transforms.InsertPartition.partition.field":"source_partition" | |
}' |
@cyberjar09 I found https://github.com/jzaralim/KeyToValueTransform and also discussion here: https://www.mail-archive.com/users@kafka.apache.org/msg37254.html. You can also use Kafka Streams, ksqlDB, etc.
hey thanks a lot for the guidance @rmoff 🎉
much appreciated! 😄
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
is there a way to add the kafka record key to the value before sending it to a sink? This is not specific to elasticsearch but any sink. I can't seem to find such a transform, and it seems to me like it should be a solved problem 🤔