Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Save ad1happy2go/f72dc434ea9bd6b36d5713c533d1b4ad to your computer and use it in GitHub Desktop.
Save ad1happy2go/f72dc434ea9bd6b36d5713c533d1b4ad to your computer and use it in GitHub Desktop.
~/spark/spark-3.2.3-bin-hadoop3.2/bin/spark-submit \
--class org.apache.hudi.utilities.deltastreamer.HoodieDeltaStreamer \
~/spark/code/hudi/packaging/hudi-utilities-bundle/target/hudi-utilities-bundle_2.12-0.13.0.jar --table-type MERGE_ON_READ \
--source-class org.apache.hudi.utilities.sources.JsonKafkaSource \
--source-ordering-field ts \
--target-base-path /tmp/issue_5916 \
--target-table stock_ticks_cow --props /var/demo/config/kafka-source.properties \
--schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider \
--continuous \
--min-sync-interval-seconds 5
base.properties-
hoodie.archive.automatic=false
hoodie.metadata.enable=false
hoodie.datasource.write.operation=insert
hoodie.compact.schedule.inline=false
hoodie.datasource.write.table.type=MERGE_ON_READ
hoodie.clean.automatic=false
hoodie.compact.inline=false
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment