Skip to content

Instantly share code, notes, and snippets.

  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
Star You must be signed in to star a gist
Embed
What would you like to do?
Azure synapse change data capture
import com.microsoft.azure.cosmos.analytics.spark.connector.datasource.CosmosOlapCDC
val configuration = Map(
"spark.cosmos.accountEndpoint" -> "https://spark-perf.documents.azure.com:443/",
"spark.cosmos.accountKey" -> "xxx",
"spark.cosmos.database" -> "cdc",
"spark.cosmos.container" -> "product",
"spark.cosmos.cdc" -> "true",
"spark.cosmos.cdc.initialRun" -> "true",
"spark.cosmos.cdc.fullFidelity" -> "false",
"spark.cosmos.cdc.keepUserDeletes" -> "true",
"spark.cosmos.cdc.keepTtlDeletes" -> "true"
)
val cdcSnapshotMetadata = CosmosOlapCDC.initializeSnapshotRead(configuration)
val cdcDf = CosmosOlapCDC.createIncrementalSnapshot(cdcSnapshotMetadata)
display(cdcDf)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment