Skip to content

Instantly share code, notes, and snippets.

@sajeetharan

sajeetharan/gist:3d1c53bdd450f6c74dbe696860bc1943

Created May 24, 2023 16:27
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save sajeetharan/3d1c53bdd450f6c74dbe696860bc1943 to your computer and use it in GitHub Desktop.
Save sajeetharan/3d1c53bdd450f6c74dbe696860bc1943 to your computer and use it in GitHub Desktop.
Download ZIP
Azure synapse change data capture
Raw
gistfile1.txt
import com.microsoft.azure.cosmos.analytics.spark.connector.datasource.CosmosOlapCDC
val configuration = Map(
"spark.cosmos.accountEndpoint" -> "https://spark-perf.documents.azure.com:443/",
"spark.cosmos.accountKey" -> "xxx",
"spark.cosmos.database" -> "cdc",
"spark.cosmos.container" -> "product",
"spark.cosmos.cdc" -> "true",
"spark.cosmos.cdc.initialRun" -> "true",
"spark.cosmos.cdc.fullFidelity" -> "false",
"spark.cosmos.cdc.keepUserDeletes" -> "true",
"spark.cosmos.cdc.keepTtlDeletes" -> "true"
)
val cdcSnapshotMetadata = CosmosOlapCDC.initializeSnapshotRead(configuration)
val cdcDf = CosmosOlapCDC.createIncrementalSnapshot(cdcSnapshotMetadata)
display(cdcDf)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment