This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Read data | |
dfCustomer = spark.read\ | |
.format("cosmos.oltp")\ | |
.option("spark.synapse.linkedService", "AdventureWorksSQL")\ | |
.option("spark.cosmos.container", "Customer")\ | |
.load() | |
display(dfCustomer.limit(10)) | |
# Write data |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Spark Streaming with Cassandra (Azure Cosmos DB) | |
1. https://github.com/mspnp/azure-databricks-streaming-analytics |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1.http://www.teradatahelp.com/2010/09/how-can-you-view-data-distribution-in.html | |
1. http://rajgopal-gurrapushala.blogspot.com/2012/11/types-of-hash-functions-used-in-teradata_734.html | |
Create Table emp | |
( | |
ID BIGINT NOT NULL GENERATED BY DEFAULT AS IDENTITY | |
(START WITH 1 | |
INCREMENT BY 1 | |
MINVALUE -999999999999999999 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
{ | |
"_id" : "F0FF0BA8-CEF4-454C-9173-FA7C546F7179", | |
"customerId" : "A852CB99-DAA1-4348-A86F-622D4478A7D0", | |
"orderDate" : "2013-07-11T00:00:00", | |
"shipDate" : "2013-07-18T00:00:00", | |
"details" : [ | |
{ | |
"sku" : "LJ-0192-M", | |
"name" : "Long-Sleeve Logo Jersey, M", | |
"price" : 49.99, |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
conda info | |
conda update -n base -c defaults conda | |
conda create --name data_ingestion python=3.6 | |
(OR) | |
conda create --name data_ingestion | |
conda activate data_ingestion | |
conda list |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
package com.iwinner.runanysql; | |
import java.io.BufferedReader; | |
import java.io.FileReader; | |
import java.io.Reader; | |
import java.sql.Connection; | |
import java.sql.DriverManager; | |
import java.sql.ResultSet; | |
import java.sql.SQLException; | |
import java.sql.Statement; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
{ | |
insertId: "wr3zn3e373bw" | |
logName: "projects/iwinner-data-318822/logs/cloudaudit.googleapis.com%2Factivity" | |
operation: { | |
first: true | |
id: "operation-1627046240424-5c7ca39319163-07a56cfa-f16ec9a5" | |
producer: "compute.googleapis.com" | |
} | |
protoPayload: { | |
@type: "type.googleapis.com/google.cloud.audit.AuditLog" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
https://morioh.com/p/4d9c2c2cb9ae?f=5c224490c513a556c9042463&fbclid=IwAR0EA40-1MS3YNSJRkdnGIjAe6of7e3kIRGVRXq1t14aT44eIx4N6ykJ4L8 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
https://medium.com/@somanathsankaran/window-functions-deep-dive-4aeffebf81e1 | |
https://medium.com/analytics-vidhya/better-spark-logging-and-monitoring-with-spark-listener-c2c5fe86547a | |
https://medium.com/@somanathsankaran/window-functions-deep-dive-4aeffebf81e1 | |
https://medium.com/analytics-vidhya/better-spark-logging-and-monitoring-with-spark-listener-c2c5fe86547a | |
Small files spark | |
https://github.com/capt-blackdron/pyspark_custom_repartitioner/blob/main/custom_repartitioner.py | |
Scd type 2 | |
https://github.com/capt-blackdron/pyspark-examples/blob/main/SCD_Type_2_Implementation_in_PySpark.ipynb | |
Spark program |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
https://stackoverflow.com/questions/59617081/cast-a-column-to-binary-datatype-in-spark | |
https://stackoverflow.com/questions/53075020/why-does-spark-infer-a-binary-instead-of-an-arraybyte-when-creating-a-datafram | |
https://stackoverflow.com/questions/41149522/extract-byte-from-spark-binarytype | |
https://stackoverflow.com/questions/47902691/filtering-by-binary-type-in-spark-sql | |
https://stackoverflow.com/questions/60627680/spark-binarytype-to-scala-java |