Matthew Powers MrPowers
🤓
MrPowers
/ delta_rs_spark_interop.py
Created
July 17, 2022 16:58
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import pathlib | |
| import shutil | |
| import deltalake as dl | |
| import pandas as pd | |
| import pyarrow.dataset as ds | |
| from pyspark.sql import SparkSession | |
| from delta import * | |
| import chispa |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| person_name | person_country | |
|---|---|---|
| a | China | |
| b | China | |
| c | China | |
| d | China | |
| e | China | |
| f | China | |
| g | China | |
| h | China | |
| i | China |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| website_url | website_type | main_language | |
|---|---|---|---|
| news.ycombinator.com | aggregator | ||
| mungingdata.com | blog | spark | |
| m.signalvnoise.com | blog | rails | |
| pgexercises.com | train | postgres | |
| codequizzes.com | train | ruby |
MrPowers
/ spark_write_to_aws.sc
Created
April 9, 2016 19:30
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| df.coalesce(1).write | |
| .format("com.databricks.spark.csv") | |
| .option("header", "true") | |
| .save("s3n://some_bucket/data/states/all_states/") |
MrPowers
/ spark_aws_credentials.sc
Created
April 9, 2016 18:43
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| val accessKeyId = System.getenv("AWS_ACCESS_KEY_ID") | |
| val secretAccessKey = System.getenv("AWS_SECRET_ACCESS_KEY") | |
| sc.hadoopConfiguration.set("fs.s3n.awsAccessKeyId", accessKeyId) | |
| sc.hadoopConfiguration.set("fs.s3n.awsSecretAccessKey", secretAccessKey) |
MrPowers
/ spark_s3_files.sc
Created
April 9, 2016 18:40
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| val df = sqlContext.read | |
| .format("com.databricks.spark.csv") | |
| .option("header", "true") | |
| .option("inferSchema", "true") | |
| .load("s3n://some_bucket/data/states/*.csv") |
MrPowers
/ spark_dataframe_multiple_gzipped_files.sc
Created
April 8, 2016 20:07
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| val df = sqlContext.read | |
| .format("com.databricks.spark.csv") | |
| .option("header", "true") | |
| .option("inferSchema", "true") | |
| .load(System.getProperty("user.home") + "/Desktop/people/*.gz") |
MrPowers
/ spark_dataframe_multiple_files.sc
Created
April 8, 2016 19:46
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| val df = sqlContext.read | |
| .format("com.databricks.spark.csv") | |
| .option("header", "true") | |
| .option("inferSchema", "true") | |
| .load(System.getProperty("user.home") + "/Desktop/people/*.csv") |
MrPowers
/ spark_dataframe_to_csv.sc
Last active
April 4, 2016 15:49
Writing a Spark DataFrame to a CSV file
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| tx_cities.coalesce(1).write | |
| .format("com.databricks.spark.csv") | |
| .option("header", "true") | |
| .save(System.getProperty("user.home") + "/Desktop/texas_cities") |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| val df = sqlContext.read | |
| .format("com.databricks.spark.csv") | |
| .option("header", "true") | |
| .option("inferSchema", "true") | |
| .load(System.getProperty("user.home") + "/Desktop/cities.csv") |