- https://www.justingarrison.com/blog/2020-08-31-remote-onboarding/
- https://twitter.com/rothgar/status/1296911972215058432
- https://twitter.com/rothgar/status/1296911975314677760
The problems I've had on-boarding remotely are
The problems I've had on-boarding remotely are
from pyspark.sql.functions import col, explode_outer, from_json, lit, concat | |
from pyspark.sql.types import StructType, ArrayType | |
def get_json_df(input_df, primary_partition_column, json_column_name, spark_session): | |
''' | |
Description: | |
This function provides the schema of json records and the dataframe to be used for flattening | |
:param input_df: [type: pyspark.sql.dataframe.DataFrame] input dataframe |