Skip to content

Instantly share code, notes, and snippets.

@RachidAZ

RachidAZ/varilable_mutation.py

Last active January 16, 2022 23:14
Show Gist options
  • Save RachidAZ/99f704c67ea408aa479a26a79b72d602 to your computer and use it in GitHub Desktop.
Save RachidAZ/99f704c67ea408aa479a26a79b72d602 to your computer and use it in GitHub Desktop.
Download ZIP
PySpark use Python variable in SQL command
Raw
varilable_mutation.py
# set the varilable in python
filePath='abfss://{container_name}@{storage_account_name}.dfs.core.windows.net/curated/Dims/Dim_X/*.csv'
spark.conf.set('f.filePath',filePath)
# use the varilable in SQL command
%sql
CREATE OR REPLACE TEMPORARY VIEW V_SomeView
USING CSV
OPTIONS (path '${f.filePath}', header "true", mode "FAILFAST")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment