Created
October 11, 2022 12:05
-
-
Save zhukovgreen/02f6369993546518d8f5c784a5163def to your computer and use it in GitHub Desktop.
Concept with transformations
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
class MyTransformation(BaseTransformation): | |
def implementation(metastore) -> DataFrame: | |
table_x = metastore.get("table_x") | |
table_y = metastore.get("table_y") | |
# do some transformation | |
return some_result_dataframe |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
name: my transformation | |
dependencies: | |
- table: table_x | |
format: hudi | |
path: s3://... | |
- table: table_y | |
ormat: hudi | |
path: s3://... | |
output: | |
table: table_out | |
format: hudi | |
path: s3://... |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment