This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
version: 2 | |
models: | |
- name: last_day_of_month_test | |
description: "A starter dbt model" | |
tests: | |
- dbt_utils.expression_is_true: | |
expression: "is_last_day_of_month = TRUE" | |
config: | |
where: "datecol = '2023-12-31'" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
{{ | |
config( | |
materialized = 'view', | |
) | |
}} | |
select datecol, {{- is_last_day_of_month('datecol') -}} as is_last_day_of_month from | |
(select '2023-12-31' as datecol union all select '2022-12-03' as datecol) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
version: 2 | |
# available only for dbt 1.8 | |
unit_tests: | |
- name: test_is_last_day_of_month | |
description: "Check of model last_day_of_month_orders correctly filter for last day of month orders" | |
model: last_day_of_month_orders | |
given: | |
- input: ref('clean_orders') | |
rows: | |
- {order_datetime: '2022-12-31'} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
select * from ref('clean_orders') where {{- is_last_day_of_month('order_datetime') -}} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
{%- macro is_last_day_of_month(datecol) %} | |
DAY(DATE_ADD(day, 1, cast({{ datecol }} AS DATE))) = 1 | |
{% endmacro -%} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
version: 2 | |
models: | |
- name: clean_orders | |
description: "raw orders after cleaning" | |
columns: | |
- name: customer_id | |
description: "The id of customers for this table" | |
tests: | |
- not_null |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
def select_all_filter_by(df, filter_column, filter_value): | |
return df.filter(f"{filter_column}='{filter_value}'") | |
def model(dbt, session): | |
orders_df = dbt.ref('clean_orders') | |
return select_all_filter_by(orders_df, 'state', 'WA') |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
def model(dbt, session): | |
dbt.config(materialized="table") | |
raw_orders_df = dbt.source('dbt_databricks', 'raw_orders') | |
all_cols = raw_orders_df.columns | |
return raw_orders_df.select([col for col in all_cols if col.startswith('order') or col.startswith('customer')]) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
import dlt | |
source = "<catalog>.<schema>." | |
@dlt.table | |
def clean_orders(): | |
raw_orders = spark.read.table(f"{source}raw_orders") | |
customers = spark.read.table(f"{source}customer_orders") | |
join_conditions = [ | |
raw_orders.customer_id == customers.customer_id, |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
{{- select_all_filter_by(ref('clean_orders'), 'city', 'Seattle') -}} |
NewerOlder