Skip to content

Instantly share code, notes, and snippets.

@jeremyyeo
Last active September 22, 2023 05:07
Show Gist options
  • Save jeremyyeo/21c384f251d2216f46e3b7ae1ffe725f to your computer and use it in GitHub Desktop.
Save jeremyyeo/21c384f251d2216f46e3b7ae1ffe725f to your computer and use it in GitHub Desktop.
Converting the timestamps in dbt #dbt

Converting the timestamps in dbt

It's not possible to change the implementation of the internal timestamps recorded by dbt's artifacts (i.e. the timestamps recorded in the on-run-end context results var or the run_results.json file) - so you'll have to manually change it yourself using some builtin modules. Keep in mind that this does not change the original values in run_results.json or in results - those WILL always be in UTC timestamps.

In the example below, we're simply converting one of the timestamps dbt records (when a model is completed) into a different timezone - then it is our responsiblity for then taking that converted value and writing it to a table in the database if we want to.

# dbt_project.yml
name: my_dbt_project
profile: all
config-version: 2
version: 1.0

models:
  my_dbt_project:
    +materialized: table

on-run-end: "{{ check_results(results) }}"
-- models/foo.sql
select 1 id

-- macros/check_results.sql
{% macro check_results(results) -%}
    {% if execute %}
    {% for r in results %}
        {% set local_tz = modules.pytz.timezone("Pacific/Auckland") %}
        {% set ts_raw = r.timing[1].completed_at %}
        {% set ts_utc_aware = modules.pytz.utc.localize(ts_raw) %}
        {% set ts_in_local_tz = ts_utc_aware.astimezone(local_tz) %}
        {% set line %}
          -----------------------------------------------------------
          node {{ r.node.unique_id }} 
          status: {{ r.status }}
          completed_at_utc:   {{ ts_utc_aware }}
          completed_at_local: {{ ts_in_local_tz }}
          -----------------------------------------------------------
        {% endset %}
        {% do log(line, True) %}
    {% endfor %}
    {% endif %}
{%- endmacro %}
$ dbt run

05:01:44  Running with dbt=1.6.3
05:01:45  Registered adapter: snowflake=1.6.2
05:01:46  Found 1 model, 1 operation, 0 sources, 0 exposures, 0 metrics, 373 macros, 0 groups, 0 semantic models
05:01:46  
05:01:52  Concurrency: 1 threads (target='sf')
05:01:52  
05:01:52  1 of 1 START sql table model dbt_jyeo.foo ...................................... [RUN]
05:01:55  1 of 1 OK created sql table model dbt_jyeo.foo ................................. [SUCCESS 1 in 3.85s]
05:01:55  
05:01:55  Running 1 on-run-end hook
05:01:55  
          -----------------------------------------------------------
          node model.my_dbt_project.foo 
          status: success
          completed_at_utc:   2023-09-22 05:01:55.337792+00:00
          completed_at_local: 2023-09-22 17:01:55.337792+12:00
          -----------------------------------------------------------
        
05:01:55  1 of 1 START hook: my_dbt_project.on-run-end.0 ................................. [RUN]
05:01:55  1 of 1 OK hook: my_dbt_project.on-run-end.0 .................................... [OK in 0.00s]
05:01:55  
05:01:55  
05:01:55  Finished running 1 table model, 1 hook in 0 hours 0 minutes and 9.45 seconds (9.45s).
05:01:55  
05:01:55  Completed successfully
05:01:55  
05:01:55  Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment