jeremyyeo

Building SCD-2 models using the default incremental materialization

1. Bootstrap the source / snapshot table source_users and the initial state of our scd2 table dim_users by running:

dbt run -m source_users dim_users

2. Copy over contents of dim_users_inc.sql replacing contents in dim_users.sql (or just rename them). Then run:

smrgit

  CREATE TEMPORARY FUNCTION

    -- In this function, we're going to be working on arrays of values.
    -- we're also going to define a set of functions 'inside' the kMeans.

    -- *heavily borrowing from https://github.com/NathanEpstein/clusters* --

    kMeans(x ARRAY<FLOAT64>,  -- ESR1 gene expression
           y ARRAY<FLOAT64>,  -- EGFR gene expression
           iterations FLOAT64,  -- the number of iterations