Timothy Carbone TimmyCarbone
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # How to schedule dbt runs with Airflow using gocardless/airflow-dbt | |
| # Folder structure is as such: | |
| # | |
| # /app | |
| # /airflow | |
| # /dags | |
| # /dbt_dag.py | |
| # /dbt | |
| # /config |
TimmyCarbone
/ distribute-streams.js
Last active
September 11, 2019 04:13
Distribute rows from readable stream to multiple streams depending on the row content
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| const { PassThrough } = require('stream'); | |
| // FUNCTIONS | |
| // -------- | |
| // Function: For each partition, we create an object containing a stream and | |
| // limits for that partition. | |
| const createBranchingStreams = (partitions) => { | |
| let streams = partitions.map((p, i) => ({ | |
| min: p.min, |