barbatus

Table of contents

• Data pre-processing
  • ETL pipeline based on dbt
• Data injection
• Aggregated values
  • Benefits (aggregations)
• PostgreSQL materialized views (instead of complex ad-hoc queries)
  • Benefits (materialized views)
• Example ETL structure
• Example: staging model (parse raw JSON/XML invoices + cleaning/deduplication)

barbatus

Reason:
Some services currently do more than CRUD—they also compute statistics via complex, on-demand queries. This creates opportunities to:

• Improve performance.
• Separate concerns by introducing a dedicated data processing pipeline for ingestion and preprocessing into query-friendly layouts.

ETL pipeline based on dbt:

raw_invoice_data (JSONB)
         │
 v

barbatus

Keybase proof

I hereby claim:

• I am barbatus on github.
• I am barbatus (https://keybase.io/barbatus) on keybase.
• I have a public key ASCRAwwKD6FxAL-1JVEJFSTZQ3RMUvrcB-fpuq-u4cTAOAo

To claim this, I am signing this object:

barbatus

const puppeteer = require('puppeteer')
const fs = require('fs')
const fetch = require('node-fetch')

function timeout(ms) {
  return new Promise(resolve => setTimeout(() => resolve(), ms));
}

const LOGIN_URL = 'https://beta.footballindex.co.uk/top-200'
const PORTFOLIO_URL = 'https://beta.footballindex.co.uk/portfolio'

barbatus

var puppeteer = require('puppeteer');
var parseUrl = require('url').parse;

var src = process.argv[2];
var dest = process.argv[3];

function timeout(ms) {
  return new Promise(resolve => setTimeout(() => resolve(), ms));
}