p5k6

select stv_active_cursors.userid, left("name", 23), sequence, "text" from stv_active_cursors join stl_utilitytext on stl_utilitytext.pid=stv_active_cursors.pid order by stv_active_cursors.name, stv_active_cursors.pid, stl_utilitytext.starttime, sequence;

p5k6

The sticky assignor serves two purposes. First, it guarantees an assignment that is as balanced as possible, meaning either:

• the numbers of topic partitions assigned to consumers differ by at most one; or
• each consumer that has 2+ fewer topic partitions than some other consumer cannot get any of those topic partitions transferred to it.

p5k6

import logging
import textwrap
from datetime import datetime, timedelta
import uuid

from elasticsearch import Elasticsearch
from airflow import DAG  # noqa
from airflow import macros  # noqa
from airflow.operators.python_operator import PythonOperator  # noqa
from pyhocon import ConfigFactory

p5k6

import logging
from collections import namedtuple

from pyhocon import ConfigFactory, ConfigTree  # noqa: F401
from typing import Iterator, Union, Dict, Any  # noqa: F401

from databuilder import Scoped
from databuilder.extractor.base_extractor import Extractor
from databuilder.extractor.sql_alchemy_extractor import SQLAlchemyExtractor
from databuilder.models.table_metadata import TableMetadata, ColumnMetadata

p5k6

re:invent

• API gateway - no longer (exclusively) at the edge, can set up links within vpcs - see here

Monday

---

ABD202 - Best Practices for Building Serverless Big Data Applications

• slide deck

p5k6

• zone entries - F1, F2, F3
• breakout by D w/forward, regroup, then 2 on 1
  • involve patty eventually
  • Don't pass at the blue line. Try to pass by red line, or take it in yourself
  • after entering - make judgement. Can I get by D? don't be afraid to slow down, create space
• dig in with skates - pushing net?
• if you're not doing anything, just waiting - practice puckhandling
• 1 on 1 on the boards - don't just throw it up the boards - esp for strong side breakouts
• Retreiving the puck in deep behind net
• D playing against F in corner

p5k6

name: qa-test-transform
frequency: daily
load_time: 08:00

description: |
  testing out new transform step with precondition

steps:
- step_type: extract-s3
  name: ops-input

p5k6

"""
ETL step wrapper for shell command activity can be executed on Ec2 / EMR, with precondition
"""
from dataduct.pipeline import S3Node
from dataduct.pipeline import ShellCommandActivity
from dataduct.pipeline import Precondition
from dataduct.s3 import S3Directory
from dataduct.s3 import S3File
from dataduct.s3 import S3Path
from dataduct.utils import constants as const

p5k6

"""
ETL step wrapper to extract data from mysql to S3, compressing it along the way
"""
import dataduct
from dataduct.config import Config
from dataduct.steps.etl_step import ETLStep
from dataduct.pipeline import CopyActivity
from dataduct.pipeline import MysqlNode
from dataduct.pipeline import PipelineObject
from dataduct.pipeline import Precondition

p5k6

Keybase proof

I hereby claim:

• I am p5k6 on github.
• I am p5k6 (https://keybase.io/p5k6) on keybase.
• I have a public key whose fingerprint is D844 1A65 94A4 665F CC35 BEA0 7DA7 7D09 8F4A 4E03

To claim this, I am signing this object: