larsyencken

[tool.poetry]
name = "owid-datautils"
version = "0.5.3"
description = "Data utils library by the Data Team at Our World in Data"
authors = ["Our World In Data <tech@ourworldindata.org>"]
license = "MIT"
packages = [{ include = "owid" }]
readme = "README.md"
repository = "https://github.com/owid/owid-datautils-py"
homepage = "https://github.com/owid/owid-datautils-py"

larsyencken

larsyencken

# just a partial config, only look at env_change
{
  hooks: {
    env_change: {
      PWD: [
        {
          # if you enter a python project
          condition: {|before, after| ["pyproject.toml" "requirements.txt"] | any {|f| $f | path exists } }
          # drop any prior virtualenv, then use a new one if it exists
          code: "

larsyencken

MySQL in Nushell

Here are some helpers for using MySQL in Nushell. You need the MySQL client installed, and also the separate MySQL Shell client (for JSON output).

The mysql-profile command is an optional helper for keeping multiple MySQL connection profiles around in your ~/.my.cnf and switching between them, instead of having to give credentials every time.

Enjoy!

larsyencken

Current dependencies

graph LR

etl --> owid-catalog-py
etl --> walden --> owid-datautils-py
walden --> owid-catalog-py
etl --> owid-datautils-py

larsyencken

#
#  python: enter virtualenv on chpwd
#
load-py-venv() {
    if [ -f .venv/bin/activate ]; then
        # enter a virtual environment that's here
        source .venv/bin/activate
    elif [ -f env/bin/activate ]; then
        source env/bin/activate
    elif [ ! -z "$VIRTUAL_ENV" ] && [ -f pyproject.toml -o -f requirements.txt ]; then

larsyencken

#!/usr/bin/env python
#
#  mysql-profile
#
#  Switch between mysql profiles.
#

from pathlib import Path
import configparser
from dataclasses import dataclass

larsyencken

from concurrent.futures import ProcessPoolExecutor

def dog():
    raise ValueError('not a dog, a cat')

def main():
    try:
        with ProcessPoolExecutor(1) as executor:
            executor.submit(dog).result()
            print('no exception')

larsyencken

sequenceDiagram
    participant cron
    participant etl_prod
    participant bulk_backport
    participant walden
    participant etl
    participant disk
    participant s3
    participant github

larsyencken

Arrow−Struck Tiger
Big Halberd
Black Whirlwind
Boat Flame
Comet
Demon Carver
Demon Face
Demon King Who Roils the World
Dragon from the Cave
Dragon from the Forest