springcoil

springcoil

springcoil

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reproducibility from the start"
   ]
  },
  {

springcoil

I want to do ETL and I want to do it in python. What tools do I have access to?

• https://gist.github.com/jiffyclub/2a252333c8dcad1b99aa

• Orca is a lightweight pandas centric library for building simple data pipelines

• Luigi - seems to be for very heavy weight projects

• Airflow from Airbnb - not used this yet

• night-shift - lightweight data flow.

• Proof

springcoil

# -*- coding: utf-8 -*-
#!/usr/bin/python
#
# Modified example of Albert Au Yeung's on matrix factorization:
# http://www.quuxlabs.com/blog/2010/09/matrix-factorization-a-simple-tutorial-and-implementation-in-python/#source-code

import numpy as np
from numba.decorators import autojit, jit

springcoil

springcoil

springcoil

springcoil

import urllib.request
import json

URL = ('http://datapoint.metoffice.gov.uk/public/data/val/wxfcs/all/json/352817'
      + '?res=3hourly&key=*api_key*')


class Manager(object):
    def __init__(self, api_key=""):
        self.api_key = api_key

springcoil