bsweger

Cloud-Based Hubs

The Hubverse is in the processing of making hub data available via publicly-accessible AWS S3 buckets.

Cloud-based hubs "mirror" the data stored in a hub's GitHub repository and provide a few advantages for data consumers:

• No need to clone a repository to access data
• Cloud-based model-output files are in parquet format, which is easier to work with and more performant

The examples here use the CDC's FluSight Forecast Hub, which is available in the following S3 bucket:

bsweger

from itertools import product
import pandas as pd
import numpy as np

def make_sample(
    n_samples: int = 2,
    n_horizons: int = 3,
    n_variants: int = 3,
    n_locations: int = 2,
    samples_joint_across: list[str] = None

bsweger

# If you come from bash you might have to change your $PATH.
# export PATH=$HOME/bin:/usr/local/bin:$PATH

# Path to your oh-my-zsh installation.
export ZSH="/home/becky/.oh-my-zsh"

# Set name of the theme to load --- if set to "random", it will
# load a random theme each time oh-my-zsh is loaded, in which case,
# to know which specific one was loaded, run: echo $RANDOM_THEME
# See https://github.com/ohmyzsh/ohmyzsh/wiki/Themes

bsweger

import pytest
import requests

@pytest.fixture
def mock_request_json():
    """Return fake API data."""
    return {'spring': [
        {
            'name': 'birds',
            'status': 'chirpin',

bsweger

### Keybase proof

I hereby claim:

  * I am bsweger on github.
  * I am bendystraw (https://keybase.io/bendystraw) on keybase.
  * I have a public key ASCmomBSZgzxe3w1YQf-eZwTB3KNg7k4VGD29NN8hhmrDAo

To claim this, I am signing this object:

bsweger

# Get SETI Institute grants from USAspending API into a pandas dataframe
import json
import requests
import pandas as pd
from pandas.io.json import json_normalize

uri = 'https://api.usaspending.gov/api/v2/search/spending_by_transaction/'
headers = {'content-type': 'application/json'}
seti_json = []
next = 1

bsweger

{
  "Ansi 3 Color" : {
    "Green Component" : 0.73333334922790527,
    "Blue Component" : 0,
    "Red Component" : 0.73333334922790527
  },
  "Tags" : [

  ],
  "Ansi 12 Color" : {

bsweger

Retrieve financial balances by agency and fiscal year

Route: /api/v2/financial_balances/agencies

Method: GET

This route retrieves financial balance information by funding agency and fiscal year

Sample Request

/api/v2/financial_balances/agencies?funding_agency=775&fiscal_year=2017

bsweger

# example of using a parameterized function as a converter when reading .csv in pandas

import pandas as pd

# a function that will be used to pad datafram column values to a specified length
# (some incoming values are multiple spaces; those should convert to Noe)
padFunction = lambda field, padTo: str(field).strip().zfill(padTo) if len(str(field).strip()) else None

# read file w/o using converters and display list of unique alloc_id values
pa = pd.read_csv(

bsweger

#tabedata.txt is a file that contains the piped markdown table rows
with open ('markdownrows.txt') as f:
    content = f.readlines()

rows = []

for c in content:
    cells = c.split('|')
    cells = ['<td>{}</td>'.format(cell.strip()) for cell in cells] 
    rows.append(cells)