Skip to content

Instantly share code, notes, and snippets.

View rfmcnally's full-sized avatar

Ryan McNally rfmcnally

  • San Francisco Bay Area
  • 05:49 (UTC -07:00)
View GitHub Profile
@rfmcnally
rfmcnally / google-sheets-to-df.py
Last active January 27, 2024 09:40
Example of fetching data from Google Sheets document and loading into Pandas DataFrame
import os
import json
from dataclassses import dataclass, asdict
import pandas as pd
from googleapiclient.discovery import build
from google.oauth2 import service_account
@dataclass
@rfmcnally
rfmcnally / keybase.md
Created December 24, 2020 01:00

Keybase proof

I hereby claim:

  • I am rfmcnally on github.
  • I am rfmcnally (https://keybase.io/rfmcnally) on keybase.
  • I have a public key ASDm8I7-V3TsLH0ncgyjmjJmQiBndd7V-5RrQFOl5WvkKAo

To claim this, I am signing this object:

@rfmcnally
rfmcnally / json-sample.json
Last active May 21, 2018 22:49
An example of the JSON format.
{ "output":
[
{"url":"https://www.yelp.com/search?find_desc=Desserts&find_loc=San+Jose,+CA&start=0","result":{"extractorData":{"url":"https://www.yelp.com/search?find_desc=Desserts&find_loc=San+Jose,+CA&start=0","data":[{"group":[{"Business":[{"href":"https://www.yelp.com/biz/milk-and-wood-san-jose?osq=Desserts","text":"Milk & Wood"}]},{"Business":[{"href":"https://www.yelp.com/biz/dzuis-cakes-and-desserts-san-jose?osq=Desserts","text":"Dzui’s Cakes & Desserts"}]},{"Business":[{"href":"https://www.yelp.com/biz/recess-italian-ice-and-desserts-san-jose?osq=Desserts","text":"Recess Italian Ice and Desserts"}]},{"Business":[{"href":"https://www.yelp.com/biz/icicles-san-jose-7?osq=Desserts","text":"ICICLES"}]},{"Business":[{"href":"https://www.yelp.com/biz/sweet-rendezvous-san-jose?osq=Desserts","text":"Sweet Rendezvous"}]},{"Business":[{"href":"https://www.yelp.com/biz/passion-t-snacks-and-desserts-san-jose?osq=Desserts","text":"Passion-T Snacks and Desserts"}]},{"Business":[{"href":"https://www.yelp.
@rfmcnally
rfmcnally / ndjson-sample.json
Created May 21, 2018 22:43
An example of the NDJSON format.
{"url":"https://www.yelp.com/search?find_desc=Desserts&find_loc=San+Jose,+CA&start=0","result":{"extractorData":{"url":"https://www.yelp.com/search?find_desc=Desserts&find_loc=San+Jose,+CA&start=0","data":[{"group":[{"Business":[{"href":"https://www.yelp.com/biz/milk-and-wood-san-jose?osq=Desserts","text":"Milk & Wood"}]},{"Business":[{"href":"https://www.yelp.com/biz/dzuis-cakes-and-desserts-san-jose?osq=Desserts","text":"Dzui’s Cakes & Desserts"}]},{"Business":[{"href":"https://www.yelp.com/biz/recess-italian-ice-and-desserts-san-jose?osq=Desserts","text":"Recess Italian Ice and Desserts"}]},{"Business":[{"href":"https://www.yelp.com/biz/icicles-san-jose-7?osq=Desserts","text":"ICICLES"}]},{"Business":[{"href":"https://www.yelp.com/biz/sweet-rendezvous-san-jose?osq=Desserts","text":"Sweet Rendezvous"}]},{"Business":[{"href":"https://www.yelp.com/biz/passion-t-snacks-and-desserts-san-jose?osq=Desserts","text":"Passion-T Snacks and Desserts"}]},{"Business":[{"href":"https://www.yelp.com/biz/vampire-penguin-fe
@rfmcnally
rfmcnally / importio-store.py
Created May 16, 2018 21:39
Sample Python classes for Import.io Store Objects
""" Classes for interacting with Import.io Object Store. Requires IMPORT_IO_API_KEY in env. """
import os
import math
import time
import json
from datetime import datetime, timedelta
from collections import OrderedDict
import requests
import pandas as pd
@rfmcnally
rfmcnally / importio-ndjson.py
Last active March 12, 2018 17:13
Sample Python class for processing NDJSON object from Import.io API
import os
import json
import requests
class Extractor(object):
""" An Import.io Extractor """
def __init__(self, guid):
self.guid = guid
def latest_json(self):
@rfmcnally
rfmcnally / importio_download.py
Created March 8, 2018 00:18
Script for downloading all CSV or Excel files from a list of Import.io Extractors
""" Script for automatically downloading attachments from Import.io API """
import os
import sys
import csv
from datetime import datetime
import requests
EXTRACTORS_FILE = 'importio_extractors.txt'
def create_folder():
@rfmcnally
rfmcnally / importio_latest_2.py
Last active March 19, 2018 18:43
Script for downloading latest CSVs from a list of Import.io Extractors with API key passed in as an environment value
""" Script for downloading latest CSVs from a list of Import.io extractors """
import os
import sys
import csv
import datetime
import requests
EXTRACTORS_FILE = 'importio_extractors.txt'
def main(text_file):
@rfmcnally
rfmcnally / importio-ndjson.js
Last active March 6, 2018 21:46
Sample JS code for processing NDJSON object from Import.io API
// This is a simple approach to downloading the data from our API
// and processing it as JSON Object. This method is only sufficient for
// small API responses (should work for responses below 100k results).
// If your response contains more results, try having a look at this Stack Overflow QA:
// http://stackoverflow.com/questions/15121584/how-to-parse-a-large-newline-delimited-json-file-by-jsonstream-module-in-node-j
// We start by getting data from our API:
let ExtractorGuid =0000000-0000-0000-0000-000000000000 // You can find this in the URL bar on the dashboard following https://dash.import.io/ 0000000-0000-0000-0000-000000000000
let apiKey = "YOUR_API_KEY"; // You can get it here https://import.io/data/account/
let API_ENDPOINT = "https://data.import.io/extractor/${ExtractorGuid}/json/latest?_apikey=${apiKey}";