hunterowens

{"executionTime":"2013-07-09 04:59:02 PM","stationBeanList":[{"id":5,"stationName":"State St & Harrison St","availableDocks":14,"totalDocks":19,"latitude":41.8739580629,"longitude":-87.6277394859,"statusValue":"In Service","statusKey":1,"availableBikes":5,"stAddress1":"State St & Harrison St","stAddress2":"","city":"","postalCode":"","location":"620 S. State St.","altitude":"","testStation":false,"lastCommunicationTime":null,"landMark":"30"},{"id":13,"stationName":"Wilton Ave & Diversey Pkwy","availableDocks":13,"totalDocks":19,"latitude":41.93250008,"longitude":-87.65268082,"statusValue":"In Service","statusKey":1,"availableBikes":3,"stAddress1":"Wilton Ave & Diversey Pkwy","stAddress2":"","city":"Chicago","postalCode":"","location":"2790 N.Wilton Ave","altitude":"","testStation":false,"lastCommunicationTime":null,"landMark":"66"},{"id":14,"stationName":"Morgan St &18th St","availableDocks":11,"totalDocks":15,"latitude":41.85807921,"longitude":-87.65140281,"statusValue":"In Service","statusKey":1,"availableB

hunterowens

###Directory Structure

• Clean, Well named set of directories. Examples include webapp, database, and models.
• No random files in the root.
• Explanation of each directory in README.
• Sub-README's in appropriate folders
• No directories named after DSSG specific info (ie, person names)
• Should your team have more than one project, each should have it its own repo.
• In your data or database folder, provide a way to re-create your database from scratch. .sql files are often appropriate for this.

hunterowens

REQUIRED WIKI SECTIONS

• Homepage

  Intro: have a sentence or two about the project: the problems its solving, the partner, and how you're solving it. Also say that "this wiki is the central place to learn about the social problem we worked on, the data we used, the methods we used to solve it, and our findings" so people know what they're looking at. List of pages in the wiki

• Problem

  An in-depth description of the problem your organization, the problem you're trying to solve, and any relevant domain knowledge. Feel free to copy from blog posts and posters, if relevant.

• Data

hunterowens

# -----------------------------
# PostgreSQL configuration file
# -----------------------------
#
# This file consists of lines of the form:
#
#   name = value
#
# (The "=" is optional.)  Whitespace may be used.  Comments are introduced with
# "#" anywhere on a line.  The complete list of parameter names and allowed

hunterowens

# PostgreSQL Client Authentication Configuration File
# ===================================================
#
# Refer to the "Client Authentication" section in the PostgreSQL
# documentation for a complete description of this file.  A short
# synopsis follows.
#
# This file controls: which hosts are allowed to connect, how clients
# are authenticated, which PostgreSQL user names they can use, which
# databases they can access.  Records take one of these forms:

hunterowens

import timeit
import pandas as pd
import requests

S3_URL = "https://s3-us-west-2.amazonaws.com/dssgtest01/crimedata.csv"
S3_PATH = "s3://dssgtest01/crimedata.csv"
LOCAL_EBS_PATH = "/mnt/vol-2a96ab12/crimedata.csv"
NFS_PATH = "/mnt/data1/SpeedTest/crimedata.csv"

#S3 Testing. using HTML

hunterowens

hunterowens

1. Portillos - Classic Chicago - Italian Beef, Hot Dogs - Farther Walk- 100 W Ontario St
2. Food Trucks - Right outside. Take a Look.
3. Food For Thought - Food Court/Salad Bar - 401 North Michigan Ave - walk towards Michigan.
4. Billy Goat Tavern - 430 N. Michigan Ave - LOWER LEVEL- The Classic Chicago Burger Place. Under Tribune Tower.
5. Water Tower Place Food Court - It's a Mall with a food court. Walk up Michigan.
6. Epic Burger - New Burger place - 227 E Ontario St
7. Sayat Nova - Sit down Armenian food. 157 E. Ohio St.
8. Chipotle - 291 E Ontario St
9. Potbelly - Local sandwich chain- 277 E Ontario St
10. Naf Naf Grill - Middle Eastern Chipotle - 326 N Michigan Ave

hunterowens

import pymssql
import pandas as pd

## instance a python db connection object- same form as psycopg2/python-mysql drivers also
conn = pymssql.connect(server="172.0.0.1", user="howens",password="some_fake_password", port=63642)  # You can lookup the port number inside SQL server. 

## Hey Look, college data
stmt = "SELECT * FROM AlumniMirror..someTable"
# Excute Query here
df = pd.read_sql(stmt,conn)

hunterowens

import xmltodict
import json
import pandas as pd

file = open('/path/to/file.xml','r')
str = file.read()
dictionary = xmltodict.parse(str)
df = pd.DataFrame(dictionary)
# Do any parsing of the XML file here, as you'll want to confirm that you are writing the correct info to csv. 
df.to_csv('/path/to/output.csv')