I like wikipedia. There must be some sort of project I could do with this data.
- Wikipedia There are accessible dumps of wikipedia data.
Hugh Brown hughdbrown
I was listening on NPR today and heard that within the UN, there are about a dozen different blocs that vote together on global warming issues:
- Switzerland alone
- Developed countries
- European group
- "77 countries plus China" ... which is actually 134 countries
- Various island nations most affected
hughdbrown
/ data-chronic-kidney-disease.md
Last active
August 31, 2015 17:35
Chronic kidney disease predictor
- UC Irving 400 rows by 25 attributes
hughdbrown
/ data-job-recommender.md
Created
August 31, 2015 22:59
Job recommender that bootstraps from list of job postings
So often, job sites give candidates job listings that are far off topic. The job title is often not applicable for the candidate, and less often, the location does not match the cadidate's location.
Can we build a better system for users by applying a recommender system to existing public listings?
- glassdoor.com API
- indeed.com web scraping
Homeaway has data on vacation rentals. The data is not nearly so worked over as AirBNB data. Possibly there is something interesting in there to disover.
- Homeaway API access The main problem with the project is that the Homeaway API is pretty opaque. I can't figure out how to get a data dump. Also, the API requires registration and advance permission.
Here is how I copied data from one S3 bucket to another:
aws s3 sync s3://bitly-challenges/hdb_sanitized s3://hughdbrown/data-capstone
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| """ | |
| Python script to backup data in src to dst using sha1 hashes of the files | |
| in a backing directory. | |
| Hugh Brown | |
| hughdbrown@yahoo.com | |
| """ | |
| from hashlib import sha1 | |
| import os |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import numpy | |
| import scipy.stats as scs | |
| def a_b_test(new_views, new_clicks, old_views, old_clicks, size=10000): | |
| new_site = scs.beta(a=new_clicks + 1, b=new_views + 1).rvs(size=size) | |
| old_site = scs.beta(a=old_clicks + 1, b=old_views + 1).rvs(size=size) | |
| return (new_site > old_site).mean() |