Skip to content

Instantly share code, notes, and snippets.

View Journalists and Numbers
Some things journalists may want to consider:
1. Anecdotes can mislead. People seeing another yet another episodic story on crime may infer that crime is increasing.
So report numbers where trustworthy numerical data are available.
2. But numbers need to be reported carefully. Most people, when reading news, do not do back of the envelope calculations to interpret data correctly.
So ill-reported numbers can mislead.
3. Rules for numbers:
a. % changes than changes in %. The former is more impressive when the base rate is low. Latter generally a better way to report things. If confused, report t1 and t2.
soodoku / Hillary_Clinton
Last active August 29, 2015 14:17
Calculating Hillary's Missing Emails
View Hillary_Clinton
55000/(365*4) ~ 37.7. That seems a touch low for Sec. of state.
1. Clinton may have used more than one private server
2. Clinton may have sent emails from other servers to unofficial accounts of other state department employees
Lower bound for missing emails from Clinton:
Take a small weighted random sample (weighting seniority more) of top state department employees.
soodoku /
Last active August 29, 2015 14:17
Get Congressional Speech Data Via CapitolWords API
Gets Congressional speech text, arranged by speaker.
Produces a csv (capitolwords.csv) with the following columns:
Uses the Sunlight foundation library:
soodoku /
Last active August 29, 2015 14:20
Salvage Corrupted CSV
What does it do?
Goes through a corrupted csv sequentially and outputs rows that are clean.
Also outputs, total n, total corrupted n
@author: Gaurav Sood
Run: python input_csv output_csv
soodoku / prop_weights.R
Created May 31, 2015 22:52
Weighting datasets by propensity scores (~YouGov Method for Sampling)
View prop_weights.R
Weighting by Propensity Scores
Last Edited: 5/31/2015
Task Outline:
1. Two datasets:
dataset 1: large pop. representative sample
dataset 2: convenient sample
2. Create weights for dataset 2 so that its marginals are close to dataset 1 on some vars.
soodoku / server_installs
Last active August 30, 2015 23:40
Basic R related installs for Initializing Scrapers on Digital Ocean Ubuntu
View server_installs
apt-get upgrade
apt-get update
sudo aptitude install emacs24
sudo aptitude install r-base
sudo aptitude install libcurl4-openssl-dev
sudo aptitude install libxml2-dev
apt-get install openjdk-7-*
R CMD javareconf -e
soodoku /
Last active November 14, 2015 05:51
Basic sentiment analysis with AFINN or custom word database
Basic Sentiment Analysis
Builds on:
Utilizes AFINN or a custom sentiment db
Example Snippets at end from:
soodoku / cong.csv
Last active November 22, 2015 20:29
Educational Qualifications of Members of the 111th Congress
View cong.csv
Name District Education Science Law
Jeff Sessions (R) AL-Senate B.A., Huntingdon College; J.D. University of Alabama School of Law 1
Richard Shelby (R) AL-Senate B.A., University of Alabama; J.D. University of Alabama School of Law 1
Jo Bonner (R) AL-1 B.A. Journalism, University of Alabama 0
Bobby Bright (D) AL-2 B.A. Political Science, Auburn University; M.S. Criminal Justice, Troy State University; J.D. Thomas Goode Jones School of Law 1
Mike Rogers (R) AL-3 B.A., Political Science; M.P.A., Jackson State University; J.D. Birmingham School of Law 1
Robert Aderholt (R) AL-4 B.A., Political Science/History, Birmingham Southern College; J.D., Samford University 1
Partker Griffith (D) AL-5 B.S.; M.D., Louisiana State University 0
Spencer Bachus (R) AL-6 B.A., Auburn University; J.D., University of Alabama 1
Artur Davis (D) AL-7 B.A., Government, Harvard University; J.D., Harvard University School of Law 1
soodoku / state_abbrev_fips.txt
Created April 12, 2016 21:03
US State Abbreviations to FIPS crosswalk
View state_abbrev_fips.txt
1 AL
2 AK
4 AZ
5 AR
6 CA
8 CO
9 CT
10 DE
11 DC
12 FL
soodoku / state_various.csv
Last active April 12, 2016 22:31
US State name, 2 letter code, Alphabetical number, Census Region, ICPSR, ICPSR 2
View state_various.csv
state code num census icpsr icpsr2
Alabama AL 1 South 41 41 AL ALABAMA
Alaska AK 2 West 81 81 AK ALASKA
Arizona AZ 3 West 61 61 AZ ARIZONA
Arkansas AR 4 South 42 42 AR ARKANSAS
California CA 5 West 71 71 CA CALIFORNIA
Colorado CO 6 West 62 62 CO COLORADO
Connecticut CT 7 Northeast 1 01 CT CONNECTICUT
Delaware DE 8 South 11 11 DE DELAWARE
District of Columbia DC 9 Northeast 55 55 DC DISTRICT OF COLUMBIA