Skip to content

Instantly share code, notes, and snippets.

🤠

Ewan Oglethorpe eoglethorpe

🤠
  • In the general biosphere
View GitHub Profile
View Classify keywords by sector.md

Creating a system for identifying keywords in textual humanitarian data

This is a brief crash course for creating a system to categorize key words in sets of data with known classifications of segments of text. Although it is written in a humanitarian context, it can be flexibly used elsewhere.

Overview: What's in the data?

A sample dataset could include the following format where it is normalized:

View Classify keywords by sector.md

Creating a system for identifying keywords in textual humanitarian data

This is a brief crash course for creating a system to categorize key words in sets of data with known classifications of segments of text. Although it is written in a humanitarian context, it can be flexibly used elsewhere.

Overview: What's in the data?

A sample dataset could include the following format where it is normalized:

View Classify keywords by sector
Creating a system for identifying keywords in textual humanitarian data
===================================================
This is a brief crash course for creating a system to categorize key words in sets of data with known classifications of segments of text. Although it is written in a humanitarian context, it can be flexibly used elsewhere.
Overview: What's in the data?
-------------------
A sample dataset could include the following format where it is normalized:
View 538_results_mar_10.json
[{
"date": "Feb. 1",
"state": "Iowa",
"del_won_hill": "23",
"num_del_tot": "44",
"del_won_bern": "21"
}, {
"date": "Feb. 9",
"state": "New Hampshire",
"del_won_hill": "9",
View parse_538_results.py
__author__ = 'eoglethorpe'
"""take election info from
http://projects.fivethirtyeight.com/election-2016/delegate-targets/democrats/
and convert it to JSON
"""
import json
import requests
View winter.sql
--add winterisation data
drop table if exists winterisation_db;
CREATE TABLE winterisation_db
("dist_code" varchar(11), "vdc_code" varchar(17), "district" varchar(15), "vdc" varchar(32), "rep_ag" varchar(26), "imp_ag" varchar(21), "target_hh" int, "type" varchar(61), "full" varchar(3), "comment" varchar(362), "act_state" varchar(9), "edit_dt" timestamp, "comments" varchar(119))
;
INSERT INTO winterisation_db
("dist_code", "vdc_code", "district", "vdc", "rep_ag", "imp_ag", "target_hh", "type", "full", "comment", "act_state", "edit_dt", "comments")
View clean.sql
--planned and neither date > delete (74)
delete from distributions
where act_desc in ('Tents', 'Tarpaulin','tarpaulin','Cash for Shelter (15,000 NPR)', 'CGI Bundle (72ft/bundle)')
and
act_status like 'Planned (funded)' and (start_dt is null and comp_dt is null)
--no status > delete (52)
delete from distributions
where act_desc in ('Tents', 'Tarpaulin','tarpaulin','Cash for Shelter (15,000 NPR)', 'CGI Bundle (72ft/bundle)')
and
View gist:558693203df79607554a
# Root logger option
log4j.rootLogger=INFO, syslog, console
#console
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console=org.apache.log4j.ConsoleAppender
#papertrail
log4j.appender.syslog=org.apache.log4j.net.SyslogAppender
log4j.appender.syslog.Facility=LOCAL7
View gist:8d8391fb1c27c01551b2
Properties props = new Properties();
Logger log4jLogger = Logger.getLogger(S3Sink.class);
props.load(getClass().getResourceAsStream("/log4j.properties"));
PropertyConfigurator.configure(props);
log4jLogger.info("log content")
You can’t perform that action at this time.