Skip to content

Instantly share code, notes, and snippets.

@davidfauth

davidfauth/FBOPigLiveAnimalNLTK.pig

Created January 21, 2014 15:46
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save davidfauth/8542526 to your computer and use it in GitHub Desktop.
Save davidfauth/8542526 to your computer and use it in GitHub Desktop.
Download ZIP
Pig code for live animals award description
Raw
FBOPigLiveAnimalNLTK.pig
/* filter awards by NIH */
activeHasAward = FILTER active_data BY (noticeType == 'Award Notice' AND contractAwardAmount IS NOT NULL AND classCode == '88 -- Live animals');
fy0004HasAward = FILTER fy00_04_data BY (noticeType == 'Award Notice' AND contractAwardAmount IS NOT NULL AND classCode == '88 -- Live animals');
fy0507HasAward = FILTER fy05_07_data BY (noticeType == 'Award Notice' AND contractAwardAmount IS NOT NULL AND classCode == '88 -- Live animals');
fy0809HasAward = FILTER fy08_09_data BY (noticeType == 'Award Notice' AND contractAwardAmount IS NOT NULL AND classCode == '88 -- Live animals');
fy1011HasAward = FILTER fy10_11_data BY (noticeType == 'Award Notice' AND contractAwardAmount IS NOT NULL AND classCode == '88 -- Live animals');
fy1213HasAward = FILTER fy12_13_data BY (noticeType == 'Award Notice' AND contractAwardAmount IS NOT NULL AND classCode == '88 -- Live animals');
/*group Data */
allAwardData = UNION activeHasAward, fy0004HasAward, fy0507HasAward, fy0809HasAward, fy1011HasAward, fy1213HasAward;
allDescrData = UNION active_descr, fy0004descr, fy0507descr, fy0809descr, fy1011descr, fy1213descr;
/* join the data */
joinedLADetails = JOIN allDescrData BY awardID, allAwardData BY noticeID;
/*remove HTML tagging from description */
B = FOREACH joinedLADetails GENERATE awardID, agencyName, awardee, contractAwardAmount, nltk_udfs.stripHTML(description) AS fboDescr;
liveAnimalResults = FOREACH B GENERATE
awardID, agencyName, awardee, contractAwardAmount,nltk_udfs.top5_bigrams(fboDescr), fboDescr;
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment