Currently, the CFO makes fiscal impact statements (FIS) available at http://app.cfo.dc.gov/services/fiscal_impact/search.asp. But, bulk data. So, scrape.
Once pdfs were obtained, they were saved into txt and then reinserted into the json. Final result = searchable json.
| cd ~ | |
| sudo apt-get update | |
| sudo apt-get install openjdk-7-jre-headless -y | |
| ### Check http://www.elasticsearch.org/download/ for latest version of ElasticSearch and replace wget link below | |
| # NEW WAY / EASY WAY | |
| wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.3.0.deb | |
| sudo dpkg -i elasticsearch-1.3.0.deb |
| [ | |
| { | |
| "date": "Tuesday, Jan 7, 2014 at 1:06 p.m.", | |
| "url": "/shows/2014-01-07/al-qaidas-new-rise-middle-east", | |
| "summary": "Iraqi armed forces are battling militants to reclaim control of the city of Fallujah in Iraq's Anbar province. For the first time since U.S. forces defeated insurgents in 2006-2007, the region bordering war-torn Syria has become a hub for an al Qaida affiliate called the Islamic State of Iraq and Syria. Experts join Kojo to understand the rise of militancy in Iraq and its traces in neighboring countries like Syria.", | |
| "guests": [ | |
| { | |
| "credentials": "Vice president, Middle East Institute", | |
| "guest": "Paul Salem" | |
| }, |
| [ | |
| { | |
| "issues": [ | |
| { | |
| "url": "IssueHome.aspx?IssueId=263", | |
| "pdf_url": "../Notice/DownLoad.aspx?IssueFileID=12530", | |
| "number": "2009 Index", | |
| "volume": "56", | |
| "date": "2009-12-31", | |
| "id": "263" |
| # Initialize | |
| sudo apt-get update | |
| mkdir /home/user-data | |
| # Install Git | |
| sudo apt-get install -y git | |
| # Install Node | |
| sudo apt-get install -y make | |
| sudo apt-get -y install build-essential g++ libssl-dev pkg-config |
| #!/usr/bin/env python3 | |
| from urllib import request, parse | |
| import json | |
| base_url = "http://lims.dccouncil.us/_layouts/15/uploader/AdminProxy.aspx/" | |
| keyword_test = "" | |
| # Generic function to get Data from LIMS | |
| def getFromLIMS(view, payload): |
| Agency,Fund,FY15 Mayors Proposed Budget | |
| ADVISORY NEIGHBORHOOD COMMISSION,GENERAL FUND,300 | |
| ADVISORY NEIGHBORHOOD COMMISSION,GENERAL FUND,2000 | |
| ADVISORY NEIGHBORHOOD COMMISSION,GENERAL FUND,5000 | |
| ADVISORY NEIGHBORHOOD COMMISSION,GENERAL FUND,5400 | |
| ADVISORY NEIGHBORHOOD COMMISSION,GENERAL FUND,23893.72 | |
| ADVISORY NEIGHBORHOOD COMMISSION,GENERAL FUND,46102.28 | |
| ADVISORY NEIGHBORHOOD COMMISSION,GENERAL FUND,163628 | |
| ADVISORY NEIGHBORHOOD COMMISSION,GENERAL FUND,677688.01 | |
| ALCOHOLIC BEVERAGE REGULATION ADMIN.,GENERAL FUND,4500 |
| levels: | |
| - form: " Sec. $x." | |
| num: 1 | |
| - form: ($x) | |
| num: a | |
| - form: ($x) | |
| num: 1 |
| levels: | |
| - form: $x. | |
| num: I | |
| - form: $x. | |
| num: A | |
| - form: ($x) | |
| num: 1 |
| levels: | |
| - form: $x. | |
| num: I | |
| - form: $x. | |
| num: A | |
| - form: ($x) | |
| num: 1 |