# models.py
from django.db import models
class Pasta(models.Model):
name = models.CharField(max_length=200)
price_per_pound = models.DecimalField(decimal_places=2)
select a.date, s.url from
opencivicdata_billaction as a
join opencivicdata_bill as b
on a.bill_id=b.id
join opencivicdata_billsource as s
on s.bill_id=b.id
where organization_id in ('ocd-organization/511270c2-f9d1-4c28-93ba-b28353454c39','ocd-organization/83c75785-fcc4-4716-8fa9-40a7385ee8be','ocd-organization/5647793a-2447-481b-8b7b-e4f63e08fbc7','ocd-organization/6d61370f-3919-41e7-b30c-d77e7614390e','ocd-organization/caa886f8-9af7-4806-a7f9-ebbc3e5b8887','ocd-organization/253d87a6-5f10-4908-9479-5ab9b197c6f1') and s.note='api' order by a.date desc;
Occasionally, you need to dump a remote database and restore it locally. Many strategies exist for achieveing this end. The following outines one path to succes.
Go to remote location (e.g., a server, your friend's computer), and run:
pg_dump -Fc -U postgres opencivicdata > /tmp/ocd.dump
Sometimes, it makes sense to isolate database dependencies when doing local development (e.g., you need to mimic a remote database). Docker saves the day.
Run the following:
docker run --name postgres96 -it -d -v pg96_data:/var/lib/postgresql/data -p 37000:5432 mdillon/postgis:9.6
Have you and your friends been making lots of changes to django-councilmatic? are you ready to cut a new release in pypi? Take these steps.
-
Summarize your changes in
RELEASENOTES.md
. -
On a branch, visit
setup.py
and adjust the version of django-councilmatic, e.g. version='0.6.9' becomes version='0.6.10' -
Push changes to GitHub and merge.
This query filters the lots included in the 2016/17 expansion. The results are not particularly interesting, since most lots in this collection have a proportion of 1.
nine | eight | seven | six | five | four | three | two | one
------+-------+-------+------+------+------+-------+------+------
3910 | 3912 | 3912 | 3912 | 3912 | 3912 | 3912 | 3912 | 3914
Note: cronjobs can collide with manual data removal from the databases. Please turn off the cron before intervening with the database, or ensure that a cron job is not running at the time of deletion.
Remove the entry from the OCD API
Login to the OCD API server (i.e., ssh ubuntu@ocd.datamade.us
), and go to the OCD app: api.opencivicdata.org
. Shell into the database via Django (workon ocd-api
+ python manage.py shell
), find the entry, and remove it:
# Example: remove a Bill in the python shell
from opencivicdata.legislative.models import Bill
What is scraping?
Every webpage contains data. Sometimes, this data proves useful in a context other than the user interface of the webpage itself. Happily, with the right tools, we can extract data from any webpage. Indeed, we might think of a webpage as a “really bad” API (credit: Forest Gregg), with which we can interact and collect information.
At DataMade, we use the lxml library to access webpages. The lxml library parses and processes HTML. It’s well-documented, expansive, and popular.