reginafcompton

The Django ORM

# models.py

from django.db import models

class Pasta(models.Model):
    name = models.CharField(max_length=200)
    price_per_pound = models.DecimalField(decimal_places=2)

reginafcompton

Get all actions sponsored by current Metro committees

select a.date, s.url from
opencivicdata_billaction as a
join opencivicdata_bill as b
on a.bill_id=b.id
join opencivicdata_billsource as s
on s.bill_id=b.id
where organization_id in ('ocd-organization/511270c2-f9d1-4c28-93ba-b28353454c39','ocd-organization/83c75785-fcc4-4716-8fa9-40a7385ee8be','ocd-organization/5647793a-2447-481b-8b7b-e4f63e08fbc7','ocd-organization/6d61370f-3919-41e7-b30c-d77e7614390e','ocd-organization/caa886f8-9af7-4806-a7f9-ebbc3e5b8887','ocd-organization/253d87a6-5f10-4908-9479-5ab9b197c6f1') and s.note='api' order by a.date desc;

reginafcompton

Dump and restore a database

Occasionally, you need to dump a remote database and restore it locally. Many strategies exist for achieveing this end. The following outines one path to succes.

Dump the database

Go to remote location (e.g., a server, your friend's computer), and run:

pg_dump -Fc -U postgres opencivicdata > /tmp/ocd.dump

reginafcompton

Dockerizing psql

Sometimes, it makes sense to isolate database dependencies when doing local development (e.g., you need to mimic a remote database). Docker saves the day.

Get postgres 9.6 with postgis

Run the following:

docker run --name postgres96 -it -d -v pg96_data:/var/lib/postgresql/data -p 37000:5432 mdillon/postgis:9.6

reginafcompton

Cutting a new pypi release: The case of django-councilmatic

Have you and your friends been making lots of changes to django-councilmatic? are you ready to cut a new release in pypi? Take these steps.

• Summarize your changes in RELEASENOTES.md.

• On a branch, visit setup.py and adjust the version of django-councilmatic, e.g. version='0.6.9' becomes version='0.6.10'

• Push changes to GitHub and merge.

reginafcompton

LIS Issue 17

Narrow query

This query filters the lots included in the 2016/17 expansion. The results are not particularly interesting, since most lots in this collection have a proportion of 1.

 nine | eight | seven | six  | five | four | three | two  | one
------+-------+-------+------+------+------+-------+------+------
 3910 |  3912 |  3912 | 3912 | 3912 | 3912 |  3912 | 3912 | 3914

reginafcompton

How to remove an entry from Councilmatic

Note: cronjobs can collide with manual data removal from the databases. Please turn off the cron before intervening with the database, or ensure that a cron job is not running at the time of deletion.

Remove the entry from the OCD API

Login to the OCD API server (i.e., ssh ubuntu@ocd.datamade.us), and go to the OCD app: api.opencivicdata.org. Shell into the database via Django (workon ocd-api + python manage.py shell), find the entry, and remove it:

# Example: remove a Bill in the python shell
from opencivicdata.legislative.models import Bill

reginafcompton

Scraping the Web

Get started

What is scraping?

Every webpage contains data. Sometimes, this data proves useful in a context other than the user interface of the webpage itself. Happily, with the right tools, we can extract data from any webpage. Indeed, we might think of a webpage as a “really bad” API (credit: Forest Gregg), with which we can interact and collect information.

At DataMade, we use the lxml library to access webpages. The lxml library parses and processes HTML. It’s well-documented, expansive, and popular.