Skip to content

Instantly share code, notes, and snippets.

@psychemedia

psychemedia/test.py

Last active April 9, 2022 07:39
Show Gist options
  • Save psychemedia/15be9d3101b6923b506e to your computer and use it in GitHub Desktop.
Save psychemedia/15be9d3101b6923b506e to your computer and use it in GitHub Desktop.
Download ZIP
UK Parliament api test and members data platform OData service test
Raw
test.py
import json, re
import urllib2
from urlparse import urlparse
from urllib import urlopen, urlencode
class UKParliamentReader():
"""
Chat to the UK Parliament API
"""
def __init__(self):
""" Need to think more about the structure of this... """
pass
def qpatch(self,query):
t=[]
for a in query.split('or'):
t.append('({})'.format(a.strip().replace(' ',' AND ')))
return ' OR '.join(t)
def search_one(self,query, typ='Research Papers',page=0,ps=100):
url='http://lda.data.parliament.uk/researchbriefings.json'
urlargs={'_view':typ,'_pageSize':ps,'_search':self.qpatch(query),'_page':page}
url='{}?{}'.format(url, urlencode(urlargs))
data =json.loads(urlopen(url).read())
response=[]
for i in data['result']['items']:
response.append("{} [http://researchbriefings.parliament.uk/ResearchBriefing/Summary/{}']".format( i['title'],i['identifier']['_value']))
return response
def search_all(self,query, typ='Research Papers',ps=100):
return
def responder(self,hook):
ukparl_token=hook['env']['ukparl_token']
ukparl_url=hook['env']['ukparl_url']
r=self.search_one(Hook['params']['text'])
r2="; \n".join(r)
payload={"channel": "#slashtest", "username": "parlibot",
"text":"I know about the following Parliamentary research papers:\n\n {}".format(r2)}
req = urllib2.Request(ukparl_url)
req.add_header('Content-Type', 'application/json')
response = urllib2.urlopen(req, json.dumps(payload))
u=UKParliamentReader()
if Hook['params']['token'] == Hook['env']['ukparl_token']:
u.responder(Hook)
Display the source blob
Display the rendered blob
Raw
ukparliament_members_odata.ipynb
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# UK Parliament Members Data Platform OData Feed\n",
"\n",
"As well as offering an [XML/JSON Search API](http://data.parliament.uk/membersdataplatform/memberquery.aspx), the *UK Parliament Members' Names Data Platform* also provides an [OData service](http://data.parliament.uk/membersdataplatform/open.aspx) from the OData endpoint http://data.parliament.uk/membersdataplatform/open/OData.svc.\n",
"\n",
"Python support for OData seems limited, although a wrapper is provided as part of the *Python package for Standards in Learning Education and Training, [pyslet](http://pyslet.readthedocs.org/en/latest/odatav2_consumer.html). \n",
"\n",
"So here's a quick exploration of the package and the UK Parliament OData service.\n",
"\n",
"Note that I'm completely new to OData, so there may be both better ways of using it, and more that can be done!"
]
},
{
"cell_type": "code",
"execution_count": 182,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"#First, we need to install the pyslet package if it isn't already installed\n",
"#!pip install pyslet"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Getting Started\n",
"\n",
"The first thing we need to do is generate a client for the OData service, which is rooted on the endpoint http://data.parliament.uk/membersdataplatform/open/OData.svc."
]
},
{
"cell_type": "code",
"execution_count": 183,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"from pyslet.odata2.client import Client\n",
"c = Client(\"http://data.parliament.uk/membersdataplatform/open/OData.svc\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The most useful way of probing the endpoint seems to be to request the available feeds. All the feeds seem to correspond to resources of type `EntitySet`, which I assume to be akin to tables..."
]
},
{
"cell_type": "code",
"execution_count": 184,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"{u'AddressTypes': <pyslet.odata2.metadata.EntitySet at 0x109984090>,\n",
" u'AnsweringBodies': <pyslet.odata2.metadata.EntitySet at 0x109994810>,\n",
" u'AreaTypes': <pyslet.odata2.metadata.EntitySet at 0x109984110>,\n",
" u'Areas': <pyslet.odata2.metadata.EntitySet at 0x1099840d0>,\n",
" u'BiographyCategories': <pyslet.odata2.metadata.EntitySet at 0x109984190>,\n",
" u'Cabinets': <pyslet.odata2.metadata.EntitySet at 0x109984210>,\n",
" u'CommitteeClerks': <pyslet.odata2.metadata.EntitySet at 0x109984290>,\n",
" u'CommitteeLayMembers': <pyslet.odata2.metadata.EntitySet at 0x109994950>,\n",
" u'CommitteeTypes': <pyslet.odata2.metadata.EntitySet at 0x109984390>,\n",
" u'Committees': <pyslet.odata2.metadata.EntitySet at 0x109984310>,\n",
" u'CommonsRegisterPublications': <pyslet.odata2.metadata.EntitySet at 0x109994850>,\n",
" u'Constituencies': <pyslet.odata2.metadata.EntitySet at 0x109984410>,\n",
" u'ConstituencyAreas': <pyslet.odata2.metadata.EntitySet at 0x109984490>,\n",
" u'ConstituencyCandidates': <pyslet.odata2.metadata.EntitySet at 0x1099946d0>,\n",
" u'ConstituencyResults': <pyslet.odata2.metadata.EntitySet at 0x109994790>,\n",
" u'ConstituencyTypes': <pyslet.odata2.metadata.EntitySet at 0x109984510>,\n",
" u'Countries': <pyslet.odata2.metadata.EntitySet at 0x109984590>,\n",
" u'Departments': <pyslet.odata2.metadata.EntitySet at 0x109984610>,\n",
" u'DisqualificationTypes': <pyslet.odata2.metadata.EntitySet at 0x109984710>,\n",
" u'ElectionTypes': <pyslet.odata2.metadata.EntitySet at 0x1099847d0>,\n",
" u'Elections': <pyslet.odata2.metadata.EntitySet at 0x109984790>,\n",
" u'EndReasons': <pyslet.odata2.metadata.EntitySet at 0x109984850>,\n",
" u'EntryTypes': <pyslet.odata2.metadata.EntitySet at 0x1099848d0>,\n",
" u'ExperienceTypes': <pyslet.odata2.metadata.EntitySet at 0x109984950>,\n",
" u'GovernmentPostDepartments': <pyslet.odata2.metadata.EntitySet at 0x1099849d0>,\n",
" u'GovernmentPosts': <pyslet.odata2.metadata.EntitySet at 0x109984a50>,\n",
" u'GovernmentRanks': <pyslet.odata2.metadata.EntitySet at 0x109984ad0>,\n",
" u'HonourLists': <pyslet.odata2.metadata.EntitySet at 0x109984bd0>,\n",
" u'HonourTypes': <pyslet.odata2.metadata.EntitySet at 0x109984cd0>,\n",
" u'HonouraryPrefixes': <pyslet.odata2.metadata.EntitySet at 0x109984b50>,\n",
" u'Honours': <pyslet.odata2.metadata.EntitySet at 0x109984c50>,\n",
" u'Houses': <pyslet.odata2.metadata.EntitySet at 0x109984d50>,\n",
" u'InterestCategories': <pyslet.odata2.metadata.EntitySet at 0x109984dd0>,\n",
" u'LordsMembershipTypes': <pyslet.odata2.metadata.EntitySet at 0x109984ed0>,\n",
" u'LordsRanks': <pyslet.odata2.metadata.EntitySet at 0x109984f50>,\n",
" u'MemberAddresses': <pyslet.odata2.metadata.EntitySet at 0x109984f90>,\n",
" u'MemberBiographies': <pyslet.odata2.metadata.EntitySet at 0x1099ad050>,\n",
" u'MemberCommitteeChairs': <pyslet.odata2.metadata.EntitySet at 0x1099ad150>,\n",
" u'MemberCommittees': <pyslet.odata2.metadata.EntitySet at 0x1099ad1d0>,\n",
" u'MemberConstituencies': <pyslet.odata2.metadata.EntitySet at 0x1099ad290>,\n",
" u'MemberDisqualifications': <pyslet.odata2.metadata.EntitySet at 0x1099ad310>,\n",
" u'MemberElectionsContesteds': <pyslet.odata2.metadata.EntitySet at 0x1099ad350>,\n",
" u'MemberExperiences': <pyslet.odata2.metadata.EntitySet at 0x1099ad3d0>,\n",
" u'MemberGovernmentPosts': <pyslet.odata2.metadata.EntitySet at 0x1099ad4d0>,\n",
" u'MemberHonours': <pyslet.odata2.metadata.EntitySet at 0x1099ad550>,\n",
" u'MemberHouseMemberships': <pyslet.odata2.metadata.EntitySet at 0x1099ad610>,\n",
" u'MemberKnownAs': <pyslet.odata2.metadata.EntitySet at 0x109994550>,\n",
" u'MemberLeaveOfAbsences': <pyslet.odata2.metadata.EntitySet at 0x1099ad6d0>,\n",
" u'MemberLordsMembershipTypes': <pyslet.odata2.metadata.EntitySet at 0x1099ad750>,\n",
" u'MemberLordsPreferredNames': <pyslet.odata2.metadata.EntitySet at 0x1099ad790>,\n",
" u'MemberMaidenSpeeches': <pyslet.odata2.metadata.EntitySet at 0x1099ad890>,\n",
" u'MemberOaths': <pyslet.odata2.metadata.EntitySet at 0x1099ad910>,\n",
" u'MemberObituaries': <pyslet.odata2.metadata.EntitySet at 0x1099ad950>,\n",
" u'MemberOppositionPosts': <pyslet.odata2.metadata.EntitySet at 0x1099ada50>,\n",
" u'MemberOtherParliaments': <pyslet.odata2.metadata.EntitySet at 0x1099adb10>,\n",
" u'MemberParliamentaryPosts': <pyslet.odata2.metadata.EntitySet at 0x1099adb90>,\n",
" u'MemberParties': <pyslet.odata2.metadata.EntitySet at 0x1099adbd0>,\n",
" u'MemberPreferredNameHonouraryPrefixes': <pyslet.odata2.metadata.EntitySet at 0x1099adc50>,\n",
" u'MemberPreferredNames': <pyslet.odata2.metadata.EntitySet at 0x1099add50>,\n",
" u'MemberStaffs': <pyslet.odata2.metadata.EntitySet at 0x1099ade10>,\n",
" u'MemberStatuses': <pyslet.odata2.metadata.EntitySet at 0x1099ade90>,\n",
" u'MemberSuspensions': <pyslet.odata2.metadata.EntitySet at 0x1099adf10>,\n",
" u'Members': <pyslet.odata2.metadata.EntitySet at 0x1099addd0>,\n",
" u'OppositionPostDepartments': <pyslet.odata2.metadata.EntitySet at 0x1099adf90>,\n",
" u'OppositionPosts': <pyslet.odata2.metadata.EntitySet at 0x109994050>,\n",
" u'OppositionRanks': <pyslet.odata2.metadata.EntitySet at 0x1099945d0>,\n",
" u'OtherParliaments': <pyslet.odata2.metadata.EntitySet at 0x1099940d0>,\n",
" u'ParliamentTypes': <pyslet.odata2.metadata.EntitySet at 0x109994250>,\n",
" u'ParliamentaryPosts': <pyslet.odata2.metadata.EntitySet at 0x109994150>,\n",
" u'ParliamentaryRanks': <pyslet.odata2.metadata.EntitySet at 0x1099941d0>,\n",
" u'Parties': <pyslet.odata2.metadata.EntitySet at 0x1099942d0>,\n",
" u'PartySubTypes': <pyslet.odata2.metadata.EntitySet at 0x109994350>,\n",
" u'PhotoOutputs': <pyslet.odata2.metadata.EntitySet at 0x1099943d0>,\n",
" u'Statuses': <pyslet.odata2.metadata.EntitySet at 0x109994450>,\n",
" u'Titles': <pyslet.odata2.metadata.EntitySet at 0x1099944d0>}"
]
},
"execution_count": 184,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"c.feeds"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Looking Inside an Entity Set\n",
"\n",
"An *entity set* resembles a relational database table and can be accessed using the `.OpenCollection()` method. The `.OpenCollection()` method brokers a connection the OData database and as such should be closed using the `.close()` method when finished with. \n",
"\n",
"Let's start by exploring the `Committees` feed."
]
},
{
"cell_type": "code",
"execution_count": 185,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"cc=c.feeds['Committees'].OpenCollection()\n",
"#Close the connection with: cc.close()\n",
"#If used within a with block, the connection will be closed automatically on leaving the block"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The collection returns a dictionary of entities. In the case of the Parliament API, these are numerically keyed. Each entity also takes the form of dictionary; so one way of exploring the data model is to look at the keys of one of the `Committee` entities..."
]
},
{
"cell_type": "code",
"execution_count": 186,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[u'Committee_Id',\n",
" u'CommitteeType_Id',\n",
" u'Name',\n",
" u'ParentCommittee_Id',\n",
" u'DateLordsAppointed',\n",
" u'DateCommonsAppointed',\n",
" u'Phone',\n",
" u'Url',\n",
" u'StartDate',\n",
" u'EndDate',\n",
" u'CreatedFromCommittee_Id',\n",
" u'IsCommons',\n",
" u'IsLords',\n",
" u'CommitteeClerks',\n",
" u'Committees1',\n",
" u'Committee1',\n",
" u'Committees11',\n",
" u'Committee2',\n",
" u'CommitteeType',\n",
" u'MemberCommittees',\n",
" u'CommitteeLayMembers']"
]
},
"execution_count": 186,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cc[1].keys()"
]
},
{
"cell_type": "code",
"execution_count": 187,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(u'Committee_Id', <pyslet.odata2.csdl.Int32Value object at 0x10b034250>)\n",
"(u'CommitteeType_Id', <pyslet.odata2.csdl.Int32Value object at 0x10b75d710>)\n",
"(u'Name', <pyslet.odata2.csdl.StringValue object at 0x10b75d150>)\n",
"(u'ParentCommittee_Id', <pyslet.odata2.csdl.Int32Value object at 0x10b034910>)\n",
"(u'DateLordsAppointed', <pyslet.odata2.csdl.DateTimeValue object at 0x10b03e050>)\n",
"(u'DateCommonsAppointed', <pyslet.odata2.csdl.DateTimeValue object at 0x10aadd410>)\n",
"(u'Phone', <pyslet.odata2.csdl.StringValue object at 0x10b7744d0>)\n",
"(u'Url', <pyslet.odata2.csdl.StringValue object at 0x10aaddc90>)\n",
"(u'StartDate', <pyslet.odata2.csdl.DateTimeValue object at 0x10b03e110>)\n",
"(u'EndDate', <pyslet.odata2.csdl.DateTimeValue object at 0x10b03e1d0>)\n",
"(u'CreatedFromCommittee_Id', <pyslet.odata2.csdl.Int32Value object at 0x10aadd650>)\n",
"(u'IsCommons', <pyslet.odata2.csdl.BooleanValue object at 0x109b43810>)\n",
"(u'IsLords', <pyslet.odata2.csdl.BooleanValue object at 0x10b76fbd0>)\n",
"(u'CommitteeClerks', <pyslet.odata2.csdl.DeferredValue object at 0x10b751a50>)\n",
"(u'Committees1', <pyslet.odata2.csdl.DeferredValue object at 0x10b751d90>)\n",
"(u'Committee1', <pyslet.odata2.csdl.DeferredValue object at 0x10b76fe90>)\n",
"(u'Committees11', <pyslet.odata2.csdl.DeferredValue object at 0x109b62b50>)\n",
"(u'Committee2', <pyslet.odata2.csdl.DeferredValue object at 0x109b4fd90>)\n",
"(u'CommitteeType', <pyslet.odata2.csdl.DeferredValue object at 0x10aadd390>)\n",
"(u'MemberCommittees', <pyslet.odata2.csdl.DeferredValue object at 0x10aaddc90>)\n",
"(u'CommitteeLayMembers', <pyslet.odata2.csdl.DeferredValue object at 0x109b4ff10>)\n"
]
}
],
"source": [
"for k in cc[1].keys(): print(k,cc[1][k])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can inspect the keys and values of a particular entity record:"
]
},
{
"cell_type": "code",
"execution_count": 188,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(u'Committee_Id', <pyslet.odata2.csdl.Int32Value object at 0x109b6dc50>, 1)\n",
"(u'CommitteeType_Id', <pyslet.odata2.csdl.Int32Value object at 0x10a5e9f50>, 3)\n",
"(u'Name', <pyslet.odata2.csdl.StringValue object at 0x1035ef910>, u'Accomodation & Works')\n",
"(u'ParentCommittee_Id', <pyslet.odata2.csdl.Int32Value object at 0x1035ef950>, None)\n",
"(u'DateLordsAppointed', <pyslet.odata2.csdl.DateTimeValue object at 0x1035ef990>, None)\n",
"(u'DateCommonsAppointed', <pyslet.odata2.csdl.DateTimeValue object at 0x1035ef9d0>, None)\n",
"(u'Phone', <pyslet.odata2.csdl.StringValue object at 0x1035efa10>, u'')\n",
"(u'Url', <pyslet.odata2.csdl.StringValue object at 0x1035efa50>, u'')\n",
"(u'StartDate', <pyslet.odata2.csdl.DateTimeValue object at 0x1035efa90>, TimePoint(date=Date(century=19, year=0, month=1, day=1),time=Time(hour=0, minute=0, second=0, zdirection=None, zhour=None, zminute=None)))\n",
"(u'EndDate', <pyslet.odata2.csdl.DateTimeValue object at 0x1035efad0>, TimePoint(date=Date(century=20, year=10, month=5, day=6),time=Time(hour=0, minute=0, second=0, zdirection=None, zhour=None, zminute=None)))\n",
"(u'CreatedFromCommittee_Id', <pyslet.odata2.csdl.Int32Value object at 0x1035efb10>, None)\n",
"(u'IsCommons', <pyslet.odata2.csdl.BooleanValue object at 0x1035efb50>, False)\n",
"(u'IsLords', <pyslet.odata2.csdl.BooleanValue object at 0x1035efb90>, True)\n"
]
}
],
"source": [
"for k, v in cc[1].data_items():\n",
" print(k, v, v.value)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"OData uses HTTP URLs to make data requests. We can view the URL for a requested resource using the `get_location()` method."
]
},
{
"cell_type": "code",
"execution_count": 189,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"http://data.parliament.uk/MembersDataPlatform/open/OData.svc/Committees(1)\n"
]
}
],
"source": [
"print(cc[1].get_location())"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can also preview various components of the URL, such as the `host` or `abs_path`:"
]
},
{
"cell_type": "code",
"execution_count": 190,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"u'/MembersDataPlatform/open/OData.svc/Committees(1)'"
]
},
"execution_count": 190,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"cc[1].get_location().abs_path"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We can retrieve the records in turn and display the values of the various record elements."
]
},
{
"cell_type": "code",
"execution_count": 191,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Accomodation & Works\n",
"Administration Committee\n",
"Speaker's Advisory Committee on Works of Art\n",
"Agriculture\n",
"Armed Forces Bill Committee\n",
"Broadcasting\n",
"Business and Enterprise Committee\n",
"Business, Enterprise and Regulatory Reform\n",
"Business, Innovation and Skills Committee\n",
"Catering\n"
]
}
],
"source": [
"ii=1\n",
"for i in cc:\n",
" print(cc[i]['Name'].value)\n",
" if ii==10: break\n",
" else: ii=ii+1"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Records are returned as paged results. We can limit the records returns using the `set_page()` method:"
]
},
{
"cell_type": "code",
"execution_count": 192,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"3 [u'Committee_Id', u'CommitteeType_Id', u'Name', u'ParentCommittee_Id', u'DateLordsAppointed', u'DateCommonsAppointed', u'Phone', u'Url', u'StartDate', u'EndDate', u'CreatedFromCommittee_Id', u'IsCommons', u'IsLords', u'CommitteeClerks', u'Committees1', u'Committee1', u'Committees11', u'Committee2', u'CommitteeType', u'MemberCommittees', u'CommitteeLayMembers'] Speaker's Advisory Committee on Works of Art\n",
"4 [u'Committee_Id', u'CommitteeType_Id', u'Name', u'ParentCommittee_Id', u'DateLordsAppointed', u'DateCommonsAppointed', u'Phone', u'Url', u'StartDate', u'EndDate', u'CreatedFromCommittee_Id', u'IsCommons', u'IsLords', u'CommitteeClerks', u'Committees1', u'Committee1', u'Committees11', u'Committee2', u'CommitteeType', u'MemberCommittees', u'CommitteeLayMembers'] Agriculture\n",
"5 [u'Committee_Id', u'CommitteeType_Id', u'Name', u'ParentCommittee_Id', u'DateLordsAppointed', u'DateCommonsAppointed', u'Phone', u'Url', u'StartDate', u'EndDate', u'CreatedFromCommittee_Id', u'IsCommons', u'IsLords', u'CommitteeClerks', u'Committees1', u'Committee1', u'Committees11', u'Committee2', u'CommitteeType', u'MemberCommittees', u'CommitteeLayMembers'] Armed Forces Bill Committee\n",
"6 [u'Committee_Id', u'CommitteeType_Id', u'Name', u'ParentCommittee_Id', u'DateLordsAppointed', u'DateCommonsAppointed', u'Phone', u'Url', u'StartDate', u'EndDate', u'CreatedFromCommittee_Id', u'IsCommons', u'IsLords', u'CommitteeClerks', u'Committees1', u'Committee1', u'Committees11', u'Committee2', u'CommitteeType', u'MemberCommittees', u'CommitteeLayMembers'] Broadcasting\n"
]
}
],
"source": [
"NUM_ITEMS=4\n",
"START_INDEX=2\n",
"\n",
"cc.set_page(NUM_ITEMS,START_INDEX)\n",
"for p in cc.iterpage(True): print p.key(),p.keys(), p['Name'].value"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Filtering Results\n",
"\n",
"When making a query, we often want to run a search over the data. The `set_filter()` method can be used to define search queries.\n",
"\n",
"*See __Section 4.5: Filter System Query Options__ of the [OData URI Conventions (OData Version 2.0) sprecification](http://www.odata.org/documentation/odata-version-2-0/uri-conventions/) for additional search constructs.*"
]
},
{
"cell_type": "code",
"execution_count": 193,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(193, [u'Committee_Id', u'CommitteeType_Id', u'Name', u'ParentCommittee_Id', u'DateLordsAppointed', u'DateCommonsAppointed', u'Phone', u'Url', u'StartDate', u'EndDate', u'CreatedFromCommittee_Id', u'IsCommons', u'IsLords', u'CommitteeClerks', u'Committees1', u'Committee1', u'Committees11', u'Committee2', u'CommitteeType', u'MemberCommittees', u'CommitteeLayMembers'], u'Science and Technology Committee (Lords)', True)\n",
"(194, [u'Committee_Id', u'CommitteeType_Id', u'Name', u'ParentCommittee_Id', u'DateLordsAppointed', u'DateCommonsAppointed', u'Phone', u'Url', u'StartDate', u'EndDate', u'CreatedFromCommittee_Id', u'IsCommons', u'IsLords', u'CommitteeClerks', u'Committees1', u'Committee1', u'Committees11', u'Committee2', u'CommitteeType', u'MemberCommittees', u'CommitteeLayMembers'], u'Science and Technology: Sub-Committee I ', True)\n",
"(253, [u'Committee_Id', u'CommitteeType_Id', u'Name', u'ParentCommittee_Id', u'DateLordsAppointed', u'DateCommonsAppointed', u'Phone', u'Url', u'StartDate', u'EndDate', u'CreatedFromCommittee_Id', u'IsCommons', u'IsLords', u'CommitteeClerks', u'Committees1', u'Committee1', u'Committees11', u'Committee2', u'CommitteeType', u'MemberCommittees', u'CommitteeLayMembers'], u'Science and Technology: Sub-Committee I (Higher education in STEM subjects)', True)\n",
"(247, [u'Committee_Id', u'CommitteeType_Id', u'Name', u'ParentCommittee_Id', u'DateLordsAppointed', u'DateCommonsAppointed', u'Phone', u'Url', u'StartDate', u'EndDate', u'CreatedFromCommittee_Id', u'IsCommons', u'IsLords', u'CommitteeClerks', u'Committees1', u'Committee1', u'Committees11', u'Committee2', u'CommitteeType', u'MemberCommittees', u'CommitteeLayMembers'], u'Select Committee on Science and Technology Sub-Committee 2', True)\n"
]
}
],
"source": [
"import pyslet.odata2.core as core\n",
"\n",
"#Construct the search string as a filter\n",
"#Note that we can search over several fields using Boolean operators.\n",
"#For example, search for \"Science\" as part of the committee name and further limit to Lords' committees\n",
"odfilter = core.CommonExpression.from_str(\"substringof('Science',Name) and IsLords eq true\")\n",
"cc.set_filter(odfilter)\n",
"for p in cc.itervalues():\n",
" print(p.key(), p.keys(),p['Name'].value,p['IsLords'].value)\n",
"cc.set_filter(None)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Ordering Search Results\n",
"\n",
"The *order* of the results can be controlled using the `orderby()` method."
]
},
{
"cell_type": "code",
"execution_count": 194,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(341, u'Palace of Westminster (Joint Committee)')\n",
"(242, u'Committee on the City of Westminster')\n"
]
}
],
"source": [
"#Set the ordering: asc or desc within a particular field (?or one or more comma separated fields?)\n",
"ordering=core.CommonExpression.OrderByFromString(\"Name desc\")\n",
"cc.set_orderby(ordering)\n",
"\n",
"#Add in a filter\n",
"odfilter = core.CommonExpression.from_str(\"substringof('Westminster',Name)\")\n",
"cc.set_filter(odfilter)\n",
"\n",
"#View the response\n",
"for p in cc.itervalues():\n",
" print(p.key(), p['Name'].value)"
]
},
{
"cell_type": "code",
"execution_count": 195,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"#Reset the filter and order limits\n",
"cc.set_filter(None)\n",
"cc.set_orderby(None)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### \"Projecting\" Search Results - Select\n",
"\n",
"In relational databases, a *projection* limits the columns that are returned from a query. Use the `select` field of the `set_expand()` method to limit the attribute data that is retrieved. The formulation is similar to the approach used in MongoDB, albeit the logic seems warped! To retrieve all data elements, set the `select` to `None`; to retrieve particular elements, set the `select` to a dictionary whose keys identify the fields you want to retrieve data for. Note that to maintain the integrity of the response dict, all keys are present, but value for the unselected elements are set to `None`. (If a selected element returns `None`, you presumably know it's value is `null` in the database?)"
]
},
{
"cell_type": "code",
"execution_count": 196,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"#Open a new collection - Members\n",
"m=c.feeds['Members'].OpenCollection()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's start by seeing the values for all value bearing fields:"
]
},
{
"cell_type": "code",
"execution_count": 197,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(u'Member_Id', 172)\n",
"(u'Forename', u'Diane')\n",
"(u'MiddleNames', u'Julie')\n",
"(u'Surname', u'Abbott')\n",
"(u'DateOfBirth', TimePoint(date=Date(century=19, year=53, month=9, day=27),time=Time(hour=0, minute=0, second=0, zdirection=None, zhour=None, zminute=None)))\n",
"(u'DateOfRetirement', None)\n",
"(u'DateOfDeath', None)\n",
"(u'TownOfBirth', u'London')\n",
"(u'BirthCountry_Id', 3)\n",
"(u'Gender', u'F')\n",
"(u'Notes', u'')\n",
"(u'DateOfAnnouncement', None)\n",
"(u'DateOfIntroduction', None)\n",
"(u'DateOfWrit', None)\n",
"(u'IsWritAm', False)\n",
"(u'NameDisplayAs', u'Ms Diane Abbott')\n",
"(u'NameListAs', u'Abbott, Ms Diane')\n",
"(u'NameFullTitle', u'Ms Diane Abbott MP')\n",
"(u'House', u'Commons')\n",
"(u'EndDate', None)\n",
"(u'Party', u'Labour')\n",
"(u'PartyAbbrev', u'Lab')\n",
"(u'PartySubType', None)\n",
"(u'MembershipFrom', u'Hackney North and Stoke Newington')\n",
"(u'CurrentStatus', u'Current Member')\n",
"(u'StartDate', TimePoint(date=Date(century=19, year=87, month=6, day=11),time=Time(hour=0, minute=0, second=0, zdirection=None, zhour=None, zminute=None)))\n",
"(u'Dods_Id', 25790)\n",
"(u'Pims_Id', 3572)\n",
"(u'Party_Id', 15)\n",
"(u'CurrentStatus_Id', 0)\n",
"(u'MembershipFrom_Id', 3506)\n",
"(u'Name_Id', 181)\n",
"(u'CurrentStatusDate', TimePoint(date=Date(century=20, year=15, month=5, day=7),time=Time(hour=0, minute=0, second=0, zdirection=None, zhour=None, zminute=None)))\n",
"(u'CurrentStatusReason', None)\n",
"(u'CurrentStatusActive', True)\n",
"(u'NameSortAs', u'Abbott')\n",
"(u'NameSortAs1', u'Diane')\n",
"(u'LayingMinisterName', None)\n"
]
}
],
"source": [
"m.set_expand(None, None)\n",
"for k, v in m[172].data_items():\n",
" print(k,v.value)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Limit the data retrieval to just the specified selection of fields:"
]
},
{
"cell_type": "code",
"execution_count": 198,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(u'Member_Id', 172)\n",
"(u'Forename', u'Diane')\n",
"(u'Surname', u'Abbott')\n"
]
}
],
"source": [
"#Get the Forename and Surname data\n",
"m.set_expand(None, {'Forename':None,'Surname':None})\n",
"for k, v in m[172].data_items():\n",
" if v.value is not None: print(k,v.value)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Note that there are also elements that are navigation objects into other collections, as well well as atomic value elements."
]
},
{
"cell_type": "code",
"execution_count": 199,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(u'Member_Id', <pyslet.odata2.csdl.Int32Value object at 0x10c229f10>)\n",
"(u'Forename', <pyslet.odata2.csdl.StringValue object at 0x10c2173d0>)\n",
"(u'MiddleNames', <pyslet.odata2.csdl.StringValue object at 0x10c1e4190>)\n",
"(u'Surname', <pyslet.odata2.csdl.StringValue object at 0x10c1e4950>)\n",
"(u'DateOfBirth', <pyslet.odata2.csdl.DateTimeValue object at 0x10c20e510>)\n",
"(u'DateOfRetirement', <pyslet.odata2.csdl.DateTimeValue object at 0x10c2172d0>)\n",
"(u'DateOfDeath', <pyslet.odata2.csdl.DateTimeValue object at 0x10c1c8a90>)\n",
"(u'TownOfBirth', <pyslet.odata2.csdl.StringValue object at 0x10c229950>)\n",
"(u'BirthCountry_Id', <pyslet.odata2.csdl.Int32Value object at 0x10c1e4d50>)\n",
"(u'Gender', <pyslet.odata2.csdl.StringValue object at 0x10c1e4190>)\n",
"(u'Notes', <pyslet.odata2.csdl.StringValue object at 0x10c20e410>)\n",
"(u'DateOfAnnouncement', <pyslet.odata2.csdl.DateTimeValue object at 0x10c217110>)\n",
"(u'DateOfIntroduction', <pyslet.odata2.csdl.DateTimeValue object at 0x10c217d90>)\n",
"(u'DateOfWrit', <pyslet.odata2.csdl.DateTimeValue object at 0x10c229510>)\n",
"(u'IsWritAm', <pyslet.odata2.csdl.BooleanValue object at 0x10c1e4ad0>)\n",
"(u'NameDisplayAs', <pyslet.odata2.csdl.StringValue object at 0x10c1e4a10>)\n",
"(u'NameListAs', <pyslet.odata2.csdl.StringValue object at 0x10c20e250>)\n",
"(u'NameFullTitle', <pyslet.odata2.csdl.StringValue object at 0x10c217dd0>)\n",
"(u'House', <pyslet.odata2.csdl.StringValue object at 0x10c217bd0>)\n",
"(u'EndDate', <pyslet.odata2.csdl.DateTimeValue object at 0x10c229910>)\n",
"(u'Party', <pyslet.odata2.csdl.StringValue object at 0x10c1e4d50>)\n",
"(u'PartyAbbrev', <pyslet.odata2.csdl.StringValue object at 0x10c1e4e10>)\n",
"(u'PartySubType', <pyslet.odata2.csdl.StringValue object at 0x10c20efd0>)\n",
"(u'MembershipFrom', <pyslet.odata2.csdl.StringValue object at 0x10c217450>)\n",
"(u'CurrentStatus', <pyslet.odata2.csdl.StringValue object at 0x10c217790>)\n",
"(u'StartDate', <pyslet.odata2.csdl.DateTimeValue object at 0x10c20e290>)\n",
"(u'Dods_Id', <pyslet.odata2.csdl.Int32Value object at 0x10c1e4790>)\n",
"(u'Pims_Id', <pyslet.odata2.csdl.Int32Value object at 0x10c1e4310>)\n",
"(u'Party_Id', <pyslet.odata2.csdl.Int32Value object at 0x10c20e5d0>)\n",
"(u'CurrentStatus_Id', <pyslet.odata2.csdl.Int32Value object at 0x10c217850>)\n",
"(u'MembershipFrom_Id', <pyslet.odata2.csdl.Int32Value object at 0x10c2176d0>)\n",
"(u'Name_Id', <pyslet.odata2.csdl.Int32Value object at 0x10c20efd0>)\n",
"(u'CurrentStatusDate', <pyslet.odata2.csdl.DateTimeValue object at 0x10c1e4310>)\n",
"(u'CurrentStatusReason', <pyslet.odata2.csdl.StringValue object at 0x10c1e4fd0>)\n",
"(u'CurrentStatusActive', <pyslet.odata2.csdl.BooleanValue object at 0x10c20e650>)\n",
"(u'NameSortAs', <pyslet.odata2.csdl.StringValue object at 0x10c217c10>)\n",
"(u'NameSortAs1', <pyslet.odata2.csdl.StringValue object at 0x10c229810>)\n",
"(u'LayingMinisterName', <pyslet.odata2.csdl.StringValue object at 0x10c181b10>)\n",
"(u'Country', <pyslet.odata2.csdl.DeferredValue object at 0x10c18a290>)\n",
"(u'MemberAddresses', <pyslet.odata2.csdl.DeferredValue object at 0x10c18af90>)\n",
"(u'MemberBiographies', <pyslet.odata2.csdl.DeferredValue object at 0x109b43f50>)\n",
"(u'MemberCommittees', <pyslet.odata2.csdl.DeferredValue object at 0x10c229650>)\n",
"(u'MemberConstituencies', <pyslet.odata2.csdl.DeferredValue object at 0x10c229790>)\n",
"(u'MemberElectionsContesteds', <pyslet.odata2.csdl.DeferredValue object at 0x10c181a90>)\n",
"(u'MemberExperiences', <pyslet.odata2.csdl.DeferredValue object at 0x10c18a610>)\n",
"(u'MemberGovernmentPosts', <pyslet.odata2.csdl.DeferredValue object at 0x10c18af90>)\n",
"(u'MemberHonours', <pyslet.odata2.csdl.DeferredValue object at 0x10c153c10>)\n",
"(u'MemberHouseMemberships', <pyslet.odata2.csdl.DeferredValue object at 0x10c2294d0>)\n",
"(u'MemberLordsMembershipTypes', <pyslet.odata2.csdl.DeferredValue object at 0x10aadd5d0>)\n",
"(u'MemberMaidenSpeeches', <pyslet.odata2.csdl.DeferredValue object at 0x10c181d90>)\n",
"(u'MemberOaths', <pyslet.odata2.csdl.DeferredValue object at 0x10c18ae50>)\n",
"(u'MemberObituaries', <pyslet.odata2.csdl.DeferredValue object at 0x10c18a490>)\n",
"(u'MemberOppositionPosts', <pyslet.odata2.csdl.DeferredValue object at 0x10c1539d0>)\n",
"(u'MemberOtherParliaments', <pyslet.odata2.csdl.DeferredValue object at 0x10c2178d0>)\n",
"(u'MemberParliamentaryPosts', <pyslet.odata2.csdl.DeferredValue object at 0x10c2293d0>)\n",
"(u'MemberParties', <pyslet.odata2.csdl.DeferredValue object at 0x10c181210>)\n",
"(u'MemberPreferredNames', <pyslet.odata2.csdl.DeferredValue object at 0x10c18a050>)\n",
"(u'MemberStatuses', <pyslet.odata2.csdl.DeferredValue object at 0x1035ef450>)\n",
"(u'MemberStaffs', <pyslet.odata2.csdl.DeferredValue object at 0x10c153f50>)\n",
"(u'MemberKnownAs', <pyslet.odata2.csdl.DeferredValue object at 0x10c1c3f90>)\n",
"(u'ConstituencyCandidates', <pyslet.odata2.csdl.DeferredValue object at 0x10c2177d0>)\n"
]
}
],
"source": [
"for k in m[172].keys():\n",
" print(k,m[172][k])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For example, we can get a link into a collection of committees that a member is associated with."
]
},
{
"cell_type": "code",
"execution_count": 200,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"[659, 660, 661, 662, 663]"
]
},
"execution_count": 200,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"#\"Navigation properties are represented as DeferredValue instances.\"\n",
"#\"All deferred values can be treated as an entity collection\"\n",
"mcm=m[172]['MemberCommittees'].OpenCollection()\n",
"mcm.keys()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The `MemberCommittees` records don't actually contain the name of the specified committee. Instead, we get another navigation object that points to the committee."
]
},
{
"cell_type": "code",
"execution_count": 201,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(u'MemberCommittee_Id', <pyslet.odata2.csdl.Int32Value object at 0x10c233950>, <pyslet.odata2.csdl.Int32Value object at 0x10c233550>)\n",
"(u'Member_Id', <pyslet.odata2.csdl.Int32Value object at 0x10c1c88d0>, <pyslet.odata2.csdl.Int32Value object at 0x10c1aae10>)\n",
"(u'Committee_Id', <pyslet.odata2.csdl.Int32Value object at 0x10c1c84d0>, <pyslet.odata2.csdl.Int32Value object at 0x10c2336d0>)\n",
"(u'ExOfficio', <pyslet.odata2.csdl.BooleanValue object at 0x10c1fc290>, <pyslet.odata2.csdl.BooleanValue object at 0x10c1a1890>)\n",
"(u'StartDate', <pyslet.odata2.csdl.DateTimeValue object at 0x10c1c8e50>, <pyslet.odata2.csdl.DateTimeValue object at 0x10c2338d0>)\n",
"(u'EndDate', <pyslet.odata2.csdl.DateTimeValue object at 0x10c233250>, <pyslet.odata2.csdl.DateTimeValue object at 0x10c1c8f90>)\n",
"(u'EndNote', <pyslet.odata2.csdl.StringValue object at 0x10c1a1950>, <pyslet.odata2.csdl.StringValue object at 0x10c1e4d10>)\n",
"(u'IsAlternate', <pyslet.odata2.csdl.BooleanValue object at 0x10c1e4bd0>, <pyslet.odata2.csdl.BooleanValue object at 0x10c1a1310>)\n",
"(u'IsCoOpted', <pyslet.odata2.csdl.BooleanValue object at 0x10c229390>, <pyslet.odata2.csdl.BooleanValue object at 0x10c1c87d0>)\n",
"(u'Committee', <pyslet.odata2.csdl.DeferredValue object at 0x10c1a1850>, <pyslet.odata2.csdl.DeferredValue object at 0x10c1e4290>)\n",
"(u'MemberCommitteeChairs', <pyslet.odata2.csdl.DeferredValue object at 0x10c217690>, <pyslet.odata2.csdl.DeferredValue object at 0x10c1e4150>)\n",
"(u'Member', <pyslet.odata2.csdl.DeferredValue object at 0x10c1a1190>, <pyslet.odata2.csdl.DeferredValue object at 0x10c1c8f50>)\n"
]
}
],
"source": [
"for k in mcm[660].keys():\n",
" print(k,mcm[660][k],mcm[660][k])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If we open *that* collection, we can look up details of the committee."
]
},
{
"cell_type": "code",
"execution_count": 202,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(u'Committee_Id', 160)\n",
"(u'CommitteeType_Id', 14)\n",
"(u'Name', u'Treasury & Civil Service Sub-Committee')\n",
"(u'ParentCommittee_Id', 159)\n",
"(u'DateLordsAppointed', None)\n",
"(u'DateCommonsAppointed', None)\n",
"(u'Phone', None)\n",
"(u'Url', None)\n",
"(u'StartDate', TimePoint(date=Date(century=19, year=79, month=5, day=3),time=Time(hour=0, minute=0, second=0, zdirection=None, zhour=None, zminute=None)))\n",
"(u'EndDate', TimePoint(date=Date(century=20, year=10, month=5, day=6),time=Time(hour=0, minute=0, second=0, zdirection=None, zhour=None, zminute=None)))\n",
"(u'CreatedFromCommittee_Id', None)\n",
"(u'IsCommons', True)\n",
"(u'IsLords', False)\n"
]
}
],
"source": [
"mcmc=mcm[660]['Committee'].OpenCollection()\n",
"\n",
"#Display the values for the committee\n",
"for k, v in mcmc[mcmc.keys()[0]].data_items():\n",
" print(k,v.value)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"So if we step back, we can find all the committees a member has been associated with"
]
},
{
"cell_type": "code",
"execution_count": 203,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(u'Committee_Id', 159)\n",
"(u'CommitteeType_Id', 2)\n",
"(u'Name', u'Treasury & Civil Service')\n",
"(u'ParentCommittee_Id', None)\n",
"(u'DateLordsAppointed', None)\n",
"(u'DateCommonsAppointed', None)\n",
"(u'Phone', None)\n",
"(u'Url', None)\n",
"(u'StartDate', TimePoint(date=Date(century=19, year=79, month=5, day=3),time=Time(hour=0, minute=0, second=0, zdirection=None, zhour=None, zminute=None)))\n",
"(u'EndDate', TimePoint(date=Date(century=20, year=10, month=5, day=6),time=Time(hour=0, minute=0, second=0, zdirection=None, zhour=None, zminute=None)))\n",
"(u'CreatedFromCommittee_Id', None)\n",
"(u'IsCommons', True)\n",
"(u'IsLords', False)\n",
"----------\n",
"(u'Committee_Id', 160)\n",
"(u'CommitteeType_Id', 14)\n",
"(u'Name', u'Treasury & Civil Service Sub-Committee')\n",
"(u'ParentCommittee_Id', 159)\n",
"(u'DateLordsAppointed', None)\n",
"(u'DateCommonsAppointed', None)\n",
"(u'Phone', None)\n",
"(u'Url', None)\n",
"(u'StartDate', TimePoint(date=Date(century=19, year=79, month=5, day=3),time=Time(hour=0, minute=0, second=0, zdirection=None, zhour=None, zminute=None)))\n",
"(u'EndDate', TimePoint(date=Date(century=20, year=10, month=5, day=6),time=Time(hour=0, minute=0, second=0, zdirection=None, zhour=None, zminute=None)))\n",
"(u'CreatedFromCommittee_Id', None)\n",
"(u'IsCommons', True)\n",
"(u'IsLords', False)\n",
"----------\n",
"(u'Committee_Id', 78)\n",
"(u'CommitteeType_Id', 2)\n",
"(u'Name', u'Foreign Affairs Committee')\n",
"(u'ParentCommittee_Id', None)\n",
"(u'DateLordsAppointed', None)\n",
"(u'DateCommonsAppointed', TimePoint(date=Date(century=20, year=15, month=7, day=8),time=Time(hour=0, minute=0, second=0, zdirection=None, zhour=None, zminute=None)))\n",
"(u'Phone', u'020 7219 3278/6109')\n",
"(u'Url', u'http://www.parliament.uk/business/committees/committees-a-z/commons-select/foreign-affairs-committee/')\n",
"(u'StartDate', TimePoint(date=Date(century=19, year=79, month=5, day=3),time=Time(hour=0, minute=0, second=0, zdirection=None, zhour=None, zminute=None)))\n",
"(u'EndDate', None)\n",
"(u'CreatedFromCommittee_Id', None)\n",
"(u'IsCommons', True)\n",
"(u'IsLords', False)\n",
"----------\n",
"(u'Committee_Id', 80)\n",
"(u'CommitteeType_Id', 14)\n",
"(u'Name', u'Foreign Affairs: Entry Clearance Sub-Committee')\n",
"(u'ParentCommittee_Id', None)\n",
"(u'DateLordsAppointed', None)\n",
"(u'DateCommonsAppointed', None)\n",
"(u'Phone', None)\n",
"(u'Url', None)\n",
"(u'StartDate', TimePoint(date=Date(century=19, year=97, month=11, day=25),time=Time(hour=0, minute=0, second=0, zdirection=None, zhour=None, zminute=None)))\n",
"(u'EndDate', TimePoint(date=Date(century=20, year=10, month=5, day=6),time=Time(hour=0, minute=0, second=0, zdirection=None, zhour=None, zminute=None)))\n",
"(u'CreatedFromCommittee_Id', None)\n",
"(u'IsCommons', True)\n",
"(u'IsLords', False)\n",
"----------\n",
"(u'Committee_Id', 158)\n",
"(u'CommitteeType_Id', 2)\n",
"(u'Name', u'Treasury Committee')\n",
"(u'ParentCommittee_Id', None)\n",
"(u'DateLordsAppointed', None)\n",
"(u'DateCommonsAppointed', TimePoint(date=Date(century=20, year=15, month=7, day=8),time=Time(hour=0, minute=0, second=0, zdirection=None, zhour=None, zminute=None)))\n",
"(u'Phone', u'020 7219 3285/5766')\n",
"(u'Url', u'http://www.parliament.uk/business/committees/committees-a-z/commons-select/treasury-committee/')\n",
"(u'StartDate', TimePoint(date=Date(century=19, year=95, month=11, day=15),time=Time(hour=0, minute=0, second=0, zdirection=None, zhour=None, zminute=None)))\n",
"(u'EndDate', None)\n",
"(u'CreatedFromCommittee_Id', None)\n",
"(u'IsCommons', True)\n",
"(u'IsLords', False)\n",
"----------\n"
]
}
],
"source": [
"#Get the MembersCommittees keys for a member\n",
"for k in mcm.keys():\n",
" #Get the Committee details\n",
" with mcm[k]['Committee'].OpenCollection() as mcmc:\n",
" for k2,v2 in mcmc[mcmc.keys()[0]].data_items():\n",
" print(k2,v2.value)\n",
" print('----------')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"There must be an easier way to do this, and the `expansion` part of the `set_expand()` method seems like it shoud be the way to do this. For example, the following construction appears to be legal and would seem to suggest that we can \"pull through\", or at least access, name information from the actual committees? But it doesn't seem to form part of the available data values?"
]
},
{
"cell_type": "code",
"execution_count": 204,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(u'Member_Id', 172)\n",
"(u'Forename', u'Diane')\n",
"(u'Surname', u'Abbott')\n"
]
}
],
"source": [
"m.set_expand({'MemberCommittees':{'Committee':None}},\n",
" {'Forename':None,'Surname':None, 'MemberCommittees':{'Committee':{'Name':None,'EndDate':None}}})\n",
"\n",
"#Show the data retrived for the selected member\n",
"for k, v in m[172].data_items():\n",
" if v.value is not None: print(k,v.value)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Alternatively, just request the value of a known attribute directly:"
]
},
{
"cell_type": "code",
"execution_count": 208,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"u'Diane'"
]
},
"execution_count": 208,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"m[172]['Forename'].value"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Using the expansion also seems to have limited the amount of data that is requested fron the `Committee` entities:"
]
},
{
"cell_type": "code",
"execution_count": 206,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(u'Committee_Id', 159)\n",
"(u'Name', u'Treasury & Civil Service')\n",
"(u'EndDate', TimePoint(date=Date(century=20, year=10, month=5, day=6),time=Time(hour=0, minute=0, second=0, zdirection=None, zhour=None, zminute=None)))\n",
"------\n",
"(u'Committee_Id', 160)\n",
"(u'Name', u'Treasury & Civil Service Sub-Committee')\n",
"(u'EndDate', TimePoint(date=Date(century=20, year=10, month=5, day=6),time=Time(hour=0, minute=0, second=0, zdirection=None, zhour=None, zminute=None)))\n",
"------\n",
"(u'Committee_Id', 78)\n",
"(u'Name', u'Foreign Affairs Committee')\n",
"(u'EndDate', None)\n",
"------\n",
"(u'Committee_Id', 80)\n",
"(u'Name', u'Foreign Affairs: Entry Clearance Sub-Committee')\n",
"(u'EndDate', TimePoint(date=Date(century=20, year=10, month=5, day=6),time=Time(hour=0, minute=0, second=0, zdirection=None, zhour=None, zminute=None)))\n",
"------\n",
"(u'Committee_Id', 158)\n",
"(u'Name', u'Treasury Committee')\n",
"(u'EndDate', None)\n",
"------\n"
]
}
],
"source": [
"mcm=m[172]['MemberCommittees'].OpenCollection()\n",
"for k in mcm.keys():\n",
" #Get the Committee details\n",
" with mcm[k]['Committee'].OpenCollection() as mcmc:\n",
" for k2,v2 in mcmc[mcmc.keys()[0]].data_items():\n",
" if k2=='EndDate' or v2.value is not None:\n",
" print(k2,v2.value)\n",
" print('------')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"I'm not sure if there's a way of filtering based on this expansion? e.g. to limit results to committees where the `EndDate` really is `None` or `null` (i.e. limit to committees that the member is stull currently a member of?)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 2",
"language": "python",
"name": "python2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.11"
}
},
"nbformat": 4,
"nbformat_minor": 0
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment