This is a very lightly edited transcript of a discussion that happened on 17 Jan 2013 over IRC in #mschat on irc.mysociety.org (times are UTC - 3 hours )

gistfile1.txt

12:58 <jpmckinney> will do :) i was also wondering if you had thoughts re: https://github.com/mysociety/popit/issues/234
13:00 <evdb> It's an interesting one. One solution is to create a position with no person - currently I think that is not allowed by the validation.
13:01 <evdb> Also bad in that you'd have to remember to create the vacant positions everytime someone leaves the position.
13:01 <jpmckinney> yeah, i think having both a "posts" and an "appointments" (positions) model may be easier in the long-run
13:02 <jpmckinney> are you familiar with http://www.w3.org/TR/vocab-org/ ? they get into the different ways of modelling memberships in orgs
13:02 <jpmckinney> * and to also delete or update the vacant positions when someone takes the position
13:03 <evdb> vocab-org <-- I was not, will look at that, thank you.
13:04 <jpmckinney> from my reading of vocab-org, any system modelling people+orgs would probably need two ways to represent memberships - one where the "post" exists without anyone holding it, and another for all other memberships, e.g. party memberships, where there is no "post"
13:05 <evdb> regarding both posts and appointments we're finding with mzalendo.com that there are some posts that are long running (like a member of parliament) but others are one offs for a particular appointment (eg "Customs official third class")
13:05 <evdb> yup - as you just said.
13:05 <matthew> I think I have some notes from a while back about this sort of thing when it was discussed prior, I'm pretty sure this was covered, I'll try and dig them out
13:07 <jpmckinney> matthew: cool, i'd be interested
13:08 <evdb> jpmckinney: note that it would probably be possible to query the data though even without having that extra model. You'd just look for all the positions with a particular title and then deduce that the lack of a position means that it is vacant...
13:09 <evdb> this moves the knowledge of which posts are ongoing out of PopIt and into your app though.
13:09 <jpmckinney> if there are 6 Councilmembers At-Large in Philadelphia, how do I know it's not 7?
13:09 <jpmckinney> ah, because i tell my app
13:10 <jpmckinney> i'd like to not have that requirement :)
13:10 <matthew> that is what the TheyWorkForYou API does for list of MPs, yes, so does assume if you need to know they're all there you know what you want (e.g. from mapit you have the list of constituencies to match up against)
13:10 <jpmckinney> especially if my app runs govt monitoring for dozens of cities
13:10 <matthew> or indeed in the 3 week period before an election, it has to return 0 results
13:11 <jpmckinney> matthew: if i understand correctly, the TWFY strategy works when it's one-to-one between elected official and boundary, but breaks down if a single boundary can have multiple representatives
13:12 <matthew> yep
13:12 <matthew> it's very old :)
13:12 <evdb> but then you could just add to your boundary store the number of representatives expected
13:13 <jpmckinney> mapit can do that?
13:13 <evdb> and have it deal with the edge cases of pre-elections etc.
13:14 <matthew>  you could do that now, sort of, storing the number in a 'code'
13:14 <jpmckinney> ah, right, those custom codes
13:14 <jpmckinney> very flexible
13:14 <mark> ;) i was just writing: <jpmckinney> mapit can do that? <-- nope, not unless you misuse codes
13:15 <jpmckinney> ah, because codes need to reflect something real in the world?
13:15 <jpmckinney> like an government-issue identifier?
13:15 <matthew> no, not at all
13:15 <matthew> they can have anything you like. i think mark says it's a misuse because it's just not technically a 'code'
13:15 <matthew> which is fair enough
13:16 <jpmckinney> i see
13:16 <jpmckinney> billy (openstates.org) stores the number of seats on each boundary, but this leads to the constituencies mongodb collection being an uncomfortable mix of constituencies and posts
13:16 <mark> yeah, i said "misuse" because i'd assumed they were intended for alternative unique identifiers for the area rather than generic metadata
13:17 <matthew> it would probably be better for something like that to be a separate table, which I'd be happy to be there if it was needed for something like that
13:18 <matthew> and/or just generalising 'code' a bit, which is also quite an old hangover, hence the poor 'types' stuff
13:19 <jpmckinney> i don't mind abusing codes, but if it's information about a post, it should be on a post object, not a boundary object. especially if that boundary is referenced by multiple posts
13:19 <matthew> sure,I was only referring to the "number of reps for this area" mentioned above
13:19 <matthew> which appears to be metadata on the area itself, not an individual post
13:19 <jpmckinney> yes, but what if that area has representatives in different orgs?
13:20 <matthew> "number of <x> reps for this area"
13:20 <jpmckinney> ah, so one code for each
13:20 <matthew> it would be custom metadata, however you'd want to do it, I would assume
13:21 <jpmckinney> so we can get the number of reps, but if we also want to know what to call those reps e.g. "Member of Parliament" we'd need another code?
13:21 <evdb> I can see how a 'posts' model would be good for that
13:22 <evdb> I'm just wary of introducing it in PopIt as it may be superfluous to many people's needs. And probably the general solution would not be specific enough for when it is actually needed.
13:22 <matthew> jpmckinney: You could do. I'm not suggesting you should or it's the best way by any means. WriteToThem has managed for 9 years or so hardcoding its list of reps that it handles.
13:23 <evdb> For example at a hospital a 'post' entry might well want to contain the qualifications needed to hold it
13:23 <matthew> but in the UK at least we are helped that each area type in mapit is for a different type of representative
13:24 <matthew> that leads to another option, that you could import the same boundary into mapit multiple times in different types, for each type of representative. Again, just saying things that are possible, all depends what you want to do, what time you have, all the usual constraints
13:25 <jpmckinney> matthew: though when the number of workarounds gets too high, it may be more time-intensive to use them than to code the alternative :)
13:25 <evdb> I'd say a third component (alongside mapit and popit) would be best. This could be separate or built into your app. Then you can tailor it to your needs - for example I imagine the number of reps changes historically
13:26 <jpmckinney> evdb: i think there is a general solution for posts that can work as well as the general solution for positions. it's mostly adding another object between people and orgs for info about posts to hang onto
13:27 <evdb> jpmckinney: you may well be right - do you mind if I copy this discussion into the comments on the github issue?
13:27 <jpmckinney> evdb: i haven't decided whether it's a question of not being needed, or a question of people not seeing its need until they start running into the hairy issues around it
13:27 <jpmckinney> sure, feel free to edit for brevity :)
13:28 <evdb> Shall do. And you're right that if this is an inevitable problem that people will run into then we should fix it at the start - like having date ranged names.
13:29 <evdb> afk for lunch I'm afraid - will add to issue later.
13:29 <jpmckinney> for context, open north (the org i lead) is working with PPF (of opengovernment.org) to create a reusable govt monitoring platform at the local level
13:29 <jpmckinney> evdb: thanks for answering all my questions!
13:29 <matthew> It sounds sensible to me, making Position just a three way many-to-many(-to-many?) table between a Post, a Position, and an Organisation.
13:30 <evdb> not at all - sorry I have to go :(
13:31 <jpmckinney> matthew: i was thinking either person -> post -> organization or person -> appointment -> post -> organization
13:31 <jpmckinney> or did you mean "between a Post, a *Area*, and an Organisation" when you wrote "between a Post, a Position, and an Organisation" ?
13:32 <matthew> i mean Person instead of Position
13:32 <matthew> a Position being a row of (Post, Person, Org)
13:32 <matthew> sorry, mistyped
13:33 <jpmckinney> ah, k, but Position doesn't need to have Org, because Post will have Org
13:33 <jpmckinney> in which case your Position is like my Appointment
13:33 <matthew> will it always? I could see you would want Post that covers multiple Orgs
13:33 <matthew> e.g. "Councillor" in multiple councils
13:33 <matthew> maybe
13:34 <jpmckinney> is there a post, that when filled by a person, that person sits on all those councils?
13:34 <matthew> I think we may be using different definitions for the word "post"
13:34 <matthew> I think I'm using position where you're using post
13:35 <matthew> I'm using Post for an example of a job e.g. "Member of Parliament" or "Councillor"
13:35 <matthew> then Org is something like "Birmingham Council" or "House of Commons"
13:35 <jpmckinney> that sounds right to me. and in my view, there'd be a different Post for each MP. you need to fill all those seats, and each seat is different
13:35 <matthew> yes, i used Position above in my string for that, yes, to match the table popit currently has
13:36 <matthew> but basically with the "title" string of the position schema becoming a separate model
13:36 <matthew> Post is the *model* of the job
13:36 <matthew> not the filler of that job
13:36 <jpmckinney> hmm, i think there's issues with that way of modelling things, let's see if i can explain
13:36 <jpmckinney> first, i don't think there's much utility to having an object that stands for a generic MP job
13:37 <jpmckinney> but there is a use to having an object that represents the "MP for Abbotsford" job, which different people can be appointed to at different times
13:37 <jpmckinney> and i think that reflects how things are in the real world
13:38 <jpmckinney> so you're not just moving the title for the position to the new model, you're putting pretty much everything besides start and end dates, and the ID of the person holding the job
13:38 <jpmckinney> so you'd attach the area ("Abbotsford") to the job ("MP for Abbotsford")
13:39 <matthew> yep
13:39 <mark> jpmckinney: how would you then search for "all members of parliament" (currently, say) if there wasn't a generic representation of that type of position?
13:40 <matthew> I think that's what I was trying to say, but badly/failing :) I agree that makes sense.
13:40 <matthew> mark: how do you do that now?
13:40 <jpmckinney> search for all Posts that reference the ID of the "House of Commons" org, then find all current Appointments to those posts, to get the list of Person IDs
13:40 <matthew> if you wanted your popit to have other types of job attached to House of Commons though...
13:41 <matthew> you'd need some way of grouping the Posts into Jobs
13:41 <matthew> that may then be going out of scope and you should use a separate instance of popit, but I could see that being useful too
13:42 <jpmckinney> we can maybe create a controlled vocabulary, and say that this Post/Job is tagged as "MP"
13:42 <mark> <matthew> mark: how do you do that now? <-- in popit, currently just assume that all current positions in the parliament are MPs :(
13:42 <jpmckinney> same goes for "find all committees of the House of Commons" - you need to be able to tag the orgs in the hierarchy as being a committee versus whatever else
13:43 <matthew> jpmckinney - okay so committees is a good example
13:43 <mark> sorry, i was wrong - i look for the position title being "Member of Parliament"
13:43 <mark> as well
13:43 <matthew> you have MPs on the committee obviously, and you also have the clerks/civil servants. So yes, some sort of tagging or 'parent' for Posts would be needed/useful there
13:44 <mark> which i guess is like: <jpmckinney> we can maybe create a controlled vocabulary, and say that this Post/Job is tagged as "MP"
13:44 <jpmckinney> yeah, that's how i read how position titles are currently used
13:46 <jpmckinney> matthew: going back, do you still think there needs to be a row "(Post, Person, Org)" (or was it (Position, Person, Org)) or can Org be on a different model?
13:46 <matthew> if you had a large popit of a) all government departments, b) ministers, c) civil servants in different roles from PermSec to nurse, d) all the current Posts of all those at all of a), you'd need some hierarchy of the Posts too to say "give me all the civil servants". But probably ignore me for now :)
13:46 <matthew> Using your terms, I think you'd want an Appointment consisting of (Person, Org, Post)?
13:47 <matthew> if I've understood right
13:47 <jpmckinney> I think (Person, Post) suffices, because the Post tracks which Org it belongs to, because Post is more specific than "MP" and is more like "MP for Avalon"
13:48 <jpmckinney> or "Councillor for Ward 1"
13:48 <jpmckinney> (in Chicago, say)
13:49 <jpmckinney> re: "give me all the civil servants", you can either tag posts as "civil servant" like you tag them "MP
13:49 <jpmckinney> though "civil servant" may be too generic a tag
13:50 <jpmckinney> i guess you can list all of the "civil servant" tags and search for those ((in which case you have to consult a hierarchy of tags to know which are civil servants i guess)
13:51 <jpmckinney> that's the sort of thing http://www.w3.org/TR/vocab-org/ addresses - i'd probably only implement a subset of what it's capable of representing
13:52 <matthew> yes, so for having Posts in more than one Org
13:53 <matthew> for your case of Reps, as you say you'd only need (Person,Post) because the Post is dependent on the Org
13:53 <matthew> but what about a popit storing e.g. theatre productions and the people involved? The Posts then are not Org-dependent, they're the same regardless of the Org
13:54 <jpmckinney> i'd say in that case to not use posts but instead a membership model closer to what positions are now
13:54 <matthew> having to replicate "Stage Manager" for every Org seems needless, when you could have Appointment of (Person, Post, Org) and then it would work fine for both situations
13:54 <jpmckinney> if a post were just a name field, i'd agree
13:54 <jpmckinney> but a post can have more metadata than that
13:55 <matthew> I'm not saying Posts couldn't be linked to Orgs as well
13:56 <jpmckinney> is Org on Appointment then a denormalization of Org on Post?
13:56 <matthew> or indeed that could perhaps be modelled as an Appointment of (null, Post, Org)
13:56 <jpmckinney> i think appointments should always have a person being appointed. if no person is appointed, then it's a post
13:57 <jpmckinney> which is why it's good to not have generic "stage manager" posts, but posts for each org
13:57 <jpmckinney> that way you can ask, "what companies have vacancies for stage managers?"
13:57 <jpmckinney> otherwise, you need to create the null appointment each time a person retires
13:58 <matthew> very true
14:01 <jpmckinney> anyhow, posts aren't always appropriate, e.g. for things like party membership, where a simpler membership model is more accurate
14:02 <matthew> and I think we would want to be able to handle those too
14:03 <matthew> if your popit is the "Conservative party" and you wanted both party members and senior leadership (with posts), say
14:04 <jpmckinney> for sure. having two ways of representing the relation between people and orgs may require more education for users, but i think it's the best compromise for modelling the data in a way that will not be a pain later, and that makes some features easier to implement
14:05 <jpmckinney> or even possible to implement
14:06 <matthew> though could you use a post for party membership?
14:06 <matthew> It would seem to work, though I've probably missed something
14:07 <jpmckinney> if post is generic like in your examples, yes, but if it's more specific like in mine, then you'd have to create a post for each party member
14:07 <matthew> Why would you have to?
14:07 <matthew> (even in your example)
14:08 <jpmckinney> hmm, let me think more carefully
14:10 <jpmckinney> from a coding perspective, it might not raise any issues
14:11 <jpmckinney> but it may confuse the meaning of that model
14:11 <jpmckinney> i try to stick to the W3C meaning of post described in http://www.w3.org/TR/vocab-org/
14:11 <jpmckinney> where "A Post represents some position within an organization that exists independently of the person or persons filling it."
14:12 <jpmckinney> party membership would not pass that criteria
14:12 <jpmckinney> people do not fill the position of "party member". there's no advertisements for vacancies for that position, etc.
14:15 <matthew> I'm not sure I wouldn't say that, but I'm not up on formal speak
14:15 <jpmckinney> in my experience, if you overload your models to mean different things in different contexts, you run into problems down the road
14:16 <matthew> party membership does exist as a concept independent of people filling it. e.g. A party member is allowed access to a conference by virtue of being a party member, not as an individual person
14:16 <matthew> sure, I just didn't really see it as a different context
14:16 <jpmckinney> sure, the membership is real, but a membership is more like an appointment. there is no intermediate object "post" in between the person and the party
14:17 <matthew> I don't see why not. There are adverts to fill party memberships, is the issue just there's not a limit?
14:17 <jpmckinney> it's a good example. i can see it going either way
14:19 <matthew> i'd always fall on the side of what is best for now in that case :)
14:20 <jpmckinney> i think i've come around. it may just be the case that if there's no additional metadata to justify a post object, i'd prefer to have more direct relations
14:20 <jpmckinney> e.g. a party membership doesn't even need a title for that membership
14:21 <jpmckinney> but if we can keep to one way of representing relations, that's good for users
14:22 <matthew> the code could presumably have things to help with directness. A title might still be helpful if your db also has other things in it though, I guess, again, depends on how you're defining/meaning membership.
14:22 <matthew> Yeah
14:23 <jpmckinney> re: title, if all i want to say is "Person X is a member of Org Y" then I just need the two Person and Org IDs
14:23 <jpmckinney> but i think we understand each other :)
14:24 <matthew> yep. all sounds good to me.
14:26 <jpmckinney> do you think adding an explicit "posts" model should be in the roadmap for popit, or implemented as another component?
14:26 <matthew> I don't see why it shouldn't be part of popit
14:26 — mark votes for "in popit"
14:29 <jpmckinney> cool, i'll let evdb copy the discussion into the github issue, but i'm happy to contribute as it progresses
14:32 <jpmckinney> i have to step out for a bit - it's been a good discussion!
14:32 <matthew> yes, thanks :) byee