anonymousreview.md

Should my conference do anonymous review?

I recently wrote a post called Anonymous review is amazing, talking about our experience with anonymous review at !!Con (it was excellent! I was surprised and delighted!). There was a discussion on the PyCon organizers list today about whether PyCon should do anonymous review, and I started thinking about this a little more carefully.

I'm going to make a few assumptions up front: our goal as conference organizers is to have

• a process that is as unbiased as possible
• speakers who will be engaging
• who come from diverse backgrounds
• where some are new speakers, and some are more experienced

Let's talk about whether anonymous review will help us with these things!

Is anonymous review less biased?

Yes.

Firstly, people believe that anonymous review is less biased.

One of our !!Con's speakers, Katherine Ye, told us:

Thank you so much for [anonymizing everything]! It’s a relief to know that I wasn’t picked for gender, race, age, or anything like that.

Kenneth Hoxworth said of RailsConf's anonymous review process:

It gave me courage that I wasn't going up against big names.

It's really important for people to have confidence in a conference's review process. Nobody wants to put time into a proposal if they're going to be dismissed because of their gender or age or race, or just because they're not famous enough. People also worry about not being accepted on their own merit.

Anonymous review helps us build confidence, and that's really valuable.

Anonymous review is also actually less biased. This study by Kathryn McKinley shows that, in peer-reviewed scientific articles, both men and women express systemic bias against women, and double-blind reviewing removed that bias.

They found nepotism and gender bias were significant factors in the evaluation process. To be judged as good as their male counter parts, female applicants had to be 2.5 times more productive.

Will anonymous review help my conference's diversity?

Maybe. EuroPython has an anonymous review process, and recently very few of their announced speakers were women. This is because very few women applied to give talks. You can't accept talks that don't exist!

A more effective way to diversify your speaker pool is through active outreach. I don't know of any evidence to show that anonymous review helps you attract a more diverse range of speakers. (is there some? I would love to know.)

Will anonymous review help me get inexperienced speakers?

Maybe.

On one hand, we have

It gave me courage that I wasn't going up against big names.

On the other hand, Douglas Napoleone pointed out:

An anonymous system has an inherent bias towards very well written proposals. Those people whom have given the most talks are those whom are best at writing proposals which are best at getting through selection committees. It becomes a feedback loop which cuts out the very speakers we want most. Knowing that a person is a new speaker with a decent proposal is key when comparing them against a proposal by someone whom has given a talk at the last 8 python conferences.

PyCon's approach is to actively encourage new speakers to apply and work with them to write better proposals, and that's been successful.

Florian Gilchner wrote about eurucamp's experience with anonymous review here:

We found that newcomers don't write worse proposals than seasoned speakers. Quite the contrary, we found that many proposals that are submitted to many conferences are unspecific and dull and would only fly by having a big name attached. Anonymous CFPs are very good at weeding out copy-pasta. We didn't accept quite a few people that would have been really shiny on the program.

and

Every year, we have at least one person we take huge bets on and get very good talks out of that. Most of the time, it's someone would [would lose out] in a direct and open battle.

But will my speakers be good?!

This is probably the scariest part. We did anonymous review for !!Con, and our speakers were very good. Our main hope was that if somebody wrote a proposal about an interesting topic, then they could give an engaging 10-minute talk. This worked. It's relevant here that our talks were all lightning talks.

We also had an anonymizer, who did an amazing job reviewing videos and telling us his impressions. This meant that we had to trust his judgement (which I do! and our speakers were great!), but having only one person watching talks introduces bias.

I'd be worried about doing anonymous review if I was organizing a conference where the talks were longer. (though it's been done successfully!)

So should you do anonymous review?

Anonymous review takes extra time. You should think about what benefits you hope that it'll bring, and what your alternatives are. There's some excellent discussion on the comments to a draft of this post. Go read it.

Some other things you can spend time on:

• doing outreach to get more applications from under-represented communities
• giving new speakers feedback on their proposals and helping them do a better job
• writing up a really good call for speakers (see JSConf EU's!)
• running brainstorming sessions to help people come up with ideas

I would do it again for !!Con, since the response to it was super positive and the talks were good.

I disagree with a lot of points. I ran CFP for eurucamp this year and we run anonymous or the third time in a row (first doesn't really count, we had 5 more submissions than slots, so everyone was accepted). We have a history of bringing new people (or with a new topic) on stage

We found that newcomers don't write worse proposals than seasoned speakers. Quite the contrary, we found that many proposals that are submitted to many conferences are unspecific and dull and would only fly by having a big name attached. Anonymous CFPs are very good at weeding out copy-pasta. We didn't accept quite a few people that would have been really shiny on the program.

Also, we had cases where we allowed seasoned speakers to switch their main topic because of that. We had a case where a speaker held a tremendous social talk, although we only knew that he was good in technical subject matter, but not delivery - turns our he changed a lot. If I read that proposal with his name attached, "no way" would have been the answer. We found that anonymity rules out "disaster avoidance" in the committee, the attempt of only having super-sure talks with super-sure speakers to "make the conference better".

In the end, our top 50 was full of new people and the top talk was by someone very unlikely.

Anonymity allows you to pick a topic and have the team rate on that instead of the people. Also, it was the smallest part of work in the process.

I do agree somewhat about implementation though:

• Just having an anonymous CFP doesn't magically make your conference diverse. It is an offer to people that would feel uneasy to be voted against well know speakers. We found that we could use if for convincement during outreach though.
• Encouraging to submit makes your conference diverse and we found that the people we encouraged really wanted to make things count.
• There is some amount of cheating. One person (and only exactly one) in our team was actually trusted to check the submitter list (without papers attached) for several metrics we wanted to hold. This is necessary for outreach.
• Any additions help, we have an active mentorship program, for example (http://cfp.eurucamp.org/mentorship). This was already running during CFP, allowing people to give feedback on proposals.
• The "well written proposal" is a red herring. There are many ways a proposal can be well written, one of them is having a very unique and special topic. That's very much possible for first-time-speakers. Most speakers on the circle have a very narrow and well-known topic area. I never found a convincing argument why seasoned speakers would win out in anonymous CFPs.
• We found that providing a good guide and clear expectations for the proposal helps many people: writing eduction is much better than speaking education, so the problem is smaller than people may think. Most people can write a text given a set of expectations.
• Every year, we have at least one person we take huge bets on and get very good talks out of that. Most of the time, it's someone would loose out in a direct and open battle.
• The work can be avoided by using one of the many open source CFP systems out there that support that work flow. We decided on 160 proposals in a long session.
• In the end: response is very good!

We wrote a blog post about that and will publish today.

Great points, and very useful information.

PyCon receives ~900 proposals for 80-94 slots. We will sometimes get 3-5 proposals from the same person. In the future we will only ever accept 1 talk from a speaker; we messed up the past two years, and there were instances where someone gave 2 talks.

One of the largest issues we face is the shear workload that puts on our 100% volunteer program committee. We need a system which will help us get the best at scale, and distributed review of the reviews; it's too much for the co-chairs.

We used to do much more mentorship, and we required that each proposal had at least one piece of feedback given to the proposer; as part of the review processes, while reviewing submitters can edit their proposal, and get feedback from the reviewers, before decisions are made. But the scale..... this happens less and less. We rely on local user groups, pyladies, scipy groups, and sprints for people to work on proposals together much more than the review feedback cycle now.

Looking at conferences which open it up for general internet review, it rarely goes well. You get the least interesting and unique talks. It ends up weeding out the controversial, unique, and unknown.

That's why we have stuck with the champion processes which bubbles up those talks which are controversial (many up and down votes), and treats talks with more than N upvotes equally, and treats both those sets as equal for consideration.

I don't like the comment that "if it ain't broke don't fix it". PyCon is fantastic and we have great talks, but we can always do better, the load on our volunteers is too high, and we are not getting enough feedback to those whom worked so hard to make what are fantastic submissions.

We just have to reject 9 for every one we accept.

Thanks for the reply! First off: PyCon is a huge inspiration for us and some of it we straight-out copy (your handling procedures are awesome). Also, I have to add that I don't think that anonymous CFP is mandatory. We just had very good experiences with it.

We wrote a lot of tooling ( https://github.com/eurucamp/call4papers ) that made review quick and easy and especially implemented the review process. Having some people just writing software was surprisingly helpful.

We tried to side-step our review problem by crowd-sourcing the assistance: we allow everyone (including submitters) to give feedback on other proposals and found that - despite the competition - it was seriously used. The feedback is anonymous, though. Also, we tried to get personal mentors for accepted speakers from the community and found that working very well. Granted, this will work worse once every conference is starting to do similar things and only works if you have a certain standing in the community. I also thee that although we also have ~9 people per speaking slot, we certainly don't work at your scale.

I don't understand the paragraph about "if it ain't broke don't fix it". I didn't want to imply that there is anything wrong with PyCon (quite the contrary!) and I am sure it keeps getting even better. I just don't agree with the quote ;).

In any case, do you happen to be in Berlin for EuroPython? I would certainly like to discuss this and it is always hard to find people that can.

Thanks for the feedback! it is very helpful. The "if it ain't broke don't fix it" comment did not come from here, but it is something that I often hear from people about PyCon and the talk selection. We have had some amazing conferences using the selection processes we have and there is an opinion out there (not shared by those on the program committee) that it should not change to an anonymous system because it works fine now. We change the processes every year, and we have reached a point where we need to make some more major changes. I would like to bring back the double blind for at least part of the processes if not all, but thankfully that is not up to me to decide :-)

Unfortunately I will not be going to EuroPython this year. I will check to see whom else from the program committee is going. Hopefully Luke or Alex are going.