I recently wrote a post called Anonymous review is amazing, talking about our experience with anonymous review at !!Con (it was excellent! I was surprised and delighted!). There was a discussion on the PyCon organizers list today about whether PyCon should do anonymous review, and I started thinking about this a little more carefully.
I'm going to make a few assumptions up front: our goal as conference organizers is to have
- a process that is as unbiased as possible
- speakers who will be engaging
- who come from diverse backgrounds
- where some are new speakers, and some are more experienced
Let's talk about whether anonymous review will help us with these things!
Yes.
Firstly, people believe that anonymous review is less biased.
One of our !!Con's speakers, Katherine Ye, told us:
Thank you so much for [anonymizing everything]! It’s a relief to know that I wasn’t picked for gender, race, age, or anything like that.
Kenneth Hoxworth said of RailsConf's anonymous review process:
It gave me courage that I wasn't going up against big names.
It's really important for people to have confidence in a conference's review process. Nobody wants to put time into a proposal if they're going to be dismissed because of their gender or age or race, or just because they're not famous enough. People also worry about not being accepted on their own merit.
Anonymous review helps us build confidence, and that's really valuable.
Anonymous review is also actually less biased. This study by Kathryn McKinley shows that, in peer-reviewed scientific articles, both men and women express systemic bias against women, and double-blind reviewing removed that bias.
They found nepotism and gender bias were significant factors in the evaluation process. To be judged as good as their male counter parts, female applicants had to be 2.5 times more productive.
Maybe. EuroPython has an anonymous review process, and recently very few of their announced speakers were women. This is because very few women applied to give talks. You can't accept talks that don't exist!
A more effective way to diversify your speaker pool is through active outreach. I don't know of any evidence to show that anonymous review helps you attract a more diverse range of speakers. (is there some? I would love to know.)
Maybe.
On one hand, we have
It gave me courage that I wasn't going up against big names.
On the other hand, Douglas Napoleone pointed out:
An anonymous system has an inherent bias towards very well written proposals. Those people whom have given the most talks are those whom are best at writing proposals which are best at getting through selection committees. It becomes a feedback loop which cuts out the very speakers we want most. Knowing that a person is a new speaker with a decent proposal is key when comparing them against a proposal by someone whom has given a talk at the last 8 python conferences.
PyCon's approach is to actively encourage new speakers to apply and work with them to write better proposals, and that's been successful.
Florian Gilchner wrote about eurucamp's experience with anonymous review here:
We found that newcomers don't write worse proposals than seasoned speakers. Quite the contrary, we found that many proposals that are submitted to many conferences are unspecific and dull and would only fly by having a big name attached. Anonymous CFPs are very good at weeding out copy-pasta. We didn't accept quite a few people that would have been really shiny on the program.
and
Every year, we have at least one person we take huge bets on and get very good talks out of that. Most of the time, it's someone would [would lose out] in a direct and open battle.
This is probably the scariest part. We did anonymous review for !!Con, and our speakers were very good. Our main hope was that if somebody wrote a proposal about an interesting topic, then they could give an engaging 10-minute talk. This worked. It's relevant here that our talks were all lightning talks.
We also had an anonymizer, who did an amazing job reviewing videos and telling us his impressions. This meant that we had to trust his judgement (which I do! and our speakers were great!), but having only one person watching talks introduces bias.
I'd be worried about doing anonymous review if I was organizing a conference where the talks were longer. (though it's been done successfully!)
Anonymous review takes extra time. You should think about what benefits you hope that it'll bring, and what your alternatives are. There's some excellent discussion on the comments to a draft of this post. Go read it.
Some other things you can spend time on:
- doing outreach to get more applications from under-represented communities
- giving new speakers feedback on their proposals and helping them do a better job
- writing up a really good call for speakers (see JSConf EU's!)
- running brainstorming sessions to help people come up with ideas
I would do it again for !!Con, since the response to it was super positive and the talks were good.
Thanks for the reply! First off: PyCon is a huge inspiration for us and some of it we straight-out copy (your handling procedures are awesome). Also, I have to add that I don't think that anonymous CFP is mandatory. We just had very good experiences with it.
We wrote a lot of tooling ( https://github.com/eurucamp/call4papers ) that made review quick and easy and especially implemented the review process. Having some people just writing software was surprisingly helpful.
We tried to side-step our review problem by crowd-sourcing the assistance: we allow everyone (including submitters) to give feedback on other proposals and found that - despite the competition - it was seriously used. The feedback is anonymous, though. Also, we tried to get personal mentors for accepted speakers from the community and found that working very well. Granted, this will work worse once every conference is starting to do similar things and only works if you have a certain standing in the community. I also thee that although we also have ~9 people per speaking slot, we certainly don't work at your scale.
I don't understand the paragraph about "if it ain't broke don't fix it". I didn't want to imply that there is anything wrong with PyCon (quite the contrary!) and I am sure it keeps getting even better. I just don't agree with the quote ;).
In any case, do you happen to be in Berlin for EuroPython? I would certainly like to discuss this and it is always hard to find people that can.