Here is how this works. Write down as many predictions as you can think of. They dont necessarily have to be things you think will happen, just things that could happen. Next, for each prediction, give your most sincere guess as to what the probability is that it will occur. If you think theres a 50% chance of it happening then you should assign it 50% (Assign 30% to 30% likely ones, 80% to 80% likely ones, etc).
Calibration predictions are a way of measuring your own objectivity about future events. If I were truly objective in my expectations about the future, then 50% of the things I expect to have 50% probability of occuring, should occur. If 90% of the things I think have a 50% chance of happening, then something is way off in how I am thinking about the world. If you were objective, then you would see things like 90% of the ones rated "90% likely" would come true.
Finding out how you've screwed up in how you think about the world is part of self improvement. But also, its just fun.
In April 2017 I wrote down a bunch of predictions for the next year (April 2018). In November 2017, I did another set of yearly predictions. This report pertains to the 80 predictions I made in that time period. Here are some of my predictions, others Ive excluded because I felt they were too personal.
So 9/9 things I rated as 100% likely to happen happened 10/16 things I rated 90% happened, etc. I did pretty good in that 60% to 80% range, and then nailed all of the things I was 100% certain about. In that 90% and 50% range I did pretty poorly. Only 28% of the things I thought were 50% likely happened. It could have been pure luck; I only gave 7 propositions a 50% chance of happening (if you flip a coin 7 times its very likely you wont get 4 heads and 3 tails). But still, lets look at the categories where I got things really wrong; where if I did better my score would have been better.
I program for a living. I made a number of predictions about my own domain of software and programming generally. It seems like I got a lot of those wrong. Let me list some examples below:
- Elm 0.19 Will be out by April 2018, 90% likely
- Elm 0.2 Will be out by April 2018, 60% likely
- There will be a serious competitor to Slack, 60% likely
- There will be a hip new front end JavaScript front end frame work at least as big as VueJS, 80%
None of these things happened, as far as I can tell (Elm 0.19 eventually came out, but later than I expected). I still mostly believe that these things will happen, its just a question of time scale I guess. I made these predictions on a 1 year time scale, but maybe a 5 year time scale would have been more reasonable.
The world is just slower than I thought I guess.
In the case of Slack prediction, I figured Slack isnt very good software, so the natural thing would be for a competitor to come in and displace them. But maybe I was kind of foolish in how I thought the economy worked. I could have failed to anticipate either, hard it would be for a competitor to make something truly better than Slack, how hard it is to enter into a market and displace Slack, or how much better positioned Slack is to improve itself than a competitor is to improve on Slack.
- I will still live in New York City : 70%
- I will leave the country : 10%
- I will have coded in Haskell : 10%
I rated it 10% likely that I would leave the country at all, and then I moved out of the US entirely. I rated it low that I would code any Haskell, but now I code at least a little Haskell every week.
I dont really know what the lesson to learn here is, but it seems like there are a lot of details about my own personal life, that I had a high degree of certainty about, that turned out to be totally off the mark. My calibration would improve quite a lot if I could anticipate my own life better.
- France will leave the EU: 10%
- Marie Le Pen will win the French election: 50%
- Angela Merkel Loses her election: 50%
- Some European nation will try and leave the EU : 50%
- Some new European nation will emerge through an independence movement: 0%
Some of these I got right, but none of those 50% ones came true.
At the time of making these predictions, I tried to be conscious about whether I was subscribing to the theory that the new right wing political developments in the US and Europe were flukey, or a new normal. I suspect when it comes to politics, developments are always portrayed as shocking, and they always feel like a disturbing new normal; but they never are. That leads me to think that most things are either a fluke, or far more normal than they feel. But, on the other hand, society does change, the causal forces behind elections and referendums are real and dont just disappear. These are contrary but compelling lines of reasoning, and all I can take away from them is "The world is changing, but slowly, and not in any obvious way big superficial political events are reflecting". So anyway, I tentatively stuck to the expectation that "for the short term, its not over" and that after April 2017 there would be further right wing victories in European politics.
And I think that position has turned out to be correct; there were further right wing victories in European politics, most evidenced by right wing parties growing in the parliaments in most every European country. But I also think I was not able to use that theory to make good predictions about what would actually happen in Europe. I suspect I was just too ignorant about how Europe works. I was not able to say "As a consequence, right wing parties will grow in every countries parliament", but I was able to say foolish things like "The odds are fair that Merkel wont be re-elected", even tho Chancellors arent elected to begin with.
Brexit was a right wing victory in the UK, but in hindsight, for the "another country will try and leave the EU" prediction, 50% was way too high of an estimation. Brexit could only have happened in the UK, just because of the dynamics in the UK compared to other EU countries, but from my naive point of view, it was easy to say "Well, the right wing wants independence in UK, so I guess the right wing all over Europe wants independence". It doesnt really work that way.