Skip to content

Instantly share code, notes, and snippets.

@g-leech
Last active April 10, 2022 18:40
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save g-leech/3e3fd37febc23251cf8c245203dfe1f6 to your computer and use it in GitHub Desktop.
Save g-leech/3e3fd37febc23251cf8c245203dfe1f6 to your computer and use it in GitHub Desktop.
Comparing Spock's predictions to a coin flip, yielding a Brier score of 0.57
import numpy as np
# impossible 0
# v unlik 10
# unlik 25
# lik 75
# vv likely 99.5
preds = [
[0, 1],
[0.75, 1],
[0.75, 0],
[0, 1],
[0, 0],
[0, 1],
[0, 1],
[0, 1],
[0.995, 1],
[0.995, 0],
[0.995, 0],
[0.25, 1],
[0.1, 1],
[0, 1],
[0.1, 0],
[0.1, 0],
[0.1, 0],
[0.995, 0],
[0.995, 0],
[0.81, 1],
[0.75, 0],
[0.825, 1],
[0.75, 1],
[0, 1],
]
def brier(ps):
ses = [(p[0] - p[1])**2 for p in ps]
return np.mean(ses)
print(brier(preds)) # 0.5699
def flip() :
return np.random.randint(0, 2)
cs = []
for i in range(10000):
coin = [[flip(), p[1]] for p in preds]
cs.append(brier(coin))
print(np.mean(cs))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment