Skip to content

Instantly share code, notes, and snippets.

@yaroslavvb

yaroslavvb/cross_entropy_hessian.py

Last active August 30, 2019 01:39
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save yaroslavvb/703f815e6a3e400a7926ff90e6f9c820 to your computer and use it in GitHub Desktop.
Save yaroslavvb/703f815e6a3e400a7926ff90e6f9c820 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
cross_entropy_hessian.py
d = 3
q = torch.rand(d)
q /= q.sum() # empirical distribution (soft labels)
theta = torch.randn(d, requires_grad=True)
p = F.softmax(theta, dim=0)
loss = -torch.sum(q*torch.log(p))
g = p - q
H = torch.diag(p) - outer(p)
torch.allclose(H, hessian(loss, theta))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment