Skip to content

Instantly share code, notes, and snippets.

@domluna

domluna/derivs.txt

Created September 27, 2016 05:29
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save domluna/8c9f54ef10f22c80d9632f5f71f5d119 to your computer and use it in GitHub Desktop.
Save domluna/8c9f54ef10f22c80d9632f5f71f5d119 to your computer and use it in GitHub Desktop.
Download ZIP
Derivatives for softmax and cross entropy
Raw
derivs.txt
# derivs
# cross entropy -1/p deriv of -log(p)
# softmax
# p * (1 - p) if i == j if this is the node we picked
# -p * p if i != j otherwise
#
# chain rule for both functions dsoftmax * dcross_entropy
# p * (1 - p) * (-1/p) = p - 1
# -p * p * (-1/p) = p
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment