iantimmis
/ softmax_cross_entropy.py
Created
October 12, 2020 03:54
Numerically stable softmax with cross entropy in numpy
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| import numpy as np | |
| def naive_softmax(logits): | |
| ''' | |
| Failure modes: | |
| * If any entry is very large, exp overflows | |
| * if all entries are very negative, all exps underflow | |
| ''' | |
| exp_logits = np.exp(logits) | |
| return exp_logits / np.sum(exp_logits) |