Skip to content

Instantly share code, notes, and snippets.

@cossio

cossio/log1msoftmax.jl

Created November 6, 2021 18:55
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save cossio/4517c3a07c601dd67502e86c240160fb to your computer and use it in GitHub Desktop.
Save cossio/4517c3a07c601dd67502e86c240160fb to your computer and use it in GitHub Desktop.
Download ZIP
Computes log(1 - softmax(X)), accurately.
Raw
log1msoftmax.jl
"""
log1msoftmax(X; dims=1)
Computes log(1 - softmax(X)), accurately.
"""
function log1msoftmax(x::AbstractArray; dims=1)
@warn "log1msoftmax can have numerical issues, https://stats.stackexchange.com/questions/469706/log1-softmaxx/469803"
#FIXME: https://stats.stackexchange.com/questions/469706/log1-softmaxx/469803?noredirect=1#comment867691_469803
m = maximum(x; dims=dims)
e = exp.(x .- m)
s = sum(e; dims=dims)
#return log.((s .- e) ./ s)
return @. log1p(-e/s)
end
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment