Skip to content

Instantly share code, notes, and snippets.

@denizyuret

denizyuret/attention.jl

Created November 13, 2020 14:04
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save denizyuret/a21c3e71e44dc4b0dcb3d68e4cd31c58 to your computer and use it in GitHub Desktop.
Save denizyuret/a21c3e71e44dc4b0dcb3d68e4cd31c58 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
attention.jl
# given some encoder states X, decoder state y, matrices Q, K, V
for i in 1:N
keys[i] = K * X[i]
values[i] = V * X[i]
end
query = Q * y
for i in 1:N
relevance[i] = query ⋅ keys[i]
end
relevance = softmax(relevance)
output = 0
for i in 1:N
output += relevance[i] * values[i]
end
return output
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment