Skip to content

Instantly share code, notes, and snippets.

@ChunML

ChunML/transformer_16.py

Created April 28, 2019 10:25
Show Gist options
  • Save ChunML/c87c8290f05f49ca9519bd25cd455654 to your computer and use it in GitHub Desktop.
Save ChunML/c87c8290f05f49ca9519bd25cd455654 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
transformer_16.py
# FFN
ffn_in = mid_sub_out
ffn_out = self.dense_2[i](self.dense_1[i](ffn_in))
ffn_out = ffn_out + ffn_in
ffn_out = self.ffn_norm[i](ffn_out)
bot_sub_in = ffn_out
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment