FWIW: I (@rondy) am not the creator of the content shared here, which is an excerpt from Edmond Lau's book. I simply copied and pasted it from another location and saved it as a personal note, before it gained popularity on news.ycombinator.com. Unfortunately, I cannot recall the exact origin of the original source, nor was I able to find the author's name, so I am can't provide the appropriate credits.
| from keras import backend as K, initializers, regularizers, constraints | |
| from keras.engine.topology import Layer | |
| def dot_product(x, kernel): | |
| """ | |
| Wrapper for dot product operation, in order to be compatible with both | |
| Theano and Tensorflow | |
| Args: |
| """Example TensorFlow code for Self-Attention mechanism. | |
| Refs: | |
| Attention Is All You Need | |
| Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin | |
| https://arxiv.org/abs/1706.03762 | |
| Transformer: A Novel Neural Network Architecture for Language Understanding | |
| https://research.googleblog.com/2017/08/transformer-novel-neural-network.html |