Explanatory (non-vectorized) code for how attention works

almost-attention.py

# This code doesn't work, and isn't intended to.
# The goal of this code is to explain how attention mechansisms work, in code.
# It is deliberately not vectorized to make it clearer.

def attention(self, X_in:List[Tensor]):
    # For every token transform previous layer's out
    for i in range(self.sequence_length):
        query[i] = self.Q * X_in[i]
        key[i]   = self.K * X_in[i]
        value[i] = self.V * X_in[i]

    # Compute output values, one at a time
    for i in range(self.sequence_length):
        this_query = query[i]
        # how relevant is each input to this out?
        for j in range(self.sequence_length):
            relevance[j] = this_query * key[j]
        # normalize relevance scores to sum to 1
        relevance = scaled_softmax(relevance)

        # compute a weighted sum of the values
        out[i] = 0 
        for j in range(self.sequence_length):
            out[i] += relevance[j] * value[j]

    return out

Hello Leo,

This code doesn't work, and isn't intended to.

great repo and talk
https://www.youtube.com/watch?v=S27pHKBEp30&t=382s
LSTM is dead. Long Live Transformers!
only can you share link to your or somebody else working plain only numpy python code to learn this new ideas.
Meaning full example from 0 to end with data input file and expected output example

Thank you very much on advance

Q,K,V are matrices to be learned.