Check out my blog on decoder phase of transformers here - Transformer Decoder: Forward Pass Mechanism and Key Insights (part 5).

Now, in this final installment of our series on Transformers, we go a step further—unveiling the mathematical underpinnings of backpropagation and how gradients guide learning in the training phase. We will also take a deep dive into the inference phase, where the Transformer generates text in an auto-regressive manner, one token at a time.

By the end of this blog, you will have a clear understanding of how Transformer decoders optimize learning and generate accurate sequences in real-world applications like machine translation, text generation, and beyond. Let’s get started! 🚀

Decoder - BackPropagation In the Training Phase

The weights and biases across all the various layers are learned during backpropagation. We will use Gradient descent algorithm for this. During the training phase the transformer’s output is a matrix of order \(M \times V_c\) where \(M\) is the number of tokens in the ground truth sequence and \(V_c\) the total number of unique tokens in the target vocabulary. This matrix essentially represents each token position in the ground truth as rows and each unique token in the vocabulary as a column. This means that the values in each row in this matrix denote the probability that the corresponding column token is the correct token for row position in the ground truth sequence.

We compute how far off the transformers predictions were from the ground truth using a loss function and we use gradient descent to compute how much each weight and bias contributes to this loss by finding the gradients of this loss with respect to each weight and bias. These gradients are then used to update the weights and biases. This is done until the loss is minimized to a desired value or for a set number of iterations called epochs.

Loss Function for Training Transformers for Machine Translation

The most commonly used loss function for training transformers on sequence-to-sequence tasks like machine translation is the Cross-Entropy Loss. This is a token-level loss that measures the difference between the predicted token probabilities and the ground truth tokens in the target sequence.

1. Cross-Entropy Loss

The loss for a single target sequence is computed as:

$$\mathcal{L} = -\frac{1}{T} \sum_{t=1}^T \sum_{i=1}^{|V_c|} y_t^{\text{true}}[i] \log(y_t^{\text{pred}}[i])$$

Where:

• \(T\) : Length of the target sequence.

• \(V_c\) : Size of the target vocabulary.

• \(y_t^{\text{true}}\) : One-hot encoded true token at time step .

• \(y_t^{\text{pred}}\) : Predicted probability distribution over the vocabulary for the t-th token (output of the softmax layer).

The prediction probabilities \(y_t^{\text{pred}}\) come from the decoder output after applying the softmax function.

• For each time step \(t\), the loss ploenalizes the model for predicting probabilities far from the ground truth token's one-hot encoding

• The average loss over all tokens in the sequence ensures equal contribution regardless of sequence length.

Why Cross-Entropy for Machine Translation?

• Multi-Class Output: At each time step, the decoder predicts the next token from a vocabulary of size \(|V_c|\), making it a multi-class classification problem.

• Alignment with Likelihood Maximization: Cross-entropy loss directly corresponds to minimizing the negative log-likelihood of the correct sequence.

• Efficient Backpropagation: Cross-entropy loss is differentiable, enabling gradient computation for weight updates during backpropagation.

How It Works for Each Position

1. At timestep \(t\):

   • The decoder generates \(y_t^{\text{pred}}\) , a probability distribution over the vocabulary using a softmax layer.

   • The ground truth token for \(t\) , \(y_t^{\text{true}}\) is a one-hot vector where the index corresponding to the correct token is 1, and the rest are 0.

2. The Cross-Entropy Loss computes:

   $$\mathcal{L}t = - \sum{i=1}^{|V_c|} y_t^{\text{true}}[i] \log(y_t^{\text{pred}}[i])$$

   • Since \(y_t^{\text{true}}\) is one-hot encoded, this simplifies to:

     $$\mathcal{L}_t = - \log(y_t^{\text{pred}}[\text{correct_index}])$$

   • Only the probability assigned to the correct token contributes to the loss.

3. The total loss over the sequence is the average of these token-level losses:

   $$\mathcal{L} = -\frac{1}{T} \sum_{t=1}^T \log(y_t^{\text{pred}}[\text{correct_index}])$$

Computing gradients

In a simplified Transformer model where we have 1 encoder, 1 attention head, 1 feedforward layer, 1 decoder, 1 masked attention head, 1 encoder-decoder attention layer, 1 feedforward layer, and 1 final output layer, the gradients that need to be computed and the corresponding chain rule equations are listed below:

Gradients to Compute in Each Layer

1. Encoder

• Multi-Head Self-Attention Layer:

  • Gradients for Query, Key, and Value weights ( \(W_Q, W_K, W_V\) ):

    $$\frac{\partial \mathcal{L}}{\partial W_Q},\frac{\partial \mathcal{L}}{\partial W_K} , \frac{\partial \mathcal{L}}{\partial W_V}$$

  • Gradient for the attention projection weight ( \(W_O\) ):

    $$\frac{\partial \mathcal{L}}{\partial W_O}$$

• Feedforward Layer:

  • Gradients for the two feedforward weights ( \(W_1, W_2\)):

    $$\frac{\partial \mathcal{L}}{\partial W_1}, \frac{\partial \mathcal{L}}{\partial W_2}$$

  • Gradients for biases ( \(b_1, b_2\)):

    $$\frac{\partial \mathcal{L}}{\partial b_1},\frac{\partial \mathcal{L}}{\partial b_2}$$

2. Decoder

• Masked Multi-Head Attention Layer:

  • Gradients for Query, Key, and Value weights ( \(W_Q^{\text{masked}}, W_K^{\text{masked}}, W_V^{\text{masked}}\)):

    $$\frac{\partial \mathcal{L}}{\partial W_Q^{\text{masked}}},\frac{\partial \mathcal{L}}{\partial W_K^{\text{masked}}},\frac{\partial \mathcal{L}}{\partial W_V^{\text{masked}}}$$

  • Gradient for the attention projection weight ( \(W_O^{\text{masked}}\)):

    $$\frac{\partial \mathcal{L}}{\partial W_O^{\text{masked}}}$$

• Encoder-Decoder Attention Layer:

  • Gradients for Query, Key, and Value weights ( \(W_Q^{\text{enc-dec}}, W_K^{\text{enc-dec}}, W_V^{\text{enc-dec}}\)):

    $$\frac{\partial \mathcal{L}}{\partial W_Q^{\text{enc-dec}}},\frac{\partial \mathcal{L}}{\partial W_K^{\text{enc-dec}}},\frac{\partial \mathcal{L}}{\partial W_V^{\text{enc-dec}}}$$

  • Gradient for the attention projection weight ( \(W_O^{\text{enc-dec}}\)):

    $$\frac{\partial \mathcal{L}}{\partial W_O^{\text{enc-dec}}}$$

• Feedforward Layer:

  • Gradients for the two feedforward weights ( \(W_1^{\text{dec}}, W_2^{\text{dec}}\)):

    $$\frac{\partial \mathcal{L}}{\partial W_1^{\text{dec}}},\frac{\partial \mathcal{L}}{\partial W_2^{\text{dec}}}$$

  • Gradients for biases ( \(b_1^{\text{dec}}, b_2^{\text{dec}}\)):

    $$\frac{\partial \mathcal{L}}{\partial b_1^{\text{dec}}},\frac{\partial \mathcal{L}}{\partial b_2^{\text{dec}}}$$

• Final Output Layer:

  • Gradient for the output projection weight ( \(W_{\text{out}}\) ):

    $$\frac{\partial \mathcal{L}}{\partial W_{\text{out}}}$$

  • Gradient for the output bias ( \(b_{\text{out}}\) ):

    $$\frac{\partial \mathcal{L}}{\partial b_{\text{out}}}$$

Chain Rule Equations for Each Gradient

1. Final Output Layer Gradients:

   • Output weights ( \(W_{\text{out}}\) ):

     $$\frac{\partial \mathcal{L}}{\partial W_{\text{out}}} = \frac{\partial \mathcal{L}}{\partial y_t^{\text{pred}}} \cdot \frac{\partial y_t^{\text{pred}}}{\partial z_t} \cdot \frac{\partial zt}{\partial W{\text{out}}}$$

     Where:

     • \(z_t\) : Logits before the softmax.

   • Output bias ( \(b_{\text{out}}\) ):

     $$\frac{\partial \mathcal{L}}{\partial b_{\text{out}}} = \frac{\partial \mathcal{L}}{\partial y_t^{\text{pred}}} \cdot \frac{\partial y_t^{\text{pred}}}{\partial z_t} \cdot \frac{\partial zt}{\partial b{\text{out}}}$$

2. Decoder Feedforward Gradients:

   • First layer weights ( \(W_1^{\text{dec}}\) ):

     $$\frac{\partial \mathcal{L}}{\partial W_1^{\text{dec}}} = \frac{\partial \mathcal{L}}{\partial h_2} \cdot \frac{\partial h_2}{\partial h_1} \cdot \frac{\partial h_1}{\partial W_1^{\text{dec}}}$$

   • Second layer weights ( \(W_2^{\text{dec}}\) ):

     $$\frac{\partial \mathcal{L}}{\partial W_2^{\text{dec}}} = \frac{\partial \mathcal{L}}{\partial h_2} \cdot \frac{\partial h_2}{\partial W_2^{\text{dec}}}$$

3. Encoder-Decoder Attention Gradients:

   • Query weights ( \(W_Q^{\text{enc-dec}}\) ):

     $$\frac{\partial \mathcal{L}}{\partial W_Q^{\text{enc-dec}}} = \frac{\partial \mathcal{L}}{\partial \alpha} \cdot \frac{\partial \alpha}{\partial Q} \cdot \frac{\partial Q}{\partial W_Q^{\text{enc-dec}}}$$

   • Similarly for \(W_K^{\text{enc-dec}}, W_V^{\text{enc-dec}},W_O^{\text{enc-dec}}\).

4. Masked Multi-Head Self-Attention Gradients:

   • Same formulas as for Encoder-Decoder Attention, but keys and values come from the decoder's \(T_{\text{input}}\).

5. Encoder Feedforward and Attention Gradients:

   • Identical to decoder feedforward and attention gradients, applied to the encoder's hidden states.

Weight and Bias Updation Rule

The weights and biases are updated using the formula

$$quantity=quantity- \alpha \times \frac{\partial \mathcal{L}}{\partial quantity}$$

Minimizing loss

The Transformer computes the forward pass again with the updated weights and biases. The losses and gradients are recomputed and the weights are updated again. This process repeats until we minimize the loss to a desired value or for a fixed number of epochs.

Now that we have understood the training phase, let’s now look at how the transformer works in the Inference Phase.

Inference Phase

During inference, the encoder in the Transformer functions the same way as during training. It performs the following operations:

• Computes the attention outputs.

• Passes them through a feedforward network (FFN).

• Adds residual connections and applies layer normalization.

The key difference between training and inference lies in how the decoder operates. Unlike training, during inference, the decoder does not have access to the ground truth target sequence. Instead, it must generate the target sequence one token at a time cyclically. This mechanism is referred to as Auto Regressive Generation.

Auto-regressive generation is a step-by-step process where a model generates output one token at a time. Each newly generated token is fed back into the model as input, along with previous tokens, until the entire sequence is produced.

How the Decoder Functions During Inference:

1. Initialization:

   The decoder begins with only the start-of-sequence (SOS) token as input.

2. Token Generation Cycle:

   • At each timestep, the decoder generates one token, appends it to the current sequence, and uses the updated sequence as the input for the next timestep.

   • This process repeats until the end-of-sequence (EOS) token is generated or a predefined maximum sequence length is reached.

Step-by-Step Breakdown Of How the Decoder Generates Text in Inference

Let’s illustrate how a Transformer decoder works during inference with an example:
Task: Translate "Varun loves cricket and finance." into French.
Assumption: The encoder has already processed the sentence and produced contextual embeddings.

Now, let’s go through the inference process step by step.

1. Step 1 (Decoder Initialization):

   • The decoder receives the SOS token as input, represented as a matrix of size \(1 \times D\), where \(D\) is the embedding dimension.

   • The masked attention layer processes this input. Since only the SOS token is present, no masking is applied at this stage.

   • The alignment scores are computed, these scores are then normalized using the softmax function, and passed through the subsequent decoder layers just like in training phase.

   • The output layer projects the final representation over the vocabulary space to generate a \(1×V_c\) matrix, where \(V_c\) is the size of the vocabulary. This matrix contains the probabilities of all tokens in the vocabulary.

   • The token with the highest probability is selected as the first output token ( \(y_1\) ), which is then appended to the SOS token.

   • Example Output for Step 1:

     • The decoder receives the input [SOS] , the decoder generates the first token (e.g., "Varun"), the decoder takes this token and appends it to the input sequence.

     • The updated sequence now becomes a matrix with:

       • First row: [SOS] token.

       • Second row: The first generated token, e.g., "Varun".

         

2. Step 2 (Next Cycle):

   • Input: The decoder now receives the updated sequence \([SOS,y_1]\), represented as a matrix of size \(2×D\).

   • The decoder computes the query (Q), key (K), and value (V) matrices for this input and updates the mask matrix to enforce causal masking.

   • The updated mask matrix for two tokens looks like this:

     

   • The masked alignment scores are calculated, normalized, and passed through the remaining layers. The next token ( \(y_2\)) is predicted and appended to the sequence.

   • Example Output for Step 2:

     • The Input to the decoder is now → [SOS, "Varun"].

     • Masked attention processes both tokens and the Encoder-decoder attention attends to the encoder output. The decoder then predicts the second token, e.g., "aime."

     • The output at this step is now [SOS, "Varun", "aime"]

3. Subsequent Steps:

   • The process repeats, with the input sequence growing at each step ( \([SOS,y_1,y_2,…]\) ), and the mask matrix dynamically expands to ensure each token attends only to itself and previous tokens.

   • For example:

     • Cycle 3: The mask matrix for three tokens would be:

       

   • The output for the example we considered given the input to the decoder is → [SOS, "Varun", "aime"] will be [SOS, "Varun", "aime", "le"].

4. Termination:

   • The process continues until the decoder generates the EOS token or the maximum sequence length is reached.

   • The final output for the example we considered will be → "Varun aime le cricket et la finance." We obtain this output by a process called detokenization wherein the generated tokens are combined together and then returned as the output.

Result: The decoder outputs the French translation: "Varun aime le cricket et la finance."

Key Difference Between Training and Inference Phase Of The Decoder

During the inference phase, the key difference in the decoder lies in the absence of the ground truth target sequence. Instead, the decoder generates the target sequence token by token, using its own previously generated tokens as input at each decoding step.

Here’s how it differs from the training phase:

1. Input to the Decoder

• Training Phase:

  • The decoder receives the entire ground truth target sequence (e.g., in French: "Varun aime le cricket et la finance.") as input for the Masked Multi-Head Attention.

  • Masking ensures that the decoder can only attend to tokens up to the current timestep.

• Inference Phase:

  • The decoder starts with only the start-of-sequence (SOS) token as input.

  • At each timestep, the decoder generates a token, appends it to the sequence, and feeds the updated sequence back as input.

  • This process repeats until the end-of-sequence (EOS) token is generated or a predefined maximum sequence length is reached.

  • For example:

    • Step 1: [SOS]

    • Step 2: [SOS, "Varun"]

    • Step 3: [SOS, "Varun", "aime"], and so on.

2. Sequence Generation

• Training Phase:

  • The entire sequence is available at once, so the decoder can compute all tokens in parallel during a single forward pass.

• Inference Phase:

  • Tokens are generated sequentially, one at a time, in an autoregressive manner.

  • At timestep \(t\), the decoder uses:

    • The encoder output.

    • The previously generated sequence .

      \([SOS, y_1, y_2, \dots, y_{t-1}]\)

    • The attention mechanism to predict the next token \(y_t\).

3. Parallel vs. Sequential Processing

• Training Phase:

  • The decoder processes the target sequence in parallel, allowing for efficient computation.

• Inference Phase:

  • The decoder operates sequentially, generating one token at a time. This is computationally slower compared to training.

4. Predictions

• Training Phase:

  • The model predicts token probabilities for all positions simultaneously, compared against the ground truth using the Cross-Entropy Loss.

• Inference Phase:

  • At each timestep, the decoder outputs a probability distribution over the vocabulary for the next token.

  • The token with the highest probability (or another strategy like beam search) is selected and added to the generated sequence.

5. Dynamic Masking

• Training Phase:

  • The model uses a static mask to compute the masked attention scores.

• Inference Phase:

  • The mask matrix dynamically adjusts to ensure that each token can only attend to itself and the tokens before it (causal masking).

  • For example:

    • Cycle 1 (only SOS token):

      $$\text{Mask} = [0]$$

    • Cycle 2 (SOS + 1st generated token):

      $$\text{Mask} = \begin{bmatrix} 0 & -\infty \ 0 & 0 \end{bmatrix}$$

    • Cycle 3 (SOS + 1st + 2nd generated tokens):

      $$\text{Mask} = \begin{bmatrix} 0 & -\infty & -\infty \ 0 & 0 & -\infty \ 0 & 0 & 0 \end{bmatrix}$$

Conclusion

And with that, we conclude our deep dive into Transformer decoders! Throughout this series, we have uncovered the inner workings of the Transformer, from how it processes input sequences to how it learns through backpropagation and generates outputs during inference.

Key Takeaways:

• The training phase involves computing gradients and updating weights using backpropagation to minimize loss.

• The inference phase follows an auto-regressive generation approach, where tokens are produced sequentially rather than in parallel.

• Masking techniques ensure that the model processes only relevant context at each step, preventing "cheating" by looking ahead.

Transformers have revolutionized NLP, powering models like GPT, BERT, and T5. But as the field evolves, we’re seeing even more efficient architectures like Mixture of Experts (MoE) models, Retrieval-Augmented Transformers (RAG), and multimodal AI systems shaping the future.

I hope this series has provided you with a solid intuitive and mathematical understanding of how Transformers work. If you have any thoughts, questions, or ideas for future deep dives, feel free to share them!

Computing the Encoder-Decoder Cross Attention

The next component of the Decoder is a Encoder-Decoder attention.

The encoder-decoder cross-attention mechanism connects the encoder’s contextualized representations of the input sequence with the decoder’s generation of the output sequence. It teaches the decoder to map the enriched encoder outputs to the ground truth passed to the decoder during teacher forcing.

This layer is specially designed to determine how much attention each token in the ground truth sequence should pay to each token and dimension in the context-rich input sequence produced by the encoder.

There can be multiple layers of encoder-decoder attention for our example we are considering one.

1. Purpose of the Encoder-Decoder Attention Layer

1.1 Bridge Between Encoder and Decoder

• The encoder processes the input (e.g., English sentence) and generates context-rich embeddings for each token.

• The decoder generates the output (e.g., French translation) using this contextual understanding.

• The encoder-decoder attention layer allows the decoder to retrieve relevant information from the encoder at each decoding step.

1.2 Focus on Relevant Parts of the Input

• Each target token should attend to the most relevant source tokens.

• For example, while translating "cricket", the decoder should focus on "criquet" in the source sequence.

2. Inputs to Encoder-Decoder Attention

Each Encoder-Decoder head takes 2 inputs:

1. The masked attention output \(Q_d\)

   • Serves as the query vector representing the partially generated target sequence.

   • The Query vector is derived from the output of the masked self-attention layer in the decoder.

   • It represents the questions being asked by the current state of the ground truth sequence (or the partially generated target sequence) to map the context-rich input sequence from the encoder to the tokens in the target sequence.

2. The encoder output.

   • Serves as the key ( \(K_d\) ) and value ( \(V_d\) ) vectors, holding contextual embeddings of the source sequence.

   • The key matrix ( \(K_d\) ) represents the metadata about the encoder’s context-rich output.The value matrix ( \(V_d\) ) represents the actual content or information in the encoder’s context-rich output that is relevant for the task.

   • The Key and Value vectors are computed from the encoder’s output.

   • These are created by multiplying the encoder's output with learned weight matrices \(W_K\) and \(W_V\) respectively, which are of size \(D × (\frac{D}{h})\) where:

     • \(D\) is the embedding dimension.

     • \(h\) is the number of attention heads.

   • These inputs are used to derive the query, key and value vectors for the encoder-decoder attention layer denoted by \(Q_d\), \(K_d\) and \(V_d\) for each such layer.

where \(Q_d\), \(K_d\) and \(V_d\) are computed as

$$\begin{aligned} Q_d &= \text{Masked Attention Output} \times W_{Qd} \\ K_d &= \text{Encoder Output} \times W_{Kd} \\ V_d &= \text{Encoder Output} \times W_{Vd} \end{aligned}$$

The Query (decoder’s contextualized understanding) interacts with the Keys (metadata from the encoder) to compute alignment scores that identify which parts of the input sequence are most relevant to the current decoding step. The Values (actual content from the encoder) provide the information needed for the decoder to generate the target token based on these alignment scores.

The Weight Matrix \(W_{Qd}\), \(W_{Kd}\) and \(W_{Vd}\) used in the example we considered are given below

$$\begin{array}{c c c} \textbf{Query Weight Matrix } (W_Q) & \textbf{Key Weight Matrix } (W_K) & \textbf{Value Weight Matrix } (W_V) \\ \begin{bmatrix} 0.3745 & 0.9507 & 0.7320 \\ 0.5987 & 0.1560 & 0.1560 \\ 0.0581 & 0.8662 & 0.6011 \end{bmatrix} & \begin{bmatrix} 0.7081 & 0.0206 & 0.9699 \\ 0.8324 & 0.2123 & 0.1818 \\ 0.1834 & 0.3042 & 0.5248 \end{bmatrix} & \begin{bmatrix} 0.4320 & 0.2912 & 0.6119 \\ 0.1395 & 0.2921 & 0.3664 \\ 0.4561 & 0.7852 & 0.1997 \end{bmatrix} \end{array}$$

The \(Q_d\), \(K_d\) and \(V_d\) vectors obtained are as follows :

$$\begin{array}{c c c} {Q_d} & {K_d} & {V_d} \\ \begin{bmatrix} 1.2040 & 1.6229 & 1.3741 \\ 1.1642 & 1.5249 & 1.3439 \\ 1.2093 & 1.6202 & 1.3871 \\ 1.1860 & 1.5555 & 1.3704 \\ 0.7762 & 1.0511 & 0.8792 \\ 0.7435 & 0.9855 & 0.8543 \\ 0.7554 & 1.0133 & 0.8733 \\ 0.7516 & 0.9917 & 0.8655 \\ -0.0720 & -0.4547 & 0.1067 \\ 0.6259 & 1.0474 & 0.6374 \\ 0.2013 & 0.1109 & 0.1203 \\ 0.3878 & 0.7875 & 0.3276 \end{bmatrix} & \begin{bmatrix} 2.0255 & 0.6438 & 1.9469 \\ 1.9487 & 0.5897 & 1.9233 \\ 2.0405 & 0.6402 & 1.9653 \\ 1.9904 & 0.6028 & 1.9579 \\ 1.2949 & 0.4174 & 1.2496 \\ 1.2471 & 0.3852 & 1.2177 \\ 1.2920 & 0.4027 & 1.2280 \\ 1.2606 & 0.3862 & 1.2350 \\ -0.0428 & -0.2864 & 0.2265 \\ 1.0807 & 0.4835 & 0.8234 \\ -0.0858 & -0.0521 & 0.4378 \\ 0.6533 & 0.3971 & 0.3785 \end{bmatrix} & \begin{bmatrix} 1.2040 & 1.6229 & 1.3741 \\ 1.1642 & 1.5249 & 1.3439 \\ 1.2093 & 1.6202 & 1.3871 \\ 1.1860 & 1.5555 & 1.3704 \\ 0.7762 & 1.0511 & 0.8792 \\ 0.7435 & 0.9855 & 0.8543 \\ 0.7554 & 1.0133 & 0.8733 \\ 0.7516 & 0.9917 & 0.8655 \\ -0.0720 & -0.4547 & 0.1067 \\ 0.6259 & 1.0474 & 0.6374 \\ 0.2013 & 0.1109 & 0.1203 \\ 0.3878 & 0.7875 & 0.3276 \end{bmatrix} \end{array}$$

3. Compute the Alignment Score

The alignment score determines how much each target token attends to each source token.

The Alignment Score is computed as follows

$$Score(Q,K)=\frac{Q.K^{T}}{\sqrt{\frac{d}{\text{No. of Heads}}}}$$

The computed alignment scores are stored in a matrix:

$$\textbf{Alignment Scores} = \begin{bmatrix} 2.85 & 2.75 & 2.87 & \dots & 1.78 & -0.12 & 1.50 & 0.18 & 0.92 \\ 4.01 & 3.86 & 4.03 & \dots & 2.50 & -0.16 & 2.11 & 0.25 & 1.29 \\ 5.30 & 5.11 & 5.33 & \dots & 3.30 & -0.22 & 2.79 & 0.34 & 1.71 \\ \vdots & \vdots & \vdots & \ddots & \vdots & \vdots & \vdots & \vdots & \vdots \\ 4.37 & 4.22 & 4.40 & \dots & 2.73 & -0.18 & 2.30 & 0.28 & 1.41 \end{bmatrix}$$

4. Normalize the output using Softmax

We apply the softmax function to normalize alignment scores into probabilities.

$$α=Softmax(Score(Q, K))$$

The computed softmax weights:

$$\alpha = \begin{bmatrix} 0.17 & 0.16 & 0.18 & 0.16 & 0.06 & 0.06 & 0.06 & 0.01 & 0.04 & 0.01 & 0.03 \\ 0.20 & 0.17 & 0.21 & 0.19 & 0.05 & 0.04 & 0.05 & 0.00 & 0.03 & 0.00 & 0.01 \\ 0.22 & 0.19 & 0.23 & 0.21 & 0.03 & 0.03 & 0.03 & 0.02 & 0.00 & 0.00 & 0.01 \\ \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots \\ 0.21 & 0.18 & 0.22 & 0.19 & 0.04 & 0.04 & 0.04 & 0.00 & 0.02 & 0.00 & 0.01 \end{bmatrix}$$

5. Computing the Attention Output

The attention output is computed as:

$$\text{Attention Output}=α⋅V_d​$$

The computed attention output matrix:

$$\text{Attention Output} = \alpha \cdot V = \begin{bmatrix} 1.02 & 1.37 & 1.16 \\ 1.08 & 1.44 & 1.23 \\ 1.12 & 1.49 & 1.28 \\ 1.12 & 1.50 & 1.29 \\ 1.10 & 1.46 & 1.26 \\ 0.99 & 1.33 & 1.13 \\ 1.09 & 1.45 & 1.25 \\ 1.09 & 1.46 & 1.25 \\ 1.09 & 1.45 & 1.25 \\ 1.09 & 1.46 & 1.25 \end{bmatrix}$$

6. Projecting the Attention Output

The attention output is projected back into the embedding space using a projection weight matrix(\(W_{edO}\)).

$$\text{Projected Output} = \text{Attention Output} \cdot W_{edO}$$

• Example Projection Matrix \(W_{edO}\) :

  $$W_{edO} = \begin{bmatrix} 0.3745 & 0.9507 & 0.7320 \ 0.5987 & 0.1560 & 0.1560 \ 0.0581 & 0.8662 & 0.6011 \end{bmatrix}$$

• The projected output obtained by projecting back the attention output to the original embedding dimension D is as follows :

  

7. Feed Forward Network (FFN)

The projected output passes through an FFN:

$$FFN_{output} = (ReLU(\text{Final Attention Output}\times W_{d1}+b_{d1})\times W_{d2} +b_{d2})$$

where \(W_{d1}\) and \(W_{d2}\) are again weight matrices and \(b_{d1}\) and \(b_{d2}\) are the biases.

Why is the Feed-Forward Network (FFN) Applied?

The Feed-Forward Network (FFN) acts as a final refinement layer that enhances the decoder’s representations before making predictions. It ensures that each token is processed individually while preserving the broader context learned from attention mechanisms.

1. Fine-Tuning Grammar & Meaning

• The FFN helps polish sentence structure by refining syntax (word order) and semantics (meaning).

• In machine translation, this ensures fluent and grammatically correct output.

2. Improving Token-Specific Details

• Unlike attention, which focuses on relationships between tokens, the FFN processes each token independently.

• This allows for word-level adjustments, making sure each token is represented accurately.

3. Capturing Complex Patterns

• The FFN introduces nonlinear transformations, enabling it to learn intricate relationships like:

  • Tense, gender, and plurality adjustments in translations.

  • Handling idioms and nuanced expressions that require deep contextual understanding.

📌 Think of the FFN as a final polish—refining each token’s representation to ensure clarity, coherence, and natural flow before making a prediction.

FeedForward Network Output

The output of the FFN after both transformations is:

$$\text{FFN Output} = \begin{bmatrix} 9.4743 & 8.7558 & 5.8188 \\ 9.9507 & 9.1566 & 6.1009 \\ 10.3006 & 9.4511 & 6.3083 \\ 10.3164 & 9.4643 & 6.3176 \\ 10.1041 & 9.2857 & 6.1918 \\ 9.2654 & 8.5799 & 5.6952 \\ 10.0444 & 9.2355 & 6.1564 \\ 10.0896 & 9.2735 & 6.1832 \\ 10.0366 & 9.2289 & 6.1518 \\ 10.0658 & 9.2535 & 6.1691 \end{bmatrix}$$

8.Residual Connection Output: Preserving and Enriching Context

The residual connection enhances the decoder’s output by adding the projected encoder-decoder attention output back to the FFN output.

Why is this Important?

1. Preserving Context from Both Sequences

   • The decoder generates the output step by step, and it must retain:

     • Token-to-token relationships within the target sequence (from masked attention).

     • Context alignment between the source and target sequences.

   • The residual connection ensures that both local (target) and global (source) contexts are maintained.

2. Enriching Contextual Representations

   • Encoder-decoder attention focuses on aligning source and target tokens.

   • The residual connection merges this alignment with the decoder’s evolving context, refining the representation for better coherence.

3. Preparing for Further Processing

   • The combined output is passed to the FFN, where:

     • It undergoes grammatical and semantic refinement.

     • Each token’s representation is further enhanced for the next processing stage.

📌 In short, the residual connection prevents information loss, strengthens alignment, and ensures the decoder’s output remains rich in context and meaning.

How is the residual connection computed ?

The residual output obtained is as follows :

$$\text{Residual Output} = \text{Projected Encoder-Decoder Attention Output} + \text{FFN Output}$$

The residual output obtained is as follows :

$$\text{Residual Output} = \begin{bmatrix} 10.7412 & 10.9458 & 7.4774 \\ 11.2877 & 11.4733 & 7.8551 \\ 11.6887 & 11.8611 & 8.1329 \\ 11.7068 & 11.8785 & 8.1454 \\ 11.4635 & 11.6433 & 7.9769 \\ 10.5013 & 10.7145 & 7.3119 \\ 11.3951 & 11.5772 & 7.9295 \\ 11.4469 & 11.6272 & 7.9654 \\ 11.3861 & 11.5685 & 7.9233 \\ 11.4196 & 11.6009 & 7.9465 \end{bmatrix}$$

9. Layer-Normalized Output

The final step involves normalizing the residual output. Layer normalization ensures that the values in a layer's output have a consistent scale and distribution, which is critical for stable and efficient training in neural networks, including transformers.

Why Is Layer Normalization Important ?

1. Consistent Scale:

   • By normalizing the outputs, layer normalization ensures that the scale of values remains consistent across tokens and dimensions, preventing instability caused by large or small activations.

2. Stable Training:

   • Normalized outputs reduce the likelihood of exploding or vanishing gradients during backpropagation, making training more stable.

3. Effective Gradient Flow:

   • Consistent scaling across layers ensures smoother and more predictable gradient updates, improving the model's ability to converge.

4. Token Independence:

   • In transformers, layer normalization is applied independently for each token. This allows the model to normalize the representation of each token without being affected by other tokens in the sequence.

The resulting matrix obtained, after applying layer normalization to the residual output is:

$$\text{Layer Normalized Output} = \begin{bmatrix} 0.6417 & 0.7705 & -1.4123 \\ 0.6506 & 0.7622 & -1.4127 \\ 0.6564 & 0.7567 & -1.4130 \\ 0.6566 & 0.7564 & -1.4130 \\ 0.6532 & 0.7597 & -1.4129 \\ 0.6375 & 0.7745 & -1.4120 \\ 0.6522 & 0.7607 & -1.4128 \\ 0.6529 & 0.7599 & -1.4129 \\ 0.6520 & 0.7608 & -1.4128 \\ 0.6525 & 0.7603 & -1.4128 \end{bmatrix}$$

Layer normalization marks the end of forward propagation in the decoder, leading to the final step—projecting the normalized output onto the target vocabulary space to predict the next token.

Final Projection to the Vocabulary Space

The final step in the decoder is to project the layer-normalized output into the target vocabulary space to generate predictions for the next token.

How Does This Projection Work?

1. Decoder Output Transformation

   • The decoder produces an output matrix of size \(M \times D\), where:

     • \(M\) is the number of tokens in the generated sequence.

     • \(D\) is the embedding dimension.

2. Mapping to Vocabulary Space

   • A learnable weight matrix \(W_{\text{vocab}} \) (of size \(D \times V_c\), where \(V_c\) is the vocabulary size) transforms the decoder's output into a probability distribution over all possible tokens.

   • Each token representation is projected from dimension \(D\) → dimension \(V_c\), resulting in a matrix of size \(M \times V_c\).

Purpose of the Projection

• Aligns the decoder’s contextualized output with the target vocabulary.

• Produces a probability distribution over the vocabulary for each token in the sequence.

• Ensures the model can sample or select the most likely next token during generation.

Computing Logits & Softmax

The projected output, referred to as logits, is computed as:

$$\text{logits} = \text{Layer Normalized Output} \times W_{\text{vocab}} + b_{\text{vocab}}$$

These logits are then passed through a softmax function to obtain probabilities for each token is

• Each row in the resulting matrix represents the probability distribution over the vocabulary for a given token position in the output sequence.

• This is nothing but the probability of each token(column) in the target vocabulary being the correct token for that position in the target sequence.

• The highest probability token is selected as the predicted output.

Why Softmax?

• Converts raw logits into interpretable probability scores.

• Ensures that the sum of probabilities across the vocabulary is 1 for each token position.

• Helps in selecting the most likely next token during training and inference.

Conclusion & Next Steps

With the forward propagation in the Transformer decoder complete, we now have a detailed understanding of how the model processes information at each step. The decoder retrieves relevant context from the encoder, applies attention mechanisms, and refines token representations before making predictions.

However, the Transformer is only as good as its training process. To improve accuracy and minimize prediction errors, we must update the model’s parameters using backpropagation. In the next and final part of this series, we will delve into the backpropagation process and inference phase, exploring how gradients are computed, weights are adjusted, and how the model ultimately learns to generate high-quality outputs.

Stay tuned for the next blog, where we unravel the final part of the crucial training phase of Transformers! 🚀

In a Transformer model, the decoder receives input from the previous decoder layer, refining and transforming it progressively through multiple layers. This process plays a crucial role in both the training phase and inference phase. While inference involves generating predictions step by step, training requires forward propagation to adjust model weights based on loss calculations.

In this blog, we will focus on the forward propagation mechanism during the training phase of the Transformer decoder. We will break down key operations, including self-attention, cross-attention, and feed-forward networks, to understand how data flows through the decoder during training.

Let’s get started!

Understanding the Decoder in the Training Phase

The Transformer decoder operates differently during the training phase compared to the inference phase. In training, it employs a strategy known as Teacher Forcing, which significantly influences how the model learns sequence relationships.

What is Teacher Forcing?

Teacher forcing is a training technique commonly used in sequence-to-sequence (Seq2Seq) models. Instead of using the model’s own predictions as input for the next step, the actual ground truth sequence (target sequence) is provided.

Why Is Teacher Forcing Used?

1. Stabilizes Training – Early in training, the model’s predictions are often inaccurate. By using the correct target sequence as input, teacher forcing prevents error accumulation and allows the model to learn effectively.

2. Faster Convergence – The model is guided by the correct sequence, helping it reach an optimal solution more quickly.

3. Focus on Learning Relationships – By eliminating errors from previous predictions, the model learns the relationships between input and output sequences rather than struggling with its own mistakes.

Decoder Operations During Forward Propagation

During training, the decoder follows a specific sequence of operations at each step:

1. Masked Self-Attention – Ensures the model attends only to previous tokens, preventing information leakage from future tokens.

2. Cross-Attention – Integrates the encoder’s output with the decoder’s partially generated sequence to align the input and output.

3. Feed-Forward Layer – Applies non-linear transformations to extract meaningful features.

4. Residual Connection – Aids in stable gradient flow, preventing vanishing gradients.

5. Layer Normalization – Stabilizes training by maintaining consistent activation distributions.

To fully grasp how the decoder processes input, let's first understand what inputs it receives and how they are prepared.

Preparing the Target Sequence for the Decoder

The decoder receives two key inputs:

1. The encoder output – Encapsulates the contextualized representation of the input sequence.

2. A partially generated ground truth sequence – A modified version of the target sequence that helps guide training.

Example: Input and Target Sequence

English Input (Source Sequence)

"My name is Vikas. I love cricket, finance, and AI."

French Translation (Target Sequence)

"Mon nom est Vikas. J'aime le cricket, la finance et l'IA."

However, the decoder does not directly use the target sequence as input. Instead, it shifts the target sequence to ensure an auto-regressive learning process.

Steps to Create the Shifted Target Sequence

To train the decoder properly, the target sequence is shifted to the right:

• A special <start> (start-of-sequence) token is added at the beginning.

• The last token is removed to maintain the same sequence length.

Example: Shifted Target Sequence

Original Target Sequence (Ground Truth)

["Mon", "nom", "est", "Vikas.", "J'aime", "le", "cricket,", "la", "finance", "et", "l'IA", "."]

Shifted Target Sequence (Decoder Input)

["<start>", "Mon", "nom", "est", "Vikas.", "J'aime", "le", "cricket,", "la", "finance", "et", "l'IA"]

Why Shift the Target Sequence?

This shifted input ensures that, at each decoding step, the model generates the next token using only the previous tokens.

• When generating "Mon", the model sees only "".

• When generating "nom", the model sees "" and "Mon".

• When generating "est", the model sees "", "Mon", and "nom".

This setup enforces an auto-regressive approach, mimicking how the decoder will function during inference while still leveraging teacher forcing for stable training.

Converting to Vector Embeddings

Before the decoder can process the target sequence, it must be converted into vector embeddings with the same dimensionality as the encoder output. This transformation ensures that each token is represented in a high-dimensional space, enabling the model to capture semantic relationships.

The vector embedding of the shifted target sequence results in a matrix of order \(M \times D\)[ Qd=Masked Attention Output×WQd Kd=Encoder Output×WKd Vd=Encoder Output×WVd ], where:

• \(M\) represents the number of tokens in the target sequence.

• \(D\) is the embedding dimension.

This embedded target sequence is denoted as \(T_{input}\).

$$T_{\text{input}} = \begin{bmatrix} 0.45 & 0.67 & -0.12 \\ 0.80 & 0.05 & -0.25 \\ 0.60 & 0.90 & 0.02 \\ 0.85 & 0.10 & 0.95 \\ 0.12 & 0.22 & 0.55 \\ -0.10 & 0.03 & -0.07 \\ 0.40 & 0.75 & -0.20 \\ 0.15 & -0.01 & 0.04 \\ -0.20 & -0.10 & 0.04 \\ 0.30 & 0.40 & 0.08 \end{bmatrix}$$

📌 For a detailed explanation of how token embeddings are generated, refer to How Transformers Work: Tokenization Embeddings and Positional Encoding Explained (Part 1)

Adding Positional Encodings

Since Transformers do not have a built-in notion of word order, we must add positional encodings to the embedded target sequence. These positional encodings inject sequential information into the embeddings, allowing the model to understand the relative position of words in a sentence.

The positional encodings are generated using the same sinusoidal function-based method discussed in the encoder. Adding these encodings results in the partially generated target sequence, which is then fed into the decoder.

Below is the matrix representing the positional encodings used in this example, along with the matrix obtained after adding them to the embeddings.

$$PE = \begin{bmatrix} 0.0000 & 1.0000 & 0.0000 \\ 0.8415 & 0.9989 & 0.0022 \\ 0.9093 & 0.9957 & 0.0043 \\ 0.1411 & 0.9903 & 0.0065 \\ -0.7568 & 0.9828 & 0.0086 \\ -0.9589 & 0.9732 & 0.0108 \\ -0.2794 & 0.9615 & 0.0129 \\ 0.6570 & 0.9477 & 0.0151 \\ 0.9894 & 0.9318 & 0.0172 \\ 0.4121 & 0.9140 & 0.0194 \end{bmatrix} \quad T_{\text{input}} + PE = \begin{bmatrix} 0.45 & 1.67 & -0.12 \\ 1.6415 & 1.0489 & -0.2478 \\ 1.5093 & 1.8957 & 0.0243 \\ 0.9911 & 1.0903 & 0.9565 \\ -0.6368 & 1.2028 & 0.5586 \\ -1.0589 & 1.0032 & -0.0592 \\ 0.1206 & 1.7115 & -0.1871 \\ 0.8070 & 0.9377 & 0.0551 \\ 0.7894 & 0.8318 & 0.0572 \\ 0.7121 & 1.3140 & 0.0994 \end{bmatrix}$$

📌 To understand how positional encodings are computed and why they are necessary, refer to How Transformers Work: Tokenization Embeddings and Positional Encoding Explained (Part 1).

Masked Self-Attention in the Decoder

Now that we have embedded the target sequence and added positional encodings, the next step is to understand how the decoder processes this sequence using Masked Self-Attention. This mechanism ensures the model predicts tokens one step at a time without seeing future words, which is crucial for language generation tasks like translation and text generation.

What is Masked Self-Attention?

Masked self-attention is a specialized form of the attention mechanism, used exclusively in the decoder of a Transformer. It ensures that, when predicting a token, the model only attends to previous tokens and not future ones.

Without masking, the model could "cheat" during training by using future tokens, which wouldn't be available during inference. Masking ensures that the Transformer learns in an auto-regressive manner, just like during actual text generation.

🔍 Analogy: Think of reading a sentence one word at a time while covering the future words with your hand. This ensures that you rely only on the words you've already seen, rather than anticipating upcoming ones.

How Does Masked Self-Attention Work?

Masked attention works by using a masking matrix that prevents the decoder from looking at future tokens. This is done by setting certain positions in the attention score matrix to \(-\infty \) (or a very large negative number) before applying the softmax function.

• When softmax is applied, these masked positions get a probability of \(0\), effectively excluding them from contributing to the final output.

• This ensures that at any given step, the model only considers previous tokens when making a prediction.

Computing Masked Self-Attention

Just like in the encoder, masked attention follows a step-by-step computation process.

1. Compute Query, Key, and Value Matrices

To compute masked attention, we first generate the Query ( \(Q\) ), Key ( \(K\) ), and Value ( \(V\) ) matrices from the input sequence.

The query weight matrix \(W_Q\), key weight matrix \(W_K\), and value weight matrix \(W_V\) are randomly initialized at the beginning of training. These matrices help the model determine which tokens should focus on others and how much importance to assign.

$$\begin{array}{ccc} \textbf{Query Weight Matrix } (W_Q) & \textbf{Key Weight Matrix } (W_K) & \textbf{Value Weight Matrix } (W_V) \\ \begin{bmatrix} 0.3745 & 0.9507 & 0.7320 \\ 0.5987 & 0.1560 & 0.1560 \\ 0.0581 & 0.8662 & 0.6011 \end{bmatrix} & \begin{bmatrix} 0.7081 & 0.0206 & 0.9699 \\ 0.8324 & 0.2123 & 0.1818 \\ 0.1834 & 0.3042 & 0.5248 \end{bmatrix} & \begin{bmatrix} 0.4320 & 0.2912 & 0.6119 \\ 0.1395 & 0.2921 & 0.3664 \\ 0.4561 & 0.7852 & 0.1997 \end{bmatrix} \end{array}$$

The Query ( \(Q\) ), Key ( \(K\) ), and Value ( \(V\) ) matrices are then obtained by multiplying the input sequence with these weight matrices.

$$\begin{array}{ccc} \textbf{Query Matrix } (Q) & \textbf{Key Matrix } (K) & \textbf{Value Matrix } (V) \\ \begin{bmatrix} 1.1613 & 0.5844 & 0.5178 \\ 1.2283 & 1.5096 & 1.2162 \\ 1.7016 & 1.7517 & 1.4151 \\ 1.0795 & 1.9409 & 1.4705 \\ 0.5140 & 0.0661 & 0.0573 \\ 0.2005 & -0.9015 & -0.6542 \\ 1.0589 & 0.2196 & 0.2428 \\ 0.8668 & 0.9613 & 0.7701 \\ 0.7969 & 0.9298 & 0.7420 \\ 1.0591 & 0.9681 & 0.7860 \end{bmatrix} & \begin{bmatrix} 1.6868 & 0.3274 & 0.6771 \\ 1.9900 & 0.1811 & 1.6528 \\ 2.6512 & 0.4410 & 1.8213 \\ 1.7848 & 0.5429 & 1.6615 \\ 0.6528 & 0.4122 & -0.1058 \\ 0.0745 & 0.1732 & -0.8757 \\ 1.4758 & 0.3090 & 0.3300 \\ 1.3621 & 0.2325 & 0.9821 \\ 1.2619 & 0.2103 & 0.9469 \\ 1.6163 & 0.3239 & 0.9818 \end{bmatrix} & \begin{bmatrix} 0.3726 & 0.5247 & 0.8632 \\ 0.7423 & 0.5899 & 1.3392 \\ 0.9275 & 1.0125 & 1.6228 \\ 1.0164 & 1.3582 & 1.1968 \\ 0.1475 & 0.6045 & 0.1626 \\ -0.3444 & -0.0618 & -0.2922 \\ 0.2055 & 0.3882 & 0.6635 \\ 0.5045 & 0.5522 & 0.8483 \\ 0.4831 & 0.5178 & 0.7992 \\ 0.5362 & 0.6693 & 0.9369 \end{bmatrix} \end{array}$$

📌 For a detailed breakdown of how these matrices are computed, refer to Understanding the Role of Query, Key, and Value Matrices in Transformer Models .

2. Compute Alignment Scores

Once we have \(Q\), \(K\) and \(V\) the next step is to compute the alignment scores using the scaled dot-product attention formula:

$$Score(Q, K) = \frac{Q \cdot K^T}{\sqrt{\frac{D}{\text{No. of Heads}}}}$$

Here:

• \(Q\): Query matrix

• \(K\): Key matrix

• \(D\): Dimensionality of the model

• \(\text{No. of Heads}\) : Number of attention heads (in this example, we assume one head)

Since we are using only one attention head, we compute a single alignment score matrix.

$$\textbf{Score(Q, K)} = \begin{bmatrix} 1.44 & 1.88 & 2.47 & \dots & 1.48 \\ 1.95 & 2.72 & 3.54 & \dots & 2.11 \\ 2.54 & 3.48 & 4.53 & \dots & 2.71 \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 1.52 & 2.06 & 2.69 & \dots & 1.61 \end{bmatrix}$$

📌 For a deeper understanding of scaled dot-product attention, check out this blog where I explain how alignment scores are computed.

3. Applying the Mask

Creating the Mask Matrix

A static triangular mask matrix is used during training to ensure that a token only attends to past and current tokens, preventing information leakage from future tokens.

Steps to Construct the Mask Matrix

1. Define Sequence Length ( \(M\) )

   • Let \(M\) be the number of tokens in the target sequence.

2. Initialize a Full Matrix

   • Create an \(M\) matrix filled with zero values.

3. Apply Masking to Future Tokens

   • Use a triangular function to set all positions where \(j > i\)(i.e., future tokens) to \(-\infty\).

   • The resulting mask matrix ensures that the model only considers current and past tokens for attention.

$$\textbf{Mask Matrix} = \begin{bmatrix} 0 & -\infty & -\infty & \dots & -\infty \\ 0 & 0 & -\infty & \dots & -\infty \\ 0 & 0 & 0 & \dots & -\infty \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & 0 & \dots & 0 \end{bmatrix}$$

This mask is then added to the alignment score matrix, ensuring that future tokens receive an attention weight of zero when softmax is applied.

Computing the Masked Alignment Scores

Once the mask matrix is created, it is applied to the alignment score matrix as follows:

$$\text{Masked Score(Q, K)} = \text{Score(Q, K)} + \text{Mask Matrix}$$

This ensures that all future token positions contribute zero probability to the final attention output. The masked attention matrix in the decoder represents the relevance of each token in the target (ground truth) sequence with respect to every other token in the sequence, but only within the context of the tokens that are available for attention up to that point.

$$\textbf{Masked Score(Q, K)} = \begin{bmatrix} 1.44 & -\infty & -\infty & \dots & -\infty \\ 1.88 & 2.72 & -\infty & \dots & -\infty \\ 2.47 & 3.54 & 4.53 & \dots & -\infty \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 1.48 & 2.11 & 2.71 & \dots & 1.61 \end{bmatrix}$$

4. Applying Softmax to Obtain Attention Weights

Now that we have the masked scores, the next step is to normalize them using the softmax function.

Softmax converts the masked alignment scores into a probability distribution that determines how much attention each token should receive. The formula used is:

$$\alpha = \text{Softmax}(\text{Masked Score}(Q, K))$$

Where:

• \(\alpha\) represents the attention weights after applying softmax.

• Higher attention scores indicate stronger relevance between tokens.

After this step, we obtain the attention weight matrix, which is used to compute the final masked attention output.

$$\textbf{Attention Weight Matrix} = \begin{bmatrix} 1.0000 & 0.0000 & 0.0000 & 0.0000 & \dots & 0.0000 \\ 0.3015 & 0.6985 & 0.0000 & 0.0000 & \dots & 0.0000 \\ 0.0845 & 0.2471 & 0.6684 & 0.0000 & \dots & 0.0000 \\ \vdots & \vdots & \vdots & \ddots & \vdots & \vdots \\ 0.0796 & 0.1497 & 0.2727 & 0.1631 & \dots & 0.0905 \end{bmatrix}$$

5. Computing the Masked Attention Output

The final masked attention output is obtained by multiplying the attention weight matrix ( \(\alpha\) ) with the Value ( \(V\) ) matrix:

$$\text{Output} = \alpha \cdot V$$

This multiplication ensures that the attention mechanism weights the Value matrix according to the computed attention scores, resulting in a context-aware representation for each token.

$$\textbf{Output} = \begin{bmatrix} 0.3726 & 0.5247 & 0.8632 \\ 0.6308 & 0.5702 & 1.1957 \\ 0.8348 & 0.8669 & 1.4885 \\ 0.8701 & 0.9780 & 1.3929 \\ 0.7138 & 0.8626 & 1.1544 \\ 0.3403 & 0.5605 & 0.6270 \\ 0.6688 & 0.8064 & 1.1348 \\ 0.7030 & 0.8258 & 1.1677 \\ 0.6746 & 0.7918 & 1.1219 \\ 0.6909 & 0.8052 & 1.1497 \end{bmatrix}$$

Final Output Projection

In case of multiple attention heads, the outputs from all heads would be concatenated. However, since our example uses only one attention head, the masked attention output matrix serves as our final result.

To prepare this output for the next layers, it is projected using a learnable weight matrix \(W_{dO}\):

$$\text{Final Output} = \text{Masked Attention Output} \cdot W_{dO}$$

This projection layer is used to blend the outputs from multiple heads (if present) into a single cohesive unit, just like in the encoder.

$$\begin{array}{c} \textbf{1. Projection Matrix } W_O \\ \text{The learned weight matrix used for the projection:} \\ W_O = \begin{bmatrix} 0.3745 & 0.9507 & 0.7320 \\ 0.5987 & 0.1560 & 0.1560 \\ 0.0581 & 0.8662 & 0.6011 \end{bmatrix} \\ \textbf{2. Final Projected Output} \\ \text{The attention output after applying the projection matrix:} \\ \text{Final Projected Output} = \begin{bmatrix} 0.5038 & 1.1838 & 0.8735 \\ 0.6471 & 1.7244 & 1.2695 \\ 0.9181 & 2.2183 & 1.6411 \\ 0.9922 & 2.1863 & 1.6267 \\ 0.8508 & 1.8131 & 1.3510 \\ 0.4994 & 0.9540 & 0.7134 \\ 0.7992 & 1.7446 & 1.2975 \\ 0.8255 & 1.8087 & 1.3454 \\ 0.7918 & 1.7366 & 1.2917 \\ 0.8076 & 1.7783 & 1.3224 \end{bmatrix} \end{array}$$

Conclusion

In this blog, we explored how the Transformer decoder processes input during training, focusing on teacher forcing, masked self-attention, and target sequence preparation. We also broke down the step-by-step computation of masked attention, ensuring the model learns in an auto-regressive manner without peeking at future tokens.

In the next installment, we'll dive into encoder-decoder cross-attention, residual connections, and feed-forward networks—key components that refine the decoder's output. Stay tuned!

• Computing attention scores to determine token relationships.

• Passing data through a feed-forward network for deeper feature extraction.

• Adding residual connections to preserve information and improve gradient flow.

• Applying layer normalization for stable and efficient training.

If you haven’t already, check out Part 2 of this series, where we break down how attention scores are computed—it’s essential for building a strong foundation!

In this blog (Part 4 of our Transformer series), we’ll develop an intuitive understanding of feed-forward networks, residual connections, and layer normalization—key elements that refine and stabilize the Transformer’s learning process.

To make the most of this post, we highly recommend reading our previous blog on attention scores, as it sets the stage for the concepts we’ll explore here.

Feed Forward Network

The final attention output is further processed in a feed forward network where it undergoes two linear transformations with a non linear activation between them.

A feedforward network (FFN) is a type of artificial neural network where information flows in one direction—from the input layer to the output layer—without looping back. It’s one of the simplest and most fundamental architectures in neural networks.

Transformers are composed of linear layers (like attention) and matrix operations.Without non-linear transformations, the model would effectively collapse into a single linear transformation, no matter how many layers are stacked.The feedforward network introduces non-linear activation functions (like ReLU or GELU), which allow the model to learn complex, non-linear relationships in the data. Attention captures relationships between tokens, while the feedforward network adds grammatical and semantic refinement, ensuring each token’s representation aligns with the task and target language requirements.

$$\text{FFN(X)}=\text{ReLU}(X \cdot W1+b1)\cdot W2+b2$$

Where,

• \(W_1 (D \times H)\) and \(W_2 (H \times D)\) are learned weight matrices.

• \(b_1 (H)\) and \(b_2 (D)\) are learned bias vectors.

• \(H\) (hidden neurons) is typically greater than \(D\).

Here, \(D\) is the embedding dimension (features per token), and \(H\) is the number of hidden neurons in the feed-forward layer.

The output shape remains \(N \times D\), ensuring token representations retain their dimensions. This transformation enhances feature extraction, enriching contextual understanding before passing to the next layer.

The weight matrices (\(W_1\) and \(W_2\)) and biases ( \(b_1\) and \(b_2\) )used in our use case are given below.

$$\begin{aligned} \textbf{First Weight Matrix } (W_1) &= \begin{bmatrix} 0.3745 & 0.9507 & 0.7320 & 0.5987 \\ 0.1560 & 0.1560 & 0.0581 & 0.8662 \\ 0.6011 & 0.7081 & 0.0206 & 0.9699 \end{bmatrix} \\[8pt] \textbf{First Bias Vector } (b_1) &= \begin{bmatrix} 0.8324, 0.2123, 0.1818, 0.1834 \end{bmatrix} \end{aligned}$$

$$\begin{aligned} \textbf{Second Weight Matrix } (W_2) &= \begin{bmatrix} 0.3042 & 0.5248 & 0.4320 \\ 0.2912 & 0.6119 & 0.1395 \\ 0.2921 & 0.3664 & 0.4561 \\ 0.7852 & 0.1997 & 0.5142 \end{bmatrix} \\[8pt] \textbf{Second Bias Vector } (b_2) &= \begin{bmatrix} 0.5924, 0.0465, 0.6075 \end{bmatrix} \end{aligned}$$

Output Obtained from the Feed-Forward Network (FFN)

The Final FFN Output obtained is as follows

$$\text{FFN Output} = \begin{bmatrix} 2.4327 & 1.7288 & 2.1912 \\ 5.7504 & 4.2261 & 4.7223 \\ 6.6566 & 4.9091 & 5.4139 \\ 3.7186 & 2.6970 & 3.1723 \\ 1.5761 & 1.0830 & 1.5374 \\ 1.6820 & 1.1630 & 1.6183 \\ 3.0062 & 2.1608 & 2.6289 \\ 5.9794 & 4.3994 & 4.8973 \\ 3.7345 & 2.7092 & 3.1845 \\ 4.6607 & 3.4069 & 3.8912 \\ 2.1759 & 1.5355 & 1.9953 \\ 2.1123 & 1.4874 & 1.9467 \end{bmatrix}$$

How Does the Feed-Forward Network (FFN) Work?

In a transformer, the Feed-Forward Network (FFN) refines token representations after attention processing. It applies two linear transformations with a nonlinear activation in between, allowing the model to capture more complex relationships.

Step-by-Step Process:

For each token’s representation:

1. Input: The FFN receives the contextualized token vector from the attention mechanism.

2. First Linear Transformation (Expansion): Expands the vector to a higher-dimensional space to capture richer feature interactions.

   $$\text{Hidden} = \text{ReLU}(X W_1 + b_1)$$

3. Nonlinear Activation: Uses ReLU (or another activation function) to introduce nonlinearity, enabling the model to learn complex patterns.

4. Second Linear Transformation (Compression): Reduces the expanded representation back to its original dimension for consistency.

   $$\text{Output} = \text{Hidden} W_2 + b_2$$

Why Is the Feed-Forward Network Necessary?

While self-attention determines which tokens are important, it operates linearly—it captures relationships between tokens but not complex within-token feature interactions. The FFN addresses this limitation by:

1. Modeling Nonlinear Relationships: Introduces nonlinearity to enhance feature extraction.

2. Dimensional Expansion & Compression: Temporarily increases the feature space, allowing deeper transformations before returning to the original size.

3. Feature Refinement: Emphasizes task-relevant features while suppressing noise.

4. Task-Specific Representation: Fine-tunes token embeddings for downstream tasks like classification, translation, or question-answering.

5. Combining Local & Global Information: Processes each token independently, ensuring that global context from attention is applied effectively at the token level.

By adding depth, expressiveness, and feature refinement, the FFN ensures that token representations are optimized for the model’s final output.

Adding Residual Connection

Residual connections play a crucial role in transformers by preserving information, balancing contextual and refined features, and stabilizing training. They are added after the self-attention mechanism and after the feed-forward network (FFN).

Why Are Residual Connections Needed?

1. Preserving Original Information

Transformers apply complex transformations that can distort or lose critical input information. Residual connections help retain the original representation by adding the input back to the transformed output.

• After Self-Attention:

  • The attention mechanism captures token-to-token relationships and dependencies (e.g., subject-verb-object structures).

  • Without residual connections, these relationships might be weakened as transformations continue.

• After the Feed-Forward Network (FFN):

  • The FFN enhances grammatical, syntactic, and semantic properties at the token level.

  • However, it does not explicitly preserve the broader sequence-wide context captured by attention.

  • Residual connections ensure that both global dependencies (from attention) and local refinements (from FFN) are retained, leading to a well-balanced token representation.

2. Preventing the Vanishing Gradient Problem

• In deep networks, gradients become smaller as they backpropagate, making it difficult to update earlier layers effectively.

• Residual connections provide a direct path for gradients, ensuring better weight updates and preventing training slowdowns.

• This technique stabilizes deep architectures, enabling transformers to learn efficiently.

How the Residual Connection Works

The residual connection simply adds the input vector ( \(X\) ) to the transformed output ( \(F(X)\)):

$$\text{Output}=X+F(X)$$

This element-wise addition allows the model to retain important input features while integrating new transformations, resulting in a more robust representation.

By combining local refinements (FFN) with global dependencies (attention) and ensuring stable gradient flow, residual connections significantly enhance the efficiency and effectiveness of transformer models.

For our use case, the residually connected output obtained is given below

$$\text{Residual Output} = \begin{bmatrix} 2.7440 & 2.3293 & 2.6400 \\ 6.7727 & 6.3734 & 6.3224 \\ 7.8748 & 7.4775 & 7.3278 \\ 4.3060 & 3.8967 & 4.0671 \\ 1.7016 & 1.2856 & 1.6900 \\ 1.8308 & 1.4146 & 1.8074 \\ 3.4410 & 3.0284 & 3.2764 \\ 7.0526 & 6.6521 & 6.5761 \\ 4.3258 & 3.9159 & 4.0846 \\ 5.4515 & 5.0446 & 5.1123 \\ 2.4321 & 2.0163 & 2.3550 \\ 2.3544 & 1.9389 & 2.2846 \end{bmatrix}$$

The residual output obtained is then normalized using layer normalization, let’s understand how layer normalization works and how it is useful in transformers.

Layer Normalization

Layer normalization standardizes the output of a layer by ensuring that the features in the token’s representation have a mean of 0 and a standard deviation of 1. It does this independently for each token, across all dimensions of its vector representation. For a token's vector representation $x = [x_1 , x_2 , … , x_D ]$ (of dimension $D$ ):

Layer normalization normalizes the values within a token (row-wise) to stabilize the learning process. The formula is:

$$\text{LayerNorm}(x) = \gamma \cdot \frac{x - \mu}{\sigma} + \beta$$

Where:

• \(x\): Input matrix (e.g., the residual output, shape $N \times D$).

• \(mu\): Mean of each token's embedding (row mean):

$$\mu = \frac{1}{D} \sum_{i=1}^{D} x_i$$

• \(\sigma\): Standard deviation of each token's embedding (row-wise standard deviation):

  $$\sigma = \sqrt{\frac{1}{D} \sum_{i=1}^{D} (x_i - \mu)^2 + \epsilon}$$

  • \(\epsilon\): Small constant for numerical stability, e.g., \(10^{-6}\)).

  • \(\gamma\) : Learnable scaling parameter (shape \(D\)).

  • \(\beta\) : Learnable shifting parameter (shape \(D\)).

Why is Normalization Needed?

When training deep neural networks (like transformers), the intermediate values (like attention outputs or FFN outputs) can vary a lot between layers or during training. This variation causes two big problems:

1. Training Instability: If the values are too big (or too small), they can cause issues like exploding or vanishing gradients.This makes it hard for the network to learn effectively.

2. Slower Learning: Networks need to adjust weights across layers to deal with the inconsistent scale of values. This slows down convergence (how fast the network learns). Layer normalization fixes these problems by normalizing the values to have a consistent scale for every token, regardless of the layer or input.

How is Layer Normalization Helpful?

1. Stabilizes Training: It keeps the values in a predictable range, which means the model learns more smoothly and consistently.Gradients (the signals used to update weights) don’t explode or vanish, so learning stays on track.

2. Token-Level Focus: In transformers, every token is processed independently.Layer normalization ensures that each token’s embedding is normalized independently, so it doesn’t depend on other tokens in the batch.

3. Faster Convergence: By keeping values well-scaled, layer normalization allows the model to converge (learn) faster during training.

4. Prepares for Next Layer: Layer normalization ensures the output is in a consistent scale, making it easier for the next layer to process it effectively.

For our example , the output obtained on applying Layer Normalization is given below.

$$\text{LayerNorm Output} = \begin{bmatrix} 1.1330 & 0.7191 & 0.6758 \\ 1.2055 & 0.7709 & -0.4830 \\ 1.1982 & 0.7851 & -0.6501 \\ 1.1851 & 0.7335 & 0.1756 \\ 1.0913 & 0.7164 & 0.9469 \\ 1.0965 & 0.7165 & 0.9165 \\ 1.1591 & 0.7239 & 0.4599 \\ 1.2041 & 0.7747 & -0.5314 \\ 1.1856 & 0.7338 & 0.1684 \\ 1.2039 & 0.7506 & -0.1775 \\ 1.1208 & 0.7178 & 0.7628 \\ 1.1175 & 0.7175 & 0.7850 \end{bmatrix}$$

Conclusion: What’s Next?

With Feed-Forward Networks, Residual Connections, and Layer Normalization, we have now completed the operations within one encoder layer. The next step depends on the model’s architecture:

• If this is the last encoder layer, the LayerNorm output is passed to the decoder for further processing.

• Otherwise, it serves as input to the next encoder layer, where the process is repeated and token representations are refined even further.

In this blog, we focused on understanding the encoder’s internal mechanisms. But what happens next? The decoder takes over!

In the next part of this series, we’ll dive into the transformer decoder, breaking down its role intuitively—from handling encoder outputs to generating meaningful sequences.

Self-attention has revolutionized deep learning models, enabling them to capture dependencies across an entire sequence, making them highly effective for NLP, computer vision, and other domains. However, a single self-attention mechanism might not be sufficient to capture multiple aspects of relationships within a sequence. This is where Multi-Head Attention (MHA) comes in.

Multi-Head Attention enhances the power of self-attention by allowing the model to attend to different parts of the input sequence simultaneously. This extension is a core component of the Transformer architecture, enabling models like BERT, GPT, and T5 to process information more effectively.

In this article, we’ll explore how Multi-Head Attention extends Self-Attention and why it is crucial in modern deep learning models.

Self-Attention in Transformers

Before diving into Multi-Head Attention, let’s briefly revisit Self-Attention.

Self-attention enables a model to weigh different words in a sequence based on their relevance to each other. Given an input sequence, the model computes Query ( \(Q\) ), Key ( \(K\) ), and Value ( \(V \) ) matrices and derives attention scores using the Scaled Dot-Product Attention formula:

$$\text{Attention}(Q, K, V) = \text{softmax} \left( \frac{QK^T}{\sqrt{d_k}} \right) V$$

This mechanism allows each word to dynamically attend to other words in the sequence, capturing relationships effectively.

If you’d like a detailed explanation of how self-attention works, check out my blog post - Transformer Encoder Explained : A Deep Dive into Attention Scores (Part 2).

Now, let’s see how Multi-Head Attention builds upon this concept.

What is Multi-Head Attention?

While self-attention is powerful, it has a limitation: a single attention mechanism might not be enough to capture different aspects of word relationships. Consider a sentence like:

"The bank approved the loan."

• "Bank" could refer to a financial institution or a riverbank depending on context.

• A single self-attention mechanism might focus on one aspect, missing the broader context.

To address this, Multi-Head Attention (MHA) applies multiple self-attention mechanisms in parallel, allowing the model to attend to different information simultaneously.

How Multi-Head Attention Works

Instead of computing attention using a single set of Q, K, V, Multi-Head Attention splits the input into multiple heads and applies self-attention independently. The results from all attention heads are then concatenated and projected into the final output.

The process involves the following steps:

1. Linear Projections: The input embeddings are transformed into multiple sets of Query ( \(Q\) ), Key ( \(K\)), and Value ( \(V\) ) matrices using different learned weight matrices.

   $$Q = X \cdot W_q \quad K = X \cdot W_k \quad V = X \cdot W_v$$

   where:

   • \(W_q, W_k, W_v\) are trainable weight matrices that create the respective projections.

   • These projections split the input into multiple attention heads, each having its own set of \(Q\), \(K\) and \(V\) matrices.

2. Independent Self-Attention: Each head applies self-attention separately on its respective \(Q\), \(K\) and \(V\).

   $$\text{Attention}(Q, K, V) = \text{softmax} \left( \frac{QK^T}{\sqrt{d_k}} \right) V$$

   where:

   • \(QK^T\) computes similarity scores between tokens.

   • The softmax function normalizes these scores to determine attention weights.

   • The attention weights are used to scale the values \(V\), determining how much focus each token should receive.

Since there are \(h\) attention heads, each computes attention separately on its own transformed \(Q, K, V\)matrices.

3. Concatenation: The outputs of all heads are concatenated together by horizontally stacking them together into a single matrix.

   $$\text{Concat(head_1​,head_2​,…,head_h​)}$$

   • If there are \(h\) heads, then there are \(h\) matrices of shape \(N \times d_k\) that are stacked together into a single matrix.

   • Horizontally stacking the attention outputs from all heads produces a matrix of shape

     $$\text{Number of Tokens}×\text{Embedding Dimension}$$

   • Why Concatenation Alone Isn't Enough

     At this stage, the stacked attention outputs from different heads are merely placed side by side, but they remain disjoint—each attention head has focused on different relationships independently, but there’s no mechanism yet to combine their insights into a unified representation.

     • Without a learned transformation, the model has no way to determine how much weight to assign to the contributions of each head.

     • Each head might capture different semantic aspects (e.g., positional dependencies, syntax, or meaning), but simply concatenating them does not blend these insights into a meaningful form for the next layer.

To address this, we apply a final transformation using a trainable projection matrix \(W^O\)​.

4. Final Linear Projection: To ensure consistency with the rest of the model, the concatenated vector is projected back to the original embedding dimension \(D\) using a weight matrix \(W^O\) of shape \(D \times D\):

   $$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \text{head}_2, \dots, \text{head}_h) W^O$$

   • \(W^O\)​ learns how to blend the outputs from all heads to create a final context-aware representation.

   • This final transformation ensures that the output remains compatible with the input dimensions for further processing in the model.

   • The concatenated vector size ( \(h \times d_k\)) is reduced back to the original embedding dimension ( \(D\)) to ensure consistency with the rest of the model.

By transforming the raw concatenated attention outputs, the model can assign appropriate importance to different attention heads, ensuring that the most relevant and meaningful information is propagated forward.

Why Is This Important?

1. Integration of Multiple Perspectives: Each attention head captures different aspects of the input sequence. The projection step ensures that these different perspectives are blended into a single, coherent representation.

2. Consistency for Subsequent Layers: By projecting the concatenated output back to the embedding dimension, the transformer maintains dimensional consistency across layers, allowing seamless integration with feed-forward layers and residual connections.

Intuitive Analogy:

Imagine a team of specialists analyzing a document:

• Each specialist (attention head) focuses on a different aspect—grammar, semantics, tone, or structure.

• They all produce individual reports (head outputs).

• These reports are then merged and summarized into a single, cohesive document (the projection step) to ensure no insights are lost and the output is streamlined for further processing.

Conclusion: The Power of Multi-Head Attention

Multi-Head Attention is a crucial enhancement of the self-attention mechanism, allowing transformers to capture multiple perspectives in parallel. By applying independent self-attention mechanisms, concatenating the outputs, and then projecting them back into the original embedding dimension, the model learns to extract richer contextual representations. This process ensures that transformers can effectively understand complex relationships between tokens, making them highly effective in NLP, computer vision, and beyond.

But Multi-Head Attention is just one piece of the puzzle. In the next part of this series, we’ll explore the Feed-Forward Network (FFN), Residual Connections, and Layer Normalization—key components that further refine the transformer’s ability to learn deep representations while stabilizing training.

Stay tuned for Part 4, where we’ll break down these mechanisms in detail! 🚀

What do Query (Q), Key (K) and Value(V) matrices denote ?

1. Query (Q): What are you looking for? The Query vector represents what a specific word (or token) is "looking for" in other words within the sequence.

2. Key (K): How do I identify information? The Key vector encodes information about a word in a way that makes it "searchable." Think of it as metadata for each token, indicating what kind of information this word provides.

3. Value (V): What information do I provide? The Value vector represents the actual content or information that the word contributes to the overall representation.

A Real-Life Analogy: The Detective Team

Imagine a team of detectives collaborating to solve a mystery. Here’s how Q, K, and V fit:

• Each detective has a specific question they want answered, like “Who was at the scene of the crime?”

  This is the Query—the focus of their search.

• The evidence they have is labeled with tags like “time of arrival” or “fingerprints.” These labels make the evidence searchable.

  This is the Key—metadata describing what the evidence can tell us.

• Each piece of evidence also contains the actual content: the fingerprint report, timestamps, or alibi details.

  This is the Value—the meaningful data itself.

How Are Q, K, and V Matrices Calculated?

The Query ( \(Q\) ), Key ( \(K\) ), and Value ( \(V\) ) matrices are computed by transforming the embedding vector ( \(E\) ) using learned weight matrices:

$$\text{Query(Q)}=E \times W_q \quad \text{Key(K)}=E \times W_k \quad \text{Value(V)}=E \times W_v$$

Where:

• \(E\) (Embedding Matrix, \( N \times D\) )

  • \(N\): Number of tokens in the input sequence.

  • \(D\): Feature dimensions describing each token (semantic meaning, syntactic role, and contextual nuance).

• \(W_q\), \(W_k\) and \(W_v\) are the learned weight matrices which are used to transform embeddings into queries, keys, and values for attention computation.

Why Multiply Weight Matrices With Embeddings?

When you multiply the embedding vector by the respective weight matrix (for Query, Key, or Value), the resulting matrix represents a transformation of the embedding. This transformation essentially adjusts the importance or emphasis assigned to each dimension of the token's representation for the specific role (Query, Key, or Value).

Let’s look at each of these matrices one by one now.

Query (Q):Weighted Questions

The Query vector can indeed be thought of as a weighted representation of the questions that need to be asked about each token, specifically in terms of how relevant or important each dimension of the token's information is to the task at hand. The Query vector emphasizes certain dimensions of the token’s embedding, based on the context and the model's learned understanding of what is important for this token at this point in the sequence. These weights essentially tell the model, "For this token, these specific aspects (dimensions) are most relevant when determining its relationship with other tokens."

How are these questions formed ?

• The "questions" encoded by the Query vector (via the weight matrix \(W_q\)) are not explicitly predetermined. Instead, they are learned during the model's training process.

• The weight matrix \(W_q\) learns to transform the token embeddings into a space where they encode "questions" about the input.

• These questions are context-sensitive, meaning the Query vector represents what this token is trying to "understand" based on its position and role in the sequence.

So essentially the query matrix \(Q\) represents all the questions that are being asked about the input sequence. These questions aim to determine:

1. Which tokens are important for understanding the sequence.

2. Which properties (dimensions) of these tokens are relevant to answering the questions.

Think of it as: "What do I need to focus on to understand this input?"

Key (K):Metadata for Retrieval

The weight matrix \(W_k\) adjusts the embedding to represent how the token provides information that others might find relevant. For instance, it might highlight dimensions like syntactic roles or semantic cues.

The Key(K) matrix represents the metadata about each token, explaining:

1. What kind of information each token provides.

2. How the dimensions of the token’s embedding describe this information.

Think of it as: "Here’s my profile—this is what I can contribute and how to interpret me."

Value Matrix (V):The Content

Represents the actual content or information that each token contributes to the input sequence. This information is what the model ultimately uses after determining relevance via Query and Key interactions. Think of it as: "Here’s the actual information I hold that can help you understand the sequence."

How is the value matrix different from the embedding matrix ?

Both the value matrix and embedding matrix do indeed represent information about each token across dimensions, but they serve different purposes and are used differently in the transformer.

The embedding vector is like a general encyclopedia entry for a token—it’s rich and multi- purpose but not directly optimized for the specific task at hand. The Value vector on the other hand is like a custom summary of that entry, tuned to the specific needs of the attention mechanism in the current sequence. The value matrix is essentially an augmented and task-specific representation of the embedding vector, tailored to the input sequence and the task at hand.

Here’s a travel guide analogy which will help you understand this better

Imagine a travel guidebook (the embedding vector):

• The guidebook contains all general information about a city (e.g., landmarks, history, transportation) - this is analogous to the embedding vector.

• If you're planning a specific trip, you'll extract only the sections relevant to your trip goals (e.g.,places to visit, public transit details, etc.) and organize them for your itinerary - similar to the value vector.

How the Value Vector Augments the Embedding Vector:

1. Embedding as a Base Representation: The embedding vector is a general-purpose encoding of a token, independent of the specific sequence or task. It contains semantic and syntactic information about the token in a learned vector space.

2. Task-Specific Transformation: The Value vector is created by multiplying the embedding vector with a learned weight matrix ( \(W_v\) ). This transformation adjusts the embedding to emphasize dimensions that are relevant for the specific task or the context of the input sequence.

3. Context-Aware Representation: The weight matrix ( \(W_v\) ) is learned during training and adapts the embedding to better reflect the token's role in the specific sequence. For example, in a translation task, certain dimensions might focus on semantic roles (e.g., subject-object relationships), while in a summarization task, different dimensions might emphasize sentence importance.

4. Preserving the Core Information: The Value vector retains the core information from the embedding vector but filters and highlights the most task-relevant and context-relevant features.

Translating to Transformers :

In a transformer, every token in a sequence acts like a detective:

1. Each token uses its Query to ask questions.

2. Every token presents its Key to show what kind of information it holds.

3. The token’s Value provides the actual content used after relevance is determined.

The model calculates the relevance between Queries and Keys (called the alignment score) to determine how much attention each token should pay to others. This process enables dynamic, context-aware information sharing across the sequence.

Why Q, K, and V Are Powerful

Using Query, Key, and Value matrices, transformers allow every token to interact dynamically with every other token in the sequence. This mechanism ensures that the model captures context effectively, making it one of the most powerful architectures for tasks like language translation, summarization, and more.

Now that we’ve broken down the roles of Query (Q), Key (K), and Value (V) in a transformer, it’s clear how these components enable the model to dynamically determine relationships between tokens. Each token asks questions (Query), presents relevant details (Key), and provides useful content (Value)—forming the foundation of the attention mechanism that makes transformers so powerful.

But how does a transformer actually decide which tokens to focus on? This is where alignment scores come into play! 🚀

🔗 Read more about alignment scores and attention computation here: Transformer Encoder Explained : A Deep Dive into Attention Scores (Part 2).

Now, in Part 2, we dive into the self-attention mechanism, breaking down how transformers compute attention scores to determine which tokens in a sequence are most relevant to each other. Let’s get started! 🚀

What is an Attention Score?

An attention score is a numerical value that determines how much importance one token in a sequence should assign to another. These scores are computed as part of the self-attention mechanism in a transformer, helping the model understand contextual relationships between tokens.

How Are Attention Scores Computed?

The process of computing attention scores involves several steps:

1. Generating Query (Q), Key (K), and Value (V) matrices

2. Computing alignment scores to measure relationships between tokens

3. Applying the softmax function to normalize these scores into probabilities

4. Applying a linear transformation to refine the attention output

We’ll start by exploring how attention scores are calculated within a single attention head and then extend this understanding to multi-head attention, which enhances the model’s ability to capture diverse contextual relationships.

1.1 Computing The Query (Q) , Key(K) and Value (V) Matrices

The attention layer in the encoder has multiple attention heads and each head has three weight matrices \(W_q\),\(W_k\) and \(W_v\) each of order \(D \times (\frac{D}{\text{No. Of Heads}})\) ( \(D\) denotes dimension size). They are initialized randomly such that for each head the values of the weight matrices are unique i.e the values of the weight matrices of no two heads are the same. The weight matrices are used to compute three matrices namely query , key and value which are used to compute the attention scores.

1. Query (Q) :

   1. A query is a vector that asks a question about the relationship between the current token and other tokens. For example, in a sentence, "The dog chased the cat," the query for "dog" might be asking, "Who or what is related to the dog?"

   2. It is computed as follows :

      $$Q= \text{Input Matrix} \times W_q$$

   3. The "questions" encoded by the Query vector (via the weight matrix \(W_q\)) are not explicitly predetermined. Instead, they are learned during the model's training process.

2. Key (K):

   1. Think of it as metadata for each token, indicating what kind of information this word provides. For example, in the sentence "The dog chased the cat," the key for "cat" might encode information like "animal" or "object being acted upon."

   2. It is computed as :

$$K =\text{Input Matrix} \times W_k$$

1. Value (V):

   1. What it represents:

      A value is the vector containing the actual information or content of a token that will be passed to the next layer if that token is deemed important. In the example "The dog chased the cat," the value for "cat" might contain its embedding and context-specific information about "cat."

   2. It is computed as :

$$V= \text{Input Matrix}\times W_v$$

In our use case because our dimension size is small we can have only one attention head.

The weight matrices( \(W_q\), \(W_k\) and \(W_v\) will be initialized to random values initially , these randomly initialized weight matrices are given below:

$$\textbf{Weight Matrices:}$$

$$\begin{array}{c c c} {W_q} & {W_k} & {W_v} \\ \begin{bmatrix} 0.3745 & 0.9507 & 0.7320 \\ 0.5987 & 0.1560 & 0.1560 \\ 0.0581 & 0.8662 & 0.6011 \end{bmatrix} & \begin{bmatrix} 0.7081 & 0.0206 & 0.9699 \\ 0.8324 & 0.2123 & 0.1818 \\ 0.1834 & 0.3042 & 0.5248 \end{bmatrix} & \begin{bmatrix} 0.4320 & 0.2912 & 0.6119 \\ 0.1395 & 0.2921 & 0.3664 \\ 0.4561 & 0.7852 & 0.1997 \end{bmatrix} \\ \end{array}$$

The \(Q\), \(K\) and \(V\) matrices obtained on by multiplying the input matrix with their corresponding learned weight matrices ( \(W_q\), \(W_k\), and \(W_v\)) are given below :

$$\begin{array}{c c c} \textbf{Query Matrix } Q & \textbf{Key Matrix } K & \textbf{Value Matrix } V \\ \begin{bmatrix} 1.1904 & 0.1467 & 0.4669 \\ 2.0582 & 0.2445 & 1.6091 \\ 2.0105 & 0.6131 & 1.5091 \\ 1.4613 & 0.2865 & 1.2076 \\ 0.3440 & 0.1828 & -0.5350 \\ 0.4377 & 0.4853 & -0.5507 \\ 1.6470 & 0.3173 & 0.5242 \\ 2.0412 & 0.2458 & 1.6666 \\ 1.5460 & 0.2431 & 1.1800 \\ 1.6145 & 0.2871 & 1.5270 \\ 1.0075 & 0.4310 & 0.0520 \\ 0.9114 & 0.2611 & 0.1972 \end{bmatrix} & \begin{bmatrix} 1.1904 & 0.1467 & 0.4669 \\ 2.0582 & 0.2445 & 1.6091 \\ 2.0105 & 0.6131 & 1.5091 \\ 1.4613 & 0.2865 & 1.2076 \\ 0.3440 & 0.1828 & -0.5350 \\ 0.4377 & 0.4853 & -0.5507 \\ 1.6470 & 0.3173 & 0.5242 \\ 2.0412 & 0.2458 & 1.6666 \\ 1.5460 & 0.2431 & 1.1800 \\ 1.6145 & 0.2871 & 1.5270 \\ 1.0075 & 0.4310 & 0.0520 \\ 0.9114 & 0.2611 & 0.1972 \end{bmatrix} & \begin{bmatrix} 0.2001 & 0.1991 & 0.6198 \\ 0.7546 & 0.6788 & 1.3418 \\ 0.9585 & 1.3736 & 1.2265 \\ 0.6461 & 0.7333 & 0.9554 \\ -0.1936 & 0.0401 & -0.0691 \\ -0.0077 & 0.6065 & -0.0870 \\ 0.2914 & 0.4476 & 0.7973 \\ 0.7873 & 0.7108 & 1.3539 \\ 0.5923 & 0.6145 & 0.9882 \\ 0.7935 & 0.8368 & 1.1251 \\ 0.1999 & 0.6229 & 0.3577 \\ 0.1711 & 0.3754 & 0.3957 \end{bmatrix} \\ \end{array}$$

NOTE :

In actual practice however, the dimension size will be much larger like 512 , 768 and 1024. So , the resultant \(Q\), \(K\) and \(V\) matrices in those cases would be of the order \(N \times (\frac{D}{\text{No. Of Heads}})\).

Since, the example considered here has only one attention head, the \(Q\), \(K\) and \(V \) matrices are of the order \(N \times D\).

📌 A deeper intuition behind the Query, Key, and Value vectors goes beyond the scope of this blog. However, you can explore it in detail in Understanding the Role of Query, Key, and Value Matrices in Transformer Models.

1.2 Computing The Alignment Scores

In transformer models, the alignment score matrix, denoted as \(Score(Q,K)\), plays a pivotal role in determining the relevance or importance of each token to every other token in a sequence.

What is the Alignment Score Matrix?

The \(Score(Q, K)\) matrix represents how relevant or important each token is to every other token in the sequence. It determines the degree of alignment between the "questions" (Queries) being asked by a token and the "metadata" (Keys) provided by all other tokens. You can think of it as a "relevance map" that guides how much attention each token should pay to every other token.

Formula for Alignment Score

$$Score(Q, K)=\frac{Q \ldotp K^{T}}{\sqrt \frac{D}{\text{No. Of Heads}}}$$

Here:

• \(Q\): Query matrix

• \(K\): Key matrix

• \(D\): Dimensionality of the model

• \(\text{No. of Heads}\) : Number of attention heads (in this case, only one)

Since there is only one head in our example, we have only one score matrix.

How It Works ?

A token's Query vector ( \(Q\) ) represents what it seeks to understand, while Key vectors (\(K\)) from other tokens describe the information they offer. The dot product of \(Q\) and \(K\) generates an alignment score, measuring how useful a token's information is to understand what the query vector seeks. This score helps determine how much attention each token should give to others.

How to Interpret the Score(Q, K) Matrix ?

The \(Score(Q, K)\)matrix provides a map of token-to-token importance. Each element reflects

1. Semantic Relationship : The dot product between \(Q\) and \(K\) captures the similarity between the query token and key tokens. A higher score indicates stronger semantic alignment.

2. Relative Importance : The scores reflect the raw importance of each key token concerning the query token. These are used to compute attention weights.

3. Contextual Focus : The alignment scores guide the query token in determining which tokens to "pay attention to" while constructing its representation.

Analogy: A Group Discussion

Think of a conversation where:

• Each participant (token) asks questions ( \(Q\) ) to understand the context.

• Every participant also shares their expertise ( \(K\) ) to provide useful information.

• The \(\text{Score}(Q, K)\) matrix acts as a relevance chart, showing who best answers whose questions.

For example, if someone asks about timelines, a participant specializing in timelines will have a high alignment score, influencing the discussion more.

Example: Alignment Score Matrix in Practice

The alignment score matrix for our use case is given below:

$$\text{Score}(Q, K) = \begin{bmatrix} 0.6727 & 1.3030 & \dots & 0.5140 \\ 1.3371 & 2.8624 & \dots & 1.0407 \\ 1.3881 & 3.0150 & \dots & 1.0887 \\ \vdots & \vdots & \ddots & \vdots \\ 0.5179 & 0.9806 & \dots & 0.3962 \end{bmatrix}$$

Properties:

• It is a square matrix of order \(N \times N\) where \(N\) is the number of tokens.

• Each cell indicates the alignment score between a pair of tokens in the sequence.

1.3. Normalizing the Alignment Score To Obtain Attention Weights

The \(\text{Score}(Q, K)\)matrix is computed for all heads are then normalized using a softmax function to get attention weights denoted by \(\alpha\). Applying softmax to the alignment scores serves to normalize them into a probability distribution, making the scores more interpretable and ensuring they can be treated as weights for aggregating the value vectors.

What is the Softmax Function?

The softmax function converts raw scores into probabilities, normalizes values within [0,1], and ensures they sum to 1.

The formula for the softmax activation function is:

$$\sigma(z_i) = \frac{e^{z_i}}{\sum_{j=1}^n e^{z_j}}$$

Where:

• \(z_i\) is the i-th input element to the softmax function.

• \(e^{z_i}\) is the exponential function applied to \(z_i\).

• \(n\) is the total number of input elements.

• \(\sum_{j=1}^n e^{z_j}\) is the sum of the exponential of all input elements, which normalizes the output.

How Softmax Enhances Alignment Scores

Before Normalization:

The raw alignment scores \(\text{Score}(Q,K)\) represent similarity between Query and Key vectors but:

• Can have arbitrary values, including negatives or large magnitudes.

• Lack a clear, interpretable scale for weighting Value vectors effectively.

After Applying Softmax:

Softmax transforms these raw scores into a probability distribution, making them more meaningful and useful for attention computation:

1. Ensuring Interpretability

   • Converts raw scores into probabilities that sum to 1 for each Query.

   • Allows the model to understand relative importance, ensuring each token's influence is bounded and weighted appropriately.

2. Determining Relative Importance

   • The exponential function amplifies significant scores while suppressing smaller ones, sharpening attention towards highly relevant tokens.

   • The resulting attention scores help the model focus on the most important tokens in the sequence.

   • This essentially implies that softmax exaggerates score differences, ensuring attention is concentrated on the most relevant tokens.

3. Capturing Relevance

   • Normalization ensures that each Query token distributes attention effectively across Key tokens.

   • Higher scores indicate stronger relevance, guiding the model to pay more attention to critical tokens while minimizing distractions.

   • This essentially implies that softmax normalizes scores into a probability distribution, ensuring proper weighting of token relevance.

Example: Attention Weights in Practice

For our use case, the attention weights ( \(\alpha\) ) are derived from the softmax-normalized alignment scores. The resulting weights are shown below:

$$\text{Attention Weights } (\alpha) = \begin{bmatrix} 0.0661 & 0.1241 & 0.1276 & \dots & 0.0999 & 0.0593 & 0.0564 \\ 0.0368 & 0.1691 & 0.1970 & \dots & 0.1197 & 0.0278 & 0.0258 \\ 0.0324 & 0.1651 & 0.2142 & \dots & 0.1275 & 0.0234 & 0.0219 \\ \vdots & \vdots & \vdots & \ddots & \vdots & \vdots & \vdots \\ 0.0661 & 0.1196 & 0.1271 & \dots & 0.1016 & 0.0609 & 0.0581 \\ 0.0685 & 0.1159 & 0.1217 & \dots & 0.0996 & 0.0635 & 0.0609 \end{bmatrix}$$

Intuitive Analogy:

Imagine you’re picking relevant answers in a quiz:

• The alignment scores are the raw relevance ratings (e.g., 7/10, 9/10, etc.) for each possible answer.

• The softmax converts these ratings into percentages (e.g., 60%, 30%, 10%), showing how much weight you give to each answer when combining them to form your understanding.

This analogy illustrates how the softmax function emphasizes the most relevant scores while normalizing the overall distribution.

1.4. Computing the final Attention Output And Applying Linear Transformation

Attention output for each head is computed as follows :

$$\text{Attention Output} = \alpha . V$$

Where:

• \(\alpha\) : Normalized attention scores (from \(\text{Softmax}(\text{Score}(Q,K))\)).

• \(V\): Value matrix containing the information content for each token.

Breaking Down the Computation

1. Attention Weights (Softmax Scores) – Define how much focus a Query token assigns to each Key token, controlling how Value vectors influence its representation.

2. Value Vectors ( \(V\) ) – Contain task-relevant semantic information from each token, essential for model understanding.

3. Weighted Summation ( \(\alpha \cdot V \) ) – Aggregates Value vectors based on attention weights, ensuring relevant tokens contribute more while refining contextual representations.

Why Multiply Attention Weights with the Value Matrix?

1. Combining Relevant Information

• Attention weights ( \(\alpha\) ) indicate how much importance each token assigns to others.

• The value matrix ( \(V\)) contains the actual content of each token.

• Multiplying \(\alpha\) with \(V\) aggregates relevant information, weighted by importance.

2. Weighted Summation

• The final token representation is a weighted sum of value vectors.

• This ensures the model focuses on the most relevant tokens while minimizing distractions.

3. Contextualization

• Value vectors ( \(V\) ) provide content, while attention weights ( \(\alpha\) ) dictate importance.

• This transformation enhances token embeddings, making them context-aware and capturing relationships in the sequence.

Analogy: Writing a Summary

Imagine summarizing a paragraph:

• Attention weights decide how much focus each sentence gets.

• Value vectors hold the actual information from the sentences.

• Weighted summation merges the most relevant details into a coherent summary.

Understanding the Attention Output Matrix

• Each row represents a token in the sequence.

• The row’s values form a vector that encodes the token’s meaning after attending to other tokens.

• This contextual representation allows the model to understand words in relation to the entire input sequence.

Matrix Dimensions

• The attention output matrix from each attention head has dimensions:

• $$N \times \frac{D}{\text{No. of Heads}}$$

• With \(H\) attention heads, we get \(H\) attention output matrices of the same size.

• These matrices are horizontally stacked to form a single attention output matrix of size: \(N \times D\)

$$\text{Attention Output} = \begin{bmatrix} 0.2131 & 0.3482 & 0.3967 \\ 0.8221 & 1.0589 & 1.3860 \\ 0.9912 & 1.2546 & 1.6513 \\ \vdots & \vdots & \vdots \\ 0.6240 & 0.8280 & 1.0567 \\ 0.1669 & 0.2925 & 0.3192 \\ 0.1561 & 0.2777 & 0.2999 \end{bmatrix}$$

Now that we’ve explored how self-attention works within a single attention head, the next step is to understand how multi-head attention enhances this process—enabling the transformer to capture multiple aspects of meaning from different perspectives.

📌 Checkout Part 3 of the series, where we’ll dive deep into Multi-Head Attention and how it allows transformers to understand context even better! 🚀

What is a Transformer ?

A Transformer is a deep learning model architecture designed to process and transform sequential data, such as text, into meaningful outputs. It is the foundation for many state-of-the-art natural language processing (NLP) models, such as GPT, BERT, and T5. Transformers are particularly effective because they rely on a mechanism called self-attention, which allows the model to weigh the importance of different words in a sequence when making predictions or translations.

To intuitively understand how a Transformer works, let's consider an example from a machine translation task, where we train a transformer to translate an English sentence into French. Below are the input and target sentences that will be used to train the transformer:

1. English Input (Source Sequence):

   "My name is Vikas. I love cricket, finance and AI."

2. French Translation (Target Sequence):

   "Mon nom est Vikas. J'aime le cricket, la finance et l'IA."

Why Were Transformers Introduced?

Transformers were designed to address the following key limitations of RNNs and encoder-decoder models with attention:

1. Sequential Processing in RNNs:

   RNNs and encoder-decoder models with attention process inputs sequentially, which limits their ability to leverage parallelism during training. Transformers eliminate this constraint by processing entire sequences simultaneously using self-attention.

2. Long-Term Dependencies:

   Even with attention mechanisms, RNNs can struggle with very long sequences due to their sequential nature and reliance on hidden states. Transformers use self-attention to directly compute dependencies between all tokens in a sequence, regardless of distance.

3. Scalability:

   Training RNN-based models with attention can be computationally intensive for long sequences. Transformers scale more efficiently due to their ability to parallelize computations over the entire sequence.

Let’s start with the first step , Tokenization

Tokenization is a fundamental step in Natural Language Processing (NLP) pipelines, where a text corpus is broken into smaller units called tokens. These tokens can be words, subwords, or characters, making the text easier for models to process.

Transformers often use Byte Pair Encoding (BPE), but for simplicity, we'll demonstrate word-based tokenization using the following corpus:

Corpus:
"My name is Vikas. I love cricket, finance, and AI."

Word Tokenization Process

1. Identifying Word Boundaries → Words are separated by spaces, punctuation, or delimiters.

2. Handling Punctuation → Symbols like periods (.) and commas ( , ) can be treated as separate tokens when needed for the task.

Tokenized Output

Applying word-based tokenization to the corpus results in:

$$\text{["My", "name", "is", "Vikas", ".", "I", "love", "cricket", ",", "finance", "and", "AI", "."]}$$

This simple yet effective tokenization method helps convert raw text into a structured format, making it easier for NLP models to analyze and process.

Creating a Vocabulary

To prepare the text for input into a transformer model, we extract the unique tokens from the tokenized output. This step creates a vocabulary, which maps each token to a unique identifier (e.g., an integer). Here is the set of unique tokens from the corpus:

["My", "name", "is", "Vikas", ".", "I", "love", "cricket", ",", "finance", "and", "AI"]

Creating Embeddings From Tokens

The next step is to convert unique tokens into a format which machine learning models can understand.

Machine learning models operate on numerical data. The process of converting high-dimensional data such as text to a numerical format (called embeddings) that can be represented in a low dimensional space is called Vectorization.

What is an Embedding Vector ?

Embedding Vectors are a High-Dimensional Representation of Tokens. Embedding vectors reside in a high-dimensional space where each token is represented by a vector. These vectors capture rich and meaningful information about each token. To achieve this, the information for each token is represented in terms of various features or properties:

Rows in the embedding matrix correspond to tokens. Columns in the embedding matrix correspond to latent features or properties.Values in the matrix indicate how much a particular token (row) relates to a specific property (column).

However, these columns (properties) are not explicitly named or predefined. Instead, they are learned during training.The initial values (scores) in the embedding matrix are random and adjusted as the model learns from the data.

What is Embedding Dimension ?

The embedding dimension is the number of ways you can describe the meaning of a token. Each column (dimension) captures one aspect of the token’s semantics.

Example Analogy: Describing a Person

Imagine describing people in a table:

Each column represents a way to describe a person (age, height, etc.). The number of columns is the embedding dimension. More columns provide more descriptive power.

Similarly for tokens , each row in an embedding vector describes that token in as many ways as the number of columns.

Each column captures a distinct semantic property, such as:

1. Does this word convey a sense of time?

2. Is this word formal or informal?

3. Does this word relate to food, emotions, or actions?

Deciding the Dimension Size The embedding dimension size determines how much information can be encoded for each token:

1. Complexity of Meaning:

   1. Small dimension (e.g., 50): Captures basic meanings. Useful for simple tasks like binary classification or topic detection.

   2. Large dimension (e.g., 512 or more): Captures rich, nuanced meanings. Necessary for complex tasks like machine translation, summarization, or question answering.

2. Analogous to Compression: Think of embedding as compressing the meaning of a token into a fixed-size vector:

   1. Low-dimensional embeddings: Coarse representations that lose some details. Analogous to a black-and-white photo.

   2. High-dimensional embeddings: Detailed representations that preserve subtle nuances. Analogous to a high-resolution color photo.

There are many techniques used to create embeddings , for our use case let us consider Word2Vec. How Word2Vec works is beyond the scope of this blog.

For the sake of simplicity , let us assume a dimension size of 3. If we train Word2Vec on all the unique tokens in our corpus , our embedding matrix might end up looking like this.

The final embedding matrix for processing is as follows

$$\text{Embedding Matrix} = \begin{bmatrix} 0.45 & 0.12 & -0.33 \\ 0.67 & 0.22 & -0.15 \\ -0.12 & 0.55 & 0.88 \\ 0.80 & -0.10 & 0.25 \\ 0.05 & 0.03 & -0.07 \\ -0.25 & 0.40 & 0.70 \\ 0.60 & 0.75 & -0.20 \\ 0.90 & 0.15 & -0.10 \\ 0.02 & -0.01 & 0.04 \\ 0.85 & -0.20 & 0.30 \\ 0.10 & 0.50 & 0.40 \\ 0.95 & 0.12 & 0.08 \end{bmatrix}$$

So the embedding matrix is of the order \(N \times D\), where \(N\) is the number of unique tokens and \(D\) is the dimension size.

Transformers solve the limitation of sequential processing and long term dependency, by processing tokens in parallel and using the self-attention mechanism.

This however comes with a limitation, that they do not have positional information about the input sequence. This means that positional encodings , have to be added to the embedding vector to give the transformer the positional context of the token.

Positional Encoding

There are many types of positional encodings , but for this example let us consider the fixed sinusoidal positional encoding.

For a sequence position \(pos\) and dimension \(i\) in the embedding vector:

$$PE(pos,2i) = \sin\left(\frac{pos}{10,000^{\frac{2i}{d}}} \right)$$

$$PE(pos,2i+1) = \cos\left(\frac{pos}{10,000^{\frac{2i}{d}}} \right)$$

where,

• \(pos\): The position of the token in the sequence (e.g., 0, 1, 2, …).

• \(i\): The dimension index of the embedding vector (e.g., 0, 1, 2, …).

• \(d\): The total dimension size of the embeddings (e.g., 512). In our case, \(d=3\).

The positional encodings are added by the operation

$$\text{Embedding Matrix} + \text{Positional Encodings Matrix}$$

Therefore , positional encodings matrix should also be of dimension \(N \times D\).

For our embedding matrix , the positional encodings is computed as

$$\text{Input Matrix} = \begin{bmatrix} 0.4500 & 1.1200 & -0.3300 \\ 1.5115 & 1.2199 & -0.1500 \\ 0.7893 & 1.5499 & 0.8800 \\ 0.9411 & 0.8999 & 0.2500 \\ -0.7068 & 1.0299 & -0.0700 \\ -1.2089 & 1.3999 & 0.7000 \\ 0.3206 & 1.7499 & -0.2000 \\ 1.5569 & 1.1498 & -0.1000 \\ 1.0094 & 0.9898 & 0.0400 \\ 1.2621 & 0.7998 & 0.3000 \\ -0.4440 & 1.4998 & 0.4000 \\ -0.0499 & 1.1197 & 0.0800 \end{bmatrix}$$

On adding the two together, we obtain the following input matrix.

$$\text{Positional Encoding Matrix} = \begin{bmatrix} 0.0000 & 1.0000 & 0.0000 \\ 0.8415 & 0.9999 & 0.0000 \\ 0.9093 & 0.9999 & 0.0000 \\ 0.1411 & 0.9999 & 0.0000 \\ -0.7568 & 0.9999 & 0.0000 \\ -0.9589 & 0.9999 & 0.0000 \\ -0.2794 & 0.9999 & 0.0000 \\ 0.6569 & 0.9998 & 0.0000 \\ 0.9894 & 0.9998 & 0.0000 \\ 0.4121 & 0.9998 & 0.0000 \\ -0.5440 & 0.9998 & 0.0000 \\ -0.9999 & 0.9997 & 0.0000 \end{bmatrix}$$

Conclusion: Laying the Foundation for Transformers

In this blog, we explored the first crucial steps in understanding how transformers work—tokenization, embeddings, and positional encoding. These processes transform raw text into a numerical format that a transformer can interpret, setting the stage for the self-attention mechanism to process and learn relationships within sequences.

By using word embeddings, transformers can represent words in a meaningful, high-dimensional space, capturing semantic similarities and contextual relationships. However, since transformers process tokens in parallel, they lack an inherent sense of order—this is why positional encodings are crucial in providing context about token positions within a sequence.

But how do transformers actually process these embeddings to understand and generate language? The key lies in Self-Attention, the core innovation behind transformers, which enables them to dynamically focus on different words in a sequence.

👉 In the next part of this series, we’ll dive deep into Self-Attention—breaking down how transformers determine relationships between tokens, compute attention scores, and generate context-rich representations. Stay tuned! 🚀