Lesson 9: Transformer — "Attention Is All You Need"

NLP from Basics to Advanced: Mastering Natural Language Processing

Part 3: Deep Learning for NLP — RNN, LSTM, to Transformer

xdev.asia

Introduction

Paper "Attention Is All You Need" (Vaswani et al., 2017) is the biggest turning point in the history of NLP. Transformer completely eliminates RNN/LSTM, uses only attention — and becomes the basis of every modern LLM: GPT-4, Gemini, Claude, LLaMA.

---

1. Why do we need a Transformer?

Problems of RNN/LSTM	Transformer solve
Sequential processing (slow)	Parallel processing (fast)
Vanishing gradient (long → forget)	Self-attention (direct connection to every location)
Fixed context window	Attention to entire sequence
Difficult to scale to GPU cluster	Easy Parallelization

---

2. Transformer architecture

┌──────────────────────────────────────────────┐
│                TRANSFORMER                    │
│                                              │
│  ┌─────────────────┐  ┌─────────────────┐   │
│  │    ENCODER       │  │    DECODER       │   │
│  │  (stack of N)    │  │  (stack of N)    │   │
│  │                  │  │                  │   │
│  │ ┌──────────────┐│  │ ┌──────────────┐ │   │
│  │ │Multi-Head    ││  │ │Masked MH     │ │   │
│  │ │Self-Attention││  │ │Self-Attention │ │   │
│  │ └──────┬───────┘│  │ └──────┬───────┘ │   │
│  │ ┌──────┴───────┐│  │ ┌──────┴───────┐ │   │
│  │ │Add & Norm    ││  │ │Add & Norm    │ │   │
│  │ └──────┬───────┘│  │ └──────┬───────┘ │   │
│  │ ┌──────┴───────┐│  │ ┌──────┴───────┐ │   │
│  │ │Feed-Forward  ││  │ │Cross-Attention│ │   │
│  │ └──────┬───────┘│  │ └──────┬───────┘ │   │
│  │ ┌──────┴───────┐│  │ ┌──────┴───────┐ │   │
│  │ │Add & Norm    ││  │ │Feed-Forward  │ │   │
│  │ └──────────────┘│  │ └──────────────┘ │   │
│  └────────┬────────┘  └────────┬────────┘   │
│           │                    │              │
│  ┌────────┴────────┐  ┌───────┴────────┐    │
│  │Positional       │  │Positional      │    │
│  │Encoding + Embed │  │Encoding + Embed│    │
│  └─────────────────┘  └────────────────┘    │
└──────────────────────────────────────────────┘

---

3. Positional Encoding

Transformer has no concept of "order" — need to add position to embeddings:

$$PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d_{model}}}\right)$$ $$PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{model}}}\right)$$

import torch
import math

class PositionalEncoding(nn.Module):
    def __init__(self, d_model, max_len=5000):
        super().__init__()
        pe = torch.zeros(max_len, d_model)
        position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)
        div_term = torch.exp(
            torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model)
        )
        pe[:, 0::2] = torch.sin(position * div_term)
        pe[:, 1::2] = torch.cos(position * div_term)
        pe = pe.unsqueeze(0)  # (1, max_len, d_model)
        self.register_buffer('pe', pe)

    def forward(self, x):
        return x + self.pe[:, :x.size(1)]

---

4. Encoder Block

class TransformerEncoderBlock(nn.Module):
    def __init__(self, d_model, num_heads, d_ff, dropout=0.1):
        super().__init__()
        self.self_attn = MultiHeadAttention(d_model, num_heads)
        self.ffn = nn.Sequential(
            nn.Linear(d_model, d_ff),
            nn.ReLU(),
            nn.Linear(d_ff, d_model),
        )
        self.norm1 = nn.LayerNorm(d_model)
        self.norm2 = nn.LayerNorm(d_model)
        self.dropout = nn.Dropout(dropout)

    def forward(self, x, mask=None):
        # Self-Attention + Residual + LayerNorm
        attn_out = self.self_attn(x, x, x, mask)
        x = self.norm1(x + self.dropout(attn_out))

        # Feed-Forward + Residual + LayerNorm
        ffn_out = self.ffn(x)
        x = self.norm2(x + self.dropout(ffn_out))
        return x

---

5. Transformer for different problems

Architecture	Usage section	Models	Math problem
Encoder-only	Encoders	BERT, RoBERTa	Classification, NER, QA
Decoder-only	Decoder	GPT, LLaMA	Text generation
Encoder-Decoder	Both	T5, BART, mBART	Translation, summarization

BERT (Encoder-only):
  Input: "The [MASK] sat on the mat"
  Output: "The cat sat on the mat"

GPT (Decoder-only):
  Input: "Once upon a time"
  Output: "Once upon a time, there was a..."

T5 (Encoder-Decoder):
  Input: "translate English to Vietnamese: Hello"
  Output: "Xin chào"

---

6. Compare Transformer vs RNN

Features	RNN/LSTM	Transformer
Parallelism	Sequential	Fully parallel
Long-range	Vanishing gradient	Direct attention
Speed ​​(training)	Slow	Much faster
Memory	O(1) per step	O(n²) attention matrix
Position	Implicit (sequence)	Explicit (positional encoding)

---

Summary

Ingredients	Role
Self-Attention	Connect every token to every token
Multi-Head	Many different perspectives
Positional Encoding	Add location information
Add & Norm	Residual connection + Layer Normalization
Feed-Forward	Nonlinear Transformation

---

Next article

Lesson 10: BERT — The first Pre-trained Language Model to completely change NLP: train once, fine-tune for every problem.