Lesson 4: Image Captioning — From Attention to Transformer

Multimodal AI: Combining Vision, Language & More

Part 2: Vision-Language Models (VLMs)

xdev.asia

Introduction

Show-Attend-Tell. Transformer-based captioning. BLIP, BLIP-2. CIDEr, METEOR metrics. Vietnamese image captioning.

---

1. Overview

Key concepts

Image Captioning — From Attention to Transformer is an important topic in the field of modern AI.

---

2. Architecture & Principles

Core Architecture

# Example implementation
import torch
import torch.nn as nn

class ExampleModel(nn.Module):
    def __init__(self, input_dim, output_dim):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(input_dim, 256),
            nn.ReLU(),
            nn.Dropout(0.2),
            nn.Linear(256, 128),
            nn.ReLU(),
            nn.Linear(128, output_dim),
        )
    
    def forward(self, x):
        return self.net(x)

---

3. Practice

Setup

pip install torch transformers datasets

Training Pipeline

# Training loop
model = ExampleModel(input_dim=768, output_dim=10)
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-4)
criterion = nn.CrossEntropyLoss()

for epoch in range(10):
    for batch in train_loader:
        optimizer.zero_grad()
        outputs = model(batch["input"])
        loss = criterion(outputs, batch["label"])
        loss.backward()
        optimizer.step()

---

4. Best Practices

Aspect	Recommendation
Data	Quality over quantity
Model	Start simple, scale up
Training	Monitor loss curves
Evaluation	Use appropriate metrics

---

Summary

Concepts	Key Takeaway
Architecture	Suitable for the problem
Training	Careful hyperparameter tuning
Evaluation	Multiple metrics