AI & LLM: From Basics to Advanced

Part 3: Training & Fine-tuning LLMs

xdev.asia

Lesson 12: PEFT — LoRA, QLoRA and Adapter Methods

1. The problem of Full Fine-Tuning

Fine-tuning the entire parameters of a large LLM poses huge challenges:

Memory required

Model	Number of parameters	VRAM (FP32)	VRAM (BF16)	VRAM + Adam (BF16)
GPT-2	117M	0.5 GB	0.25 GB	~1 GB
LLaMA-7B	7B	28 GB	14 GB	~56 GB
LLaMA-13B	13B	52 GB	26 GB	~104 GB
LLaMA-70B	70B	280 GB	140 GB	~560 GB

Adam optimizer saves momentum and variance for each parameter — 3 times the model weight memory. LLaMA-7B needs 4× A100 80GB for full fine-tune!

Other issues

• Catastrophic forgetting: model "forgets" pre-training knowledge
• Storage: each fine-tuned version = full model copy (14GB for 7B)
• Cost: thousands of USD for one training run

---

2. Parameter-Efficient Fine-Tuning (PEFT) Overview

PEFT is a family of techniques that only updates a small subset of parameters, leaving the rest unchanged:

Full Fine-Tuning:  cập nhật 100% params  → VRAM cao, chậm, tốn kém
PEFT:              cập nhật 0.1% - 5%    → VRAM thấp, nhanh, tiết kiệm

Main PEFT methods:

1. Adapter Layers: add small layers between transformer layers
2. Prefix Tuning: add trainable tokens to the beginning of the sequence
3. Prompt Tuning: only fine-tune soft prompt embeddings
4. LoRA: decomposes the weight update matrix into the product of two low rank matrices

---

3. LoRA: Low-Rank Adaptation

Mathematics of LoRA

Core idea: weight updates in fine-tuning often have low rankings.

Instead of learning directly:

W_new = W_original + ΔW   (ΔW có kích thước d×d, rất lớn)

LoRA factorize ΔW to:

ΔW = B × A

In which:

• W_original ∈ ℝ^(d×d): frozen, not updated
• A ∈ ℝ^(r×d): trainable, random Gaussian initialization
• B ∈ ℝ^(d×r): trainable, initialized to 0 (so that ΔW = 0 initially)
• r is rank (usually 4-64), r << d

Forward pass: h = W_original × x + (B × A) × x × (alpha/r)

Benefits of parameters:

d = 4096 (LLaMA-7B hidden dim)

Full ΔW:   4096 × 4096 = 16,777,216 tham số
LoRA r=16: (4096×16) + (16×4096) = 131,072 tham số
→ Giảm 128 lần!

Rank and Alpha

lora_config = LoraConfig(
    r=16,          # Rank: càng cao → càng nhiều params nhưng expressive hơn
    lora_alpha=32, # Scaling factor: alpha/r = scale của LoRA
    # Thường đặt alpha = 2*r hoặc alpha = r
    lora_dropout=0.05,  # Regularization
)

Instructions for choosing rank:

Rank (r)	Use cases	Number of trainable params (7B model)
4	Simple task, little data	~4M (~0.06%)
8	Good balance	~8M (~0.12%)
16	More complex tasks	~17M (~0.24%)
64	Large domain adaptation	~67M (~0.95%)

---

4. Select Target Modules

LoRA is applied to linear layers in the attention mechanism:

# Với LLaMA/Mistral architecture:
target_modules = [
    "q_proj",   # Query projection
    "v_proj",   # Value projection
    "k_proj",   # Key projection
    "o_proj",   # Output projection
    # Tùy chọn thêm:
    "gate_proj", # MLP gate
    "up_proj",   # MLP up
    "down_proj", # MLP down
]

Strategy:

# Minimal (nhanh, ít VRAM):
target_modules = ["q_proj", "v_proj"]

# Standard (cân bằng tốt):
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj"]

# Full attention + MLP (tốt nhất, nhưng chậm hơn):
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                  "gate_proj", "up_proj", "down_proj"]

---

5. QLoRA: 4-bit Quantization + LoRA

QLoRA (Dettmers et al., 2023) combines two techniques:

1. NF4 (NormalFloat 4-bit): quantize model weights down to 4-bit
2. Double Quantization: quantize both quantization constants
3. Paged Optimizers: optimizer memory management with CPU offloading

Why is NF4 better than INT4?

NF4 is designed for a normal distribution of model weights:

• LLM weights are normally distributed normally (Gaussian)
• NF4 allocates more bins in high density areas
• Significantly reduced quantization error compared to INT4

FP16 → NF4: giảm 75% bộ nhớ, chất lượng gần như giữ nguyên

VRAM Comparison (LLaMA-7B)

Method	VRAM Training	Quality
Full FT (FP16)	~56 GB	Baseline
LoRA (FP16)	~28 GB	~98% baseline
QLoRA (NF4)	~10 GB	~97% baseline

---

6. BitsAndBytes: Load Model 4-bit

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

# Cấu hình 4-bit quantization
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,              # Bật 4-bit loading
    bnb_4bit_quant_type="nf4",      # Dùng NF4 (tốt hơn fp4)
    bnb_4bit_compute_dtype=torch.bfloat16,  # Dtype cho compute
    bnb_4bit_use_double_quant=True, # Double quantization (tiết kiệm thêm ~0.4 bit/param)
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3-8B",
    quantization_config=bnb_config,
    device_map="auto",  # Tự động phân bổ layers lên GPU/CPU
)

# Kiểm tra bộ nhớ
print(f"Model footprint: {model.get_memory_footprint() / 1e9:.2f} GB")
# → ~4.5 GB thay vì ~16 GB (BF16)

---

7. Prefix Tuning and Prompt Tuning

Prompt Tuning

Just fine-tune the soft prompt embeddings (virtual embeddings tokens are added to the input):

from peft import PromptTuningConfig, TaskType, get_peft_model

config = PromptTuningConfig(
    task_type=TaskType.CAUSAL_LM,
    num_virtual_tokens=20,  # Số soft prompt tokens
    tokenizer_name_or_path="gpt2",
)
model = get_peft_model(model, config)
# trainable params: 15,360 / 124,475,648 → 0.01%!

Prefix Tuning

Add trainable "prefix" to the key and value of each attention layer:

from peft import PrefixTuningConfig

config = PrefixTuningConfig(
    task_type=TaskType.CAUSAL_LM,
    num_virtual_tokens=30,
    encoder_hidden_size=512,  # MLP để tạo prefix
)

Prefix Tuning is better than Prompt Tuning but worse than LoRA in many tasks. Use when very few trainable params are needed.

---

8. Full code: QLoRA Fine-tuning LLaMA with PEFT + TRL

import torch
from datasets import load_dataset
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
)
from peft import (
    LoraConfig,
    get_peft_model,
    prepare_model_for_kbit_training,
    TaskType,
)
from trl import SFTTrainer, SFTConfig

# ============================================================
# 1. Cấu hình
# ============================================================
MODEL_ID = "meta-llama/Meta-Llama-3-8B"
OUTPUT_DIR = "./llama3-8b-qlora-vi"
MAX_SEQ_LENGTH = 2048

# ============================================================
# 2. Load Tokenizer
# ============================================================
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"

# ============================================================
# 3. Load Model với 4-bit Quantization
# ============================================================
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    quantization_config=bnb_config,
    device_map="auto",
    attn_implementation="flash_attention_2",  # Faster attention
)

# Chuẩn bị model cho k-bit training
model = prepare_model_for_kbit_training(model)

# ============================================================
# 4. Cấu hình LoRA
# ============================================================
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
    lora_dropout=0.05,
    bias="none",
    task_type=TaskType.CAUSAL_LM,
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()

# ============================================================
# 5. Dataset: Vietnamese instruction data
# ============================================================
dataset = load_dataset("tatsu-lab/alpaca", split="train[:10000]")

def format_chat(example):
    messages = [
        {"role": "system", "content": "Bạn là trợ lý AI hữu ích, trả lời bằng tiếng Việt."},
        {"role": "user", "content": example["instruction"]
                                    + (f"\n\n{example['input']}" if example["input"] else "")},
        {"role": "assistant", "content": example["output"]},
    ]
    text = tokenizer.apply_chat_template(messages, tokenize=False)
    return {"text": text}

dataset = dataset.map(format_chat, remove_columns=dataset.column_names)

# ============================================================
# 6. Training Configuration
# ============================================================
training_args = SFTConfig(
    output_dir=OUTPUT_DIR,
    num_train_epochs=2,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=8,      # Effective batch = 16
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.05,
    bf16=True,
    gradient_checkpointing=True,
    gradient_checkpointing_kwargs={"use_reentrant": False},
    max_seq_length=MAX_SEQ_LENGTH,
    dataset_text_field="text",
    logging_steps=10,
    save_strategy="epoch",
    optim="paged_adamw_8bit",           # Paged optimizer cho QLoRA
    report_to="tensorboard",
)

# ============================================================
# 7. Train
# ============================================================
trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset,
    tokenizer=tokenizer,
)

trainer.train()

# ============================================================
# 8. Lưu LoRA weights (chỉ ~80MB, không phải toàn bộ model!)
# ============================================================
trainer.save_model(OUTPUT_DIR + "/lora-weights")
tokenizer.save_pretrained(OUTPUT_DIR + "/lora-weights")

# ============================================================
# 9. Merge LoRA vào base model (cho inference)
# ============================================================
from peft import PeftModel

base_model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
merged_model = PeftModel.from_pretrained(base_model, OUTPUT_DIR + "/lora-weights")
merged_model = merged_model.merge_and_unload()
merged_model.save_pretrained(OUTPUT_DIR + "/merged")
print("Done! Merged model saved.")

---

9. Comparison: Full FT vs LoRA vs QLoRA

Criteria	Full Fine-Tuning	LoRA (BF16)	QLoRA (NF4)
VRAM (7B model)	~56 GB	~28 GB	~10 GB
Required GPU	4× A100	2× A100	1× RTX 3090
Training speed	Fastest	Fast	~30% slower
Quality	100% (baseline)	~98-99%	~96-98%
Checkpoint size	14 GB	~80 MB	~80 MB
Easy to test	Difficult	Easy	Easiest
Catastrophic forgetting	Cao	Low	Low

When to use what?

• Full FT: Has many GPUs, large dataset, needs maximum quality
• LoRA: A100/H100 GPU, good balance between quality and resources
• QLoRA: consumer GPU (RTX 3090/4090), personal research, rapid prototype

---

Summary

• Full fine-tuning is not practical with model 7B+ on regular GPUs
• LoRA factorize weight updates to B × A with low rank — reduce 99%+ trainable params
• QLoRA combines NF4 quantization with LoRA — fine-tune 70B model on 2× RTX 3090
• target_modules important: start with q_proj, v_proj, expand if necessary
• After training, merge LoRA into the base model for inference without overhead

The next article will go into RLHF and Alignment — the process of training the model to not only follow instructions but also to be safe and consistent with human values.