xDev Asia

Compare tokenization methods: whitespace, BPE, WordPiece, Unigram, SentencePiece. Vocabulary size and trade-offs. Tokenizer training from scratch. Hugging Face Tokenizers library. Vietnamese and specific tokenization challenges.

NLP from Basics to Advanced: Mastering Natural Language Processing

Part 1: NLP Foundations — Understanding Language Through a Computer Lens

xdev.asia

Introduction

Tokenization is the first and most important step in every NLP pipeline. It determines how the model "looks" at the text — and directly affects performance.

💡 All modern LLMs (GPT-4, Gemini, Claude, LLaMA) use subword tokenization — and this article will explain why.

1. Why is Word-level Tokenization not enough?

OOV (Out-of-Vocabulary) Problem

vocab = {"hello", "world", "natural", "language"}

# Gặp từ mới → OOV!
text = "unhappiness"  # Không có trong vocab → [UNK]

# Word-level vocab cần KHỔNG LỒ
# Tiếng Anh: ~170,000 từ
# + Tên riêng, thuật ngữ, viết tắt → 500,000+
# + Đa ngôn ngữ → Hàng triệu từ

3 main problems

Problem	Word-level	Subword	Character-level
Vocab size	Huge (500K+)	Moderate (32K–128K)	Very small (256)
OOV	Many [UNK]	Rarely	Never
Semantic Meaning	Good	Good	Poor (each character)
Sequence length	Short	Moderate	Very long

2. Byte-Pair Encoding (BPE)

2.1 Algorithm

Bước 1: Bắt đầu với tất cả characters làm vocab
         Vocab: {a, b, c, ..., z, _}

Bước 2: Đếm tần suất các cặp adjacent tokens
         "l o w" → (l,o): 5, (o,w): 5
         "l o w e r" → (l,o): 5, (o,w): 5, (w,e): 2, (e,r): 2
         "n e w e r" → (n,e): 1, (e,w): 1, (w,e): 2, (e,r): 2

Bước 3: Merge cặp có tần suất cao nhất
         (l,o) → "lo"     Vocab: {a, b, ..., z, _, lo}

Bước 4: Lặp lại bước 2-3 cho đến khi đạt vocab size mong muốn
         "lo w" → (lo,w) → "low"
         Vocab: {a, b, ..., z, _, lo, low, ...}

2.2 BPE in practice

from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import Whitespace

# 1. Khởi tạo BPE tokenizer
tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
tokenizer.pre_tokenizer = Whitespace()

# 2. Training
trainer = BpeTrainer(
    vocab_size=30000,
    special_tokens=["[UNK]", "[PAD]", "[CLS]", "[SEP]", "[MASK]"],
    min_frequency=2,
)
tokenizer.train(files=["corpus.txt"], trainer=trainer)

# 3. Tokenize
output = tokenizer.encode("Xử lý ngôn ngữ tự nhiên rất thú vị")
print(output.tokens)
# ['X', 'ử', 'lý', 'ngôn', 'ngữ', 'tự', 'nhiên', 'rất', 'thú', 'vị']

Used by: GPT-2, GPT-3, GPT-4, LLaMA, RoBERTa

3. WordPiece

WordPiece is similar to BPE but uses likelihood instead of frequency:

BPE:       Merge cặp có tần suất CAO nhất
WordPiece: Merge cặp tối đa hóa LIKELIHOOD của training data

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")

text = "unhappiness is unbelievable"
tokens = tokenizer.tokenize(text)
print(tokens)
# ['un', '##happiness', 'is', 'un', '##believable']
# '##' = tiếp nối từ trước (không phải đầu từ)

Used by: BERT, DistilBERT, PhoBERT

4. SentencePiece & Unigram

SentencePiece

Treat text as raw bytes — no space pre-tokenization required:

import sentencepiece as spm

# Train
spm.SentencePieceTrainer.Train(
    input='corpus.txt',
    model_prefix='my_model',
    vocab_size=32000,
    model_type='unigram',  # hoặc 'bpe'
)

# Load & use
sp = spm.SentencePieceProcessor()
sp.Load('my_model.model')

text = "Xử lý ngôn ngữ tự nhiên"
tokens = sp.EncodeAsPieces(text)
print(tokens)
# ['▁Xử', '▁lý', '▁ngôn', '▁ngữ', '▁tự', '▁nhiên']
# '▁' = đầu từ mới

Used by: T5, ALBERT, XLNet, LLaMA (combined BPE)

5. Comparative Summary

Method	How to merge	Pre-tokenize?	Symbol	Models
BPE	Highest Frequency	Need (whitespace)	—	GPT, LLaMA, RoBERTa
WordPiece	Highest Likelihood	Need (whitespace)	`##` (continuation)	BERT, PhoBERT
Unigram	Type of token has little impact	No need	`▁` (word start)	T5, ALBERT
SentencePiece	BPE or Unigram	No need	`▁` (word start)	T5, LLaMA

6. Hugging Face Tokenizers — Practice

from transformers import AutoTokenizer

# So sánh tokenization giữa các model
models = [
    "bert-base-uncased",
    "gpt2",
    "google/flan-t5-base",
]

text = "Tokenization is surprisingly important for NLP"

for model_name in models:
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    tokens = tokenizer.tokenize(text)
    ids = tokenizer.encode(text)
    print(f"\n{model_name}:")
    print(f"  Tokens ({len(tokens)}): {tokens}")
    print(f"  IDs: {ids}")

7. Tokenization for Vietnamese

from transformers import AutoTokenizer

# PhoBERT tokenizer (WordPiece, có word segmentation)
phobert_tok = AutoTokenizer.from_pretrained("vinai/phobert-base-v2")
text = "Xử lý ngôn ngữ tự nhiên rất thú vị"
print(phobert_tok.tokenize(text))

# Gemma/LLaMA tokenizer (BPE, byte-level)
gemma_tok = AutoTokenizer.from_pretrained("google/gemma-2b")
print(gemma_tok.tokenize(text))
# Tiếng Việt thường bị tách thành nhiều subword hơn tiếng Anh
# → Sequence dài hơn → Tốn cost inference hơn!

🇻🇳 Insight: GPT-4 / Gemini's tokenizer splits Vietnamese into more tokens than English (~1.5-2x), resulting in higher API costs.

Summary

Key points	Details
Word-level	Vocab is too big, many OOV → not feasible
BPE	Merge by frequency, most common (GPT, LLaMA)
WordPiece	Merge by likelihood, use `##` (BERT)
SentencePiece	No need for pre-tokenization, language-agnostic
Vietnamese	English Tokenizer splits into more tokens → pay attention to cost

Lesson 4: Bag of Words, TF-IDF & N-grams — Classic text representation method but still effective in many problems.

Lesson 3: Tokenization Deep Dive — From Word to BPE, WordPiece, SentencePiece