xDev Asia

Data cleaning pipeline: dedup, filtering, quality scoring. Data augmentation techniques. Handling imbalance and edge cases. Tokenization deep-dive. Train/Val/Test split.

Fine-tuning LLM: The Art of AI Tuning

Part 2: Data Preparation — The foundation of all success

xdev.asia

Introduction

Raw data is rarely ready for fine-tuning. This article builds a professional data processing pipeline.

1. Data Cleaning Pipeline

class DataCleaner:
    def __init__(self):
        self.stats = {"total": 0, "removed": 0, "cleaned": 0}
    
    def clean(self, examples):
        results = []
        for ex in examples:
            self.stats["total"] += 1
            
            # Step 1: Remove duplicates
            if self.is_duplicate(ex): continue
            
            # Step 2: Filter too short/long
            if not self.valid_length(ex, min_tokens=20, max_tokens=4096): continue
            
            # Step 3: Quality scoring
            if self.quality_score(ex) < 0.7: continue
            
            # Step 4: Format validation
            ex = self.normalize_format(ex)
            
            results.append(ex)
            self.stats["cleaned"] += 1
        
        return results

2. Tokenization Deep-dive

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o-mini")

def analyze_dataset(examples):
    token_counts = []
    for ex in examples:
        text = json.dumps(ex, ensure_ascii=False)
        tokens = len(enc.encode(text))
        token_counts.append(tokens)
    
    print(f"Total examples: {len(token_counts)}")
    print(f"Total tokens: {sum(token_counts):,}")
    print(f"Avg tokens/example: {sum(token_counts)/len(token_counts):.0f}")
    print(f"Training cost (3 epochs, $3/1M): ${sum(token_counts)*3*3/1_000_000:.2f}")

3. Data Augmentation

Paraphrasing: Rewrite the question/answer keeping the meaning intact
Language mixing: Add example of English-Vietnamese mixing
Edge cases: Create examples for rare situations
Negative examples: Teach the model "what not to answer"

Summary

Data cleaning pipeline: dedup → filter → quality score → normalize
Tokenization analysis helps calculate accurate costs
Augmentation increases diversity without needing to collect more
Train/Val/Test split: 80/10/10 or 90/5/5

Exercises