xDev Asia

Distillation from large → small model. Hyperparameter optimization. Integrated evaluation pipeline. A/B testing. Multi-task fine-tuning. Cost optimization.

Fine-tuning LLM: The Art of AI Tuning

Part 3: Fine-tuning on Google Gemini / Vertex AI

xdev.asia

Introduction

Fine-tuning once is a prototype, fine-tuning for production requires iteration, evaluation, and optimization.

1. Distillation: Large model → Small model

# Step 1: Dùng Gemini Pro (lớn) để generate training data
# Step 2: Fine-tune Gemini Flash (nhỏ) trên data đó
# → Flash performs gần Pro nhưng rẻ hơn 10x

def distill_dataset(prompts, teacher_model="gemini-2.0-pro"):
    training_data = []
    for prompt in prompts:
        response = client.models.generate_content(
            model=teacher_model, contents=prompt
        )
        training_data.append({
            "messages": [
                {"role": "user", "content": prompt},
                {"role": "model", "content": response.text}
            ]
        })
    return training_data

2. Hyperparameter Optimization

Param	Default	When to increase	When to reduce
Epochs	3	Small dataset (<200)	Large dataset (>2000)
Learning rate	1.0	Slow learning model	Model "forgets" old knowledge

3. A/B Testing Framework

import random

def ab_test(query, model_a, model_b, n_trials=100):
    results = {"a_wins": 0, "b_wins": 0, "tie": 0}
    for _ in range(n_trials):
        resp_a = call_model(model_a, query)
        resp_b = call_model(model_b, query)
        winner = llm_judge(query, resp_a, resp_b)
        results[winner] += 1
    return results

Summary

Distillation: distill large → small model knowledge, saving 10x inference
Hyperparameter tuning: epochs, learning rate, batch size
A/B testing required before moving to production
Multi-task fine-tuning: 1 model, many capabilities

Exercises