Lesson 16: Capstone — Fine-tune Model for actual Use Cases

Fine-tuning LLM: The Art of AI Tuning

Part 6: Production & Best Practices

xdev.asia

Introduction

Here's the summary — you'll build an end-to-end fine-tuning project from A → Z.

---

1. Project: Vietnamese Code Review Assistant

Architecture

┌────────────────────────────────────────────────┐
│           FINE-TUNING PIPELINE                  │
│                                                │
│  Data Collection  → Data Cleaning → Training   │
│  (GitHub PRs,       (Dedup,         (Gemini    │
│   code reviews)     quality score)   Flash SFT)│
│                                                │
│  Evaluation → A/B Testing → Production Deploy  │
│  (ROUGE,      (Base vs FT,   (Vertex AI       │
│   LLM-Judge,   100 queries)   Endpoint)        │
│   Golden Set)                                  │
└────────────────────────────────────────────────┘

Components Checklist

• Select use case & define success metrics
• Collect 200+ training examples
• Data cleaning & quality scoring
• Fine-tune on Gemini Flash (Vertex AI)
• Fine-tune on open-source (LoRA, for comparison)
• Multi-layer evaluation pipeline
• Catastrophic forgetting check
• A/B testing base vs fine-tuned
• Cost analysis & ROI report
• Deploy production endpoint
• Monitoring setup

2. Step-by-step

Phase 1: Data (2–4 hours)

• Collect 200+ examples
• Clean, format, split (80/10/10)
• Quality review random 20 samples

Phase 2: Training (1–2 hours)

• Fine-tune Gemini Flash on Vertex AI
• Fine-tune LLaMA с LoRA (comparison)
• 3 experiments: epochs 2, 3, 5

Phase 3: Evaluation (2–3 hours)

• Automated metrics: ROUGE, BERTScore
• LLM-as-Judge: 50 test cases
• Golden test set: 30 curated cases
• Catastrophic forgetting: 20 general questions

Phase 4: Production (1 hour)

• Deploy best model
• Monitoring setup
• Cost analysis report

---

3. Best Practices Summary

✅ DO:
- Start with prompt engineering (free!)
- Invest 70% time in data quality
- Use multi-layer evaluation
- Version control everything
- Calculate ROI before and after
- Monitor in production

❌ DON'T:
- Fine-tune without trying PE/RAG first
- Use raw, unclean data
- Evaluate by "vibes" — use metrics
- Ignore catastrophic forgetting
- Skip A/B testing
- Forget about ongoing maintenance cost

---

🎉 Congratulations!

You have completed Fine-tuning LLM: The Art of AI Tuning! You can:

1. The Right Decision: Fine-tune vs RAG vs Prompt Engineering
2. Fine-tune on 3 platforms: Google Gemini, OpenAI, LoRA open-source
3. Scientific Evaluation: BLEU, ROUGE, BERTScore, LLM-as-Judge, Human Eval
4. Cost calculation: ROI calculator, budget planning, cost optimization
5. Deploy production: Monitoring, A/B testing, drift detection

Final exercise

1. Complete the end-to-end capstone project
2. Write an evaluation report (3+ pages) with specific metrics
3. Publish the model or share findings with the community
4. Identify next use case to fine-tune