xDev Asia

Handle a variety of document sources: PDF, DOCX, HTML/Web, YouTube transcripts, code repos. LangChain document loaders vs LlamaIndex readers.

Real Battle RAG: From Basic to Advanced

Part 2: Document Processing Pipeline

xdev.asia

Introduction

RAG pipeline starts from data — and data can be in any format: PDF, Word, web page, YouTube video, code repo... This lesson teaches you how to load all common document types into RAG pipeline.

Rule: Garbage in = Garbage out. Document loading high quality = RAG high quality.

1. Overview of Document Loaders

1.1 Popular data sources

Source	Format	Challenge
PDF	.pdf	Table, figure, multi-column, scan
Word	.docx, .doc	Formatting, headers/footers
Web Pages	HTML	JavaScript rendering, noise
YouTube	Video/Audio	Need transcript, timestamp
Code Repository	.py, .js, .md	Structure, dependencies
CSV/Excel	.csv, .xlsx	Tabular data → text
Database	SQL	Query, schema understanding

1.2 LangChain Document Loaders

LangChain provides a rich ecosystem of document loaders:

from langchain_community.document_loaders import (
    PyPDFLoader,          # PDF
    Docx2txtLoader,       # Word
    WebBaseLoader,        # Web pages
    YoutubeLoader,        # YouTube
    GitLoader,            # Git repos
    CSVLoader,            # CSV files
    TextLoader,           # Plain text
    UnstructuredExcelLoader,  # Excel
)

2. PDF — The most popular document type

2.1 PyPDFLoader — Basics

"""Load PDF đơn giản nhất — PyPDFLoader"""
from langchain_community.document_loaders import PyPDFLoader

# Load từng trang
loader = PyPDFLoader("company_handbook.pdf")
pages = loader.load()

print(f"Số trang: {len(pages)}")
for i, page in enumerate(pages[:3]):
    print(f"\n--- Trang {i+1} ---")
    print(f"Nội dung: {page.page_content[:200]}...")
    print(f"Metadata: {page.metadata}")
    # metadata = {"source": "company.pdf", "page": 0}

2.2 Advanced PDF — Handling tables and figures

PyPDF is simple but weak with tables and figures. Use Unstructured for complex PDFs:

"""Unstructured: xử lý PDF phức tạp (bảng, hình, multi-column)"""
from langchain_community.document_loaders import UnstructuredPDFLoader

# Partition strategy: "hi_res" cho chất lượng cao
loader = UnstructuredPDFLoader(
    "financial_report.pdf",
    mode="elements",           # Tách từng element (text, table, image)
    strategy="hi_res",         # OCR + layout analysis
)
elements = loader.load()

# Lọc theo loại element
for elem in elements[:10]:
    elem_type = elem.metadata.get("category", "unknown")
    print(f"[{elem_type}] {elem.page_content[:100]}...")

2.3 Compare PDF loaders

Loader	Table	Figure	OCR (scan)	Speed	Settings
PyPDFLoader	❌	❌	❌	⚡ Fast	`pip install pypdf`
PyMuPDFLoader	⚠️	❌	❌	⚡ Fast	`pip install pymupdf`
UnstructuredPDFLoader	✅	✅	✅	🐢 Slow	`pip install unstructured[pdf]`
Amazon Textract	✅	✅	✅	⚡	AWS accounts

Recommendation: Start with PyPDFLoader. If the PDF has tables/figures, use Unstructured.

3. Web Pages

3.1 Load website

"""Load nội dung từ trang web"""
from langchain_community.document_loaders import WebBaseLoader

# Load 1 trang
loader = WebBaseLoader("https://blog.xdev.asia/ai/")
docs = loader.load()

print(f"Content length: {len(docs[0].page_content)} chars")
print(docs[0].page_content[:500])

# Load nhiều trang
urls = [
    "https://blog.xdev.asia/blog/post-1",
    "https://blog.xdev.asia/blog/post-2",
    "https://blog.xdev.asia/blog/post-3",
]
loader = WebBaseLoader(urls)
docs = loader.load()
print(f"Loaded {len(docs)} pages")

3.2 Crawl website (Sitemap)

"""Crawl tất cả trang từ sitemap"""
from langchain_community.document_loaders.sitemap import SitemapLoader

loader = SitemapLoader(
    web_path="https://blog.xdev.asia/sitemap.xml",
    filter_urls=["https://blog.xdev.asia/blog/"],  # Chỉ blog
)
docs = loader.load()
print(f"Crawled {len(docs)} pages from blog")

4. YouTube Transcripts

"""Load transcript từ YouTube video"""
from langchain_community.document_loaders import YoutubeLoader

# Load transcript (auto-generated hoặc manual)
loader = YoutubeLoader.from_youtube_url(
    "https://www.youtube.com/watch?v=dQw4w9WgXcQ",
    add_video_info=True,         # Thêm title, description
    language=["vi", "en"],       # Ưu tiên tiếng Việt
)
docs = loader.load()

print(f"Title: {docs[0].metadata.get('title')}")
print(f"Duration: {docs[0].metadata.get('length')} seconds")
print(f"Transcript: {docs[0].page_content[:300]}...")

5. Code Repositories

"""Load code từ Git repo"""
from langchain_community.document_loaders import GitLoader

# Clone + load tất cả files
loader = GitLoader(
    clone_url="https://github.com/user/project",
    repo_path="./temp_repo",
    branch="main",
    file_filter=lambda f: f.endswith((".py", ".md", ".yaml")),
)
docs = loader.load()

print(f"Loaded {len(docs)} files")
for doc in docs[:5]:
    print(f"  {doc.metadata['file_path']} ({len(doc.page_content)} chars)")

6. Batch Processing — Processing multiple files

6.1 Load document folder

"""Load tất cả tài liệu trong folder"""
from langchain_community.document_loaders import DirectoryLoader, PyPDFLoader
from pathlib import Path

# Load tất cả PDF trong folder
loader = DirectoryLoader(
    "./company_docs/",
    glob="**/*.pdf",           # Recursive, chỉ PDF
    loader_cls=PyPDFLoader,
    show_progress=True,
    use_multithreading=True,   # Parallel loading
)
docs = loader.load()
print(f"Loaded {len(docs)} pages from PDF files")

# Thêm custom metadata
for doc in docs:
    filename = Path(doc.metadata["source"]).stem
    doc.metadata["department"] = "HR" if "hr" in filename.lower() else "General"
    doc.metadata["doc_type"] = "policy"

6.2 Multi-format loader

"""Unified loader cho nhiều format"""
from langchain_community.document_loaders import (
    PyPDFLoader, Docx2txtLoader, TextLoader, CSVLoader
)
from pathlib import Path

def load_document(file_path: str):
    """Load document dựa trên extension."""
    ext = Path(file_path).suffix.lower()
    
    loaders = {
        ".pdf": PyPDFLoader,
        ".docx": Docx2txtLoader,
        ".txt": TextLoader,
        ".md": TextLoader,
        ".csv": CSVLoader,
    }
    
    loader_cls = loaders.get(ext)
    if not loader_cls:
        raise ValueError(f"Unsupported format: {ext}")
    
    return loader_cls(file_path).load()

# Usage
all_docs = []
for f in Path("./docs/").glob("**/*"):
    if f.suffix in [".pdf", ".docx", ".txt", ".md", ".csv"]:
        try:
            docs = load_document(str(f))
            all_docs.extend(docs)
            print(f"✅ {f.name}: {len(docs)} chunks")
        except Exception as e:
            print(f"❌ {f.name}: {e}")

print(f"\nTotal: {len(all_docs)} documents loaded")

💡 Exercise 6: Write a script to load all documents in a folder (mix PDF + DOCX + TXT). Print out: file name, number of pages/chunks, total characters. Handle errors gracefully.

7. Tips & Best Practices

Tip	Explanation
Clean before loading	Delete headers/footers, page numbers, watermarks
Add metadata	Source, date, category — used for filtering later
Handle encoding	Vietnamese: UTF-8. Check `chardet` if unknown
Batch processing	Use multithread for large folders (100+ files)
Test quality	Spot-check 10% documents after loading
Error handling	Log files are corrupted, do not skip silently

Summary

Source	Loader	Notes
PDF	PyPDFLoader / Unstructured	Unstructured for tables/figures
Web	WebBaseLoader / SitemapLoader	Crawl full site via sitemap
YouTube	YoutubeLoader	Auto-transcript + metadata
Code	GitLoader	File filter for extensions
Folder	DirectoryLoader	Multithread, recursive

General exercises

✅ Complete small exercises (6)
Build Ingestion Pipeline: Write function ingest(folder_path) Load all files in the folder → return a list of documents with complete metadata.
PDF Benchmark: Take a complex PDF (with tables + pictures). Compare output between PyPDFLoader vs Unstructured. How does the quality differ?
Web Crawler: Crawl 10 blog posts from a favorite website → load into ChromaDB → create "Chat with Blog".

Next article: Chunking Strategies — how to divide documents into optimal chunks for retrieval.

Lesson 4: Document Loading — PDF, DOCX, Web, YouTube, Code