Lesson 7: ControlNet & Image-to-Image — Controlling Output

Generative AI: Create Images & Videos with AI

Part 3: Practicing Advanced Image Generation

xdev.asia

Introduction

Text-to-image is powerful but difficult to control precisely the layout, pose, or style. ControlNet solves this problem: adds spatial conditioning (canny edge, depth, pose) to control output while preserving creativity.

---

1. Image-to-Image Pipeline

from diffusers import StableDiffusionImg2ImgPipeline
from PIL import Image

pipe = StableDiffusionImg2ImgPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
)
pipe.to("cuda")

init_image = Image.open("sketch.png").resize((1024, 1024))

image = pipe(
    prompt="a detailed digital painting of a castle",
    image=init_image,
    strength=0.75,      # 0.0 = no change, 1.0 = full generation
    guidance_scale=7.5,
    num_inference_steps=30,
).images[0]

Strength parameter

strength=0.3  →  Giữ nguyên phần lớn input, chỉ enhance
strength=0.5  →  Balance giữa input và generation
strength=0.75 →  Generate nhiều, giữ composition
strength=1.0  →  Hoàn toàn mới (giống txt2img)

---

2. ControlNet — Spatial Control

from diffusers import StableDiffusionXLControlNetPipeline, ControlNetModel
import cv2
import numpy as np

# Load ControlNet model (canny edge)
controlnet = ControlNetModel.from_pretrained(
    "diffusers/controlnet-canny-sdxl-1.0",
    torch_dtype=torch.float16,
)

pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    controlnet=controlnet,
    torch_dtype=torch.float16,
)
pipe.to("cuda")

# Extract canny edges from reference image
image = cv2.imread("reference.jpg")
edges = cv2.Canny(image, 100, 200)
control_image = Image.fromarray(edges)

# Generate with ControlNet
result = pipe(
    prompt="a futuristic city, cyberpunk style",
    image=control_image,
    controlnet_conditioning_scale=0.7,  # strength of control
    num_inference_steps=30,
).images[0]

---

3. ControlNet Types

Canny Edge

# Detect edges → preserve structure/outlines
edges = cv2.Canny(image, 100, 200)
# Use case: maintain composition, change style

Depth Map

from transformers import pipeline
depth_estimator = pipeline("depth-estimation", model="Intel/dpt-hybrid-midas")
depth = depth_estimator(image)["depth"]
# Use case: maintain 3D structure, change scene

OpenPose (Human Pose)

# Detect skeleton → control body pose
from controlnet_aux import OpenposeDetector
openpose = OpenposeDetector.from_pretrained("lllyasviel/ControlNet")
pose = openpose(image)
# Use case: maintain pose, change character appearance

Segmentation Map

# Semantic regions → control spatial layout
from controlnet_aux import SamDetector
sam = SamDetector.from_pretrained("ybelkada/segment-anything")
seg_map = sam(image)
# Use case: control where each element goes

Scribble / Line Art

# Hand-drawn sketches → detailed images
from controlnet_aux import PidiNetDetector
pidi = PidiNetDetector.from_pretrained("lllyasviel/Annotators")
scribble = pidi(image, safe=True)

---

4. Multi-ControlNet

from diffusers import StableDiffusionXLControlNetPipeline, ControlNetModel

# Combine multiple controlnets
controlnets = [
    ControlNetModel.from_pretrained("controlnet-canny-sdxl-1.0"),
    ControlNetModel.from_pretrained("controlnet-depth-sdxl-1.0"),
]

pipe = StableDiffusionXLControlNetPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    controlnet=controlnets,
    torch_dtype=torch.float16,
)

result = pipe(
    prompt="...",
    image=[canny_image, depth_image],
    controlnet_conditioning_scale=[0.7, 0.5],  # per-controlnet weight
).images[0]

---

5. IP-Adapter — Style Transfer

from diffusers import StableDiffusionXLPipeline
from diffusers.utils import load_image

pipe = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
)
pipe.load_ip_adapter(
    "h94/IP-Adapter",
    subfolder="sdxl_models",
    weight_name="ip-adapter_sdxl.bin"
)

# Style image determines the aesthetic
style_image = load_image("style_reference.jpg")

pipe.set_ip_adapter_scale(0.6)  # strength of style transfer

result = pipe(
    prompt="a cat sitting on a windowsill",
    ip_adapter_image=style_image,
    num_inference_steps=30,
).images[0]

---

6. ControlNet conditioning_scale Guide

scale=0.0  →  Ignore control (pure text-to-image)
scale=0.3  →  Subtle guidance, high creativity
scale=0.5  →  Balanced control and creativity
scale=0.7  →  Strong control (recommended)
scale=1.0  →  Strict adherence to control image
scale=1.5+ →  Over-constrained (artifacts)

---

Summary

Engineering	Input → Output
img2img	Reference image → styled variation
ControlNet Canny	Edge lines → preserve structure
ControlNet Depth	Depth map → maintain 3D layout
ControlNet Pose	Skeleton → control body pose
IP-Adapter	Style image → transfer aesthetic
Multi-ControlNet	Multiple controls → combined

📌 Next article: Inpainting, Outpainting & Image Editing with AI.