レッスン 18: サーキット ブレーカーと再試行パターン

クラウドネイティブのマイクロサービスアーキテクチャ

パート 6: 回復力パターン

xdev.asia

はじめに

マイクロサービスでは、障害は正常です。サービスはタイムアウト、クラッシュ、過負荷になります。問題は「失敗するかどうか」ではありません。しかし、「失敗はどのように処理されるのでしょうか?」

保護メカニズムがないと、サービスに障害が発生すると、カスケード障害によってシステム全体がダウンする可能性があります。この記事では、回復力のあるシステムを構築するための重要なパターンを紹介します。

---

1. カスケード障害 — 中心的な問題

1.1 典型的なカスケード障害シナリオ

Normal flow:
API Gateway → Order Service → Payment Service → DB

Payment Service chậm (DB overloaded):
        ↓
Order Service gọi Payment, chờ... timeout sau 30s
Threads của Order Service bị block 30s mỗi request
        ↓
New requests vào Order Service không có thread để xử lý
Order Service trở nên không phản hồi
        ↓
API Gateway không nhận response từ Order Service
API Gateway threads bị block
        ↓
Toàn bộ hệ thống đóng băng

カスケード障害が発生する理由: 失敗したダウンストリーム サービスへの呼び出しによってリソース (スレッド、接続、メモリ) が保持されている。

1.2 全体的なソリューション

┌─────────────────────────────────────────────────────┐
│              Resiliency Patterns                    │
│                                                     │
│  Timeout         — Không chờ vô thời hạn            │
│  Retry           — Thử lại khi lỗi tạm thời         │
│  Circuit Breaker — Ngừng gọi khi downstream lỗi    │
│  Bulkhead        — Cô lập resources theo group      │
│  Fallback        — Trả về giá trị mặc định          │
└─────────────────────────────────────────────────────┘

---

2. タイムアウトパターン

2.1 すべてのネットワーク呼び出しのタイムアウト

ルール: すべてのアウトバウンドネットワーク呼び出しにはタイムアウトが必要です。

// Không có timeout — NGUY HIỂM
HttpClient client = HttpClient.newHttpClient();
HttpResponse resp = client.send(request, BodyHandlers.ofString());

// Có timeout — ĐÚNG
HttpClient client = HttpClient.newBuilder()
    .connectTimeout(Duration.ofSeconds(2))
    .build();

HttpResponse resp = client.send(
    request.timeout(Duration.ofSeconds(5)),
    BodyHandlers.ofString()
);

2.2 タイムアウト階層

User Request Timeout:     30s  (tổng budget cho request)
API Gateway Timeout:      25s  (dư 5s buffer)
Service Call Timeout:     10s  (mỗi downstream call)
DB Query Timeout:          5s
Cache Timeout:             1s

Luật: Mỗi layer phải có timeout nhỏ hơn layer cha
để không bao giờ "child đã timeout trước khi parent biết"

2.3 タイムアウトはどれくらいに設定すればよいですか?

Phân tích baseline:
1. Đo p99 latency của operation khi bình thường: 150ms
2. Thêm buffer: 150ms × 3 = 450ms
3. Set timeout: 500ms

Không đặt quá lớn: timeout = 30s → thread bị block 30s
Không đặt quá nhỏ: timeout = 50ms → false positives khi GC pause

---

3. 再試行パターン

3.1 いつ再試行する必要がありますか?

Nên retry:
✅ Network timeout (tạm thời)
✅ 503 Service Unavailable
✅ 429 Too Many Requests
✅ Connection reset

Không retry:
❌ 400 Bad Request (lỗi input, retry cũng vô ích)
❌ 401/403 Unauthorized (cần fix auth, không phải retry)
❌ 404 Not Found
❌ Operation gây side effects (đừng retry POST payment!)

3.2 指数関数的バックオフ

Không có backoff — THUNDERING HERD:
  Attempt 1: T+0ms   → Fail
  Attempt 2: T+0ms   → Fail (tất cả clients retry cùng lúc → server càng quá tải)
  Attempt 3: T+0ms   → Fail

Exponential Backoff:
  Attempt 1: T+0ms      → Fail
  Attempt 2: T+100ms    → Fail
  Attempt 3: T+400ms    → Fail
  Attempt 4: T+1600ms   → Success ✓

  delay(n) = baseDelay × 2^(n-1)

3.3 ジッター — リトライの分散

Không có Jitter — vẫn thundering herd:
  10.000 clients đều retry tại T+100ms → spike

Với Jitter (Full Jitter):
  delay(n) = random(0, baseDelay × 2^(n-1))
  Clients retry tại các thời điểm khác nhau → hệ thống phục hồi dần

3.4 Resilience4j を使用したデプロイメント (Java)

// Config
RetryConfig retryConfig = RetryConfig.custom()
    .maxAttempts(4)
    .waitDuration(Duration.ofMillis(100))
    .intervalFunction(IntervalFunction.ofExponentialRandomBackoff(
        Duration.ofMillis(100),  // initial delay
        2.0,                     // multiplier
        Duration.ofSeconds(10)   // max delay
    ))
    // Chỉ retry với các exception phù hợp
    .retryOnException(e ->
        e instanceof ConnectTimeoutException ||
        (e instanceof HttpStatusCodeException se && se.getStatusCode() == 503)
    )
    // Không retry nếu đây là non-retryable error
    .ignoreExceptions(BadRequestException.class, UnauthorizedException.class)
    .build();

RetryRegistry registry = RetryRegistry.of(retryConfig);
Retry retry = registry.retry("payment-service");

// Sử dụng
Supplier<PaymentResponse> supplier = Retry.decorateSupplier(
    retry,
    () -> paymentClient.charge(request)
);

// Log retry attempts
retry.getEventPublisher().onRetry(event ->
    log.warn("Retry attempt {} for payment-service: {}",
        event.getNumberOfRetryAttempts(),
        event.getLastThrowable().getMessage())
);

---

4. サーキットブレーカーのパターン

4.1 3 つの状態

  ┌─────────────────────────────────────────────────────────────────┐
  │                                                                 │
  │         CLOSED                                                  │
  │    (Normal operation)                                           │
  │    - Requests flow through                                      │
  │    - Count failures                                             │
  │         │                                                       │
  │         │ failures > threshold (e.g., 50% in 10s window)       │
  │         ▼                                                       │
  │          OPEN                                                   │
  │    (Fail fast)                                                   │
  │    - Requests rejected immediately                              │
  │    - Return fallback response                                   │
  │    - No calls to downstream                                     │
  │         │                                                       │
  │         │ after wait duration (e.g., 30s)                      │
  │         ▼                                                       │
  │       HALF-OPEN                                                 │
  │    (Testing recovery)                                           │
  │    - Allow N probe requests through                             │
  │    - If success → CLOSED                                        │
  │    - If failure → OPEN again                                    │
  │                                                                 │
  └─────────────────────────────────────────────────────────────────┘

4.2 Resilience4j による実装

CircuitBreakerConfig cbConfig = CircuitBreakerConfig.custom()
    // Sliding window: đánh giá trên 10 calls gần nhất
    .slidingWindowType(SlidingWindowType.COUNT_BASED)
    .slidingWindowSize(10)
    // Mở circuit khi > 50% calls fail
    .failureRateThreshold(50)
    // Mở circuit khi > 50% calls slow (> 2s)
    .slowCallRateThreshold(50)
    .slowCallDurationThreshold(Duration.ofSeconds(2))
    // Ở trạng thái OPEN trong 30s trước khi thử lại
    .waitDurationInOpenState(Duration.ofSeconds(30))
    // Cho 5 probe requests khi HALF-OPEN
    .permittedNumberOfCallsInHalfOpenState(5)
    // Cần ít nhất 10 calls trước khi đánh giá
    .minimumNumberOfCalls(10)
    .build();

CircuitBreakerRegistry registry = CircuitBreakerRegistry.of(cbConfig);
CircuitBreaker cb = registry.circuitBreaker("payment-service");

// Kết hợp Circuit Breaker + Retry + Timeout
CircuitBreaker circuitBreaker = registry.circuitBreaker("payment-service");
TimeLimiter timeLimiter = TimeLimiter.of(Duration.ofSeconds(2));
Retry retry = RetryRegistry.ofDefaults().retry("payment-service");

// Thứ tự quan trọng: Retry → CircuitBreaker → TimeLimiter
Supplier<CompletableFuture<PaymentResponse>> supplier = TimeLimiter
    .decorateFutureSupplier(timeLimiter,
        CircuitBreaker.decorateSupplier(circuitBreaker,
            () -> CompletableFuture.supplyAsync(
                () -> paymentClient.charge(request)
            )
        )
    );

// Fallback
Try<PaymentResponse> result = Try.ofSupplier(
    Retry.decorateSupplier(retry, supplier.get()::get)
).recover(throwable -> {
    log.error("Payment service unavailable, using fallback", throwable);
    return PaymentResponse.pending(request.getOrderId());
});

4.3 アノテーションベース (より簡単)

@Service
public class PaymentServiceClient {

    @CircuitBreaker(name = "payment-service", fallbackMethod = "paymentFallback")
    @Retry(name = "payment-service")
    @TimeLimiter(name = "payment-service")
    public CompletableFuture<PaymentResponse> charge(ChargeRequest request) {
        return CompletableFuture.supplyAsync(
            () -> paymentClient.post("/charge", request, PaymentResponse.class)
        );
    }

    // Fallback phải có cùng signature + Throwable parameter
    public CompletableFuture<PaymentResponse> paymentFallback(
            ChargeRequest request, CallNotPermittedException e) {
        log.warn("Circuit OPEN for payment-service, returning pending response");
        return CompletableFuture.completedFuture(
            PaymentResponse.pending(request.getOrderId())
        );
    }

    public CompletableFuture<PaymentResponse> paymentFallback(
            ChargeRequest request, TimeoutException e) {
        log.error("Payment service timeout", e);
        return CompletableFuture.completedFuture(
            PaymentResponse.timeout(request.getOrderId())
        );
    }
}

# application.yml
resilience4j:
  circuitbreaker:
    instances:
      payment-service:
        sliding-window-size: 10
        failure-rate-threshold: 50
        wait-duration-in-open-state: 30s
        permitted-number-of-calls-in-half-open-state: 5
        minimum-number-of-calls: 10
        register-health-indicator: true  # Expose vào /actuator/health
  retry:
    instances:
      payment-service:
        max-attempts: 3
        wait-duration: 100ms
        exponential-backoff-multiplier: 2
        enable-exponential-backoff: true
  timelimiter:
    instances:
      payment-service:
        timeout-duration: 2s

---

5. フォールバック戦略

5.1 フォールバックの種類

1. Static Fallback
   → Trả về hardcoded giá trị mặc định
   → Ví dụ: Recommendation service down → trả top 10 sản phẩm best-seller cố định

2. Cache Fallback
   → Trả về data từ cache (có thể stale)
   → Ví dụ: Product catalog down → trả data từ Redis cache (refresh mỗi 1 phút)

3. Graceful Degradation
   → Disable tính năng, hiển thị thông báo
   → Ví dụ: Review service down → ẩn phần reviews, vẫn cho phép đặt hàng

4. Pending/Async Fallback
   → Nhận request, lưu vào queue xử lý sau
   → Ví dụ: Notification service down → lưu vào database, retry sau

5.2 実践例

@Service
public class ProductRecommendationService {

    @CircuitBreaker(name = "recommendation", fallbackMethod = "getCachedRecommendations")
    public List<Product> getRecommendations(String userId) {
        return recommendationEngine.getPersonalized(userId);
    }

    // Cache fallback — trả data cũ hơn thay vì lỗi
    public List<Product> getCachedRecommendations(String userId, Exception e) {
        log.warn("Recommendation engine unavailable, using cached data");

        List<Product> cached = cache.get("recommendations:" + userId);
        if (cached != null) return cached;

        // Nếu không có cache cá nhân hóa → dùng best-sellers
        return cache.get("best-sellers:top-10");
    }
}

---

6. 回復力のテスト

6.1 Resilience4j を使用した単体テスト

@Test
void whenPaymentServiceDown_shouldRetryAndFallback() {
    // Mock payment service liên tục fail
    when(paymentClient.charge(any()))
        .thenThrow(new ConnectTimeoutException("Connection refused"))
        .thenThrow(new ConnectTimeoutException("Connection refused"))
        .thenReturn(PaymentResponse.success());

    PaymentResponse result = subject.charge(request);

    // Verify retry happened
    verify(paymentClient, times(3)).charge(any());

    // Verify result is correct
    assertThat(result.getStatus()).isEqualTo("SUCCESS");
}

@Test
void whenCircuitIsOpen_shouldReturnFallbackImmediately() {
    // Force circuit OPEN
    CircuitBreaker cb = CircuitBreakerRegistry.ofDefaults()
        .circuitBreaker("payment-service");
    cb.transitionToOpenState();

    long start = System.currentTimeMillis();
    PaymentResponse result = subject.charge(request);
    long duration = System.currentTimeMillis() - start;

    // Phải return ngay lập tức (fail fast)
    assertThat(duration).isLessThan(50);
    // Phải là fallback response
    assertThat(result.getStatus()).isEqualTo("PENDING");
}

---

7. サーキットブレーカーの監視

Prometheus 経由でサーキット ブレーカーの状態を公開します。

# Spring Boot Actuator + Micrometer tự động export
# Metrics có sẵn:
resilience4j_circuitbreaker_state{name="payment-service"}  # 0=CLOSED, 1=OPEN, 2=HALF_OPEN
resilience4j_circuitbreaker_calls_total{name, kind, result}
resilience4j_circuitbreaker_failure_rate{name}

# Alert khi circuit OPEN
resilience4j_circuitbreaker_state{name="payment-service"} == 1

# Grafana panel: Circuit Breaker Status
- CLOSED: ✅ Xanh lá
- HALF-OPEN: ⚠️ Vàng
- OPEN: 🔴 Đỏ + Alert

---

概要

パターン	問題が解決しました	いつ使用するか
タイムアウト	スレッドの飢餓	すべてのネットワーク通話
再試行 + バックオフ + ジッター	一時的な障害	冪等な操作
サーキットブレーカー	カスケード障害	外部サービス呼び出し
フォールバック	劣化したエクスペリエンス	失敗が避けられないとき

適用順序: タイムアウト → 再試行 → サーキット ブレーカー → フォールバック

次の記事: バルクヘッド、レート制限、ヘルス チェック パターン