簡介
「一切都會失敗，一直如此。」— 沃納‧沃格爾斯，亞馬遜首席技術長。高可用性 (HA) 並不是為了防止故障，而是為了在發生故障時系統繼續運作。
1. 可用性指標
1.1 九號
Availability Downtime/year Downtime/month Downtime/week
99% 3.65 days 7.31 hours 1.68 hours
99.9% 8.77 hours 43.8 minutes 10.1 minutes
99.95% 4.38 hours 21.9 minutes 5.04 minutes
99.99% 52.6 minutes 4.38 minutes 1.01 minutes
99.999% 5.26 minutes 26.3 seconds 6.05 seconds
Availability = Uptime / (Uptime + Downtime)
MTBF = Mean Time Between Failures
MTTR = Mean Time To Recover
Availability = MTBF / (MTBF + MTTR)
Tăng MTBF → Ít failures hơn (khó)
Giảm MTTR → Recovery nhanh hơn (dễ hơn!)
1.2 複雜系統的可用性
Sequential (cả 2 phải up):
A(99.9%) ──► B(99.9%)
System = 99.9% × 99.9% = 99.8%
Thêm components → Availability GIẢM!
Parallel (1 trong 2 up là đủ):
A(99.9%) ──┐
├──► System
B(99.9%) ──┘
System = 1 - (0.1% × 0.1%) = 99.9999%
Thêm redundancy → Availability TĂNG!
2. 冗餘模式
2.1 主動-被動（故障轉移）
Normal:
Traffic ──► Active Server (processing)
Passive Server (standby, syncing data)
Failover:
Traffic ──► Active Server ✗ (down!)
Passive Server → Promoted to Active
Traffic ──► New Active Server (was passive)
Types:
Hot Standby: Passive chạy sẵn, failover nhanh (<30s)
Warm Standby: Passive chạy nhưng không sync real-time
Cold Standby: Passive tắt, bật lên khi cần (phút-giờ)
2.2 主動-主動
Traffic ──► Load Balancer
├──► Server A (processing)
└──► Server B (processing)
Cả 2 servers đều nhận traffic
Nếu A down → B nhận 100% traffic
Không cần failover (tự động)
Tốt hơn Active-Passive nhưng phức tạp hơn
- Session management
- Data consistency
- Split-brain problem
2.3 多層冗餘
┌──────────────────────────────────────────┐
│ Region: Vietnam │
│ │
│ AZ-1 AZ-2 │
│ ┌────────────┐ ┌────────────┐ │
│ │ LB (active)│ │ LB (active)│ │
│ ├────────────┤ ├────────────┤ │
│ │ App × 3 │ │ App × 3 │ │
│ ├────────────┤ ├────────────┤ │
│ │ DB Primary │←───►│ DB Replica │ │
│ └────────────┘ └────────────┘ │
└──────────────────────────────────────────┘
Redundancy levels:
Process: Multiple app instances
Server: Multiple AZs (Availability Zones)
Region: Multi-region (cho global services)
3. 健康檢查
Types:
1. Liveness: "App còn sống không?"
GET /healthz → 200 OK
Fail → Restart container
2. Readiness: "App sẵn sàng nhận traffic?"
GET /readyz → 200 OK (DB connected, cache warm)
Fail → Remove from load balancer
3. Deep health check:
GET /health/detailed
{
"status": "healthy",
"checks": {
"database": { "status": "up", "latency": "5ms" },
"redis": { "status": "up", "latency": "1ms" },
"disk": { "status": "up", "free": "50GB" },
"memory": { "status": "warning", "used": "85%" }
}
}
Health Check Cascade:
Tránh: Service A health check gọi Service B
→ Service B slow → Service A "unhealthy" → Cascading!
Correct: Health check chỉ check LOCAL resources
4. 優雅降級
Khi một component fail, hệ thống vẫn hoạt động
với reduced functionality
Ví dụ: E-commerce
Normal:
Product page: title + description + reviews + recommendations
Review Service down:
Product page: title + description + "Reviews temporarily unavailable"
Recommendation Service down:
Product page: title + description + reviews + "Popular products" (static)
Search Service down:
Homepage: Categories navigation + "Search coming back soon"
Patterns:
1. Feature flags: Disable features instantly
2. Fallback values: Default/cached responses
3. Read-only mode: Disable writes, serve reads
4. Static content: Serve cached HTML khi backend down
5. 混沌工程
5.1 原則
"Inject failures INTENTIONALLY to discover weaknesses
BEFORE they surprise you in production"
Steps:
1. Define "steady state" (normal behavior)
2. Hypothesize: "System survives X failure"
3. Inject failure (kill server, add latency, corrupt data)
4. Observe: Did system behave as expected?
5. Fix: Address discovered weaknesses
Tools:
- Chaos Monkey (Netflix): Kill random instances
- Litmus Chaos: K8s chaos engineering
- Gremlin: Enterprise chaos platform
- Toxiproxy: Simulate network conditions
5.2 混沌實驗
Experiment 1: Kill an instance
Action: Terminate 1 of 3 app servers
Expected: LB routes to remaining 2, no user impact
Verify: Error rate unchanged, latency < 2x
Experiment 2: Network partition
Action: Block traffic between App and Database
Expected: Circuit breaker opens, cached responses served
Verify: Graceful error message, no crash
Experiment 3: Dependency slow
Action: Add 5s latency to Payment Service
Expected: Timeout after 3s, show "Try again later"
Verify: Other features unaffected (bulkhead)
Experiment 4: Disk full
Action: Fill disk to 100%
Expected: Alert triggered, log rotation kicks in
Verify: App doesn't crash, monitoring shows disk alert
6. 故障檢查表設計
□ Mọi external call có timeout
□ Circuit breakers cho downstream services
□ Retry với exponential backoff + jitter
□ Health check endpoints (liveness + readiness)
□ Graceful shutdown (drain connections)
□ Graceful degradation (fallbacks)
□ Data replication (ít nhất 2 copies)
□ Multi-AZ deployment
□ Auto-scaling configured
□ Runbooks cho common failures
□ Chaos experiments scheduled
□ Monitoring + alerting configured
□ Backup + restore tested regularly
總結
|戰略| MTBF 影響 |平均修復時間 (MTTR) 影響複雜性 | |----------|-------------|-------------|------------| | 冗餘 |中性| ⬇️ 快速故障轉移 |中等| |健康檢查| ⬆️ 及早發現 | ⬇️ 自動恢復 |低| |優雅的惡化|中性| ⬇️ 部分服務 |中等| |混沌工程| ⬆️ 尋找弱點 | ⬇️ 更好的操作手冊 |高| | 自動縮放| ⬆️ 處理尖刺 |中性|中等|
練習
-
可用性計算： 系統包括：LB（99.99%）→ 3台App Server（各99.9%，雙活）→ DB Primary（99.95%）+ DB Replica（99.95%，熱備）。計算總體可用性。
-
故障轉移設計： PostgreSQL 叢集：1 個主節點 + 2 個副本節點。主要崩潰。編寫詳細的故障轉移流程：偵測、升級、重連、驗證。
-
混沌計畫： 為電商系統設計5個混沌實驗（API、資料庫、Redis、S3、支付網關）。對於每個實驗：行動、假設、預期行為、回溯計畫。