簡介
生產準備並不是最後一步，而是從第一天起就必須滿足的標準。課程的最後一課將所有知識綜合成一個實用的清單、建議的技術堆疊和一個從零到生產部署微服務系統的 4 階段路線圖。
1. 架構決策清單
1.1 什麼時候應該使用微服務？
✅ Phù hợp với Microservices khi:
□ Team > 20 người, chia thành nhiều team nhỏ (2-pizza rule)
□ Cần scale độc lập từng phần (payment cần scale nhiều hơn catalog)
□ Các domain có tốc độ thay đổi khác nhau (checkout vs reporting)
□ Cần polyglot technology (ML service = Python, core = Java)
□ Traffic peak asymmetric (flash sale chỉ ảnh hưởng order service)
❌ Không nên dùng Microservices khi:
□ Team < 10 người — overhead > benefit
□ Domain chưa rõ ràng — microservices đòi hỏi stable boundaries
□ Không có DevOps maturity — không có CI/CD, monitoring, container platform
□ Startup phase — premature optimization, YAGNI
□ Simple CRUD app — monolith đơn giản hơn nhiều
1.2 服務分解決策
Domain → Bounded Context → Service
Câu hỏi trước khi tách service:
1. Service có thể deploy độc lập không?
2. Service có team riêng maintain không?
3. Có cần scale độc lập không?
4. Domain boundary có rõ ràng không?
5. Data coupling có low không?
Nếu ≥ 4 câu YES → tách service là hợp lý
Nếu < 3 câu YES → giữ trong monolith hoặc module
2. 技術堆疊推薦
2.1 核心平台
┌─────────────────────────────────────────────────────────────┐
│ Recommended Stack 2024 │
├──────────────────────┬──────────────────────────────────────┤
│ Container Runtime │ containerd (không dùng Docker) │
│ Orchestration │ Kubernetes (EKS/GKE/AKS hoặc k3s) │
│ Service Mesh │ Istio (lớn) / Cilium (nhỏ/đơn giản) │
│ API Gateway │ Kong / Traefik / Nginx │
│ Service Registry │ Kubernetes DNS + Endpoints │
├──────────────────────┼──────────────────────────────────────┤
│ Message Broker │ Apache Kafka (high throughput) │
│ │ NATS JetStream (lightweight) │
├──────────────────────┼──────────────────────────────────────┤
│ Observability │ Prometheus + Grafana (metrics) │
│ │ Grafana Loki (logs) │
│ │ Tempo / Jaeger (traces) │
│ │ OpenTelemetry (instrumentation) │
├──────────────────────┼──────────────────────────────────────┤
│ Storage │ PostgreSQL (relational) │
│ │ MongoDB (document) │
│ │ Redis (cache/session) │
│ │ Elasticsearch (search) │
├──────────────────────┼──────────────────────────────────────┤
│ CI/CD │ GitHub Actions / GitLab CI │
│ GitOps │ ArgoCD │
│ Secrets │ HashiCorp Vault │
│ Registry │ Harbor │
└──────────────────────┴──────────────────────────────────────┘
2.2 語言/框架推薦
Backend Services:
- Java/Kotlin + Spring Boot 3.x → enterprise, mature ecosystem
- Go → performance-critical, CLI tools
- Node.js/TypeScript → BFF, real-time, high I/O
- Python → ML services, data processing
Frontend/BFF:
- Next.js (React) → SSR + BFF pattern
- GraphQL (Apollo/gqlgen) → flexible API aggregation
Database:
- PostgreSQL 16+ → default choice for relational
- Redis 7+ → cache, pub/sub, streams
- Kafka 3.x → event streaming, CDC
3. 4階段實施路線圖
第一階段：基礎（一月至二月）
Mục tiêu: Core infrastructure, CI/CD, observability cơ bản
INFRA:
□ Kubernetes cluster (managed: EKS/GKE/AKS hoặc self-managed: k3s)
□ Container registry (Harbor hoặc ECR/GCR/ACR)
□ DNS và Load Balancer (cert-manager cho TLS)
□ Namespace strategy (dev / staging / prod)
CI/CD:
□ Git repository structure (monorepo vs polyrepo — quyết định sớm)
□ GitHub Actions / GitLab CI pipelines
□ Docker build + push pipeline
□ ArgoCD install + app-of-apps pattern
OBSERVABILITY:
□ Prometheus + Grafana (Kube Prometheus Stack)
□ Grafana Loki (log aggregation)
□ Basic alerting rules (CPU, memory, pod restarts)
BASELINE METRICS:
□ DORA metrics tracking (deploy frequency, lead time, MTTR, change failure rate)
DELIVERABLE: Có thể deploy hello-world service trong < 5 phút
第 2 階段：核心平台（3 月至 5 月）
Mục tiêu: Service communication, data management, security cơ bản
SERVICES:
□ API Gateway (Kong/Traefik)
□ Service-to-service communication pattern (REST/gRPC/events)
□ Message broker (Kafka hoặc NATS)
□ Service discovery (Kubernetes DNS)
DATA:
□ Database per service (mỗi service có DB riêng)
□ Data migration strategy (Flyway / Liquibase)
□ Read replicas cho read-heavy services
□ Redis caching layer
SECURITY:
□ mTLS giữa services (Istio / Cilium) hoặc manual gRPC TLS
□ OAuth2 + JWT authentication (Keycloak)
□ Network Policies
□ Pod Security Standards (baseline)
□ Vault cho secrets management
RESILIENCE:
□ Circuit breaker (Resilience4j)
□ Retry với exponential backoff
□ Resource limits trên tất cả containers
□ Readiness + Liveness + Startup probes
DELIVERABLE: Core business services deploy thành công, auth hoạt động
第 3 階段：進階（6 月至 9 月）
Mục tiêu: Distributed systems patterns, advanced observability, reliability
PATTERNS:
□ Saga pattern cho distributed transactions
□ Event Sourcing cho audit-critical services
□ CQRS (nếu read/write model khác nhau)
□ Outbox pattern cho reliable event publishing
OBSERVABILITY (nâng cao):
□ Distributed tracing (OpenTelemetry + Tempo/Jaeger)
□ Custom Grafana dashboards (RED method per service)
□ SLI/SLO definitions và error budget alerts
□ Synthetic monitoring (Blackbox Exporter)
DEPLOYMENT:
□ Canary deployments (Argo Rollouts)
□ Feature flags (Unleash / LaunchDarkly)
□ Blue/Green cho stateful services
SECURITY (nâng cao):
□ Dynamic secrets (Vault DB secrets)
□ Container image signing (Cosign)
□ Vulnerability scanning tích hợp CI (Trivy)
□ Falco runtime security
CHAOS:
□ Chaos Engineering baseline (LitmusChaos)
□ Quarterly Game Days
DELIVERABLE: SLOs defined cho tất cả core services, chaos testing hoạt động
第 4 階段：最佳化（10 月以上）
Mục tiêu: Performance, cost optimization, advanced automation
PERFORMANCE:
□ Profiling services (async profiler / pprof)
□ Database query optimization (slow query log, indexes)
□ Caching strategy tinh chỉnh (cache hit rates > 80%)
□ Connection pool tuning
AUTO-SCALING:
□ HPA cho tất cả services (CPU + custom metrics)
□ VPA recommendations (resource right-sizing)
□ KEDA cho event-driven autoscaling (Kafka lag)
□ Cluster Autoscaler / Karpenter
COST:
□ Spot instances cho non-critical workloads
□ Namespace resource quotas
□ Idle resource cleanup (K8s Goldilocks)
□ Cost allocation by service / team
MULTI-CLUSTER (nếu cần):
□ Multi-region deployment
□ Global load balancing
□ Cross-region replication
DELIVERABLE: Infrastructure cost giảm ≥ 20%, SLO maintained ≥ 99.9%
4. 容量規劃
4.1 尺寸服務
Rule of thumb cho JVM services:
- Memory request: heap_max * 1.5 + 200Mi (JVM overhead)
Ví dụ: heap 512Mi → request 968Mi ≈ 1Gi
- CPU request: throughput / 1000 * 2 (2ms per req avg)
Ví dụ: 500 RPS → CPU request: 1.0 cores
Go services:
- Memory: 50–200Mi (much lower than JVM)
- CPU: throughput / 2000 (faster per req)
Node.js:
- Memory: 256–512Mi
- CPU: throughput / 1500
4.2 負載測試基線
# k6 load test baseline trước production
cat <<EOF > load-test.js
import http from 'k6/http';
import { check, sleep } from 'k6';
export const options = {
stages: [
{ duration: '2m', target: 100 }, // Ramp up
{ duration: '5m', target: 100 }, // Steady state
{ duration: '2m', target: 500 }, // Spike test
{ duration: '5m', target: 500 }, // Hold spike
{ duration: '2m', target: 0 }, // Ramp down
],
thresholds: {
http_req_duration: ['p(95)<500'], // 95th percentile < 500ms
http_req_failed: ['rate<0.01'], // Error rate < 1%
},
};
export default () => {
const r = http.get('https://api.example.com/orders');
check(r, { 'status is 200': (r) => r.status === 200 });
sleep(1);
};
EOF
k6 run load-test.js
5. 運行手冊模板
每項服務都必須有一個運作手冊。最小模板：
# Runbook: Order Service
## Service Overview
- Owner: Order Team
- PagerDuty: order-service
- Dashboard: https://grafana/d/order-service
- Source: https://github.com/company/order-service
## SLO
- Availability: 99.9% (43.8 min downtime/month)
- Latency: P99 < 500ms
## Common Incidents
### High Error Rate
**Alert**: `OrderServiceErrorRate > 5%`
**Procedure**:
1. `kubectl logs -n prod -l app=order-service --tail=100`
2. Kiểm tra database connection: `kubectl exec -it postgres-0 -- psql -U orders -c "SELECT count(*) FROM pg_stat_activity"`
3. Nếu DB connection pool exhausted: `kubectl rollout restart deploy/order-service`
4. Nếu vẫn fail: activate circuit breaker manually, alert on-call
### High Latency
**Alert**: `OrderServiceP99Latency > 2s`
**Procedure**:
1. Mở trace dashboard, filter by slow spans
2. Kiểm tra slow queries: `kubectl exec postgres-0 -- psql -c "SELECT * FROM pg_stat_statements ORDER BY total_time DESC LIMIT 10"`
3. Kiểm tra Kafka consumer lag: `kafka-consumer-groups.sh --describe --group order-consumer`
### Pod CrashLoopBackOff
**Procedure**:
1. `kubectl describe pod <pod> -n prod`
2. `kubectl logs <pod> -n prod --previous`
3. Nếu OOMKilled: tăng memory limit hoặc profile memory leak
4. Nếu config error: kiểm tra ConfigMap/Secret đúng chưa
## Rollback Procedure
```巴什
# 將 ArgoCD 應用程式回滾到先前的版本
argocd 應用程式回滾訂單服務 --revision PREVIOUS_SYNC_ID
# 或回滾 Helm 版本
helm rollback order-service -n prod 0 # 0 = 之前的修訂版
\```
## Escalation
- L1 (0–15 min): On-call engineer
- L2 (15–30 min): Engineer lead
- L3 (30+ min): CTO notification
6. 災難復原計劃
6.1 RTO/RPO 目標
Tier 1 — Payment, Authentication:
RTO: 1 hour (phục hồi trong 1 giờ)
RPO: 15 phút (mất tối đa 15 phút data)
→ Multi-AZ, database replication, automated failover
Tier 2 — Order, Inventory:
RTO: 4 giờ
RPO: 1 giờ
→ DB backup mỗi giờ, manual failover procedure
Tier 3 — Reporting, Analytics:
RTO: 24 giờ
RPO: 24 giờ
→ Daily backup là đủ
6.2 災難復原清單
Backup:
□ Database backup tự động (Velero cho PVC, pg_dump cho PostgreSQL)
□ Backup retention theo tier (Tier 1: 30 ngày, Tier 2: 7 ngày, Tier 3: 3 ngày)
□ Backup restore test mỗi tháng (automated DR drill)
□ Cross-region backup storage
Application State:
□ Stateless services (state trong database/cache — easy to restore)
□ Kafka topics replicated across brokers
□ Redis sentinel / cluster mode cho high availability
Infrastructure:
□ Infrastructure as Code (Terraform / Pulumi) — rebuilt từ code
□ GitOps state (ArgoCD) — redeploy từ Git
□ Secrets backed up trong Vault (HA mode)
Recovery Drills:
□ DR drill quarterly: simulate AZ failure
□ Data restore test monthly: restore từ backup vào staging
□ Chaos day biannual: full DR walkthrough
7. DORA 指標與團隊健康狀況
7.1 DORA 指標追蹤
Elite Performance (mục tiêu):
- Deploy Frequency: Multiple deploys/day
- Lead Time for Change: < 1 giờ (code merged → production)
- MTTR: < 1 giờ
- Change Failure Rate: < 5%
Đo lường:
- Deploy Frequency: count(ArgoCD sync) per day
- Lead Time: git tag timestamp → ArgoCD sync timestamp
- MTTR: PagerDuty alert triggered → resolved
- Change Failure Rate: rollbacks / total deploys
8.整體架構－最終視圖
Internet
│
[CloudFlare WAF]
│
[Global Load Balancer]
│
┌─────────────┴─────────────┐
│ │
[Region A] [Region B]
│
[Kubernetes Cluster]
│
┌─────────┴─────────────────────┐
│ Ingress / API Gateway │
│ (Kong + cert-manager + TLS) │
└─────────┬──────────────────────┘
│
┌─────────┴──────────────┐
│ Service Mesh (Istio) │
│ mTLS between all pods │
└─────────┬──────────────┘
│
┌─────────┴────────────────────────────┐
│ Core Services │
│ │
│ [Auth] [Order] [Payment] [Catalog]│
│ └────────┬──────────┘ │
│ [Kafka] │
└─────────┬─────────────────┬──────────┘
│ │
┌─────────┴───┐ ┌────────┴───────┐
│ Data Layer │ │ Platform │
│ PostgreSQL │ │ Vault │
│ Redis │ │ ArgoCD │
│ Elasticsearch │ Harbor │
└─────────────┘ │ Prometheus │
│ Grafana/Loki │
└────────────────┘
9. 課程總結
您已完成雲端原生微服務架構 - 從理論到生產就緒的實作：
| 部分 | 主題 | 要點 |
|---|---|---|
| 1 | 雲端原生基礎 | 12 要素應用程式、容器、Kubernetes 基礎知識 |
| 2 | 設計與傳播 | 領域驅動設計、REST/gRPC/事件 |
| 3 | 資料管理 | 每個服務的資料庫、Saga、CQRS、事件溯源 |
| 4 | 服務網格 | Istio、mTLS、流量管理 |
| 5 | 可觀察性 | 普羅米修斯、Grafana、Loki、OpenTelemetry |
| 6 | 彈性 | 斷路器、艙壁、速率限制、混亂 |
| 7 | CI/CD 與部署 | GitOps、ArgoCD、Canary、藍/綠 |
| 8 | 安全與生產 | JWT、Vault、容器安全、運作手冊 |
您的下一個：
- 選擇要從單體 → 微服務轉換的服務
- 設定 Kubernetes 叢集（k3s 本機或託管雲端）
- 實施 CI/CD 管道和可觀察性堆疊
- 一次應用一種模式－不要大爆炸遷移
- 衡量 DORA 指標－迭代改進
微服務不是目的地——它是您的團隊更快、更安全、更永續地交付價值的一種手段。