xDev Asia

Capstone project tổng hợp: thiết kế, deploy và vận hành hệ thống e-commerce microservices hoàn chỉnh trên K8s HA on-premises, áp dụng toàn bộ kiến thức từ 49 bài trước.

Deploy Microservices On-Premises với Kubernetes HA

Phần 12: Production Operations & Capstone Project

xdev.asia

🎯 MỤC TIÊU BÀI HỌC

✅ Thiết kế kiến trúc e-commerce microservices
✅ Deploy toàn bộ platform trên K8s HA on-premises
✅ Áp dụng tất cả best practices từ 49 bài trước
✅ Production-ready: security, observability, DR
✅ Performance testing và go-live

PHẦN 1: SYSTEM ARCHITECTURE


graph TB
    GW["🌐 Istio Gateway
TLS + CORS"]

    GW --> US["👤 User Service
Go"]
    GW --> PS["📦 Product Service
Go"]
    GW --> OS["🛒 Order Service
Go"]

    OS --> IS["📊 Inventory Service
Go"]
    OS --> PAY["💳 Payment Service
Go"]
    PAY --> NS["📧 Notification Service
Node.js
email · SMS · push"]
    OS --> NS

    subgraph DATA["💾 Data Stores"]
        PG["🐘 PostgreSQL
HA Cluster"]
        RD["⚡ Redis
Sentinel"]
        RMQ["🐰 RabbitMQ
Quorum Queues"]
        KF["📡 Kafka
Events"]
    end

    US & PS & OS & IS & PAY --> PG
    US & PS & OS --> RD
    OS & PAY --> RMQ
    OS & PAY --> KF
    NS --> RMQ

    subgraph PLATFORM["🏗️ Platform Layer"]
        direction LR
        K8S["☸ K8s HA
3 masters + 4 workers"]
        CEPH["💿 Rook-Ceph"]
        ISTIO["🔗 Istio Mesh"]
        ARGO["🔄 ArgoCD GitOps"]
        OBS["📊 Prometheus + Loki + Tempo"]
        SEC["🛡️ Kyverno + Falco"]
        VEL["💼 Velero Backup/DR"]
    end

    style GW fill:#1d4ed8,stroke:#60a5fa,color:#fff
    style DATA fill:#1e3a5f,stroke:#3b82f6,color:#e2e8f0
    style PLATFORM fill:#1e293b,stroke:#475569,color:#e2e8f0
    style NS fill:#7c3aed,stroke:#a78bfa,color:#fff

PHẦN 2: MICROSERVICES DESIGN

Service	Language	Database	Async Events
User Service	Go	PostgreSQL (users DB)	user.created, user.updated
Product Service	Go	PostgreSQL (products DB)	product.updated
Inventory Service	Go	PostgreSQL (inventory DB)	stock.reserved, stock.released
Order Service	Go	PostgreSQL (orders DB)	order.created, order.completed
Payment Service	Go	PostgreSQL (payments DB)	payment.processed, payment.failed
Notification Service	Node.js	Redis (queue)	Consumes order/payment events

# GitOps repo structure:
ecommerce-gitops/
├── apps/
│   ├── user-service/
│   │   ├── base/
│   │   │   ├── deployment.yaml
│   │   │   ├── service.yaml
│   │   │   ├── hpa.yaml
│   │   │   ├── pdb.yaml
│   │   │   └── kustomization.yaml
│   │   └── overlays/
│   │       ├── staging/
│   │       └── production/
│   ├── product-service/
│   ├── order-service/
│   ├── payment-service/
│   ├── inventory-service/
│   └── notification-service/
├── infrastructure/
│   ├── namespaces/
│   ├── networking/
│   │   ├── istio-gateway.yaml
│   │   ├── virtualservices.yaml
│   │   └── network-policies.yaml
│   ├── databases/
│   │   ├── postgresql-cluster.yaml
│   │   ├── redis-sentinel.yaml
│   │   ├── rabbitmq-cluster.yaml
│   │   └── kafka-cluster.yaml
│   └── observability/
│       ├── servicemonitors.yaml
│       ├── prometheusrules.yaml
│       └── grafana-dashboards.yaml
├── argocd/
│   ├── applicationset.yaml
│   └── appproject.yaml
└── helmfile.yaml

PHẦN 3: DEPLOYMENT STEPS

# Step 1: Prepare infrastructure (Bài 1-4)
# ✅ 7 nodes provisioned, OS hardened, HAProxy + keepalived

# Step 2: K8s HA cluster (Bài 5-10)
# ✅ 3 master + 4 worker, Cilium CNI, MetalLB, etcd backup

# Step 3: Storage (Bài 11-15)
# ✅ Rook-Ceph cluster, StorageClasses (block + filesystem)

# Step 4: Databases (Bài 16-23)
# ✅ PostgreSQL HA, Redis Sentinel, RabbitMQ cluster, Kafka

# Step 5: Service Mesh (Bài 24-27)
# ✅ Istio + Gateway, mTLS, AuthorizationPolicy

# Step 6: GitOps (Bài 28-31)
# ✅ ArgoCD, Helm charts, Vault secrets, CI/CD pipeline

# Step 7: Observability (Bài 32-35)
# ✅ Prometheus, Loki, Tempo, Grafana, SLO alerts

# Step 8: Security (Bài 36-39)
# ✅ RBAC, Kyverno, Falco, Harbor

# Step 9: Deploy services
kubectl apply -f argocd/applicationset.yaml
# ArgoCD auto-deploys all services from GitOps repo

# ArgoCD ApplicationSet (deploy all services):
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: ecommerce-services
  namespace: argocd
spec:
  generators:
    - git:
        repoURL: https://github.com/company/ecommerce-gitops
        revision: main
        directories:
          - path: apps/*
  template:
    metadata:
      name: '{{path.basename}}'
    spec:
      project: ecommerce
      source:
        repoURL: https://github.com/company/ecommerce-gitops
        targetRevision: main
        path: '{{path}}/overlays/production'
      destination:
        server: https://kubernetes.default.svc
        namespace: production
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

PHẦN 4: TESTING & VALIDATION

# Integration test flow:
# 1. Create user → 2. Get products → 3. Place order
#    → 4. Check inventory reserved → 5. Process payment
#    → 6. Verify notification sent

# Load test:
k6 run ecommerce-load-test.js
# Target: 500 orders/min, P99 < 1s, errors < 0.1%

# Chaos test:
# Round 1: Kill order-service pod (verify auto-recovery)
# Round 2: Simulate payment timeout (verify retry + DLQ)
# Round 3: Database failover (verify < 5s downtime)
# Round 4: Node failure (verify pod rescheduling)

# Security validation:
# - Kyverno: deploy privileged pod → blocked ✅
# - Falco: kubectl exec → alert fired ✅
# - Harbor: push image with critical CVE → scan detected ✅
# - NetworkPolicy: cross-namespace access → blocked ✅

PHẦN 5: CAPSTONE EVALUATION CHECKLIST

#	Requirement	Points	Status
1	K8s HA cluster (3 masters, 4 workers)	10	☐
2	Rook-Ceph distributed storage	10	☐
3	PostgreSQL HA + automated backup	10	☐
4	Message queues (RabbitMQ/Kafka)	5	☐
5	Redis caching layer	5	☐
6	Istio service mesh + mTLS	10	☐
7	ArgoCD GitOps deployment	10	☐
8	CI/CD pipeline (build + scan + sign)	5	☐
9	Prometheus + Grafana monitoring	5	☐
10	Loki centralized logging	5	☐
11	Tempo distributed tracing	5	☐
12	SLO/Error budget alerts	5	☐
13	Security (RBAC + Kyverno + Falco)	5	☐
14	Velero backup + DR tested	5	☐
15	Load test passing (500 orders/min)	5	☐
	Total	100

💡 KEY TAKEAWAYS

Capstone: Tích hợp tất cả 49 bài trước vào 1 hệ thống thực tế
GitOps: Toàn bộ infrastructure + apps as code
Production-ready: Security, observability, reliability, DR
Testing: Integration + load + chaos + security validation
Operations: SLO monitoring, incident response, capacity planning

🎓 KẾT THÚC KHÓA HỌC

Chúc mừng bạn đã hoàn thành 50 bài học của khóa "Deploy Microservices On-Premises với Kubernetes HA"!

Bạn đã nắm vững:

☑️ Infrastructure planning & Linux system tuning
☑️ Kubernetes HA cluster setup & operations
☑️ Distributed storage (Rook-Ceph)
☑️ Database HA (PostgreSQL, Redis, RabbitMQ, Kafka)
☑️ Service Mesh (Istio) & API Gateway
☑️ GitOps (ArgoCD) & CI/CD & Secrets Management
☑️ Full observability stack (Prometheus, Loki, Tempo, Grafana)
☑️ Security hardening (RBAC, Kyverno, Falco, Harbor)
☑️ Deployment patterns & Auto-scaling
☑️ Disaster Recovery & Chaos Engineering
☑️ Production operations & Troubleshooting

Keep learning, keep building! 🚀

BÀI 50: CAPSTONE PROJECT — E-COMMERCE MICROSERVICES PLATFORM