第 24 課：高可用性、叢集和多站點

從基礎到進階的鑰匙斗篷__HTMLTAG_59___

第 7 部分：生產、HA 和 Kubernetes

xdev.asia

HTMLTAG_67__HTMLTAG_68___1。 HA 架構概述___HTMLTAG_69__HTMLTAG_70

Keycloak 高可用性可確保身份驗證/授權系統持續運行，即使一個或多個節點發生故障也是如此。 HA 基於三個支柱：叢集 (Infinispan)、資料庫複製 和 負載平衡.

預編碼_0

HTMLTAG_79__HTMLTAG_80___2。 Infinispan 快取類型___HTMLTAG_81__HTMLTAG_82

HTMLTAG_83__HTMLTAG_84___2.1 本機快取與分散式/複製快取___HTMLTAG_85__HTMLTAG_86

Keycloak 使用兩種類型的 Infinispan 快取來實現不同的目的：

HTMLTAG_161__HTMLTAG_162___2.2 Keycloak 快取架構詳細資訊___HTMLTAG_163__HTMLTAG_164

預編碼_1

HTMLTAG_165__HTMLTAG_166___3。嵌入式與外部 Infinispan___HTMLTAG_167__HTMLTAG_168

___HTMLTAG_169__HTMLTAG_170___3.1 嵌入式 Infinispan（預設）HTMLTAG_171__HTMLTAG_172

預設情況下，Keycloak 將 Infinispan 嵌入 JVM 進程中。節點透過發現協定相互發現並自動形成叢集：

預編碼_2

HTMLTAG_175__HTMLTAG_176___3.2 外部Infinispan___HTMLTAG_177__HTMLTAG_178

對於多站點部署或需要單獨的快取層管理時，請使用外部 Infinispan 伺服器：

預編碼_3

HTMLTAG_235__HTMLTAG_236___4。快取堆疊配置___HTMLTAG_237__HTMLTAG_238

___HTMLTAG_239__HTMLTAG_240___4.1 Kubernetes (KUBE_PING)HTMLTAG_241__HTMLTAG_242

在 Kubernetes 上，使用 KUBE_PING (dns.DNS_PING) 讓 Keycloak Pod 透過 Kubernetes API 或 DNS 相互發現：

預編碼_4

Kubernetes Keycloak 需要 無頭服務 才能進行 DNS 發現：

預編碼_5

預編碼_6

___HTMLTAG_251__HTMLTAG_252___4.2 JDBC-PING（用於虛擬機器部署）HTMLTAG_253__HTMLTAG_254

在虛擬機器上部署時（不含 Kubernetes），使用 JDBC_PING 讓節點透過共用資料庫相互偵測：

預編碼_7

HTMLTAG_259__HTMLTAG_260___4.3 DNS-PING___HTMLTAG_261__HTMLTAG_262

預編碼_8

HTMLTAG_263__HTMLTAG_264___5。外部 Infinispan 伺服器設定___HTMLTAG_265__HTMLTAG_266

HTMLTAG_267__HTMLTAG_268___5.1 Infinispan 伺服器設定___HTMLTAG_269__HTMLTAG_270

預編碼_9

HTMLTAG_271__HTMLTAG_272___5.2 Keycloak 遠端快取設定___HTMLTAG_273__HTMLTAG_274

配置 Keycloak 以連接到外部 Infinispan：

預編碼_10

HTMLTAG_277__HTMLTAG_278___6。多站點部署___HTMLTAG_279__HTMLTAG_280

HTMLTAG_281__HTMLTAG_282___6.1 主動-被動模式___HTMLTAG_283__HTMLTAG_284

在主動被動模式下，僅 主站點 提供流量。備份站點處於待機狀態，準備在主站點出現問題時接管：

┌──────────────────────────────────────────────────────────────────────┐
│                  Active-Passive Multi-site                          │
│                                                                      │
│  ┌────────────────────────┐      ┌────────────────────────┐         │
│  │     Site A (Active)    │      │   Site B (Passive)     │         │
│  │                        │      │                        │         │
│  │  ┌──────┐  ┌──────┐   │      │  ┌──────┐  ┌──────┐   │         │
│  │  │ KC-1 │  │ KC-2 │   │      │  │ KC-3 │  │ KC-4 │   │         │
│  │  └──┬───┘  └──┬───┘   │      │  └──┬───┘  └──┬───┘   │         │
│  │     │         │        │      │     │         │        │         │
│  │  ┌──▼─────────▼───┐   │      │  ┌──▼─────────▼───┐   │         │
│  │  │ Infinispan      │◄─────────►│ Infinispan      │   │         │
│  │  │ (External)      │  Cross-  │ │ (External)      │   │         │
│  │  └────────────────┘  Site    │  └────────────────┘   │         │
│  │                       Repl.  │                        │         │
│  │  ┌────────────────┐   │      │  ┌────────────────┐   │         │
│  │  │ PostgreSQL     │◄─────────►│ PostgreSQL     │   │         │
│  │  │ (Primary)      │  Repl.  │  │ (Standby)      │   │         │
│  │  └────────────────┘   │      │  └────────────────┘   │         │
│  └────────────────────────┘      └────────────────────────┘         │
│                                                                      │
│  DNS: auth.example.com ──► Site A (failover to Site B)              │
└──────────────────────────────────────────────────────────────────────┘

HTMLTAG_289__HTMLTAG_290___6.2 主動-主動模式___HTMLTAG_291__HTMLTAG_292

在主動-主動模式下，兩個網站__HTMLTAG_295___同時提供流量。更複雜但充分利用資源：

┌──────────────────────────────────────────────────────────────────────┐
│                  Active-Active Multi-site                           │
│                                                                      │
│  ┌────────────────────────┐      ┌────────────────────────┐         │
│  │    Site A (Active)     │      │    Site B (Active)     │         │
│  │                        │      │                        │         │
│  │  ┌──────┐  ┌──────┐   │      │  ┌──────┐  ┌──────┐   │         │
│  │  │ KC-1 │  │ KC-2 │   │      │  │ KC-3 │  │ KC-4 │   │         │
│  │  └──┬───┘  └──┬───┘   │      │  └──┬───┘  └──┬───┘   │         │
│  │     └────┬────┘        │      │     └────┬────┘        │         │
│  │  ┌───────▼────────┐   │      │  ┌───────▼────────┐   │         │
│  │  │ Infinispan      │◄═══════════►│ Infinispan      │   │         │
│  │  │ (Cross-site)    │  RELAY   │  │ (Cross-site)    │   │         │
│  │  └────────────────┘   │      │  └────────────────┘   │         │
│  │                        │      │                        │         │
│  │  ┌────────────────┐   │      │  ┌────────────────┐   │         │
│  │  │ PostgreSQL     │◄═══════════►│ PostgreSQL     │   │         │
│  │  │ (Multi-master) │  Sync    │  │ (Multi-master) │   │         │
│  │  └────────────────┘   │      │  └────────────────┘   │         │
│  └────────────────────────┘      └────────────────────────┘         │
│                                                                      │
│  GSLB: auth.example.com ──► Site A (50%) + Site B (50%)             │
└──────────────────────────────────────────────────────────────────────┘

<!-- infinispan-xsite.xml - Site A Infinispan Server -->
<infinispan xmlns="urn:infinispan:config:15.0"
            xmlns:server="urn:infinispan:server:15.0">

    <jgroups>
        <!-- Local cluster stack -->
        <stack name="tcp" extends="tcp">
            <!-- Node discovery within same site -->
        </stack>

        <!-- Cross-site (relay) stack -->
        <stack name="relay">
            <TCP bind_addr="match-interface:eth0" bind_port="7900"/>
            <TCPPING initial_hosts="infinispan-siteA:7900,infinispan-siteB:7900"
                     port_range="0"/>
            <MERGE3/>
            <FD_ALL/>
            <VERIFY_SUSPECT/>
            <pbcast.NAKACK2/>
            <UNICAST3/>
            <pbcast.STABLE/>
            <pbcast.GMS/>
        </stack>
    </jgroups>

    <cache-container name="keycloak" statistics="true">
        <transport cluster="keycloak-cluster"
                   stack="tcp"
                   node-name="${infinispan.node.name}"
                   site="${infinispan.site.name:siteA}">
            <!-- Relay configuration for cross-site -->
            <relay site="${infinispan.site.name:siteA}">
                <remote-site name="siteA" stack="relay"/>
                <remote-site name="siteB" stack="relay"/>
            </relay>
        </transport>

        <!-- Sessions cache with cross-site backup -->
        <distributed-cache name="sessions" owners="2" mode="SYNC">
            <encoding>
                <key media-type="application/x-protostream"/>
                <value media-type="application/x-protostream"/>
            </encoding>
            <backups>
                <backup site="siteB" strategy="ASYNC"
                        failure-policy="WARN"
                        timeout="15000"/>
            </backups>
        </distributed-cache>

        <distributed-cache name="authenticationSessions" owners="2" mode="SYNC">
            <encoding>
                <key media-type="application/x-protostream"/>
                <value media-type="application/x-protostream"/>
            </encoding>
            <backups>
                <backup site="siteB" strategy="ASYNC"
                        failure-policy="WARN"
                        timeout="15000"/>
            </backups>
        </distributed-cache>

        <!-- Thêm các distributed caches khác tương tự... -->
    </cache-container>
</infinispan>

___HTMLTAG_371__HTMLTAG_372___Patroni 管理 PostgreSQL HA（自動故障轉移），PgBouncer 提供連接池：

┌─────────────────────────────────────────────────────────────┐
│            PostgreSQL HA with Patroni + PgBouncer           │
│                                                             │
│  ┌──────────────┐                                           │
│  │  Keycloak     │                                          │
│  │  Instances    │                                          │
│  └──────┬───────┘                                           │
│         │                                                   │
│  ┌──────▼───────┐                                           │
│  │  PgBouncer    │  ←── Connection pooling                  │
│  │  (VIP/DNS)    │      Transaction mode                    │
│  └──────┬───────┘                                           │
│         │                                                   │
│  ┌──────┴────────────────────────┐                          │
│  │               │                │                          │
│  ▼               ▼                ▼                          │
│ ┌─────────┐  ┌─────────┐  ┌─────────┐                      │
│ │ PG Node1│  │ PG Node2│  │ PG Node3│                      │
│ │(Primary)│  │(Replica)│  │(Replica)│                      │
│ │ Patroni │  │ Patroni │  │ Patroni │                      │
│ └────┬────┘  └────┬────┘  └────┬────┘                      │
│      │            │            │                            │
│      └────────────┼────────────┘                            │
│                   ▼                                         │
│            ┌─────────────┐                                  │
│            │    etcd      │  ←── Consensus store            │
│            │   cluster    │      (leader election)          │
│            └─────────────┘                                  │
└─────────────────────────────────────────────────────────────┘

HTMLTAG_377__HTMLTAG_378___7.2 Patroni 設定___HTMLTAG_379__HTMLTAG_380

# patroni.yml - Patroni configuration cho Keycloak
scope: keycloak-cluster
name: pg-node1

restapi:
  listen: 0.0.0.0:8008
  connect_address: pg-node1:8008

etcd3:
  hosts:
    - etcd1:2379
    - etcd2:2379
    - etcd3:2379

bootstrap:
  dcs:
    ttl: 30
    loop_wait: 10
    retry_timeout: 10
    maximum_lag_on_failover: 1048576
    synchronous_mode: true            # Synchronous replication
    synchronous_mode_strict: false
    postgresql:
      use_pg_rewind: true
      parameters:
        max_connections: 400
        shared_buffers: 2GB
        effective_cache_size: 6GB
        work_mem: 16MB
        wal_level: replica
        max_wal_senders: 5
        max_replication_slots: 5
        hot_standby: "on"
        synchronous_commit: "on"      # Synchronous cho data safety

  initdb:
    - encoding: UTF8
    - data-checksums

postgresql:
  listen: 0.0.0.0:5432
  connect_address: pg-node1:5432
  data_dir: /var/lib/postgresql/data
  authentication:
    superuser:
      username: postgres
      password: postgres_password
    replication:
      username: replicator
      password: replicator_password
    rewind:
      username: rewinder
      password: rewinder_password
  parameters:
    unix_socket_directories: "/var/run/postgresql"

HTMLTAG_381__HTMLTAG_382___7.3 PgBouncer 設定___HTMLTAG_383__HTMLTAG_384

# pgbouncer.ini
[databases]
keycloak = host=pg-primary port=5432 dbname=keycloak
           auth_user=keycloak

[pgbouncer]
listen_addr = 0.0.0.0
listen_port = 6432
auth_type = md5
auth_file = /etc/pgbouncer/userlist.txt

# Connection pooling
pool_mode = transaction         # Transaction mode cho Keycloak
max_client_conn = 1000
default_pool_size = 100
min_pool_size = 25
reserve_pool_size = 10
reserve_pool_timeout = 3

# Timeouts
server_connect_timeout = 15
server_idle_timeout = 600
server_lifetime = 3600
client_idle_timeout = 0
client_login_timeout = 60
query_timeout = 0
query_wait_timeout = 120

# Logging
log_connections = 1
log_disconnections = 1
log_pooler_errors = 1
stats_period = 60

HTMLTAG_385__HTMLTAG_386___8。負載平衡器配置___HTMLTAG_387__HTMLTAG_388

HTMLTAG_389__HTMLTAG_390___8.1 黏性會話___HTMLTAG_391__HTMLTAG_392

Keycloak 需要黏性會話進行身分驗證流程（登入、註冊）。基於 Cookie 的會話關聯性 KC_ROUTE（以前稱為 KEYCLOAK_SESSION）：

HTMLTAG_401__HTMLTAG_402___8.2 HAProxy 設定___HTMLTAG_403__HTMLTAG_404

# haproxy.cfg cho Keycloak HA
global
    log stdout format raw daemon
    maxconn 4096

defaults
    mode http
    log global
    option httplog
    option dontlognull
    option http-server-close
    option forwardfor except 127.0.0.0/8
    retries 3
    timeout http-request 10s
    timeout queue 60s
    timeout connect 10s
    timeout client 60s
    timeout server 60s
    timeout http-keep-alive 10s
    timeout check 10s

# Frontend HTTPS
frontend keycloak_frontend
    bind *:443 ssl crt /etc/haproxy/certs/auth.example.com.pem
    http-request set-header X-Forwarded-Proto https
    http-request set-header X-Forwarded-Port 443

    # Health check endpoint (không cần sticky)
    acl is_health path_beg /health
    acl is_metrics path_beg /metrics

    use_backend keycloak_health if is_health
    use_backend keycloak_health if is_metrics
    default_backend keycloak_backend

# Backend với sticky sessions
backend keycloak_backend
    balance roundrobin
    # Sticky session dựa trên KC_ROUTE cookie
    cookie KC_ROUTE insert indirect nocache httponly secure
    option httpchk GET /health/ready
    http-check expect status 200

    # Keycloak servers
    server kc1 keycloak-1:8443 ssl verify none check inter 10s \
           fall 3 rise 2 cookie kc1
    server kc2 keycloak-2:8443 ssl verify none check inter 10s \
           fall 3 rise 2 cookie kc2
    server kc3 keycloak-3:8443 ssl verify none check inter 10s \
           fall 3 rise 2 cookie kc3

# Backend cho health/metrics (không cần sticky)
backend keycloak_health
    balance roundrobin
    option httpchk GET /health/ready
    server kc1 keycloak-1:8443 ssl verify none check
    server kc2 keycloak-2:8443 ssl verify none check
    server kc3 keycloak-3:8443 ssl verify none check

# Stats dashboard
listen stats
    bind *:8404
    stats enable
    stats uri /stats
    stats refresh 10s
    stats auth admin:admin_password

HTMLTAG_405__HTMLTAG_406___8.3 Nginx 負載平衡器___HTMLTAG_407__HTMLTAG_408

# nginx.conf - Keycloak load balancer
upstream keycloak_cluster {
    # Sticky sessions via cookie
    sticky cookie KC_ROUTE expires=1h domain=.example.com httponly secure;

    server keycloak-1:8443 max_fails=3 fail_timeout=30s;
    server keycloak-2:8443 max_fails=3 fail_timeout=30s;
    server keycloak-3:8443 max_fails=3 fail_timeout=30s;
}

server {
    listen 443 ssl http2;
    server_name auth.example.com;

    ssl_certificate     /etc/nginx/certs/tls.crt;
    ssl_certificate_key /etc/nginx/certs/tls.key;

    # Buffer sizes cho Keycloak
    proxy_buffer_size        128k;
    proxy_buffers            4 256k;
    proxy_busy_buffers_size  256k;

    location / {
        proxy_pass https://keycloak_cluster;
        proxy_set_header Host               $host;
        proxy_set_header X-Real-IP          $remote_addr;
        proxy_set_header X-Forwarded-For    $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto  $scheme;
        proxy_set_header X-Forwarded-Host   $host;
        proxy_set_header X-Forwarded-Port   $server_port;

        # WebSocket support
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }

    # Health check (no sticky needed)
    location /health {
        proxy_pass https://keycloak_cluster;
        proxy_set_header Host $host;
    }
}

HTMLTAG_409__HTMLTAG_410___9。裂腦處理___HTMLTAG_411__HTMLTAG_412

HTMLTAG_413__HTMLTAG_414___9.1 裂腦場景___HTMLTAG_415__HTMLTAG_416

當網路分區將叢集分成 2 個以上分區時，就會發生腦裂，每個分區都認為對方已死亡：

┌─────────────────────────────────────────────────────────────┐
│                    Split-Brain Scenario                      │
│                                                             │
│  Partition A              ║  Partition B                    │
│  ┌──────┐  ┌──────┐      ║  ┌──────┐  ┌──────┐           │
│  │ KC-1 │◄─►│ KC-2 │     ║  │ KC-3 │◄─►│ KC-4 │           │
│  └──────┘  └──────┘      ║  └──────┘  └──────┘           │
│       ↕                   ║       ↕                         │
│  ┌──────────┐             ║  ┌──────────┐                  │
│  │ PG Primary│            ║  │ PG Replica│  ← Có thể       │
│  └──────────┘             ║  └──────────┘    promote sai!  │
│                           ║                                 │
│  Network Partition ═══════╝                                 │
└─────────────────────────────────────────────────────────────┘

HTMLTAG_419__HTMLTAG_420___9.2 合併策略___HTMLTAG_421__HTMLTAG_422

預編碼_20

HTMLTAG_467__HTMLTAG_468___10。災難復原策略___HTMLTAG_469__HTMLTAG_470

HTMLTAG_471__HTMLTAG_472___10.1 RPO/RTO 目標___HTMLTAG_473__HTMLTAG_474

HTMLTAG_521__HTMLTAG_522___10.2 備份與還原___HTMLTAG_523__HTMLTAG_524

#!/bin/bash
# backup-keycloak.sh - Automated backup script

BACKUP_DIR="/backups/keycloak/$(date +%Y%m%d_%H%M%S)"
mkdir -p "$BACKUP_DIR"

# 1. Database backup (PostgreSQL)
echo "=== Backing up database ==="
pg_dump -h db-host -U keycloak -d keycloak \
  --format=custom \
  --compress=9 \
  --file="$BACKUP_DIR/keycloak-db.dump"

# 2. Realm export via Admin API
echo "=== Exporting realms ==="
# Lấy admin token
ADMIN_TOKEN=$(curl -s \
  -d "client_id=admin-cli" \
  -d "username=admin" \
  -d "password=admin_password" \
  -d "grant_type=password" \
  "https://auth.example.com/realms/master/protocol/openid-connect/token" \
  | jq -r '.access_token')

# Export từng realm
for REALM in $(curl -s \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  "https://auth.example.com/admin/realms" \
  | jq -r '.[].realm'); do

  echo "Exporting realm: $REALM"
  curl -s \
    -H "Authorization: Bearer $ADMIN_TOKEN" \
    "https://auth.example.com/admin/realms/$REALM/partial-export?exportClients=true&exportGroupsAndRoles=true" \
    -o "$BACKUP_DIR/realm-$REALM.json"
done

# 3. Compress backup
echo "=== Compressing backup ==="
tar -czf "$BACKUP_DIR.tar.gz" -C "$(dirname $BACKUP_DIR)" "$(basename $BACKUP_DIR)"
rm -rf "$BACKUP_DIR"

# 4. Upload to S3 (optional)
# aws s3 cp "$BACKUP_DIR.tar.gz" "s3://my-backups/keycloak/"

echo "=== Backup completed: $BACKUP_DIR.tar.gz ==="

#!/bin/bash
# restore-keycloak.sh - Restore from backup

BACKUP_FILE="$1"
if [ -z "$BACKUP_FILE" ]; then
  echo "Usage: $0 "
  exit 1
fi

# 1. Extract backup
RESTORE_DIR="/tmp/keycloak-restore"
mkdir -p "$RESTORE_DIR"
tar -xzf "$BACKUP_FILE" -C "$RESTORE_DIR"

# 2. Restore database
echo "=== Restoring database ==="
DB_DUMP=$(find "$RESTORE_DIR" -name "*.dump" | head -1)
pg_restore -h db-host -U keycloak -d keycloak \
  --clean --if-exists \
  "$DB_DUMP"

# 3. Restart Keycloak
echo "=== Restarting Keycloak ==="
# Docker Compose
docker compose restart keycloak

# Hoặc Kubernetes
# kubectl rollout restart deployment/keycloak -n keycloak

echo "=== Restore completed ==="

HTMLTAG_525__HTMLTAG_526___10.3 故障轉移自動化___HTMLTAG_527__HTMLTAG_528

#!/bin/bash
# failover-check.sh - Health check và auto-failover script

PRIMARY_URL="https://auth-primary.example.com/health/ready"
BACKUP_URL="https://auth-backup.example.com/health/ready"
DNS_ZONE="example.com"
DNS_RECORD="auth"
MAX_FAILURES=3
FAILURE_COUNT=0
CHECK_INTERVAL=10

while true; do
  # Check primary health
  HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" \
    --connect-timeout 5 --max-time 10 \
    "$PRIMARY_URL")

  if [ "$HTTP_CODE" != "200" ]; then
    FAILURE_COUNT=$((FAILURE_COUNT + 1))
    echo "[$(date)] Primary UNHEALTHY ($HTTP_CODE) - failure $FAILURE_COUNT/$MAX_FAILURES"

    if [ "$FAILURE_COUNT" -ge "$MAX_FAILURES" ]; then
      echo "[$(date)] FAILOVER: Switching DNS to backup site"

      # Check backup is healthy first
      BACKUP_CODE=$(curl -s -o /dev/null -w "%{http_code}" \
        --connect-timeout 5 --max-time 10 \
        "$BACKUP_URL")

      if [ "$BACKUP_CODE" == "200" ]; then
        # Update DNS (example with AWS Route53)
        # aws route53 change-resource-record-sets ...

        # Send alert
        echo "CRITICAL: Keycloak failover executed at $(date)" | \
          mail -s "Keycloak Failover Alert" [email protected]

        FAILURE_COUNT=0
      else
        echo "[$(date)] CRITICAL: Both sites are down!"
      fi
    fi
  else
    if [ "$FAILURE_COUNT" -gt 0 ]; then
      echo "[$(date)] Primary recovered"
    fi
    FAILURE_COUNT=0
  fi

  sleep "$CHECK_INTERVAL"
done

HTMLTAG_529__HTMLTAG_530___11。 Docker Compose HA 叢集___HTMLTAG_531__HTMLTAG_532

具有 2 個 Keycloak 節點 + 外部 Infinispan + PostgreSQL 的完整範例：

預編碼_24

HTMLTAG_535__HTMLTAG_536___12。 Infinispan CLI 與監控___HTMLTAG_537__HTMLTAG_538

預編碼_25