Bài 23: Triển khai Production và Performance Tuning

Keycloak từ Cơ bản đến Nâng cao

Phần 7: Production, HA và Kubernetes

xdev.asia

1. Production Deployment Checklist

Triển khai Keycloak trên production đòi hỏi cấu hình kỹ lưỡng về database, networking, JVM, caching và security. Bài này cung cấp hướng dẫn toàn diện từ database selection đến load testing.

┌─────────────────────────────────────────────────────────────┐
│                  Production Checklist                       │
├─────────────────────────────────────────────────────────────┤
│  ✅ Database: PostgreSQL + Connection Pool tuning           │
│  ✅ Build: kc.sh build --optimized                          │
│  ✅ Hostname: hostname-v2 provider configured               │
│  ✅ Proxy: X-Forwarded-* / PROXY protocol                   │
│  ✅ TLS: Certificates configured                            │
│  ✅ JVM: Heap, GC, container-aware settings                 │
│  ✅ Caching: Infinispan local caches tuned                  │
│  ✅ Metrics: /metrics endpoint enabled                      │
│  ✅ Health: /health/ready, /health/live enabled              │
│  ✅ Load Test: Gatling benchmark passed                     │
└─────────────────────────────────────────────────────────────┘

2. Database Selection và Configuration

2.1 Database Support Matrix

Database	Vendor Flag	Recommended	Ghi chú
PostgreSQL	postgres	✅ Yes	Best performance, được Keycloak team test nhiều nhất
MySQL	mysql	⚠️ OK	Cần InnoDB, utf8mb4 charset
MariaDB	mariadb	⚠️ OK	Tương tự MySQL
Oracle	oracle	⚠️ OK	Enterprise license cần thiết
Microsoft SQL Server	mssql	⚠️ OK	Windows environment
H2 (embedded)	dev-file/dev-mem	❌ No	Chỉ dùng cho development

2.2 PostgreSQL Configuration

# Cấu hình database cơ bản
bin/kc.sh start \
  --db=postgres \
  --db-url="jdbc:postgresql://db-host:5432/keycloak" \
  --db-username=keycloak \
  --db-password=secure_password_here \
  --db-schema=public

Với environment variables (recommended cho containers):

# Environment variables cho database
export KC_DB=postgres
export KC_DB_URL="jdbc:postgresql://db-host:5432/keycloak"
export KC_DB_USERNAME=keycloak
export KC_DB_PASSWORD=secure_password_here
export KC_DB_SCHEMA=public

# JDBC URL với advanced parameters
export KC_DB_URL="jdbc:postgresql://db-host:5432/keycloak?ssl=true&sslmode=verify-full&sslrootcert=/certs/ca.crt"

2.3 Connection Pool Tuning (Agroal)

Keycloak sử dụng Agroal connection pool (Quarkus default). Tuning connection pool là yếu tố quan trọng ảnh hưởng đến performance:

# Connection pool configuration
bin/kc.sh start \
  --db=postgres \
  --db-url="jdbc:postgresql://db-host:5432/keycloak" \
  --db-username=keycloak \
  --db-password=secure_password_here \
  --db-pool-initial-size=25 \
  --db-pool-min-size=25 \
  --db-pool-max-size=100

Parameter	Default	Production Recommended	Mô tả
--db-pool-initial-size	0	25	Số connections khởi tạo ban đầu
--db-pool-min-size	0	25	Số connections tối thiểu duy trì
--db-pool-max-size	100	50–100	Số connections tối đa

Nguyên tắc sizing connection pool:

Tổng connections = Số Keycloak instances × db-pool-max-size

Ví dụ: 3 instances × 100 max = 300 connections
→ PostgreSQL max_connections ≥ 300 + buffer (20%)
→ Đặt max_connections = 360

Cấu hình PostgreSQL phía server (postgresql.conf):

# postgresql.conf - tối ưu cho Keycloak
max_connections = 400
shared_buffers = 2GB
effective_cache_size = 6GB
work_mem = 16MB
maintenance_work_mem = 512MB

# Connection timeout
idle_in_transaction_session_timeout = 30000  # 30 seconds
statement_timeout = 60000                     # 60 seconds

# WAL configuration
wal_level = replica
max_wal_senders = 5
wal_keep_size = 1GB

3. Build Optimization

3.1 Build vs Start Phases

Keycloak có 2 phases: build (đóng gói config) và start (chạy server). Trong production, luôn sử dụng --optimized để tách rời 2 phases, giúp startup nhanh hơn đáng kể.

┌──────────────────────────────────────────────────────────────┐
│              Keycloak Build & Start Phases                   │
├──────────────────────────────────────────────────────────────┤
│                                                              │
│  Phase 1: BUILD (chạy 1 lần, hoặc khi config thay đổi)     │
│  ┌────────────────────────────────────────────────────────┐  │
│  │ kc.sh build                                           │  │
│  │ - Parse configuration                                 │  │
│  │ - Install providers/extensions                        │  │
│  │ - Quarkus augmentation (ahead-of-time optimization)   │  │
│  │ - Persist build options                               │  │
│  └────────────────────────────────────────────────────────┘  │
│                           ↓                                  │
│  Phase 2: START (mỗi lần khởi động)                         │
│  ┌────────────────────────────────────────────────────────┐  │
│  │ kc.sh start --optimized                               │  │
│  │ - Skip build phase → Fast startup                     │  │
│  │ - Use pre-built configuration                         │  │
│  │ - Apply runtime-only options                          │  │
│  └────────────────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────────────────┘

# Build phase - chạy khi build Docker image
bin/kc.sh build \
  --db=postgres \
  --features=token-exchange,admin-fine-grained-authz \
  --health-enabled=true \
  --metrics-enabled=true \
  --http-relative-path=/auth

# Start phase - chạy khi container start
bin/kc.sh start --optimized \
  --db-url="jdbc:postgresql://db-host:5432/keycloak" \
  --db-username=keycloak \
  --db-password=secure_password_here \
  --hostname=auth.example.com \
  --https-certificate-file=/certs/tls.crt \
  --https-certificate-key-file=/certs/tls.key

3.2 Production Dockerfile

# Multi-stage Dockerfile cho Keycloak Production
FROM quay.io/keycloak/keycloak:26.0 AS builder

# Build phase - pre-build configuration
ENV KC_DB=postgres
ENV KC_HEALTH_ENABLED=true
ENV KC_METRICS_ENABLED=true
ENV KC_FEATURES=token-exchange,admin-fine-grained-authz
ENV KC_HTTP_RELATIVE_PATH=/auth

# Thêm custom providers nếu có
# COPY --chown=keycloak:keycloak my-provider.jar /opt/keycloak/providers/

# Thêm custom themes nếu có
# COPY --chown=keycloak:keycloak my-theme/ /opt/keycloak/themes/my-theme/

RUN /opt/keycloak/bin/kc.sh build

# Runtime stage
FROM quay.io/keycloak/keycloak:26.0

COPY --from=builder /opt/keycloak/ /opt/keycloak/

ENTRYPOINT ["/opt/keycloak/bin/kc.sh"]
CMD ["start", "--optimized"]

4. Hostname Configuration

4.1 Hostname v2 Provider

Keycloak sử dụng hostname-v2 provider (default từ Keycloak 25+) để xác định URL cho tất cả endpoints (frontend, backend, admin):

# Hostname configuration cơ bản
bin/kc.sh start --optimized \
  --hostname=auth.example.com \
  --hostname-admin=admin-auth.example.com

Parameter	Mô tả	Ví dụ
--hostname	Hostname cho frontend URLs (login pages, well-known endpoints)	auth.example.com
--hostname-admin	Hostname riêng cho Admin Console (nếu khác frontend). Không set = dùng chung --hostname	admin-auth.internal.com
--hostname-strict	Chỉ cho phép request đến hostname đã cấu hình. Default: true	true
--hostname-backchannel-dynamic	Backend URL dùng request hostname thay vì fixed hostname. Default: false	false

4.2 Hostname Scenarios

# Scenario 1: Single domain cho tất cả
bin/kc.sh start --optimized \
  --hostname=auth.example.com

# Scenario 2: Tách Admin Console ra domain riêng (khuyến nghị production)
bin/kc.sh start --optimized \
  --hostname=auth.example.com \
  --hostname-admin=admin-auth.internal.example.com

# Scenario 3: Edge proxy determine hostname từ request
bin/kc.sh start --optimized \
  --hostname-strict=false \
  --proxy-headers=xforwarded

# Scenario 4: Backchannel dynamic (backend-to-backend dùng internal URL)
bin/kc.sh start --optimized \
  --hostname=auth.example.com \
  --hostname-backchannel-dynamic=true

5. Proxy Configuration

5.1 Proxy Headers

Khi Keycloak nằm sau reverse proxy (Nginx, HAProxy, AWS ALB...), cần cấu hình proxy headers để Keycloak nhận đúng client IP, protocol và hostname:

# Option 1: X-Forwarded-* headers (phổ biến nhất)
bin/kc.sh start --optimized \
  --proxy-headers=xforwarded

# Option 2: RFC 7239 Forwarded header
bin/kc.sh start --optimized \
  --proxy-headers=forwarded

Header	Mục đích	Flag
X-Forwarded-For	Client IP address	xforwarded
X-Forwarded-Proto	Original protocol (http/https)	xforwarded
X-Forwarded-Host	Original hostname	xforwarded
X-Forwarded-Port	Original port	xforwarded
Forwarded	RFC 7239 combined header	forwarded

5.2 Nginx Reverse Proxy Config

# /etc/nginx/conf.d/keycloak.conf
upstream keycloak_backend {
    server keycloak-1:8443;
    server keycloak-2:8443;
    # Sticky session based on KEYCLOAK_SESSION cookie
    sticky cookie KEYCLOAK_ROUTE expires=1h domain=.example.com httponly secure;
}

server {
    listen 443 ssl http2;
    server_name auth.example.com;

    ssl_certificate     /etc/nginx/certs/tls.crt;
    ssl_certificate_key /etc/nginx/certs/tls.key;

    # Security headers
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
    add_header X-Content-Type-Options nosniff;
    add_header X-Frame-Options SAMEORIGIN;

    # Buffer sizes cho Keycloak (tokens có thể lớn)
    proxy_buffer_size        128k;
    proxy_buffers            4 256k;
    proxy_busy_buffers_size  256k;
    large_client_header_buffers 4 16k;

    location / {
        proxy_pass https://keycloak_backend;

        # X-Forwarded headers
        proxy_set_header Host               $host;
        proxy_set_header X-Real-IP          $remote_addr;
        proxy_set_header X-Forwarded-For    $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto  $scheme;
        proxy_set_header X-Forwarded-Host   $host;
        proxy_set_header X-Forwarded-Port   $server_port;

        # WebSocket support (cho Admin Console)
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        # Timeouts
        proxy_connect_timeout 60s;
        proxy_send_timeout    60s;
        proxy_read_timeout    60s;
    }
}

5.3 HTTP/2 Support

# Enable HTTP/2 (default đã enabled khi dùng HTTPS)
bin/kc.sh start --optimized \
  --hostname=auth.example.com \
  --https-certificate-file=/certs/tls.crt \
  --https-certificate-key-file=/certs/tls.key \
  --http-enabled=false

# Nếu cần HTTP/2 cleartext (h2c) cho internal communication
bin/kc.sh start --optimized \
  --hostname=auth.example.com \
  --http-enabled=true \
  --http-port=8080

6. JVM Tuning

6.1 Heap Configuration

Keycloak chạy trên Quarkus/JVM, vì vậy JVM tuning ảnh hưởng trực tiếp đến performance và stability:

# JVM Heap - sử dụng JAVA_OPTS_KC_HEAP (Keycloak 25+)
export JAVA_OPTS_KC_HEAP="-XX:InitialRAMPercentage=50.0 -XX:MaxRAMPercentage=70.0"

# Hoặc set fixed heap size
export JAVA_OPTS_KC_HEAP="-Xms512m -Xmx2g"

6.2 Garbage Collector Selection

# Option 1: G1GC (recommended cho heap ≤ 4GB)
export JAVA_OPTS_APPEND="-XX:+UseG1GC \
  -XX:MaxGCPauseMillis=200 \
  -XX:G1HeapRegionSize=16m \
  -XX:+ParallelRefProcEnabled \
  -XX:+UseStringDeduplication"

# Option 2: ZGC (recommended cho heap > 4GB, low-latency)
export JAVA_OPTS_APPEND="-XX:+UseZGC \
  -XX:+ZGenerational \
  -XX:ConcGCThreads=2"

# Option 3: Shenandoah GC (alternative low-latency)
export JAVA_OPTS_APPEND="-XX:+UseShenandoahGC \
  -XX:ShenandoahGCHeuristics=compact"

GC Algorithm	Best For	Heap Size	Pause Time
G1GC	General purpose, balanced throughput/latency	≤ 4GB	~200ms
ZGC	Low-latency, large heap	> 4GB	< 1ms
Shenandoah	Low-latency, concurrent	> 2GB	< 10ms

6.3 Container-Aware JVM Settings

# Container-aware JVM settings (Docker/Kubernetes)
export JAVA_OPTS_APPEND=" \
  -XX:+UseContainerSupport \
  -XX:MaxRAMPercentage=70.0 \
  -XX:InitialRAMPercentage=50.0 \
  -XX:MinRAMPercentage=50.0 \
  -XX:ActiveProcessorCount=2 \
  -XX:+UseG1GC \
  -XX:MaxGCPauseMillis=200 \
  -Djava.net.preferIPv4Stack=true \
  -Djava.awt.headless=true \
  -Dfile.encoding=UTF-8"

6.4 Complete JVM Configuration

# Production JVM configuration hoàn chỉnh
export JAVA_OPTS_KC_HEAP="-XX:InitialRAMPercentage=50.0 -XX:MaxRAMPercentage=70.0"

export JAVA_OPTS_APPEND=" \
  -XX:+UseG1GC \
  -XX:MaxGCPauseMillis=200 \
  -XX:G1HeapRegionSize=16m \
  -XX:+ParallelRefProcEnabled \
  -XX:+UseStringDeduplication \
  -XX:+UseContainerSupport \
  -XX:ActiveProcessorCount=2 \
  -XX:MetaspaceSize=256m \
  -XX:MaxMetaspaceSize=512m \
  -Djava.net.preferIPv4Stack=true \
  -Djava.awt.headless=true \
  -Dfile.encoding=UTF-8 \
  -XX:+ExitOnOutOfMemoryError \
  -XX:+HeapDumpOnOutOfMemoryError \
  -XX:HeapDumpPath=/opt/keycloak/dumps/"

7. Quarkus Thread Pool và Vert.x

Keycloak chạy trên Quarkus (Vert.x event loop + worker thread pool). Hiểu mô hình này giúp tuning chính xác:

┌───────────────────────────────────────────────────────┐
│                  Keycloak / Quarkus                    │
│                                                       │
│  ┌─────────────────────────────────────────────────┐  │
│  │          Vert.x Event Loop Threads              │  │
│  │  (IO_THREADS = 2 × CPU cores, default)          │  │
│  │  - Accept connections                           │  │
│  │  - Parse HTTP requests                          │  │
│  │  - Non-blocking operations                      │  │
│  └──────────────┬──────────────────────────────────┘  │
│                  │ delegate blocking work              │
│  ┌──────────────▼──────────────────────────────────┐  │
│  │          Worker Thread Pool                     │  │
│  │  (WORKER_THREADS = 8 × CPU cores, default)      │  │
│  │  - Database queries                             │  │
│  │  - LDAP calls                                   │  │
│  │  - Token signing/validation                     │  │
│  │  - Template rendering                           │  │
│  └─────────────────────────────────────────────────┘  │
└───────────────────────────────────────────────────────┘

# Quarkus thread pool configuration (qua Java system properties)
export JAVA_OPTS_APPEND=" \
  -Dquarkus.thread-pool.max-threads=200 \
  -Dquarkus.thread-pool.queue-size=1000 \
  -Dquarkus.thread-pool.growth-resistance=0.1 \
  -Dquarkus.vertx.event-loops-pool-size=4"

Parameter	Default	Mô tả
quarkus.thread-pool.max-threads	200 (hoặc 8 × CPU max)	Max worker threads
quarkus.thread-pool.queue-size	unbounded	Queue size khi tất cả threads busy
quarkus.thread-pool.growth-resistance	0	0–1, thread pool grow resistance
quarkus.vertx.event-loops-pool-size	2 × CPU	Vert.x event loop threads

8. Infinispan Local Cache Tuning

8.1 Keycloak Cache Architecture

Keycloak sử dụng Infinispan local caches để cache metadata (realms, users, keys...) nhằm giảm database queries. Tuning cache size và lifespan ảnh hưởng lớn đến performance:

┌─────────────────────────────────────────────────────────────┐
│                 Infinispan Local Caches                      │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐  │
│  │    realms     │  │    users     │  │  authorization   │  │
│  │  (realm cfg)  │  │ (user data)  │  │ (permissions)    │  │
│  │  max: 10000   │  │  max: 10000  │  │  max: 10000      │  │
│  └──────────────┘  └──────────────┘  └──────────────────┘  │
│                                                             │
│  ┌──────────────┐  ┌──────────────────────────────────────┐ │
│  │    keys       │  │        Revision Caches               │ │
│  │  (crypto)     │  │  (realmRevisions, userRevisions,     │ │
│  │  max: 1000    │  │   authorizationRevisions)            │ │
│  └──────────────┘  └──────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘

8.2 Custom Cache Configuration

Tạo file cache-ispn.xml tùy chỉnh:

<?xml version="1.0" encoding="UTF-8"?>
<infinispan
    xmlns="urn:infinispan:config:15.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="urn:infinispan:config:15.0
        https://infinispan.org/schemas/infinispan-config-15.0.xsd">

    <cache-container name="keycloak">
        <!-- Realm cache - cache metadata realm -->
        <local-cache name="realms">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="20000"/>
        </local-cache>

        <!-- User cache - cache user data -->
        <local-cache name="users">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="20000"/>
        </local-cache>

        <!-- Authorization cache -->
        <local-cache name="authorization">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="20000"/>
        </local-cache>

        <!-- Keys cache - signing/encryption keys -->
        <local-cache name="keys">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="5000"/>
            <expiration max-idle="3600000"/> <!-- 1 hour -->
        </local-cache>

        <!-- Revision caches - invalidation tracking -->
        <local-cache name="realmRevisions">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="40000"/>
        </local-cache>

        <local-cache name="userRevisions">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="40000"/>
        </local-cache>

        <local-cache name="authorizationRevisions">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="40000"/>
        </local-cache>
    </cache-container>
</infinispan>

# Sử dụng custom cache config
bin/kc.sh build --cache=ispn --cache-config-file=cache-ispn.xml
bin/kc.sh start --optimized

9. Metrics và Health Checks

9.1 Metrics Endpoint

# Enable metrics (cần set khi build)
bin/kc.sh build --metrics-enabled=true
bin/kc.sh start --optimized

Metrics endpoint: https://auth.example.com/metrics (Prometheus format)

# Ví dụ metrics output
# HELP keycloak_logins_total Total successful logins
# TYPE keycloak_logins_total counter
keycloak_logins_total{realm="my-realm",provider="keycloak",client_id="my-app"} 1523

# HELP keycloak_failed_login_attempts_total Total failed login attempts
# TYPE keycloak_failed_login_attempts_total counter
keycloak_failed_login_attempts_total{realm="my-realm",provider="keycloak",client_id="my-app",error="invalid_user_credentials"} 42

# HELP vendor_memoryPool_usage_max_bytes Peak JVM memory pool usage
# TYPE vendor_memoryPool_usage_max_bytes gauge
vendor_memoryPool_usage_max_bytes{name="G1 Old Gen"} 524288000

# HELP vendor_cpu_processCpuLoad JVM Process CPU load
# TYPE vendor_cpu_processCpuLoad gauge
vendor_cpu_processCpuLoad 0.15

9.2 Health Check Endpoints

# Enable health checks (cần set khi build)
bin/kc.sh build --health-enabled=true

Endpoint	Mục đích	Kubernetes Probe
/health/ready	Readiness - sẵn sàng nhận traffic	readinessProbe
/health/live	Liveness - process còn hoạt động	livenessProbe
/health/started	Startup - đã khởi động xong	startupProbe
/health	Combined health status	—

// GET /health/ready - Response khi healthy
{
  "status": "UP",
  "checks": [
    {
      "name": "Keycloak database connections health check",
      "status": "UP"
    }
  ]
}

// GET /health/ready - Response khi unhealthy
{
  "status": "DOWN",
  "checks": [
    {
      "name": "Keycloak database connections health check",
      "status": "DOWN",
      "data": {
        "message": "Unable to connect to database"
      }
    }
  ]
}

10. Load Testing với Gatling

10.1 Keycloak Benchmark Project

Keycloak cung cấp official benchmark project dựa trên Gatling framework:

# Clone Keycloak benchmark
git clone https://github.com/keycloak/keycloak-benchmark.git
cd keycloak-benchmark

# Build benchmark
mvn clean install -DskipTests

# Run token endpoint benchmark
cd benchmark
mvn gatling:test \
  -Dgatling.simulationClass=keycloak.scenario.authentication.ClientSecret \
  -Dkeycloak.server.url=https://auth.example.com \
  -Drealm=benchmark-realm \
  -DclientId=benchmark-client \
  -DclientSecret=benchmark-secret \
  -DusersPerSec=50 \
  -DrampUpPeriod=30 \
  -DwarmUpPeriod=60 \
  -DmeasurementPeriod=180

10.2 Custom Gatling Simulation

// KeycloakLoadTest.scala
package keycloak.benchmark

import io.gatling.core.Predef._
import io.gatling.http.Predef._
import scala.concurrent.duration._

class KeycloakTokenEndpointSimulation extends Simulation {

  val keycloakUrl = System.getProperty("keycloak.url", "https://auth.example.com")
  val realm = System.getProperty("realm", "my-realm")
  val clientId = System.getProperty("clientId", "benchmark-client")
  val clientSecret = System.getProperty("clientSecret", "benchmark-secret")

  val httpProtocol = http
    .baseUrl(keycloakUrl)
    .acceptHeader("application/json")
    .contentTypeHeader("application/x-www-form-urlencoded")
    .disableFollowRedirect

  // Scenario 1: Client Credentials Grant (service-to-service)
  val clientCredentialsScenario = scenario("Client Credentials Flow")
    .exec(
      http("Token Request - Client Credentials")
        .post(s"/realms/$realm/protocol/openid-connect/token")
        .formParam("grant_type", "client_credentials")
        .formParam("client_id", clientId)
        .formParam("client_secret", clientSecret)
        .check(status.is(200))
        .check(jsonPath("$.access_token").exists)
        .check(responseTimeInMillis.lte(500))
    )

  // Scenario 2: Resource Owner Password Grant (user login)
  val passwordGrantScenario = scenario("Resource Owner Password Flow")
    .exec(
      http("Token Request - Password Grant")
        .post(s"/realms/$realm/protocol/openid-connect/token")
        .formParam("grant_type", "password")
        .formParam("client_id", clientId)
        .formParam("client_secret", clientSecret)
        .formParam("username", "test-user")
        .formParam("password", "test-password")
        .check(status.is(200))
        .check(jsonPath("$.access_token").saveAs("accessToken"))
    )
    .pause(1.second)
    .exec(
      http("Token Introspection")
        .post(s"/realms/$realm/protocol/openid-connect/token/introspect")
        .formParam("token", "${accessToken}")
        .formParam("client_id", clientId)
        .formParam("client_secret", clientSecret)
        .check(status.is(200))
        .check(jsonPath("$.active").is("true"))
    )

  // Scenario 3: UserInfo endpoint
  val userInfoScenario = scenario("UserInfo Flow")
    .exec(
      http("Get Token")
        .post(s"/realms/$realm/protocol/openid-connect/token")
        .formParam("grant_type", "password")
        .formParam("client_id", clientId)
        .formParam("client_secret", clientSecret)
        .formParam("username", "test-user")
        .formParam("password", "test-password")
        .check(status.is(200))
        .check(jsonPath("$.access_token").saveAs("accessToken"))
    )
    .exec(
      http("UserInfo")
        .get(s"/realms/$realm/protocol/openid-connect/userinfo")
        .header("Authorization", "Bearer ${accessToken}")
        .check(status.is(200))
    )

  setUp(
    clientCredentialsScenario.inject(
      rampUsersPerSec(1).to(100).during(60.seconds),  // Ramp up
      constantUsersPerSec(100).during(180.seconds),     // Sustained load
      rampUsersPerSec(100).to(1).during(30.seconds)    // Ramp down
    ),
    passwordGrantScenario.inject(
      rampUsersPerSec(1).to(50).during(60.seconds),
      constantUsersPerSec(50).during(180.seconds),
      rampUsersPerSec(50).to(1).during(30.seconds)
    )
  ).protocols(httpProtocol)
    .assertions(
      global.responseTime.percentile3.lt(500),  // p95 < 500ms
      global.responseTime.percentile4.lt(1000), // p99 < 1000ms
      global.successfulRequests.percent.gt(99)  // > 99% success rate
    )
}

11. Production Checklist Summary

Category	Item	Status
Database	PostgreSQL với connection pool tuned	☐
Database	Database backup automated (pg_dump / pg_basebackup)	☐
Database	Database replication configured	☐
Build	kc.sh build + start --optimized	☐
Build	Multi-stage Dockerfile	☐
Hostname	--hostname configured (hostname-v2)	☐
Hostname	--hostname-admin set to separate domain	☐
TLS	Valid TLS certificates (Let's Encrypt / CA-signed)	☐
TLS	TLS 1.2+ enforced, weak ciphers disabled	☐
Proxy	--proxy-headers configured correctly	☐
JVM	Heap sized (50–70% container memory)	☐
JVM	GC algorithm selected (G1GC/ZGC)	☐
JVM	Container-aware flags enabled	☐
JVM	-XX:+ExitOnOutOfMemoryError set	☐
Cache	Infinispan local caches tuned	☐
Observability	Metrics endpoint enabled (/metrics)	☐
Observability	Health checks enabled (/health/*)	☐
Observability	Logging configured (JSON format, log level)	☐
Security	Admin Console on separate domain/network	☐
Security	Default admin credentials changed	☐
Security	Brute force protection enabled	☐
Security	CORS configured correctly	☐
Load Test	Gatling benchmark passed (p95 < 500ms)	☐
Load Test	Capacity planning documented	☐
Backup	Realm export automated	☐
Backup	Disaster recovery plan tested	☐

# Complete production start command
bin/kc.sh start --optimized \
  --db=postgres \
  --db-url="jdbc:postgresql://db-host:5432/keycloak?ssl=true" \
  --db-username=keycloak \
  --db-password="${KC_DB_PASSWORD}" \
  --db-pool-initial-size=25 \
  --db-pool-min-size=25 \
  --db-pool-max-size=100 \
  --hostname=auth.example.com \
  --hostname-admin=admin-auth.internal.example.com \
  --hostname-strict=true \
  --proxy-headers=xforwarded \
  --https-certificate-file=/certs/tls.crt \
  --https-certificate-key-file=/certs/tls.key \
  --http-enabled=false \
  --health-enabled=true \
  --metrics-enabled=true \
  --log=console \
  --log-level=info \
  --log-console-output=json

# Docker Compose cho production
# docker-compose.production.yml

version: "3.9"

services:
  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: keycloak
      POSTGRES_USER: keycloak
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
    volumes:
      - pgdata:/var/lib/postgresql/data
      - ./postgresql.conf:/etc/postgresql/postgresql.conf
    command: postgres -c config_file=/etc/postgresql/postgresql.conf
    secrets:
      - db_password
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U keycloak"]
      interval: 10s
      timeout: 5s
      retries: 5
    deploy:
      resources:
        limits:
          cpus: "2.0"
          memory: 4G

  keycloak:
    image: my-registry/keycloak-production:26.0
    build:
      context: .
      dockerfile: Dockerfile.keycloak
    environment:
      KC_DB_URL: jdbc:postgresql://postgres:5432/keycloak
      KC_DB_USERNAME: keycloak
      KC_DB_PASSWORD_FILE: /run/secrets/db_password
      KC_DB_POOL_INITIAL_SIZE: "25"
      KC_DB_POOL_MIN_SIZE: "25"
      KC_DB_POOL_MAX_SIZE: "100"
      KC_HOSTNAME: auth.example.com
      KC_HOSTNAME_ADMIN: admin-auth.internal.example.com
      KC_PROXY_HEADERS: xforwarded
      KC_HTTPS_CERTIFICATE_FILE: /certs/tls.crt
      KC_HTTPS_CERTIFICATE_KEY_FILE: /certs/tls.key
      KC_HTTP_ENABLED: "false"
      KC_LOG: console
      KC_LOG_LEVEL: info
      KC_LOG_CONSOLE_OUTPUT: json
      JAVA_OPTS_KC_HEAP: "-XX:InitialRAMPercentage=50.0 -XX:MaxRAMPercentage=70.0"
      JAVA_OPTS_APPEND: >-
        -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+UseContainerSupport
        -XX:+ExitOnOutOfMemoryError -Djava.net.preferIPv4Stack=true
    volumes:
      - ./certs:/certs:ro
    secrets:
      - db_password
    ports:
      - "8443:8443"
    depends_on:
      postgres:
        condition: service_healthy
    healthcheck:
      test: ["CMD-SHELL", "curl -sf https://localhost:8443/health/ready || exit 1"]
      interval: 15s
      timeout: 5s
      retries: 5
      start_period: 120s
    deploy:
      resources:
        limits:
          cpus: "2.0"
          memory: 2G

volumes:
  pgdata:

secrets:
  db_password:
    file: ./secrets/db_password.txt