Lesson 23: Deploying Production and Performance Tuning

Keycloak from Basic to Advanced

Part 7: Production, HA and Kubernetes

xdev.asia

1. Production Deployment Checklist

Deploying Keycloak in production requires careful configuration of database, networking, JVM, caching and security. This article provides comprehensive guidance from database selection to load testing.

┌─────────────────────────────────────────────────────────────┐
│                  Production Checklist                       │
├─────────────────────────────────────────────────────────────┤
│  ✅ Database: PostgreSQL + Connection Pool tuning           │
│  ✅ Build: kc.sh build --optimized                          │
│  ✅ Hostname: hostname-v2 provider configured               │
│  ✅ Proxy: X-Forwarded-* / PROXY protocol                   │
│  ✅ TLS: Certificates configured                            │
│  ✅ JVM: Heap, GC, container-aware settings                 │
│  ✅ Caching: Infinispan local caches tuned                  │
│  ✅ Metrics: /metrics endpoint enabled                      │
│  ✅ Health: /health/ready, /health/live enabled              │
│  ✅ Load Test: Gatling benchmark passed                     │
└─────────────────────────────────────────────────────────────┘

2. Database Selection and Configuration

2.1 Database Support Matrix

Database	Vendor Flag	Recommended	Note
PostgreSQL	postgres	✅ Yes	Best performance, most tested by Keycloak team
MySQL	mysql	⚠️ OK	Need InnoDB, utf8mb4 charset
MariaDB	mariadb	⚠️ OK	Similar to MySQL
Oracle	oracle	⚠️ OK	Enterprise license required
Microsoft SQL Server	mssql	⚠️ OK	Windows environment
H2 (embedded)	dev-file/dev-mem	❌ No	For development only

2.2 PostgreSQL Configuration

# Cấu hình database cơ bản
bin/kc.sh start \
  --db=postgres \
  --db-url="jdbc:postgresql://db-host:5432/keycloak" \
  --db-username=keycloak \
  --db-password=secure_password_here \
  --db-schema=public

With environment variables (recommended for containers):

# Environment variables cho database
export KC_DB=postgres
export KC_DB_URL="jdbc:postgresql://db-host:5432/keycloak"
export KC_DB_USERNAME=keycloak
export KC_DB_PASSWORD=secure_password_here
export KC_DB_SCHEMA=public

# JDBC URL với advanced parameters
export KC_DB_URL="jdbc:postgresql://db-host:5432/keycloak?ssl=true&sslmode=verify-full&sslrootcert=/certs/ca.crt"

2.3 Connection Pool Tuning (Agroal)

Keycloak uses Agroal connection pool (Quarkus default). Tuning connection pool is an important factor affecting performance:

# Connection pool configuration
bin/kc.sh start \
  --db=postgres \
  --db-url="jdbc:postgresql://db-host:5432/keycloak" \
  --db-username=keycloak \
  --db-password=secure_password_here \
  --db-pool-initial-size=25 \
  --db-pool-min-size=25 \
  --db-pool-max-size=100

Parameter	Default	Production Recommended	Description
--db-pool-initial-size	0	25	Number of initial initial connections
--db-pool-min-size	0	25	Minimum number of connections maintained
--db-pool-max-size	100	50–100	Max connections

Principle for sizing connection pool:

Tổng connections = Số Keycloak instances × db-pool-max-size

Ví dụ: 3 instances × 100 max = 300 connections
→ PostgreSQL max_connections ≥ 300 + buffer (20%)
→ Đặt max_connections = 360

Server-side PostgreSQL configuration (postgresql.conf):

# postgresql.conf - tối ưu cho Keycloak
max_connections = 400
shared_buffers = 2GB
effective_cache_size = 6GB
work_mem = 16MB
maintenance_work_mem = 512MB

# Connection timeout
idle_in_transaction_session_timeout = 30000  # 30 seconds
statement_timeout = 60000                     # 60 seconds

# WAL configuration
wal_level = replica
max_wal_senders = 5
wal_keep_size = 1GB

3. Build Optimization

3.1 Build vs Start Phases

Keycloak has 2 phases: build (packaging config) and start (running server). In production, always use --optimized to separate the two phases, making startup significantly faster.

┌──────────────────────────────────────────────────────────────┐
│              Keycloak Build & Start Phases                   │
├──────────────────────────────────────────────────────────────┤
│                                                              │
│  Phase 1: BUILD (chạy 1 lần, hoặc khi config thay đổi)     │
│  ┌────────────────────────────────────────────────────────┐  │
│  │ kc.sh build                                           │  │
│  │ - Parse configuration                                 │  │
│  │ - Install providers/extensions                        │  │
│  │ - Quarkus augmentation (ahead-of-time optimization)   │  │
│  │ - Persist build options                               │  │
│  └────────────────────────────────────────────────────────┘  │
│                           ↓                                  │
│  Phase 2: START (mỗi lần khởi động)                         │
│  ┌────────────────────────────────────────────────────────┐  │
│  │ kc.sh start --optimized                               │  │
│  │ - Skip build phase → Fast startup                     │  │
│  │ - Use pre-built configuration                         │  │
│  │ - Apply runtime-only options                          │  │
│  └────────────────────────────────────────────────────────┘  │
└──────────────────────────────────────────────────────────────┘

# Build phase - chạy khi build Docker image
bin/kc.sh build \
  --db=postgres \
  --features=token-exchange,admin-fine-grained-authz \
  --health-enabled=true \
  --metrics-enabled=true \
  --http-relative-path=/auth

# Start phase - chạy khi container start
bin/kc.sh start --optimized \
  --db-url="jdbc:postgresql://db-host:5432/keycloak" \
  --db-username=keycloak \
  --db-password=secure_password_here \
  --hostname=auth.example.com \
  --https-certificate-file=/certs/tls.crt \
  --https-certificate-key-file=/certs/tls.key

3.2 Production Dockerfile

# Multi-stage Dockerfile cho Keycloak Production
FROM quay.io/keycloak/keycloak:26.0 AS builder

# Build phase - pre-build configuration
ENV KC_DB=postgres
ENV KC_HEALTH_ENABLED=true
ENV KC_METRICS_ENABLED=true
ENV KC_FEATURES=token-exchange,admin-fine-grained-authz
ENV KC_HTTP_RELATIVE_PATH=/auth

# Thêm custom providers nếu có
# COPY --chown=keycloak:keycloak my-provider.jar /opt/keycloak/providers/

# Thêm custom themes nếu có
# COPY --chown=keycloak:keycloak my-theme/ /opt/keycloak/themes/my-theme/

RUN /opt/keycloak/bin/kc.sh build

# Runtime stage
FROM quay.io/keycloak/keycloak:26.0

COPY --from=builder /opt/keycloak/ /opt/keycloak/

ENTRYPOINT ["/opt/keycloak/bin/kc.sh"]
CMD ["start", "--optimized"]

4. Hostname Configuration

4.1 Hostname v2 Provider

Keycloak uses the hostname-v2 provider (default from Keycloak 25+) to define URLs for all endpoints (frontend, backend, admin):

# Hostname configuration cơ bản
bin/kc.sh start --optimized \
  --hostname=auth.example.com \
  --hostname-admin=admin-auth.example.com

Parameter	Description	Example
--hostname	Hostname cho frontend URLs (login pages, well-known endpoints)	auth.example.com
--hostname-admin	Special hostname for Admin Console (if different from frontend). No set = shared --hostname	admin-auth.internal.com
--hostname-strict	Only allow requests to the configured hostname. Default: true	true
--hostname-backchannel-dynamic	Backend URL uses request hostname instead of fixed hostname. Default: false	false

4.2 Hostname Scenarios

# Scenario 1: Single domain cho tất cả
bin/kc.sh start --optimized \
  --hostname=auth.example.com

# Scenario 2: Tách Admin Console ra domain riêng (khuyến nghị production)
bin/kc.sh start --optimized \
  --hostname=auth.example.com \
  --hostname-admin=admin-auth.internal.example.com

# Scenario 3: Edge proxy determine hostname từ request
bin/kc.sh start --optimized \
  --hostname-strict=false \
  --proxy-headers=xforwarded

# Scenario 4: Backchannel dynamic (backend-to-backend dùng internal URL)
bin/kc.sh start --optimized \
  --hostname=auth.example.com \
  --hostname-backchannel-dynamic=true

5. Proxy Configuration

5.1 Proxy Headers

When Keycloak is behind a reverse proxy (Nginx, HAProxy, AWS ALB...), it is necessary to configure proxy headers so that Keycloak receives the correct client IP, protocol and hostname:

# Option 1: X-Forwarded-* headers (phổ biến nhất)
bin/kc.sh start --optimized \
  --proxy-headers=xforwarded

# Option 2: RFC 7239 Forwarded header
bin/kc.sh start --optimized \
  --proxy-headers=forwarded

Header	Purpose	Flag
X-Forwarded-For	Client IP address	xforwarded
X-Forwarded-Proto	Original protocol (http/https)	xforwarded
X-Forwarded-Host	Original hostname	xforwarded
X-Forwarded-Port	Original port	xforwarded
Forwarded	RFC 7239 combined header	forwarded

5.2 Nginx Reverse Proxy Config

# /etc/nginx/conf.d/keycloak.conf
upstream keycloak_backend {
    server keycloak-1:8443;
    server keycloak-2:8443;
    # Sticky session based on KEYCLOAK_SESSION cookie
    sticky cookie KEYCLOAK_ROUTE expires=1h domain=.example.com httponly secure;
}

server {
    listen 443 ssl http2;
    server_name auth.example.com;

    ssl_certificate     /etc/nginx/certs/tls.crt;
    ssl_certificate_key /etc/nginx/certs/tls.key;

    # Security headers
    add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
    add_header X-Content-Type-Options nosniff;
    add_header X-Frame-Options SAMEORIGIN;

    # Buffer sizes cho Keycloak (tokens có thể lớn)
    proxy_buffer_size        128k;
    proxy_buffers            4 256k;
    proxy_busy_buffers_size  256k;
    large_client_header_buffers 4 16k;

    location / {
        proxy_pass https://keycloak_backend;

        # X-Forwarded headers
        proxy_set_header Host               $host;
        proxy_set_header X-Real-IP          $remote_addr;
        proxy_set_header X-Forwarded-For    $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto  $scheme;
        proxy_set_header X-Forwarded-Host   $host;
        proxy_set_header X-Forwarded-Port   $server_port;

        # WebSocket support (cho Admin Console)
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        # Timeouts
        proxy_connect_timeout 60s;
        proxy_send_timeout    60s;
        proxy_read_timeout    60s;
    }
}

5.3 HTTP/2 Support

# Enable HTTP/2 (default đã enabled khi dùng HTTPS)
bin/kc.sh start --optimized \
  --hostname=auth.example.com \
  --https-certificate-file=/certs/tls.crt \
  --https-certificate-key-file=/certs/tls.key \
  --http-enabled=false

# Nếu cần HTTP/2 cleartext (h2c) cho internal communication
bin/kc.sh start --optimized \
  --hostname=auth.example.com \
  --http-enabled=true \
  --http-port=8080

6. JVM Tuning

6.1 Heap Configuration

Keycloak runs on Quarkus/JVM, so JVM tuning directly affects performance and stability:

# JVM Heap - sử dụng JAVA_OPTS_KC_HEAP (Keycloak 25+)
export JAVA_OPTS_KC_HEAP="-XX:InitialRAMPercentage=50.0 -XX:MaxRAMPercentage=70.0"

# Hoặc set fixed heap size
export JAVA_OPTS_KC_HEAP="-Xms512m -Xmx2g"

6.2 Garbage Collector Selection

# Option 1: G1GC (recommended cho heap ≤ 4GB)
export JAVA_OPTS_APPEND="-XX:+UseG1GC \
  -XX:MaxGCPauseMillis=200 \
  -XX:G1HeapRegionSize=16m \
  -XX:+ParallelRefProcEnabled \
  -XX:+UseStringDeduplication"

# Option 2: ZGC (recommended cho heap > 4GB, low-latency)
export JAVA_OPTS_APPEND="-XX:+UseZGC \
  -XX:+ZGenerational \
  -XX:ConcGCThreads=2"

# Option 3: Shenandoah GC (alternative low-latency)
export JAVA_OPTS_APPEND="-XX:+UseShenandoahGC \
  -XX:ShenandoahGCHeuristics=compact"

GC Algorithm	Best For	Heap Size	Pause Time
G1GC	General purpose, balanced throughput/latency	≤ 4GB	~200ms
ZGC	Low-latency, large heap	> 4GB	< 1ms
Shenandoah	Low-latency, concurrent	> 2GB	< 10ms

6.3 Container-Aware JVM Settings

# Container-aware JVM settings (Docker/Kubernetes)
export JAVA_OPTS_APPEND=" \
  -XX:+UseContainerSupport \
  -XX:MaxRAMPercentage=70.0 \
  -XX:InitialRAMPercentage=50.0 \
  -XX:MinRAMPercentage=50.0 \
  -XX:ActiveProcessorCount=2 \
  -XX:+UseG1GC \
  -XX:MaxGCPauseMillis=200 \
  -Djava.net.preferIPv4Stack=true \
  -Djava.awt.headless=true \
  -Dfile.encoding=UTF-8"

6.4 Complete JVM Configuration

# Production JVM configuration hoàn chỉnh
export JAVA_OPTS_KC_HEAP="-XX:InitialRAMPercentage=50.0 -XX:MaxRAMPercentage=70.0"

export JAVA_OPTS_APPEND=" \
  -XX:+UseG1GC \
  -XX:MaxGCPauseMillis=200 \
  -XX:G1HeapRegionSize=16m \
  -XX:+ParallelRefProcEnabled \
  -XX:+UseStringDeduplication \
  -XX:+UseContainerSupport \
  -XX:ActiveProcessorCount=2 \
  -XX:MetaspaceSize=256m \
  -XX:MaxMetaspaceSize=512m \
  -Djava.net.preferIPv4Stack=true \
  -Djava.awt.headless=true \
  -Dfile.encoding=UTF-8 \
  -XX:+ExitOnOutOfMemoryError \
  -XX:+HeapDumpOnOutOfMemoryError \
  -XX:HeapDumpPath=/opt/keycloak/dumps/"

7. Quarkus Thread Pool and Vert.x

Keycloak runs on Quarkus (Vert.x event loop + worker thread pool). Understanding this pattern helps to tune correctly:

┌───────────────────────────────────────────────────────┐
│                  Keycloak / Quarkus                    │
│                                                       │
│  ┌─────────────────────────────────────────────────┐  │
│  │          Vert.x Event Loop Threads              │  │
│  │  (IO_THREADS = 2 × CPU cores, default)          │  │
│  │  - Accept connections                           │  │
│  │  - Parse HTTP requests                          │  │
│  │  - Non-blocking operations                      │  │
│  └──────────────┬──────────────────────────────────┘  │
│                  │ delegate blocking work              │
│  ┌──────────────▼──────────────────────────────────┐  │
│  │          Worker Thread Pool                     │  │
│  │  (WORKER_THREADS = 8 × CPU cores, default)      │  │
│  │  - Database queries                             │  │
│  │  - LDAP calls                                   │  │
│  │  - Token signing/validation                     │  │
│  │  - Template rendering                           │  │
│  └─────────────────────────────────────────────────┘  │
└───────────────────────────────────────────────────────┘

# Quarkus thread pool configuration (qua Java system properties)
export JAVA_OPTS_APPEND=" \
  -Dquarkus.thread-pool.max-threads=200 \
  -Dquarkus.thread-pool.queue-size=1000 \
  -Dquarkus.thread-pool.growth-resistance=0.1 \
  -Dquarkus.vertx.event-loops-pool-size=4"

Parameter	Default	Description
quarkus.thread-pool.max-threads	200 (or 8 × CPU max)	Max worker threads
quarkus.thread-pool.queue-size	unbounded	Queue size when all threads are busy
quarkus.thread-pool.growth-resistance	0	0–1, thread pool grow resistance
quarkus.vertx.event-loops-pool-size	2 × CPU	Vert.x event loop threads

8. Infinispan Local Cache Tuning

8.1 Keycloak Cache Architecture

Keycloak uses Infinispan local caches to cache metadata (realms, users, keys...) to reduce database queries. Tuning cache size and lifespan greatly affects performance:

┌─────────────────────────────────────────────────────────────┐
│                 Infinispan Local Caches                      │
├─────────────────────────────────────────────────────────────┤
│                                                             │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐  │
│  │    realms     │  │    users     │  │  authorization   │  │
│  │  (realm cfg)  │  │ (user data)  │  │ (permissions)    │  │
│  │  max: 10000   │  │  max: 10000  │  │  max: 10000      │  │
│  └──────────────┘  └──────────────┘  └──────────────────┘  │
│                                                             │
│  ┌──────────────┐  ┌──────────────────────────────────────┐ │
│  │    keys       │  │        Revision Caches               │ │
│  │  (crypto)     │  │  (realmRevisions, userRevisions,     │ │
│  │  max: 1000    │  │   authorizationRevisions)            │ │
│  └──────────────┘  └──────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘

8.2 Custom Cache Configuration

Create custom cache-ispn.xml file:

<?xml version="1.0" encoding="UTF-8"?>
<infinispan
    xmlns="urn:infinispan:config:15.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="urn:infinispan:config:15.0
        https://infinispan.org/schemas/infinispan-config-15.0.xsd">

    <cache-container name="keycloak">
        <!-- Realm cache - cache metadata realm -->
        <local-cache name="realms">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="20000"/>
        </local-cache>

        <!-- User cache - cache user data -->
        <local-cache name="users">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="20000"/>
        </local-cache>

        <!-- Authorization cache -->
        <local-cache name="authorization">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="20000"/>
        </local-cache>

        <!-- Keys cache - signing/encryption keys -->
        <local-cache name="keys">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="5000"/>
            <expiration max-idle="3600000"/> <!-- 1 hour -->
        </local-cache>

        <!-- Revision caches - invalidation tracking -->
        <local-cache name="realmRevisions">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="40000"/>
        </local-cache>

        <local-cache name="userRevisions">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="40000"/>
        </local-cache>

        <local-cache name="authorizationRevisions">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="40000"/>
        </local-cache>
    </cache-container>
</infinispan>

# Sử dụng custom cache config
bin/kc.sh build --cache=ispn --cache-config-file=cache-ispn.xml
bin/kc.sh start --optimized

9. Metrics and Health Checks

9.1 Metrics Endpoint

# Enable metrics (cần set khi build)
bin/kc.sh build --metrics-enabled=true
bin/kc.sh start --optimized

Metrics endpoint: https://auth.example.com/metrics (Prometheus format)

# Ví dụ metrics output
# HELP keycloak_logins_total Total successful logins
# TYPE keycloak_logins_total counter
keycloak_logins_total{realm="my-realm",provider="keycloak",client_id="my-app"} 1523

# HELP keycloak_failed_login_attempts_total Total failed login attempts
# TYPE keycloak_failed_login_attempts_total counter
keycloak_failed_login_attempts_total{realm="my-realm",provider="keycloak",client_id="my-app",error="invalid_user_credentials"} 42

# HELP vendor_memoryPool_usage_max_bytes Peak JVM memory pool usage
# TYPE vendor_memoryPool_usage_max_bytes gauge
vendor_memoryPool_usage_max_bytes{name="G1 Old Gen"} 524288000

# HELP vendor_cpu_processCpuLoad JVM Process CPU load
# TYPE vendor_cpu_processCpuLoad gauge
vendor_cpu_processCpuLoad 0.15

9.2 Health Check Endpoints

# Enable health checks (cần set khi build)
bin/kc.sh build --health-enabled=true

Endpoint	Purpose	Kubernetes Probe
/health/ready	Readiness - ready to receive traffic	readinessProbe
/health/live	Liveness - active process	livenessProbe
/health/started	Startup - started up	startupProbe
/health	Combined health status	—

// GET /health/ready - Response khi healthy
{
  "status": "UP",
  "checks": [
    {
      "name": "Keycloak database connections health check",
      "status": "UP"
    }
  ]
}

// GET /health/ready - Response khi unhealthy
{
  "status": "DOWN",
  "checks": [
    {
      "name": "Keycloak database connections health check",
      "status": "DOWN",
      "data": {
        "message": "Unable to connect to database"
      }
    }
  ]
}

10. Load Testing with Gatling

10.1 Keycloak Benchmark Project

Keycloak provides an official benchmark project based on the Gatling framework:

# Clone Keycloak benchmark
git clone https://github.com/keycloak/keycloak-benchmark.git
cd keycloak-benchmark

# Build benchmark
mvn clean install -DskipTests

# Run token endpoint benchmark
cd benchmark
mvn gatling:test \
  -Dgatling.simulationClass=keycloak.scenario.authentication.ClientSecret \
  -Dkeycloak.server.url=https://auth.example.com \
  -Drealm=benchmark-realm \
  -DclientId=benchmark-client \
  -DclientSecret=benchmark-secret \
  -DusersPerSec=50 \
  -DrampUpPeriod=30 \
  -DwarmUpPeriod=60 \
  -DmeasurementPeriod=180

10.2 Custom Gatling Simulation

// KeycloakLoadTest.scala
package keycloak.benchmark

import io.gatling.core.Predef._
import io.gatling.http.Predef._
import scala.concurrent.duration._

class KeycloakTokenEndpointSimulation extends Simulation {

  val keycloakUrl = System.getProperty("keycloak.url", "https://auth.example.com")
  val realm = System.getProperty("realm", "my-realm")
  val clientId = System.getProperty("clientId", "benchmark-client")
  val clientSecret = System.getProperty("clientSecret", "benchmark-secret")

  val httpProtocol = http
    .baseUrl(keycloakUrl)
    .acceptHeader("application/json")
    .contentTypeHeader("application/x-www-form-urlencoded")
    .disableFollowRedirect

  // Scenario 1: Client Credentials Grant (service-to-service)
  val clientCredentialsScenario = scenario("Client Credentials Flow")
    .exec(
      http("Token Request - Client Credentials")
        .post(s"/realms/$realm/protocol/openid-connect/token")
        .formParam("grant_type", "client_credentials")
        .formParam("client_id", clientId)
        .formParam("client_secret", clientSecret)
        .check(status.is(200))
        .check(jsonPath("$.access_token").exists)
        .check(responseTimeInMillis.lte(500))
    )

  // Scenario 2: Resource Owner Password Grant (user login)
  val passwordGrantScenario = scenario("Resource Owner Password Flow")
    .exec(
      http("Token Request - Password Grant")
        .post(s"/realms/$realm/protocol/openid-connect/token")
        .formParam("grant_type", "password")
        .formParam("client_id", clientId)
        .formParam("client_secret", clientSecret)
        .formParam("username", "test-user")
        .formParam("password", "test-password")
        .check(status.is(200))
        .check(jsonPath("$.access_token").saveAs("accessToken"))
    )
    .pause(1.second)
    .exec(
      http("Token Introspection")
        .post(s"/realms/$realm/protocol/openid-connect/token/introspect")
        .formParam("token", "${accessToken}")
        .formParam("client_id", clientId)
        .formParam("client_secret", clientSecret)
        .check(status.is(200))
        .check(jsonPath("$.active").is("true"))
    )

  // Scenario 3: UserInfo endpoint
  val userInfoScenario = scenario("UserInfo Flow")
    .exec(
      http("Get Token")
        .post(s"/realms/$realm/protocol/openid-connect/token")
        .formParam("grant_type", "password")
        .formParam("client_id", clientId)
        .formParam("client_secret", clientSecret)
        .formParam("username", "test-user")
        .formParam("password", "test-password")
        .check(status.is(200))
        .check(jsonPath("$.access_token").saveAs("accessToken"))
    )
    .exec(
      http("UserInfo")
        .get(s"/realms/$realm/protocol/openid-connect/userinfo")
        .header("Authorization", "Bearer ${accessToken}")
        .check(status.is(200))
    )

  setUp(
    clientCredentialsScenario.inject(
      rampUsersPerSec(1).to(100).during(60.seconds),  // Ramp up
      constantUsersPerSec(100).during(180.seconds),     // Sustained load
      rampUsersPerSec(100).to(1).during(30.seconds)    // Ramp down
    ),
    passwordGrantScenario.inject(
      rampUsersPerSec(1).to(50).during(60.seconds),
      constantUsersPerSec(50).during(180.seconds),
      rampUsersPerSec(50).to(1).during(30.seconds)
    )
  ).protocols(httpProtocol)
    .assertions(
      global.responseTime.percentile3.lt(500),  // p95 < 500ms
      global.responseTime.percentile4.lt(1000), // p99 < 1000ms
      global.successfulRequests.percent.gt(99)  // > 99% success rate
    )
}

11. Production Checklist Summary

Category	Item	Status
Database	PostgreSQL with connection pool tuned	☐
Database	Database backup automated (pg_dump / pg_basebackup)	☐
Database	Database replication configured	☐
Build	kc.sh build + start --optimized	☐
Build	Multi-stage Dockerfile	☐
Hostname	--hostname configured (hostname-v2)	☐
Hostname	--hostname-admin set to separate domain	☐
TLS	Valid TLS certificates (Let's Encrypt / CA-signed)	☐
TLS	TLS 1.2+ enforced, weak ciphers disabled	☐
Proxy	--proxy-headers configured correctly	☐
JVM	Heap sized (50–70% container memory)	☐
JVM	GC algorithm selected (G1GC/ZGC)	☐
JVM	Container-aware flags enabled	☐
JVM	-XX:+ExitOnOutOfMemoryError set	☐
Cache	Infinispan local caches tuned	☐
Observability	Metrics endpoint enabled (/metrics)	☐
Observability	Health checks enabled (/health/*)	☐
Observability	Logging configured (JSON format, log level)	☐
Security	Admin Console on separate domain/network	☐
Security	Default admin credentials changed	☐
Security	Brute force protection enabled	☐
Security	CORS configured correctly	☐
Load Test	Gatling benchmark passed (p95 < 500ms)	☐
Load Test	Capacity planning documented	☐
Backup	Realm export automated	☐
Backup	Disaster recovery plan tested	☐

# Complete production start command
bin/kc.sh start --optimized \
  --db=postgres \
  --db-url="jdbc:postgresql://db-host:5432/keycloak?ssl=true" \
  --db-username=keycloak \
  --db-password="${KC_DB_PASSWORD}" \
  --db-pool-initial-size=25 \
  --db-pool-min-size=25 \
  --db-pool-max-size=100 \
  --hostname=auth.example.com \
  --hostname-admin=admin-auth.internal.example.com \
  --hostname-strict=true \
  --proxy-headers=xforwarded \
  --https-certificate-file=/certs/tls.crt \
  --https-certificate-key-file=/certs/tls.key \
  --http-enabled=false \
  --health-enabled=true \
  --metrics-enabled=true \
  --log=console \
  --log-level=info \
  --log-console-output=json

# Docker Compose cho production
# docker-compose.production.yml

version: "3.9"

services:
  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: keycloak
      POSTGRES_USER: keycloak
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
    volumes:
      - pgdata:/var/lib/postgresql/data
      - ./postgresql.conf:/etc/postgresql/postgresql.conf
    command: postgres -c config_file=/etc/postgresql/postgresql.conf
    secrets:
      - db_password
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U keycloak"]
      interval: 10s
      timeout: 5s
      retries: 5
    deploy:
      resources:
        limits:
          cpus: "2.0"
          memory: 4G

  keycloak:
    image: my-registry/keycloak-production:26.0
    build:
      context: .
      dockerfile: Dockerfile.keycloak
    environment:
      KC_DB_URL: jdbc:postgresql://postgres:5432/keycloak
      KC_DB_USERNAME: keycloak
      KC_DB_PASSWORD_FILE: /run/secrets/db_password
      KC_DB_POOL_INITIAL_SIZE: "25"
      KC_DB_POOL_MIN_SIZE: "25"
      KC_DB_POOL_MAX_SIZE: "100"
      KC_HOSTNAME: auth.example.com
      KC_HOSTNAME_ADMIN: admin-auth.internal.example.com
      KC_PROXY_HEADERS: xforwarded
      KC_HTTPS_CERTIFICATE_FILE: /certs/tls.crt
      KC_HTTPS_CERTIFICATE_KEY_FILE: /certs/tls.key
      KC_HTTP_ENABLED: "false"
      KC_LOG: console
      KC_LOG_LEVEL: info
      KC_LOG_CONSOLE_OUTPUT: json
      JAVA_OPTS_KC_HEAP: "-XX:InitialRAMPercentage=50.0 -XX:MaxRAMPercentage=70.0"
      JAVA_OPTS_APPEND: >-
        -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+UseContainerSupport
        -XX:+ExitOnOutOfMemoryError -Djava.net.preferIPv4Stack=true
    volumes:
      - ./certs:/certs:ro
    secrets:
      - db_password
    ports:
      - "8443:8443"
    depends_on:
      postgres:
        condition: service_healthy
    healthcheck:
      test: ["CMD-SHELL", "curl -sf https://localhost:8443/health/ready || exit 1"]
      interval: 15s
      timeout: 5s
      retries: 5
      start_period: 120s
    deploy:
      resources:
        limits:
          cpus: "2.0"
          memory: 2G

volumes:
  pgdata:

secrets:
  db_password:
    file: ./secrets/db_password.txt