xDev Asia

Raft Integrated Storage deep dive, Autopilot, HA architecture patterns, performance standby, read replicas concepts, TLS end-to-end, production hardening checklist, OS tuning cho Vault.

HashiCorp Vault from Basic to Advanced

Part 7: Production, Enterprise and Operations

xdev.asia

1. Raft Integrated Storage Deep Dive

From Vault 1.4+, Integrated Storage (Raft) is the recommended storage backend. It eliminates the dependency on external storage like Consul, simplifying architecture and operations.

Raft Consensus Architecture

┌─────────────────────────────────────────────────────────────┐
│                    Vault HA Cluster                         │
│                                                             │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      │
│  │  Node 1       │  │  Node 2       │  │  Node 3       │   │
│  │  (Leader)     │  │  (Follower)   │  │  (Follower)   │   │
│  │  ┌──────────┐ │  │  ┌──────────┐ │  │  ┌──────────┐ │   │
│  │  │  Raft    │ │  │  │  Raft    │ │  │  │  Raft    │ │   │
│  │  │  Log     │◄├──├─▶│  Log     │◄├──├─▶│  Log     │ │   │
│  │  └──────────┘ │  │  └──────────┘ │  │  └──────────┘ │   │
│  │  ┌──────────┐ │  │  ┌──────────┐ │  │  ┌──────────┐ │   │
│  │  │  BoltDB  │ │  │  │  BoltDB  │ │  │  │  BoltDB  │ │   │
│  │  └──────────┘ │  │  └──────────┘ │  │  └──────────┘ │   │
│  └──────────────┘  └──────────────┘  └──────────────┘      │
└─────────────────────────────────────────────────────────────┘

Raft Storage Configuration

# vault-config.hcl
storage "raft" {
  path    = "/opt/vault/data"
  node_id = "vault-node-1"

  retry_join {
    leader_api_addr         = "https://vault-node-2.company.com:8200"
    leader_ca_cert_file     = "/opt/vault/tls/ca.pem"
    leader_client_cert_file = "/opt/vault/tls/client.pem"
    leader_client_key_file  = "/opt/vault/tls/client-key.pem"
  }

  retry_join {
    leader_api_addr         = "https://vault-node-3.company.com:8200"
    leader_ca_cert_file     = "/opt/vault/tls/ca.pem"
    leader_client_cert_file = "/opt/vault/tls/client.pem"
    leader_client_key_file  = "/opt/vault/tls/client-key.pem"
  }

  # Performance tuning
  performance_multiplier = 1  # Optimized for fast networks (default: 5)
}

listener "tcp" {
  address            = "0.0.0.0:8200"
  cluster_address    = "0.0.0.0:8201"
  tls_cert_file      = "/opt/vault/tls/server.pem"
  tls_key_file       = "/opt/vault/tls/server-key.pem"
  tls_client_ca_file = "/opt/vault/tls/ca.pem"
}

cluster_addr = "https://vault-node-1.company.com:8201"
api_addr     = "https://vault-node-1.company.com:8200"

seal "awskms" {
  region     = "ap-southeast-1"
  kms_key_id = "alias/vault-unseal"
}

Autopilot — Automatic cluster management

# Xem Autopilot state
vault operator raft autopilot state

# Cấu hình Autopilot
vault operator raft autopilot set-config \
  -cleanup-dead-servers=true \
  -dead-server-last-contact-threshold=24h \
  -min-quorum=3 \
  -server-stabilization-time=10s

# Snapshot tự động (Enterprise)
vault operator raft autopilot snapshot-config set \
  -interval=1h \
  -retain=72 \
  -path-prefix="raft/snapshots"

Manual Snapshots

# Tạo snapshot
vault operator raft snapshot save /backup/vault-$(date +%Y%m%d%H%M).snap

# Restore snapshot
vault operator raft snapshot restore /backup/vault-20250101.snap

# List peers
vault operator raft list-peers

# Remove dead peer
vault operator raft remove-peer vault-node-4

2. HA Architecture Patterns

3-Node Cluster (Standard)

Characteristic	Value
Node number	3
Fault tolerance	1 node failure
Quorum	2
Suitable	Staging, Small/Medium Production

5-Node Cluster (Enterprise / Large scale)

Characteristic	Value
Node number	5
Fault tolerance	2 node failures
Quorum	3
Suitable	Large production, multi-AZ

Multi-AZ Deployment


Region: ap-southeast-1
┌────────────────┐  ┌────────────────┐  ┌────────────────┐
│      AZ-1      │  │      AZ-2      │  │      AZ-3      │
│                │  │                │  │                │
│  ┌──────────┐  │  │  ┌──────────┐  │  │  ┌──────────┐  │
│  │ Vault-1  │  │  │  │ Vault-2  │  │  │  │ Vault-3  │  │
│  │ (Leader) │  │  │  │(Follower)│  │  │  │(Follower)│  │
│  └──────────┘  │  │  └──────────┘  │  │  └──────────┘  │
│                │  │                │  │                │
│  ┌──────────┐  │  │                │  │  ┌──────────┐  │
│  │   LB     │  │  │  ┌──────────┐  │  │  │   LB     │  │
│  └──────────┘  │  │  │   LB     │  │  │  └──────────┘  │
│                │  │  └──────────┘  │  │                │
└────────────────┘  └────────────────┘  └────────────────┘
         ▲                  ▲                  ▲
         └──────────────────┴──────────────────┘
                        NLB / DNS

3. TLS End-to-End

Create Certificates with Vault PKI

# Root CA
vault secrets enable -path=pki-root pki
vault secrets tune -max-lease-ttl=87600h pki-root
vault write pki-root/root/generate/internal \
  common_name="Company Root CA" \
  ttl=87600h

# Intermediate CA cho Vault cluster
vault secrets enable -path=pki-vault pki
vault secrets tune -max-lease-ttl=43800h pki-vault

vault write pki-vault/intermediate/generate/internal \
  common_name="Vault Intermediate CA" \
  | jq -r '.data.csr' > vault-intermediate.csr

vault write pki-root/root/sign-intermediate \
  [email protected] \
  ttl=43800h \
  | jq -r '.data.certificate' > vault-intermediate.pem

vault write pki-vault/intermediate/set-signed \
  [email protected]

# Tạo role cho Vault server certs
vault write pki-vault/roles/vault-server \
  allowed_domains="vault.company.com,company.internal" \
  allow_subdomains=true \
  max_ttl=8760h

# Issue cert
vault write pki-vault/issue/vault-server \
  common_name="vault-node-1.company.internal" \
  alt_names="vault.company.com" \
  ip_sans="10.0.1.10" \
  ttl=720h

4. Production Hardening Checklist

OS-Level Tuning

# /etc/sysctl.d/99-vault.conf
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_tw_reuse = 1
vm.swappiness = 0
vm.dirty_ratio = 10

# Apply
sysctl --system

# File descriptor limits
# /etc/security/limits.d/vault.conf
vault soft nofile 65536
vault hard nofile 65536
vault soft nproc 65536
vault hard nproc 65536

Systemd Service Hardening

# /etc/systemd/system/vault.service
[Unit]
Description=HashiCorp Vault
Documentation=https://www.vaultproject.io/docs
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/vault.d/vault.hcl

[Service]
Type=notify
User=vault
Group=vault
ExecStart=/usr/bin/vault server -config=/etc/vault.d/vault.hcl
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
LimitNOFILE=65536
LimitMEMLOCK=infinity

# Security hardening
ProtectSystem=full
ProtectHome=true
PrivateTmp=true
PrivateDevices=true
NoNewPrivileges=true
CapabilityBoundingSet=CAP_IPC_LOCK CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_IPC_LOCK

[Install]
WantedBy=multi-user.target

Vault Configuration Hardening

# Production vault.hcl
ui            = true
disable_mlock = false
log_level     = "warn"

# Disable unused features
raw_storage_endpoint = false

listener "tcp" {
  address            = "0.0.0.0:8200"
  cluster_address    = "0.0.0.0:8201"
  tls_cert_file      = "/opt/vault/tls/server.pem"
  tls_key_file       = "/opt/vault/tls/server-key.pem"
  tls_client_ca_file = "/opt/vault/tls/ca.pem"
  tls_min_version    = "tls13"
  tls_require_and_verify_client_cert = false

  # Telemetry
  telemetry {
    unauthenticated_metrics_access = false
  }
}

# Audit device — BẮT BUỘC trong production
# Cấu hình sau khi unseal qua CLI

General Checklist

#	Item	Status
1	TLS end-to-end (API + Cluster)	☐
2	Auto-unseal (KMS / HSM)	☐
3	Audit device enabled	☐
4	Initial root token revoked	☐
5	mlock enabled (disable_mlock = false)	☐
6	File descriptor limits increased	☐
7	Swap disabled or swappiness = 0	☐
8	Dedicated machine / VM	☐
9	Firewall — open only 8200, 8201	☐
10	Network segmentation	☐
11	Automated backup schedule	☐
12	Monitoring & alerting	☐
13	Log rotation configured	☐
14	Least-privilege policies	☐

5. Summary

Raft Integrated Storage — Built-in HA storage, no Consul needed
Autopilot — Automatically manage cluster membership, dead server cleanup
Multi-AZ — Spread nodes across AZs cho fault tolerance
TLS end-to-end — Required in production
Hardening — OS tuning, systemd security, mlock, firewall, audit

Lesson 23: High Availability, Integrated Storage and Production Hardening