1. Raft Integrated Storage Deep Dive
Từ Vault 1.4+, Integrated Storage (Raft) là storage backend được khuyên dùng. Nó loại bỏ sự phụ thuộc vào external storage như Consul, đơn giản hóa kiến trúc và vận hành.
Kiến trúc Raft Consensus
┌─────────────────────────────────────────────────────────────┐
│ Vault HA Cluster │
│ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Node 1 │ │ Node 2 │ │ Node 3 │ │
│ │ (Leader) │ │ (Follower) │ │ (Follower) │ │
│ │ ┌──────────┐ │ │ ┌──────────┐ │ │ ┌──────────┐ │ │
│ │ │ Raft │ │ │ │ Raft │ │ │ │ Raft │ │ │
│ │ │ Log │◄├──├─▶│ Log │◄├──├─▶│ Log │ │ │
│ │ └──────────┘ │ │ └──────────┘ │ │ └──────────┘ │ │
│ │ ┌──────────┐ │ │ ┌──────────┐ │ │ ┌──────────┐ │ │
│ │ │ BoltDB │ │ │ │ BoltDB │ │ │ │ BoltDB │ │ │
│ │ └──────────┘ │ │ └──────────┘ │ │ └──────────┘ │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
└─────────────────────────────────────────────────────────────┘
Cấu hình Raft Storage
# vault-config.hcl
storage "raft" {
path = "/opt/vault/data"
node_id = "vault-node-1"
retry_join {
leader_api_addr = "https://vault-node-2.company.com:8200"
leader_ca_cert_file = "/opt/vault/tls/ca.pem"
leader_client_cert_file = "/opt/vault/tls/client.pem"
leader_client_key_file = "/opt/vault/tls/client-key.pem"
}
retry_join {
leader_api_addr = "https://vault-node-3.company.com:8200"
leader_ca_cert_file = "/opt/vault/tls/ca.pem"
leader_client_cert_file = "/opt/vault/tls/client.pem"
leader_client_key_file = "/opt/vault/tls/client-key.pem"
}
# Performance tuning
performance_multiplier = 1 # Optimized for fast networks (default: 5)
}
listener "tcp" {
address = "0.0.0.0:8200"
cluster_address = "0.0.0.0:8201"
tls_cert_file = "/opt/vault/tls/server.pem"
tls_key_file = "/opt/vault/tls/server-key.pem"
tls_client_ca_file = "/opt/vault/tls/ca.pem"
}
cluster_addr = "https://vault-node-1.company.com:8201"
api_addr = "https://vault-node-1.company.com:8200"
seal "awskms" {
region = "ap-southeast-1"
kms_key_id = "alias/vault-unseal"
}
Autopilot — Quản lý cluster tự động
# Xem Autopilot state
vault operator raft autopilot state
# Cấu hình Autopilot
vault operator raft autopilot set-config \
-cleanup-dead-servers=true \
-dead-server-last-contact-threshold=24h \
-min-quorum=3 \
-server-stabilization-time=10s
# Snapshot tự động (Enterprise)
vault operator raft autopilot snapshot-config set \
-interval=1h \
-retain=72 \
-path-prefix="raft/snapshots"
Manual Snapshots
# Tạo snapshot
vault operator raft snapshot save /backup/vault-$(date +%Y%m%d%H%M).snap
# Restore snapshot
vault operator raft snapshot restore /backup/vault-20250101.snap
# List peers
vault operator raft list-peers
# Remove dead peer
vault operator raft remove-peer vault-node-4
2. HA Architecture Patterns
3-Node Cluster (Tiêu chuẩn)
| Đặc tính | Giá trị |
|---|---|
| Số node | 3 |
| Fault tolerance | 1 node failure |
| Quorum | 2 |
| Phù hợp | Staging, Production nhỏ/trung bình |
5-Node Cluster (Enterprise / Large scale)
| Đặc tính | Giá trị |
|---|---|
| Số node | 5 |
| Fault tolerance | 2 node failures |
| Quorum | 3 |
| Phù hợp | Large production, multi-AZ |
Multi-AZ Deployment
Region: ap-southeast-1
┌────────────────┐ ┌────────────────┐ ┌────────────────┐
│ AZ-1 │ │ AZ-2 │ │ AZ-3 │
│ │ │ │ │ │
│ ┌──────────┐ │ │ ┌──────────┐ │ │ ┌──────────┐ │
│ │ Vault-1 │ │ │ │ Vault-2 │ │ │ │ Vault-3 │ │
│ │ (Leader) │ │ │ │(Follower)│ │ │ │(Follower)│ │
│ └──────────┘ │ │ └──────────┘ │ │ └──────────┘ │
│ │ │ │ │ │
│ ┌──────────┐ │ │ │ │ ┌──────────┐ │
│ │ LB │ │ │ ┌──────────┐ │ │ │ LB │ │
│ └──────────┘ │ │ │ LB │ │ │ └──────────┘ │
│ │ │ └──────────┘ │ │ │
└────────────────┘ └────────────────┘ └────────────────┘
▲ ▲ ▲
└──────────────────┴──────────────────┘
NLB / DNS
3. TLS End-to-End
Tạo Certificates với Vault PKI
# Root CA
vault secrets enable -path=pki-root pki
vault secrets tune -max-lease-ttl=87600h pki-root
vault write pki-root/root/generate/internal \
common_name="Company Root CA" \
ttl=87600h
# Intermediate CA cho Vault cluster
vault secrets enable -path=pki-vault pki
vault secrets tune -max-lease-ttl=43800h pki-vault
vault write pki-vault/intermediate/generate/internal \
common_name="Vault Intermediate CA" \
| jq -r '.data.csr' > vault-intermediate.csr
vault write pki-root/root/sign-intermediate \
[email protected] \
ttl=43800h \
| jq -r '.data.certificate' > vault-intermediate.pem
vault write pki-vault/intermediate/set-signed \
[email protected]
# Tạo role cho Vault server certs
vault write pki-vault/roles/vault-server \
allowed_domains="vault.company.com,company.internal" \
allow_subdomains=true \
max_ttl=8760h
# Issue cert
vault write pki-vault/issue/vault-server \
common_name="vault-node-1.company.internal" \
alt_names="vault.company.com" \
ip_sans="10.0.1.10" \
ttl=720h
4. Production Hardening Checklist
OS-Level Tuning
# /etc/sysctl.d/99-vault.conf
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_tw_reuse = 1
vm.swappiness = 0
vm.dirty_ratio = 10
# Apply
sysctl --system
# File descriptor limits
# /etc/security/limits.d/vault.conf
vault soft nofile 65536
vault hard nofile 65536
vault soft nproc 65536
vault hard nproc 65536
Systemd Service Hardening
# /etc/systemd/system/vault.service
[Unit]
Description=HashiCorp Vault
Documentation=https://www.vaultproject.io/docs
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/vault.d/vault.hcl
[Service]
Type=notify
User=vault
Group=vault
ExecStart=/usr/bin/vault server -config=/etc/vault.d/vault.hcl
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
LimitNOFILE=65536
LimitMEMLOCK=infinity
# Security hardening
ProtectSystem=full
ProtectHome=true
PrivateTmp=true
PrivateDevices=true
NoNewPrivileges=true
CapabilityBoundingSet=CAP_IPC_LOCK CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_IPC_LOCK
[Install]
WantedBy=multi-user.target
Vault Configuration Hardening
# Production vault.hcl
ui = true
disable_mlock = false
log_level = "warn"
# Disable unused features
raw_storage_endpoint = false
listener "tcp" {
address = "0.0.0.0:8200"
cluster_address = "0.0.0.0:8201"
tls_cert_file = "/opt/vault/tls/server.pem"
tls_key_file = "/opt/vault/tls/server-key.pem"
tls_client_ca_file = "/opt/vault/tls/ca.pem"
tls_min_version = "tls13"
tls_require_and_verify_client_cert = false
# Telemetry
telemetry {
unauthenticated_metrics_access = false
}
}
# Audit device — BẮT BUỘC trong production
# Cấu hình sau khi unseal qua CLI
Checklist tổng hợp
| # | Hạng mục | Trạng thái |
|---|---|---|
| 1 | TLS end-to-end (API + Cluster) | ☐ |
| 2 | Auto-unseal (KMS / HSM) | ☐ |
| 3 | Audit device enabled | ☐ |
| 4 | Initial root token revoked | ☐ |
| 5 | mlock enabled (disable_mlock = false) | ☐ |
| 6 | File descriptor limits tăng | ☐ |
| 7 | Swap disabled hoặc swappiness = 0 | ☐ |
| 8 | Dedicated machine / VM | ☐ |
| 9 | Firewall — chỉ mở 8200, 8201 | ☐ |
| 10 | Network segmentation | ☐ |
| 11 | Automated backup schedule | ☐ |
| 12 | Monitoring & alerting | ☐ |
| 13 | Log rotation configured | ☐ |
| 14 | Least-privilege policies | ☐ |
5. Tổng kết
Raft Integrated Storage — Built-in HA storage, không cần Consul
Autopilot — Tự động quản lý cluster membership, dead server cleanup
Multi-AZ — Spread nodes across AZs cho fault tolerance
TLS end-to-end — Bắt buộc trong production
Hardening — OS tuning, systemd security, mlock, firewall, audit