xDev Asia

Prometheus + Grafana monitoring, audit device configuration, audit log analysis, Raft backup/restore, common issues troubleshooting, upgrade strategies.

HashiCorp Vault từ Cơ bản đến Nâng cao

Phần 7: Production, Enterprise và Vận hành

xdev.asia

1. Monitoring với Prometheus + Grafana

Enable Telemetry

# vault.hcl
telemetry {
  prometheus_retention_time = "30s"
  disable_hostname          = true
  
  # StatsD (optional nếu dùng StatsD exporter)
  # statsd_address = "statsd:8125"
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = false
  # ...

  telemetry {
    unauthenticated_metrics_access = true  # Hoặc false + dùng token
  }
}

Prometheus Scrape Config

# prometheus.yml
scrape_configs:
  - job_name: 'vault'
    metrics_path: '/v1/sys/metrics'
    params:
      format: ['prometheus']
    scheme: https
    tls_config:
      ca_file: /etc/prometheus/vault-ca.pem
    # Nếu unauthenticated_metrics_access = false:
    # bearer_token_file: /etc/prometheus/vault-token
    static_configs:
      - targets:
          - 'vault-node-1:8200'
          - 'vault-node-2:8200'
          - 'vault-node-3:8200'
        labels:
          cluster: 'production'

Key Metrics để Monitor

Metric	Mô tả	Alert threshold
`vault.core.handle_request.count`	Tổng số requests	Trend analysis
`vault.core.handle_request.duration`	Request latency	P99 > 500ms
`vault.token.count`	Active tokens	Sudden spike
`vault.expire.num_leases`	Active leases	> 256000
`vault.runtime.alloc_bytes`	Memory allocation	> 80% RAM
`vault.runtime.gc_pause_ns`	GC pause	> 2s
`vault.raft.leader.lastContact`	Time since leader contact	> 200ms
`vault.raft.commitTime`	Raft commit time	> 25ms
`vault.seal.unseal`	Unseal events	Unexpected count
`vault.audit.log_response_failure`	Audit log failures	Any > 0

Grafana Dashboard

{
  "dashboard_id": "vault-overview",
  "panels": [
    {
      "title": "Request Rate",
      "query": "rate(vault_core_handle_request_count[5m])"
    },
    {
      "title": "Request Latency P99",
      "query": "histogram_quantile(0.99, rate(vault_core_handle_request_duration_bucket[5m]))"
    },
    {
      "title": "Active Leases",
      "query": "vault_expire_num_leases"
    },
    {
      "title": "Raft Leader Last Contact",
      "query": "vault_raft_leader_lastContact"
    },
    {
      "title": "Memory Usage",
      "query": "vault_runtime_alloc_bytes / 1024 / 1024"
    }
  ]
}

Alerting Rules

# vault-alerts.yml
groups:
  - name: vault
    rules:
      - alert: VaultSealed
        expr: vault_core_unsealed == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Vault node is sealed"

      - alert: VaultHighLatency
        expr: histogram_quantile(0.99, rate(vault_core_handle_request_duration_bucket[5m])) > 0.5
        for: 5m
        labels:
          severity: warning

      - alert: VaultLeadershipLost
        expr: vault_raft_leader_lastContact > 200
        for: 30s
        labels:
          severity: critical

      - alert: VaultAuditFailure
        expr: increase(vault_audit_log_response_failure[5m]) > 0
        labels:
          severity: critical
          
      - alert: VaultTooManyLeases
        expr: vault_expire_num_leases > 200000
        for: 10m
        labels:
          severity: warning

2. Audit Logging

Cấu hình Audit Devices

# File audit device
vault audit enable file file_path=/var/log/vault/audit.log

# Syslog audit device
vault audit enable syslog tag="vault" facility="LOCAL0"

# Socket audit device (cho log aggregator)
vault audit enable socket \
  address="logstash.company.com:9000" \
  socket_type="tcp"

# Luôn enable ít nhất 2 audit devices!
# Vault sẽ BLOCK tất cả requests nếu không có audit device nào hoạt động

# List audit devices
vault audit list -detailed

Audit Log Format

{
  "time": "2025-01-15T10:30:00.000Z",
  "type": "request",
  "auth": {
    "client_token": "hmac-sha256:abc123...",
    "accessor": "hmac-sha256:def456...",
    "display_name": "approle-cicd",
    "policies": ["cicd-deploy", "default"],
    "metadata": {
      "role_name": "cicd-pipeline"
    },
    "entity_id": "entity-uuid-here",
    "token_type": "service"
  },
  "request": {
    "id": "request-uuid",
    "operation": "read",
    "mount_type": "kv",
    "path": "secret/data/production/db",
    "remote_address": "10.0.1.50",
    "remote_port": 45678
  }
}

Log Analysis Queries

# Tìm failed requests
cat /var/log/vault/audit.log | \
  jq -r 'select(.type == "response" and .response.data == null) | 
  [.time, .request.path, .auth.display_name, .error] | @tsv'

# Tìm policy denials
cat /var/log/vault/audit.log | \
  jq -r 'select(.error != null and (.error | contains("permission denied"))) |
  [.time, .request.path, .request.operation, .auth.display_name] | @tsv'

# Top 10 paths accessed
cat /var/log/vault/audit.log | \
  jq -r 'select(.type == "request") | .request.path' | \
  sort | uniq -c | sort -rn | head -10

# Requests by identity
cat /var/log/vault/audit.log | \
  jq -r 'select(.type == "request") | .auth.display_name' | \
  sort | uniq -c | sort -rn

3. Backup và Restore

Raft Snapshot Backup

#!/bin/bash
# vault-backup.sh

VAULT_ADDR="https://vault.company.com:8200"
BACKUP_DIR="/backup/vault"
RETENTION_DAYS=30
DATE=$(date +%Y%m%d_%H%M%S)

# Tạo snapshot
vault operator raft snapshot save \
  "${BACKUP_DIR}/vault-snapshot-${DATE}.snap"

# Verify snapshot
vault operator raft snapshot inspect \
  "${BACKUP_DIR}/vault-snapshot-${DATE}.snap"

# Upload to S3
aws s3 cp "${BACKUP_DIR}/vault-snapshot-${DATE}.snap" \
  "s3://company-backup/vault/vault-snapshot-${DATE}.snap" \
  --sse aws:kms

# Cleanup old backups
find "${BACKUP_DIR}" -name "*.snap" -mtime +${RETENTION_DAYS} -delete

# Cron job — backup mỗi giờ
0 * * * * /opt/vault/scripts/vault-backup.sh >> /var/log/vault-backup.log 2>&1

Restore

# Restore snapshot — CHÚ Ý: overwrite toàn bộ data
vault operator raft snapshot restore \
  /backup/vault/vault-snapshot-20250115_100000.snap

# Force restore (khi cluster đã thay đổi)
vault operator raft snapshot restore -force \
  /backup/vault/vault-snapshot-20250115_100000.snap

4. Troubleshooting

Common Issues

Issue	Nguyên nhân	Giải pháp
Vault sealed sau restart	Không có auto-unseal	Cấu hình KMS/HSM seal
503 Service Unavailable	Node đang standby / DR secondary	Route requests tới active node
Lease count tăng liên tục	Client không revoke leases	Tidy leases, kiểm tra TTL
Token store quá lớn	Orphan tokens không bị revoke	Token tidy, reduce TTL
Raft leader flapping	Network instability	Tăng heartbeat timeout
Audit device blocking	Disk full / log destination down	Free disk, fix log dest
High memory usage	Too many leases/tokens	Tidy and tune TTLs

Debug Commands

# Server status
vault status
vault status -format=json

# Raft cluster info
vault operator raft list-peers
vault operator raft autopilot state

# Leader info
vault operator step-down  # Force leader election

# Key status
vault operator key-status

# Lease management
vault lease list -prefix "database/"
vault lease revoke -prefix "database/creds/app-role/"

# Token tidy
vault write sys/tidy/tidy-token-store \
  safety_buffer="72h"

# Lease tidy
vault write sys/leases/tidy \
  tidy_type="irrevocable"

# Debug bundle (support)
vault debug -duration=2m -targets=metrics,server-status,replication-status

# Read internal counters
vault read sys/internal/counters/tokens
vault read sys/internal/counters/requests

Upgrade Strategy

# 1. Backup trước khi upgrade
vault operator raft snapshot save /backup/pre-upgrade.snap

# 2. Read release notes + upgrade guide

# 3. Upgrade standby nodes trước (rolling upgrade)
# Node 3 (standby)
systemctl stop vault
# Replace binary
cp vault-new /usr/bin/vault
systemctl start vault

# Verify node rejoins cluster
vault operator raft list-peers

# 4. Repeat cho Node 2

# 5. Step-down leader, upgrade cuối cùng
vault operator step-down
# Wait for new leader election
systemctl stop vault
cp vault-new /usr/bin/vault
systemctl start vault

# 6. Verify
vault status
vault operator raft list-peers
vault operator raft autopilot state

5. Tổng kết

Monitoring — Prometheus metrics + Grafana dashboards + alerting là bắt buộc
Audit — Luôn enable ≥ 2 audit devices, analyze logs regularly
Backup — Raft snapshots tự động, lưu offsite, test restore định kỳ
Troubleshooting — Biết các debug commands, common issues và cách xử lý
Upgrade — Rolling upgrade cho HA cluster, luôn backup trước

Đến đây, bạn đã hoàn thành toàn bộ series HashiCorp Vault từ Cơ bản đến Nâng cao. Từ setup, secrets engines, auth methods, policies, agent, Kubernetes, CI/CD, đến production HA, Enterprise, monitoring và vận hành.

Bài 25: Monitoring, Audit Logging, Backup/Restore và Troubleshooting