xDev Asia

Prometheus + Grafana monitoring, audit device configuration, audit log analysis, Raft backup/restore, common issues troubleshooting, upgrade strategies.

HashiCorp Vault from Basic to Advanced

Part 7: Production, Enterprise and Operations

xdev.asia

1. Monitoring with Prometheus + Grafana

Enable Telemetry

# vault.hcl
telemetry {
  prometheus_retention_time = "30s"
  disable_hostname          = true
  
  # StatsD (optional nếu dùng StatsD exporter)
  # statsd_address = "statsd:8125"
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = false
  # ...

  telemetry {
    unauthenticated_metrics_access = true  # Hoặc false + dùng token
  }
}

Prometheus Scrape Config

# prometheus.yml
scrape_configs:
  - job_name: 'vault'
    metrics_path: '/v1/sys/metrics'
    params:
      format: ['prometheus']
    scheme: https
    tls_config:
      ca_file: /etc/prometheus/vault-ca.pem
    # Nếu unauthenticated_metrics_access = false:
    # bearer_token_file: /etc/prometheus/vault-token
    static_configs:
      - targets:
          - 'vault-node-1:8200'
          - 'vault-node-2:8200'
          - 'vault-node-3:8200'
        labels:
          cluster: 'production'

Key Metrics to Monitor

Metric	Description	Alert threshold
`vault.core.handle_request.count`	Total requests	Trend analysis
`vault.core.handle_request.duration`	Request latency	P99 > 500ms
`vault.token.count`	Active tokens	Sudden spike
`vault.expire.num_leases`	Active leases	> 256000
`vault.runtime.alloc_bytes`	Memory allocation	> 80% RAM
`vault.runtime.gc_pause_ns`	GC pause	> 2s
`vault.raft.leader.lastContact`	Time since leader contact	> 200ms
`vault.raft.commitTime`	Raft commit time	> 25ms
`vault.seal.unseal`	Unseal events	Unexpected count
`vault.audit.log_response_failure`	Audit log failures	Any > 0

Grafana Dashboard

{
  "dashboard_id": "vault-overview",
  "panels": [
    {
      "title": "Request Rate",
      "query": "rate(vault_core_handle_request_count[5m])"
    },
    {
      "title": "Request Latency P99",
      "query": "histogram_quantile(0.99, rate(vault_core_handle_request_duration_bucket[5m]))"
    },
    {
      "title": "Active Leases",
      "query": "vault_expire_num_leases"
    },
    {
      "title": "Raft Leader Last Contact",
      "query": "vault_raft_leader_lastContact"
    },
    {
      "title": "Memory Usage",
      "query": "vault_runtime_alloc_bytes / 1024 / 1024"
    }
  ]
}

Alerting Rules

# vault-alerts.yml
groups:
  - name: vault
    rules:
      - alert: VaultSealed
        expr: vault_core_unsealed == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Vault node is sealed"

      - alert: VaultHighLatency
        expr: histogram_quantile(0.99, rate(vault_core_handle_request_duration_bucket[5m])) > 0.5
        for: 5m
        labels:
          severity: warning

      - alert: VaultLeadershipLost
        expr: vault_raft_leader_lastContact > 200
        for: 30s
        labels:
          severity: critical

      - alert: VaultAuditFailure
        expr: increase(vault_audit_log_response_failure[5m]) > 0
        labels:
          severity: critical
          
      - alert: VaultTooManyLeases
        expr: vault_expire_num_leases > 200000
        for: 10m
        labels:
          severity: warning

2. Audit Logging

Configure Audit Devices

# File audit device
vault audit enable file file_path=/var/log/vault/audit.log

# Syslog audit device
vault audit enable syslog tag="vault" facility="LOCAL0"

# Socket audit device (cho log aggregator)
vault audit enable socket \
  address="logstash.company.com:9000" \
  socket_type="tcp"

# Luôn enable ít nhất 2 audit devices!
# Vault sẽ BLOCK tất cả requests nếu không có audit device nào hoạt động

# List audit devices
vault audit list -detailed

Audit Log Format

{
  "time": "2025-01-15T10:30:00.000Z",
  "type": "request",
  "auth": {
    "client_token": "hmac-sha256:abc123...",
    "accessor": "hmac-sha256:def456...",
    "display_name": "approle-cicd",
    "policies": ["cicd-deploy", "default"],
    "metadata": {
      "role_name": "cicd-pipeline"
    },
    "entity_id": "entity-uuid-here",
    "token_type": "service"
  },
  "request": {
    "id": "request-uuid",
    "operation": "read",
    "mount_type": "kv",
    "path": "secret/data/production/db",
    "remote_address": "10.0.1.50",
    "remote_port": 45678
  }
}

Log Analysis Queries

# Tìm failed requests
cat /var/log/vault/audit.log | \
  jq -r 'select(.type == "response" and .response.data == null) | 
  [.time, .request.path, .auth.display_name, .error] | @tsv'

# Tìm policy denials
cat /var/log/vault/audit.log | \
  jq -r 'select(.error != null and (.error | contains("permission denied"))) |
  [.time, .request.path, .request.operation, .auth.display_name] | @tsv'

# Top 10 paths accessed
cat /var/log/vault/audit.log | \
  jq -r 'select(.type == "request") | .request.path' | \
  sort | uniq -c | sort -rn | head -10

# Requests by identity
cat /var/log/vault/audit.log | \
  jq -r 'select(.type == "request") | .auth.display_name' | \
  sort | uniq -c | sort -rn

3. Backup and Restore

Raft Snapshot Backup

#!/bin/bash
# vault-backup.sh

VAULT_ADDR="https://vault.company.com:8200"
BACKUP_DIR="/backup/vault"
RETENTION_DAYS=30
DATE=$(date +%Y%m%d_%H%M%S)

# Tạo snapshot
vault operator raft snapshot save \
  "${BACKUP_DIR}/vault-snapshot-${DATE}.snap"

# Verify snapshot
vault operator raft snapshot inspect \
  "${BACKUP_DIR}/vault-snapshot-${DATE}.snap"

# Upload to S3
aws s3 cp "${BACKUP_DIR}/vault-snapshot-${DATE}.snap" \
  "s3://company-backup/vault/vault-snapshot-${DATE}.snap" \
  --sse aws:kms

# Cleanup old backups
find "${BACKUP_DIR}" -name "*.snap" -mtime +${RETENTION_DAYS} -delete

# Cron job — backup mỗi giờ
0 * * * * /opt/vault/scripts/vault-backup.sh >> /var/log/vault-backup.log 2>&1

Restore

# Restore snapshot — CHÚ Ý: overwrite toàn bộ data
vault operator raft snapshot restore \
  /backup/vault/vault-snapshot-20250115_100000.snap

# Force restore (khi cluster đã thay đổi)
vault operator raft snapshot restore -force \
  /backup/vault/vault-snapshot-20250115_100000.snap

4. Troubleshooting

Common Issues

Issue	Cause	Solution
Vault sealed after restart	No auto-unseal	KMS/HSM configuration sealed
503 Service Unavailable	Node is standby / DR secondary	Route requests to active node
Lease count increases continuously	Client does not revoke leases	Tidy leases, check TTL
Token store is too large	Orphan tokens are not revoke	Token tidy, reduce TTL
Raft leader flapping	Network instability	Increased heartbeat timeout
Audit device blocking	Disk full / log destination down	Free disk, fix log dest
High memory usage	Too many leases/tokens	Tidy and tune TTLs

Debug Commands

# Server status
vault status
vault status -format=json

# Raft cluster info
vault operator raft list-peers
vault operator raft autopilot state

# Leader info
vault operator step-down  # Force leader election

# Key status
vault operator key-status

# Lease management
vault lease list -prefix "database/"
vault lease revoke -prefix "database/creds/app-role/"

# Token tidy
vault write sys/tidy/tidy-token-store \
  safety_buffer="72h"

# Lease tidy
vault write sys/leases/tidy \
  tidy_type="irrevocable"

# Debug bundle (support)
vault debug -duration=2m -targets=metrics,server-status,replication-status

# Read internal counters
vault read sys/internal/counters/tokens
vault read sys/internal/counters/requests

Upgrade Strategy

# 1. Backup trước khi upgrade
vault operator raft snapshot save /backup/pre-upgrade.snap

# 2. Read release notes + upgrade guide

# 3. Upgrade standby nodes trước (rolling upgrade)
# Node 3 (standby)
systemctl stop vault
# Replace binary
cp vault-new /usr/bin/vault
systemctl start vault

# Verify node rejoins cluster
vault operator raft list-peers

# 4. Repeat cho Node 2

# 5. Step-down leader, upgrade cuối cùng
vault operator step-down
# Wait for new leader election
systemctl stop vault
cp vault-new /usr/bin/vault
systemctl start vault

# 6. Verify
vault status
vault operator raft list-peers
vault operator raft autopilot state

5. Summary

Monitoring — Prometheus metrics + Grafana dashboards + alerting is required
Audit — Always enable ≥ 2 audit devices, analyze logs regularly
Backup — Automatic Raft snapshots, offsite save, periodic test restore
Troubleshooting — Know debug commands, common issues and how to handle them
Upgrade — Rolling upgrade for HA cluster, always backup first

At this point, you have completed the entire HashiCorp Vault series from Basic to Advanced. From setup, secrets engines, auth methods, policies, agents, Kubernetes, CI/CD, to production HA, Enterprise, monitoring and operations.

Lesson 25: Monitoring, Audit Logging, Backup/Restore and Troubleshooting