xDev Asia

Prometheus + Grafana monitoring, audit device configuration, audit log analysis, Raft backup/restore, common issues troubleshooting, upgrade strategies.

HashiCorp Vault の基本から上級まで

パート 7: 本番、エンタープライズ、運用

xdev.asia

1。 Prometheus + Grafana

によるモニタリング

Enable Telemetry

# vault.hcl
telemetry {
  prometheus_retention_time = "30s"
  disable_hostname          = true
  
  # StatsD (optional nếu dùng StatsD exporter)
  # statsd_address = "statsd:8125"
}

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_disable = false
  # ...

  telemetry {
    unauthenticated_metrics_access = true  # Hoặc false + dùng token
  }
}

Prometheus Scrape Config

# prometheus.yml
scrape_configs:
  - job_name: 'vault'
    metrics_path: '/v1/sys/metrics'
    params:
      format: ['prometheus']
    scheme: https
    tls_config:
      ca_file: /etc/prometheus/vault-ca.pem
    # Nếu unauthenticated_metrics_access = false:
    # bearer_token_file: /etc/prometheus/vault-token
    static_configs:
      - targets:
          - 'vault-node-1:8200'
          - 'vault-node-2:8200'
          - 'vault-node-3:8200'
        labels:
          cluster: 'production'

監視すべき主要なメトリクス

Metric	Mô tả	Alert threshold
`vault.core.handle_request.count`	総リクエスト	傾向分析
`vault.core.handle_request.duration`	Request latency	P99 > 500ms
`vault.token.count`	Active tokens	Sudden spike
`vault.expire.num_leases`	Active leases	> 256000
`vault.runtime.alloc_bytes`	Memory allocation	> 80% RAM
`vault.runtime.gc_pause_ns`	GC pause	> 2s
`vault.raft.leader.lastContact`	Time since leader contact	> 200ms
`vault.raft.commitTime`	Raft commit time	> 25ms
`vault.seal.unseal`	Unseal events	Unexpected count
`vault.audit.log_response_failure`	Audit log failures	Any > 0

Grafana Dashboard

{
  "dashboard_id": "vault-overview",
  "panels": [
    {
      "title": "Request Rate",
      "query": "rate(vault_core_handle_request_count[5m])"
    },
    {
      "title": "Request Latency P99",
      "query": "histogram_quantile(0.99, rate(vault_core_handle_request_duration_bucket[5m]))"
    },
    {
      "title": "Active Leases",
      "query": "vault_expire_num_leases"
    },
    {
      "title": "Raft Leader Last Contact",
      "query": "vault_raft_leader_lastContact"
    },
    {
      "title": "Memory Usage",
      "query": "vault_runtime_alloc_bytes / 1024 / 1024"
    }
  ]
}

Alerting Rules

# vault-alerts.yml
groups:
  - name: vault
    rules:
      - alert: VaultSealed
        expr: vault_core_unsealed == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Vault node is sealed"

      - alert: VaultHighLatency
        expr: histogram_quantile(0.99, rate(vault_core_handle_request_duration_bucket[5m])) > 0.5
        for: 5m
        labels:
          severity: warning

      - alert: VaultLeadershipLost
        expr: vault_raft_leader_lastContact > 200
        for: 30s
        labels:
          severity: critical

      - alert: VaultAuditFailure
        expr: increase(vault_audit_log_response_failure[5m]) > 0
        labels:
          severity: critical
          
      - alert: VaultTooManyLeases
        expr: vault_expire_num_leases > 200000
        for: 10m
        labels:
          severity: warning

2. Audit Logging

監査デバイスの構成

# File audit device
vault audit enable file file_path=/var/log/vault/audit.log

# Syslog audit device
vault audit enable syslog tag="vault" facility="LOCAL0"

# Socket audit device (cho log aggregator)
vault audit enable socket \
  address="logstash.company.com:9000" \
  socket_type="tcp"

# Luôn enable ít nhất 2 audit devices!
# Vault sẽ BLOCK tất cả requests nếu không có audit device nào hoạt động

# List audit devices
vault audit list -detailed

Audit Log Format

{
  "time": "2025-01-15T10:30:00.000Z",
  "type": "request",
  "auth": {
    "client_token": "hmac-sha256:abc123...",
    "accessor": "hmac-sha256:def456...",
    "display_name": "approle-cicd",
    "policies": ["cicd-deploy", "default"],
    "metadata": {
      "role_name": "cicd-pipeline"
    },
    "entity_id": "entity-uuid-here",
    "token_type": "service"
  },
  "request": {
    "id": "request-uuid",
    "operation": "read",
    "mount_type": "kv",
    "path": "secret/data/production/db",
    "remote_address": "10.0.1.50",
    "remote_port": 45678
  }
}

Log Analysis Queries

# Tìm failed requests
cat /var/log/vault/audit.log | \
  jq -r 'select(.type == "response" and .response.data == null) | 
  [.time, .request.path, .auth.display_name, .error] | @tsv'

# Tìm policy denials
cat /var/log/vault/audit.log | \
  jq -r 'select(.error != null and (.error | contains("permission denied"))) |
  [.time, .request.path, .request.operation, .auth.display_name] | @tsv'

# Top 10 paths accessed
cat /var/log/vault/audit.log | \
  jq -r 'select(.type == "request") | .request.path' | \
  sort | uniq -c | sort -rn | head -10

# Requests by identity
cat /var/log/vault/audit.log | \
  jq -r 'select(.type == "request") | .auth.display_name' | \
  sort | uniq -c | sort -rn

3。バックアップと復元

Raft Snapshot Backup

#!/bin/bash
# vault-backup.sh

VAULT_ADDR="https://vault.company.com:8200"
BACKUP_DIR="/backup/vault"
RETENTION_DAYS=30
DATE=$(date +%Y%m%d_%H%M%S)

# Tạo snapshot
vault operator raft snapshot save \
  "${BACKUP_DIR}/vault-snapshot-${DATE}.snap"

# Verify snapshot
vault operator raft snapshot inspect \
  "${BACKUP_DIR}/vault-snapshot-${DATE}.snap"

# Upload to S3
aws s3 cp "${BACKUP_DIR}/vault-snapshot-${DATE}.snap" \
  "s3://company-backup/vault/vault-snapshot-${DATE}.snap" \
  --sse aws:kms

# Cleanup old backups
find "${BACKUP_DIR}" -name "*.snap" -mtime +${RETENTION_DAYS} -delete

# Cron job — backup mỗi giờ
0 * * * * /opt/vault/scripts/vault-backup.sh >> /var/log/vault-backup.log 2>&1

Restore

# Restore snapshot — CHÚ Ý: overwrite toàn bộ data
vault operator raft snapshot restore \
  /backup/vault/vault-snapshot-20250115_100000.snap

# Force restore (khi cluster đã thay đổi)
vault operator raft snapshot restore -force \
  /backup/vault/vault-snapshot-20250115_100000.snap

4. Troubleshooting

Common Issues

Issue	Nguyên nhân	Giải pháp
再起動後にコンテナーがシールされます	自動シール解除なし	KMS/HSM 構成がシールされます
503 サービスが利用できません	ノードはスタンバイ/DR セカンダリ	リクエストをアクティブノードにルーティングします
リース数が継続的に増加します	クライアントがリースを取り消しません	リースを整理してください、TTLを確認してください
トークンストアが大きすぎます	孤立したトークンは取り消されません	トークンを整理し、TTL を削減してください
ラフトリーダーの羽ばたき	ネットワークの不安定	ハートビートタイムアウトの増加
Audit device blocking	Disk full / log destination down	Free disk, fix log dest
High memory usage	Too many leases/tokens	Tidy and tune TTLs

Debug Commands

# Server status
vault status
vault status -format=json

# Raft cluster info
vault operator raft list-peers
vault operator raft autopilot state

# Leader info
vault operator step-down  # Force leader election

# Key status
vault operator key-status

# Lease management
vault lease list -prefix "database/"
vault lease revoke -prefix "database/creds/app-role/"

# Token tidy
vault write sys/tidy/tidy-token-store \
  safety_buffer="72h"

# Lease tidy
vault write sys/leases/tidy \
  tidy_type="irrevocable"

# Debug bundle (support)
vault debug -duration=2m -targets=metrics,server-status,replication-status

# Read internal counters
vault read sys/internal/counters/tokens
vault read sys/internal/counters/requests

Upgrade Strategy

# 1. Backup trước khi upgrade
vault operator raft snapshot save /backup/pre-upgrade.snap

# 2. Read release notes + upgrade guide

# 3. Upgrade standby nodes trước (rolling upgrade)
# Node 3 (standby)
systemctl stop vault
# Replace binary
cp vault-new /usr/bin/vault
systemctl start vault

# Verify node rejoins cluster
vault operator raft list-peers

# 4. Repeat cho Node 2

# 5. Step-down leader, upgrade cuối cùng
vault operator step-down
# Wait for new leader election
systemctl stop vault
cp vault-new /usr/bin/vault
systemctl start vault

# 6. Verify
vault status
vault operator raft list-peers
vault operator raft autopilot state

5。概要

Monitoring — Prometheus メトリクス + Grafana ダッシュボード + アラートが必要
Audit — 常に 2 つ以上の監査デバイスを有効にし、ログを定期的に分析します
Backup — 自動 Raft スナップショット、オフサイト保存、定期テスト復元
トラブルシューティング — デバッグコマンド、一般的な問題、およびそれらの対処方法を理解する
Upgrade — HA クラスターのローリングアップグレード、常に最初にバックアップ

この時点で、HashiCorp Vault シリーズ全体を Basic から Advanced まで完了しました。セットアップ、シークレットエンジン、認証方法、ポリシー、エージェント、Kubernetes、CI/CD から、運用 HA、エンタープライズ、監視、運用まで。

レッスン 25: 監視、監査ログ、バックアップ/復元、およびトラブルシューティング