Lesson 20: Vault on Kubernetes - Helm, Operator and CSI

HashiCorp Vault from Basic to Advanced

Part 5: Vault Agent, Proxy and Kubernetes Integration

xdev.asia

1. Deploy Vault on Kubernetes using Helm

Deployment modes

Mode	Description	Use case
Dev	Single pod, in-memory storage	Development, testing
Standalone	Single pod, persistent storage	Small deployments
HA	Multi-pod, Integrated Storage (Raft)	Production
External	Vault runs outside K8s, only deploy Agent Injector	External Vault server

Helm Install (HA Mode)

# Add Helm repo
helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

# Install Vault HA với Integrated Storage
helm install vault hashicorp/vault \
  --namespace vault \
  --create-namespace \
  --set server.ha.enabled=true \
  --set server.ha.replicas=3 \
  --set server.ha.raft.enabled=true \
  --set server.dataStorage.size=10Gi \
  --set server.resources.requests.memory=256Mi \
  --set server.resources.requests.cpu=250m \
  --set server.resources.limits.memory=512Mi \
  --set server.resources.limits.cpu=500m \
  --set ui.enabled=true \
  --set ui.serviceType=ClusterIP

Custom Values

# values-production.yaml
server:
  ha:
    enabled: true
    replicas: 3
    raft:
      enabled: true
      config: |
        ui = true

        listener "tcp" {
          tls_disable = 0
          address = "[::]:8200"
          cluster_address = "[::]:8201"
          tls_cert_file = "/vault/userconfig/vault-tls/tls.crt"
          tls_key_file  = "/vault/userconfig/vault-tls/tls.key"
        }

        storage "raft" {
          path = "/vault/data"
          retry_join {
            leader_api_addr = "https://vault-0.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault-tls/ca.crt"
          }
          retry_join {
            leader_api_addr = "https://vault-1.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault-tls/ca.crt"
          }
          retry_join {
            leader_api_addr = "https://vault-2.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault-tls/ca.crt"
          }
        }

        service_registration "kubernetes" {}

  dataStorage:
    size: 20Gi
    storageClass: gp3

  extraVolumes:
    - type: secret
      name: vault-tls

  resources:
    requests:
      memory: 512Mi
      cpu: 500m
    limits:
      memory: 1Gi
      cpu: 1000m

  ingress:
    enabled: true
    hosts:
      - host: vault.company.com

ui:
  enabled: true

injector:
  enabled: true
  replicas: 2

# Install với custom values
helm install vault hashicorp/vault \
  --namespace vault \
  --create-namespace \
  -f values-production.yaml

Init and Unseal

# Init trên vault-0
kubectl exec -n vault vault-0 -- vault operator init \
  -key-shares=5 \
  -key-threshold=3 \
  -format=json > vault-init.json

# Unseal vault-0
kubectl exec -n vault vault-0 -- vault operator unseal <key-1>
kubectl exec -n vault vault-0 -- vault operator unseal <key-2>
kubectl exec -n vault vault-0 -- vault operator unseal <key-3>

# Join vault-1 và vault-2 vào Raft cluster
kubectl exec -n vault vault-1 -- vault operator raft join \
  https://vault-0.vault-internal:8200

kubectl exec -n vault vault-2 -- vault operator raft join \
  https://vault-0.vault-internal:8200

# Unseal vault-1 và vault-2
kubectl exec -n vault vault-1 -- vault operator unseal <key-1>
# ... (repeat for each node)

2. Vault Secrets Operator (VSO)

VSO is the official Kubernetes operator, automatically syncing secrets from Vault to Kubernetes Secrets. This is the recommended method for Kubernetes-native secret management.

Install VSO

helm install vault-secrets-operator hashicorp/vault-secrets-operator \
  --namespace vault-secrets-operator-system \
  --create-namespace

Custom Resources

# 1. VaultConnection — kết nối đến Vault server
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultConnection
metadata:
  name: vault-connection
  namespace: app
spec:
  address: https://vault.company.com:8200
  skipTLSVerify: false
  caCertSecretRef: vault-ca-cert

---
# 2. VaultAuth — xác thực với Vault
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: vault-auth
  namespace: app
spec:
  vaultConnectionRef: vault-connection
  method: kubernetes
  mount: kubernetes
  kubernetes:
    role: webapp
    serviceAccount: webapp-sa

---
# 3. VaultStaticSecret — sync KV secret
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: app-db-secret
  namespace: app
spec:
  vaultAuthRef: vault-auth
  mount: secret
  type: kv-v2
  path: production/db
  refreshAfter: 60s
  destination:
    name: app-db-credentials
    create: true
    labels:
      app: webapp
    transformation:
      excludeRaw: true
      templates:
        connection-string:
          text: "postgresql://{{ .Secrets.username }}:{{ .Secrets.password }}@{{ .Secrets.host }}:5432/{{ .Secrets.database }}"

---
# 4. VaultDynamicSecret — dynamic database credentials
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultDynamicSecret
metadata:
  name: app-dynamic-db
  namespace: app
spec:
  vaultAuthRef: vault-auth
  mount: database
  path: creds/app-role
  renewalPercent: 67
  destination:
    name: app-dynamic-db-credentials
    create: true

---
# 5. VaultPKISecret — auto-issue certificates
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultPKISecret
metadata:
  name: app-tls-cert
  namespace: app
spec:
  vaultAuthRef: vault-auth
  mount: pki
  role: app-cert
  commonName: app.company.com
  altNames:
    - app.internal.company.com
  ttl: 24h
  expiryOffset: 1h
  destination:
    name: app-tls
    create: true

Use in Pod

apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
  namespace: app
spec:
  template:
    spec:
      serviceAccountName: webapp-sa
      containers:
        - name: webapp
          image: myapp:latest
          envFrom:
            - secretRef:
                name: app-db-credentials
          volumeMounts:
            - name: tls
              mountPath: /etc/tls
              readOnly: true
      volumes:
        - name: tls
          secret:
            secretName: app-tls

3. Vault CSI Provider

Vault CSI Provider uses Secrets Store CSI Driver to mount secrets directly into Pod volumes, without creating Kubernetes Secrets.

# SecretProviderClass
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: vault-db-creds
  namespace: app
spec:
  provider: vault
  parameters:
    vaultAddress: "https://vault.company.com:8200"
    roleName: "webapp"
    objects: |
      - objectName: "db-username"
        secretPath: "secret/data/production/db"
        secretKey: "username"
      - objectName: "db-password"
        secretPath: "secret/data/production/db"
        secretKey: "password"
  # Optionally sync to K8s Secret
  secretObjects:
    - secretName: app-db-synced
      type: Opaque
      data:
        - objectName: db-username
          key: username
        - objectName: db-password
          key: password

---
# Pod sử dụng CSI volume
apiVersion: v1
kind: Pod
metadata:
  name: webapp
spec:
  serviceAccountName: webapp-sa
  containers:
    - name: webapp
      image: myapp:latest
      volumeMounts:
        - name: secrets
          mountPath: /mnt/secrets
          readOnly: true
  volumes:
    - name: secrets
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: vault-db-creds

4. Vault Agent Injector

Agent Injector uses Kubernetes Mutating Webhook to automatically inject Vault Agent sidecar into Pods using annotations.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "webapp"
        vault.hashicorp.com/agent-inject-secret-db.txt: "secret/data/production/db"
        vault.hashicorp.com/agent-inject-template-db.txt: |
          {{- with secret "secret/data/production/db" -}}
          DB_HOST={{ .Data.data.host }}
          DB_USER={{ .Data.data.username }}
          DB_PASS={{ .Data.data.password }}
          {{- end }}
    spec:
      serviceAccountName: webapp-sa
      containers:
        - name: webapp
          image: myapp:latest
          # Secrets available at /vault/secrets/db.txt

5. Compare VSO vs CSI vs Agent Injector

Criteria	VSO	CSI Provider	Agent Injector
Mechanism	Operator + CRDs	CSI Driver	Mutating Webhook
K8s Secrets	Create K8s Secrets	Optional sync	No
Dynamic secrets	✅ (VaultDynamicSecret)	Restrictions	✅ (templates)
PKI certificates	✅ (VaultPKISecret)	Restrictions	✅ (templates)
Auto-rotation	✅ (refreshAfter)	Individual configuration	✅ (sidecar)
Sidecar required	No	No	Yes
Resource overhead	Low (general operator)	Low	High (1 sidecar/pod)
Recommendation	✅ Recommended	OK cho simple	Legacy

6. Best Practices cho Kubernetes

• Priority VSO for new deployments — native K8s experience, less overhead

• Use dedicated ServiceAccount for each workload

• Enable Auto-unseal with Cloud KMS for production

• Configure Pod Security for Vault pods (non-root, read-only filesystem)

• Use NetworkPolicy to restrict access to Vault

• Mount secrets into emptyDir (memory-backed) instead of persistent volume

7. Summary

• Helm chart — deploy Vault HA with Integrated Storage on K8s

• VSO — recommended, CRD-based, automatically sync secrets to K8s Secrets

• CSI Provider — mount secrets directly to Pod volumes

• Agent Injector — legacy, sidecar-based, annotation-driven

The next section will focus on integrating Vault with real applications — Spring Boot, Node.js, Terraform, Ansible, and CI/CD pipelines.