Chuyển đến nội dung chính
BlogCoursesHL7Cert PrepRoadmapGameDomainAboutMCPNEW
Topics
DatabaseDevOpsKiến trúc hệ thốngLinuxProgrammingSecurity

Lesson 8: PKI Secrets Engine - Certificate Authority

PKI deep dive, Root CA, Intermediate CA, Certificate roles, Issue/Sign, CRL, OCSP, Auto-rotation, ACME, PKI certificate counter (1.21), cert-manager integration, mTLS.

🔒 DevSecOps — Lesson 8 Lesson 8: PKI Secrets Engine - Certificate Authority

HashiCorp Vault from Basic to Advanced

Part 2: Secrets Engines - Managing Secrets

xdev.asia

Introduce

PKI (Public Key Infrastructure) Secrets Engine turns Vault into a complete Certificate Authority (CA), capable of automatically creating, signing, and managing X.509 certificates. Instead of purchasing certificates from a third party or managing a separate CA server, Vault provides PKI-as-a-Service with a simple API.

Why need PKI Secrets Engine?

  • Full automation: Issue certificates via API/CLI, CI/CD integration
  • Short-lived certificates: Reduces the risk of certificates being compromised
  • mTLS: Secure service-to-service communication
  • Centralized management: Manage all certificates from one place
  • ACME support: Compatible with Let's Encrypt protocol

1. PKI architecture — Root CA and Intermediate CA

1.1. 2-storey model (recommended)

┌─────────────────────────────────────────┐
│            Root CA (Offline)             │ ← Vault PKI mount: pki/
│         Validity: 10-20 years           │
│    Chỉ dùng để ký Intermediate CA       │
└────────────────┬────────────────────────┘
                 │ Signs
                 ▼
┌─────────────────────────────────────────┐
│        Intermediate CA (Online)          │ ← Vault PKI mount: pki_int/
│         Validity: 3-5 years             │
│    Dùng để issue certificates           │
└────────────────┬────────────────────────┘
                 │ Issues
                 ▼
┌─────────────────────────────────────────┐
│         Leaf Certificates               │
│    Validity: 30 days - 1 year           │
│    TLS, mTLS, code signing, etc.        │
└─────────────────────────────────────────┘

Why use 2 floors?

  • Root CA private key is better protected (rarely used)
  • If Intermediate CA is compromised, just revoke and create a new one
  • Does not affect the Root CA trust chain

2. Set up Root CA

2.1. Enable PKI cho Root CA

# Enable PKI engine cho Root CA
vault secrets enable -path=pki pki

# Cấu hình max lease TTL cho Root CA (20 năm)
vault secrets tune -max-lease-ttl=175200h pki

2.2. Generate Root Certificate

# Generate Root CA certificate
vault write -format=json pki/root/generate/internal \
  common_name="XDev Root Certificate Authority" \
  organization="XDev Asia" \
  ou="Infrastructure Security" \
  country="VN" \
  locality="Ho Chi Minh City" \
  issuer_name="root-2024" \
  ttl=175200h \
  key_type="rsa" \
  key_bits=4096 | tee /tmp/root-ca.json

# Extract Root CA certificate
jq -r '.data.certificate' /tmp/root-ca.json > /tmp/root-ca.pem

# Xem thông tin certificate
openssl x509 -in /tmp/root-ca.pem -text -noout | head -20

# Output:
# Certificate:
#     Data:
#         Version: 3 (0x2)
#         Serial Number: ...
#         Signature Algorithm: sha256WithRSAEncryption
#         Issuer: C = VN, L = Ho Chi Minh City, OU = Infrastructure Security, O = XDev Asia, CN = XDev Root Certificate Authority
#         Validity
#             Not Before: Jan 15 10:00:00 2024 GMT
#             Not After : Jan 15 10:00:00 2044 GMT

2.3. Configure URLs for Root CA

# Cấu hình CRL và Issuing URLs
vault write pki/config/urls \
  issuing_certificates="https://vault.xdev.asia/v1/pki/ca" \
  crl_distribution_points="https://vault.xdev.asia/v1/pki/crl" \
  ocsp_servers="https://vault.xdev.asia/v1/pki/ocsp"

3. Set up Intermediate CA

3.1. Enable PKI cho Intermediate CA

# Enable PKI engine cho Intermediate CA
vault secrets enable -path=pki_int pki

# Cấu hình max lease TTL (5 năm)
vault secrets tune -max-lease-ttl=43800h pki_int

3.2. Generate CSR cho Intermediate CA

# Generate CSR (Certificate Signing Request)
vault write -format=json pki_int/intermediate/generate/internal \
  common_name="XDev Intermediate Certificate Authority" \
  organization="XDev Asia" \
  ou="Infrastructure Security" \
  country="VN" \
  issuer_name="intermediate-2024" \
  key_type="rsa" \
  key_bits=4096 | tee /tmp/intermediate-csr.json

# Extract CSR
jq -r '.data.csr' /tmp/intermediate-csr.json > /tmp/intermediate.csr

3.3. Sign Intermediate CSR using Root CA

# Root CA ký CSR của Intermediate CA
vault write -format=json pki/root/sign-intermediate \
  csr=@/tmp/intermediate.csr \
  format=pem_bundle \
  ttl=43800h | tee /tmp/intermediate-cert.json

# Extract signed certificate
jq -r '.data.certificate' /tmp/intermediate-cert.json > /tmp/intermediate.pem

3.4. Import signed certificate into Intermediate CA

# Set signed certificate cho Intermediate CA
vault write pki_int/intermediate/set-signed \
  certificate=@/tmp/intermediate.pem

3.5. Configure URLs for Intermediate CA

vault write pki_int/config/urls \
  issuing_certificates="https://vault.xdev.asia/v1/pki_int/ca" \
  crl_distribution_points="https://vault.xdev.asia/v1/pki_int/crl" \
  ocsp_servers="https://vault.xdev.asia/v1/pki_int/ocsp"

4. Certificate Roles

Roles defines the template for issued certificates. Each role specifies domain patterns, TTL, key types, and allowed extensions.

4.1. Role cho Web Server TLS

vault write pki_int/roles/web-server \
  allowed_domains="xdev.asia,internal.xdev.asia" \
  allow_subdomains=true \
  allow_bare_domains=false \
  allow_wildcard_certificates=true \
  max_ttl="2160h" \
  ttl="720h" \
  key_type="rsa" \
  key_bits=2048 \
  key_usage="DigitalSignature,KeyEncipherment" \
  ext_key_usage="ServerAuth" \
  organization="XDev Asia" \
  country="VN" \
  require_cn=true \
  server_flag=true \
  client_flag=false

4.2. Role cho mTLS Client

vault write pki_int/roles/mtls-client \
  allowed_domains="services.internal" \
  allow_subdomains=true \
  max_ttl="720h" \
  ttl="168h" \
  key_type="ec" \
  key_bits=256 \
  key_usage="DigitalSignature" \
  ext_key_usage="ClientAuth" \
  require_cn=true \
  server_flag=false \
  client_flag=true \
  no_store=true

4.3. Roles for mTLS (both Server and Client)

vault write pki_int/roles/mtls-service \
  allowed_domains="services.internal" \
  allow_subdomains=true \
  allow_ip_sans=true \
  allowed_other_sans="" \
  max_ttl="720h" \
  ttl="168h" \
  key_type="ec" \
  key_bits=256 \
  key_usage="DigitalSignature,KeyEncipherment" \
  ext_key_usage="ServerAuth,ClientAuth" \
  require_cn=true \
  server_flag=true \
  client_flag=true

4.4. Role cho Internal Services (short-lived)

vault write pki_int/roles/internal-service \
  allowed_domains="svc.cluster.local,internal.xdev.asia" \
  allow_subdomains=true \
  allow_ip_sans=true \
  max_ttl="72h" \
  ttl="24h" \
  key_type="ec" \
  key_bits=256 \
  generate_lease=true \
  no_store=false

5. Issue Certificates

5.1. Issue certificate cho web server

# Issue certificate
vault write -format=json pki_int/issue/web-server \
  common_name="api.xdev.asia" \
  alt_names="api-v2.xdev.asia,api-internal.xdev.asia" \
  ip_sans="10.0.1.100" \
  ttl="720h" | tee /tmp/api-cert.json

# Extract certificate components
jq -r '.data.certificate' /tmp/api-cert.json > /tmp/api.crt
jq -r '.data.private_key' /tmp/api-cert.json > /tmp/api.key
jq -r '.data.ca_chain[]' /tmp/api-cert.json > /tmp/ca-chain.pem
jq -r '.data.issuing_ca' /tmp/api-cert.json > /tmp/issuing-ca.pem

# Xem thông tin certificate
openssl x509 -in /tmp/api.crt -text -noout

# Verify certificate chain
openssl verify -CAfile /tmp/ca-chain.pem /tmp/api.crt
# /tmp/api.crt: OK

5.2. Issue wildcard certificate

vault write -format=json pki_int/issue/web-server \
  common_name="*.xdev.asia" \
  ttl="720h" | tee /tmp/wildcard-cert.json

5.3. Issue certificate cho mTLS service

vault write -format=json pki_int/issue/mtls-service \
  common_name="payment-service.services.internal" \
  ip_sans="10.0.2.50" \
  ttl="168h" | tee /tmp/payment-svc-cert.json

# Extract cho service sử dụng
jq -r '.data.certificate' /tmp/payment-svc-cert.json > /etc/tls/tls.crt
jq -r '.data.private_key' /tmp/payment-svc-cert.json > /etc/tls/tls.key
jq -r '.data.ca_chain[]' /tmp/payment-svc-cert.json > /etc/tls/ca.crt

6. Sign CSR — Sign Certificate Request externally

When the application generates its own private key and sends the CSR for signing:

# Application tạo private key và CSR
openssl req -new -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 \
  -keyout /tmp/app.key -out /tmp/app.csr -nodes \
  -subj "/CN=myapp.services.internal/O=XDev Asia"

# Vault ký CSR
vault write -format=json pki_int/sign/internal-service \
  csr=@/tmp/app.csr \
  common_name="myapp.services.internal" \
  ttl="24h" | tee /tmp/signed-cert.json

# Extract signed certificate
jq -r '.data.certificate' /tmp/signed-cert.json > /tmp/app.crt

7. CRL (Certificate Revocation List)

7.1. CRL configuration

# Cấu hình CRL settings
vault write pki_int/config/crl \
  expiry="72h" \
  disable=false \
  auto_rebuild=true \
  auto_rebuild_grace_period="12h" \
  enable_delta=true \
  delta_rebuild_interval="15m"

7.2. Revoke Certificate

# Revoke bằng serial number
vault write pki_int/revoke \
  serial_number="39:dd:2e:90:b7:23:1f:8d:d3:7d:31:c5:1b:da:84:d0:5b:65:31:58"

# Revoke bằng certificate PEM
vault write pki_int/revoke \
  certificate=@/tmp/api.crt

# Xem CRL hiện tại
curl -s "$VAULT_ADDR/v1/pki_int/crl" | openssl crl -inform DER -text -noout

7.3. Tidy — Cleanup expired certificates

# Tidy certificates database
vault write pki_int/tidy \
  tidy_cert_store=true \
  tidy_revoked_certs=true \
  tidy_revoked_cert_issuer_associations=true \
  tidy_expired_issuers=true \
  safety_buffer="72h" \
  issuer_safety_buffer="720h"

# Kiểm tra trạng thái tidy
vault read pki_int/tidy-status

8. OCSP (Online Certificate Status Protocol)

OCSP allows clients to check certificate revocation status in real-time.

8.1. Enable OCSP

# OCSP đã được enable mặc định khi cấu hình ocsp_servers URL
# Kiểm tra OCSP response
openssl ocsp \
  -issuer /tmp/issuing-ca.pem \
  -cert /tmp/api.crt \
  -url "$VAULT_ADDR/v1/pki_int/ocsp" \
  -resp_text

# Output:
# Response Status: successful (0x0)
# ...
# Cert Status: good
# This Update: Jan 15 12:00:00 2024 GMT
# Next Update: Jan 16 12:00:00 2024 GMT

9. ACME Protocol Support

From Vault 1.14+, the PKI engine supports the ACME protocol — the same protocol that Let's Encrypt uses. This allows integration with ACME clients such as certbot, cert-manager, etc.

9.1. Enable ACME

# Cấu hình cluster path (bắt buộc cho ACME)
vault write pki_int/config/cluster \
  path="https://vault.xdev.asia/v1/pki_int" \
  aia_path="https://vault.xdev.asia/v1/pki_int"

# Enable ACME
vault write pki_int/config/acme \
  enabled=true \
  allowed_roles="web-server,internal-service" \
  allow_role_ext_key_usage=true \
  default_directory_policy="role:web-server"

9.2. Use certbot with ACME Vault

# Sử dụng certbot để request certificate từ Vault
certbot certonly \
  --server "https://vault.xdev.asia/v1/pki_int/acme/directory" \
  --standalone \
  --non-interactive \
  --agree-tos \
  --email [email protected] \
  -d "api.xdev.asia"

10. PKI Certificate Counter (Vault 1.21+)

Vault 1.21 adds the feature to count issued certificates, helping with monitoring and compliance.

# Đọc certificate count
vault read pki_int/certificates/count

# Output:
# Key      Value
# ---      -----
# count    1547

# List certificates (serial numbers)
vault list pki_int/certs

# Đọc thông tin certificate cụ thể
vault read pki_int/cert/<serial-number>

11. Integrate cert-manager (Kubernetes)

11.1. Install cert-manager Vault Issuer

# vault-issuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: vault-issuer
spec:
  vault:
    server: https://vault.xdev.asia
    path: pki_int/sign/internal-service
    auth:
      kubernetes:
        role: cert-manager
        mountPath: /v1/auth/kubernetes
        serviceAccountRef:
          name: cert-manager-vault

11.2. Request certificate qua cert-manager

# certificate.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: api-tls
  namespace: default
spec:
  secretName: api-tls-secret
  duration: 720h
  renewBefore: 168h
  issuerRef:
    name: vault-issuer
    kind: ClusterIssuer
  commonName: api.services.internal
  dnsNames:
    - api.services.internal
    - api.default.svc.cluster.local
  ipAddresses:
    - 10.0.2.100
  privateKey:
    algorithm: ECDSA
    size: 256

11.3. Use in Pod

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
spec:
  template:
    spec:
      containers:
        - name: api
          image: myapp/api:latest
          volumeMounts:
            - name: tls
              mountPath: /etc/tls
              readOnly: true
          env:
            - name: TLS_CERT
              value: /etc/tls/tls.crt
            - name: TLS_KEY
              value: /etc/tls/tls.key
            - name: CA_CERT
              value: /etc/tls/ca.crt
      volumes:
        - name: tls
          secret:
            secretName: api-tls-secret

12. Thiết lập mTLS hoàn chỉnh

12.1. Issue Server Certificate

# Server certificate
vault write -format=json pki_int/issue/mtls-service \
  common_name="server.services.internal" \
  ttl="168h" > /tmp/server-cert.json

jq -r '.data.certificate' /tmp/server-cert.json > /etc/tls/server.crt
jq -r '.data.private_key' /tmp/server-cert.json > /etc/tls/server.key
jq -r '.data.ca_chain[]' /tmp/server-cert.json > /etc/tls/ca-bundle.crt

12.2. Issue Client Certificate

# Client certificate
vault write -format=json pki_int/issue/mtls-client \
  common_name="client-app.services.internal" \
  ttl="168h" > /tmp/client-cert.json

jq -r '.data.certificate' /tmp/client-cert.json > /etc/tls/client.crt
jq -r '.data.private_key' /tmp/client-cert.json > /etc/tls/client.key

12.3. Test mTLS with curl

# Server cần verify client certificate
# Client gửi request với certificate
curl --cacert /etc/tls/ca-bundle.crt \
  --cert /etc/tls/client.crt \
  --key /etc/tls/client.key \
  https://server.services.internal:8443/api/health

12.4. Nginx mTLS Configuration

server {
    listen 8443 ssl;
    server_name server.services.internal;

    # Server certificate
    ssl_certificate     /etc/tls/server.crt;
    ssl_certificate_key /etc/tls/server.key;

    # mTLS — require client certificate
    ssl_client_certificate /etc/tls/ca-bundle.crt;
    ssl_verify_client on;
    ssl_verify_depth 2;

    # TLS settings
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers on;

    location / {
        proxy_pass http://backend:8080;
        proxy_set_header X-Client-CN $ssl_client_s_dn_cn;
    }
}

13. Auto-Rotation Script

#!/bin/bash
# cert-renew.sh — Tự động renew certificate trước khi hết hạn

set -euo pipefail

CERT_PATH="/etc/tls/server.crt"
KEY_PATH="/etc/tls/server.key"
CA_PATH="/etc/tls/ca-bundle.crt"
ROLE="web-server"
CN="api.xdev.asia"
RENEW_BEFORE_DAYS=7

# Kiểm tra certificate còn bao nhiêu ngày
if [ -f "$CERT_PATH" ]; then
  EXPIRY=$(openssl x509 -in "$CERT_PATH" -enddate -noout | cut -d= -f2)
  EXPIRY_EPOCH=$(date -d "$EXPIRY" +%s 2>/dev/null || date -j -f "%b %d %T %Y %Z" "$EXPIRY" +%s)
  NOW_EPOCH=$(date +%s)
  DAYS_LEFT=$(( (EXPIRY_EPOCH - NOW_EPOCH) / 86400 ))
  
  echo "Certificate expires in $DAYS_LEFT days"
  
  if [ "$DAYS_LEFT" -gt "$RENEW_BEFORE_DAYS" ]; then
    echo "Certificate still valid. No renewal needed."
    exit 0
  fi
fi

echo "Renewing certificate..."

# Issue new certificate
RESULT=$(vault write -format=json "pki_int/issue/${ROLE}" \
  common_name="$CN" \
  ttl="720h")

echo "$RESULT" | jq -r '.data.certificate' > "$CERT_PATH"
echo "$RESULT" | jq -r '.data.private_key' > "$KEY_PATH"
echo "$RESULT" | jq -r '.data.ca_chain[]' > "$CA_PATH"

chmod 644 "$CERT_PATH" "$CA_PATH"
chmod 600 "$KEY_PATH"

echo "Certificate renewed successfully!"

# Reload service (nginx example)
nginx -s reload 2>/dev/null || true

14. Policies cho PKI

# Policy cho application: chỉ issue certificates
path "pki_int/issue/web-server" {
  capabilities = ["create", "update"]
}

path "pki_int/sign/web-server" {
  capabilities = ["create", "update"]
}

# Policy cho cert-manager
path "pki_int/sign/internal-service" {
  capabilities = ["create", "update"]
}

# Policy cho PKI admin
path "pki_int/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

path "pki/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}

# Read CA certificates (public)
path "pki/ca/pem" {
  capabilities = ["read"]
}

path "pki_int/ca/pem" {
  capabilities = ["read"]
}

15. Best Practices

15.1. Certificate Lifecycle

  • ✅ Root CA TTL: 10-20 years, offline storage
  • ✅ Intermediate CA TTL: 3-5 years
  • ✅ Leaf certificates TTL: 30 days - 1 year (the shorter the better)
  • ✅ Renew certificates when 1/3 of their lifespan remains
  • ✅ Use ECDSA P-256 for leaf certs (faster than RSA)

15.2. Security

  • ✅ Do not export Root CA private key
  • ✅ Separate PKI mounts for Root and Intermediate
  • ✅ Enable CRL and OCSP
  • ✅ Regular tidy operations
  • ✅ Monitor certificate count and expiration
  • ✅ Restrict role permissions theo team/service
  • ✅ Use no_store=true for high-volume releasing

Summary

In this lesson, you set up a complete PKI infrastructure with Vault:

  1. Root CA + Intermediate CA — standard 2-tier architecture
  2. Certificate Roles — templates cho web server, mTLS, internal services
  3. Issue and Sign certificates — automatically via CLI/API
  4. CRL and OCSP — certificate revocation
  5. ACME support — compatible with Let's Encrypt protocol
  6. cert-manager integration — PKI automation in Kubernetes
  7. mTLS — service-to-service security
  8. Auto-rotation — automatically renew certificates

The next article will explore Transit Secrets Engine — Encryption as a Service, where Vault helps you encrypt data without managing encryption keys.