Introduction
Leaked credentials are one of the leading causes of data breaches — hardcoded passwords in code, commit secrets to Git, unprotected environment variables.
This article presents a proper secrets management system and security layers for the container runtime.
1. Problem with regular Secrets
1.1 Common anti-patterns
# ❌ Hardcode trong code
DB_PASSWORD="super_secret_123"
# ❌ Commit vào Git
echo "DB_PASSWORD=xxx" >> .env
git add .env && git commit -m "add config"
# ❌ Kubernetes Secret mặc định (base64, không phải encryption)
kubectl get secret db-secret -o yaml
# data:
# password: c3VwZXJfc2VjcmV0 ← base64 decode = plaintext!
# ❌ Log secrets vô tình
logger.info("Connecting to DB with password: {}", password)
1.2 Secret sprawl
Trong một hệ thống lớn:
- DB passwords (mỗi service có DB riêng)
- API keys (external services)
- TLS certificates
- SSH keys
- OAuth2 client secrets
- Encryption keys
- Webhook tokens
Vấn đề:
- Ai biết secret nào?
- Secret có bị rotate không?
- Secret có bị shared không cần thiết?
- Audit trail: ai access secret lúc mấy giờ?
2. HashiCorp Vault
2.1 Vault Architecture
┌─────────────────────────────────────────────────────────────┐
│ HashiCorp Vault │
│ │
│ ┌─────────────────┐ ┌──────────────────────────────────┐ │
│ │ Auth Methods │ │ Secret Engines │ │
│ │ - Kubernetes │ │ ┌────────┐ ┌─────┐ ┌─────────┐ │ │
│ │ - JWT/OIDC │ │ │ KV v2 │ │ PKI │ │Database │ │ │
│ │ - AppRole │ │ │(static)│ │ │ │(dynamic)│ │ │
│ │ - AWS IAM │ │ └────────┘ └─────┘ └─────────┘ │ │
│ └─────────────────┘ └──────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ Audit Log (every access logged) │ │
│ └─────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────┘
2.2 Dynamic Secrets — Database
Vault creates temporary database credentials, automatically revokes after TTL:
# Cấu hình Vault Database Secret Engine
vault secrets enable database
vault write database/config/order-db \
plugin_name=postgresql-database-plugin \
connection_url="postgresql://{{username}}:{{password}}@postgres:5432/orders" \
username="vault_root" \
password="vault_root_password"
# Tạo role — Vault tạo user với template này
vault write database/roles/order-service \
db_name=order-db \
creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';
GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
default_ttl="1h" \
max_ttl="24h"
# Service yêu cầu credentials
vault read database/creds/order-service
# Key Value
# --- -----
# lease_id database/creds/order-service/abc123
# lease_duration 1h
# password A1B2-randomly-generated
# username v-k8s-order-x7j2k
# Sau 1 giờ, Vault tự động revoke user này!
Benefits of Dynamic Secrets:
- Each service instance has its own credentials
- Credentials expire automatically → breach impact limited
- Never reuse passwords
- Full audit trail
2.3 Vault Agent Sidecar (Kubernetes)
# Annotate pod để Vault Agent inject secrets
apiVersion: v1
kind: Pod
metadata:
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/role: "order-service"
# Inject DB credentials vào file
vault.hashicorp.com/agent-inject-secret-db: "database/creds/order-service"
vault.hashicorp.com/agent-inject-template-db: |
{{- with secret "database/creds/order-service" -}}
spring.datasource.username={{ .Data.username }}
spring.datasource.password={{ .Data.password }}
{{- end }}
spec:
serviceAccountName: order-service # Service account cho Kubernetes auth
containers:
- name: order-service
image: order-service:latest
env:
- name: SPRING_CONFIG_IMPORT
value: "file:/vault/secrets/db"
# Vault Agent sẽ mount /vault/secrets/ với credentials
3. Kubernetes Secrets — Best Practices
3.1 Sealed Secrets (Bitnami)
Allow encrypted commit secrets to Git:
# Cài Sealed Secrets Controller
helm install sealed-secrets \
sealed-secrets/sealed-secrets \
--namespace kube-system
# Tạo SealedSecret từ Kubernetes Secret
kubectl create secret generic db-secret \
--from-literal=password="super_secret" \
--dry-run=client -o yaml | \
kubeseal --format yaml > sealed-db-secret.yaml
# sealed-db-secret.yaml an toàn để commit vào Git
# Chỉ cluster có private key mới decrypt được
# sealed-db-secret.yaml (safe to commit)
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
name: db-secret
spec:
encryptedData:
password: AgBh3f2... ← RSA encrypted, safe to store in Git
3.2 External Secrets Operator
Sync secrets from external providers (Vault, AWS Secrets Manager, GCP Secret Manager):
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: order-db-secret
spec:
refreshInterval: 1h
secretStoreRef:
name: vault-backend
kind: ClusterSecretStore
target:
name: order-db-k8s-secret # Kubernetes Secret được tạo/update
creationPolicy: Owner
data:
- secretKey: db-password # Key trong Kubernetes Secret
remoteRef:
key: secret/order-service # Path trong Vault
property: db_password
# ClusterSecretStore — cấu hình kết nối Vault
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
name: vault-backend
spec:
provider:
vault:
server: "http://vault.platform:8200"
path: "secret"
version: "v2"
auth:
kubernetes:
mountPath: "kubernetes"
role: "secret-reader"
4. Container Image Security
4.1 Secure Dockerfile
# Stage 1: Build
FROM eclipse-temurin:21-jdk AS build
WORKDIR /app
COPY . .
RUN ./gradlew bootJar --no-daemon
# Stage 2: Run — minimal base image
FROM eclipse-temurin:21-jre-alpine # Alpine: nhỏ hơn, attack surface nhỏ hơn
# Tạo non-root user
RUN addgroup -S appgroup && adduser -S appuser -G appgroup
USER appuser # Không chạy root!
WORKDIR /app
# Copy chỉ artifact cần thiết
COPY --from=build --chown=appuser:appgroup /app/build/libs/*.jar app.jar
# Drop capabilities
RUN apk add --no-cache curl # Chỉ install những gì cần thiết
EXPOSE 8080
# Health check tích hợp
HEALTHCHECK --interval=30s --timeout=3s --retries=3 \
CMD curl -f http://localhost:8080/actuator/health/liveness || exit 1
ENTRYPOINT ["java", \
"-XX:+UseContainerSupport", \
"-XX:MaxRAMPercentage=75.0", \
"-jar", "app.jar"]
4.2 Image Scanning with Trivy
# Scan image trước khi push
trivy image \
--exit-code 1 \
--severity CRITICAL,HIGH \
--ignore-unfixed \
order-service:latest
# Output:
# Total: 3 (HIGH: 2, CRITICAL: 1)
# ┌────────────────────────┬──────┬────────────┬───────────────────────┐
# │ Library │ CVE │ Severity │ Fixed Version │
# ├────────────────────────┼──────┼────────────┼───────────────────────┤
# │ org.springframework... │ ... │ CRITICAL │ 6.1.12 (update now!) │
# CI Pipeline (GitHub Actions)
- name: Trivy vulnerability scan
uses: aquasecurity/trivy-action@master
with:
image-ref: ${{ env.IMAGE }}
format: 'table'
exit-code: '1' # Fail CI nếu có CRITICAL/HIGH
severity: 'CRITICAL,HIGH'
ignore-unfixed: true # Bỏ qua CVE chưa có fix
- name: Upload SARIF to GitHub Security
if: always()
uses: github/codeql-action/upload-sarif@v3
with:
sarif_file: trivy-results.sarif
4.3 Supply Chain Security — Cosign
Sign and verify container images:
# Ký image sau khi push (trong CI)
cosign sign \
--key cosign.key \
registry.example.com/order-service:sha-abc123
# Verify image trước khi deploy (trong admission webhook)
cosign verify \
--key cosign.pub \
registry.example.com/order-service:sha-abc123
# Kiểm tra SBOM (Software Bill of Materials)
cosign download sbom registry.example.com/order-service:sha-abc123
5. Pod Security Standards
5.1 Pod Security Admission
# Enforce restricted policy cho tất cả pods trong namespace
apiVersion: v1
kind: Namespace
metadata:
name: services-prod
labels:
# Enforce: reject non-compliant pods
pod-security.kubernetes.io/enforce: restricted
# Warn: log warning nhưng không reject
pod-security.kubernetes.io/warn: restricted
# Audit: log audit event
pod-security.kubernetes.io/audit: restricted
Policy Levels:
privileged: No restrictions (used for infra, not for apps)baseline: Prevent common privilege escalationsrestricted: Hardened, follow security best practices
5.2 Security Context in Pod/Container
spec:
securityContext:
runAsNonRoot: true # Không chạy root
runAsUser: 10001
runAsGroup: 10001
fsGroup: 10001
seccompProfile:
type: RuntimeDefault # Seccomp filter mặc định
containers:
- name: order-service
securityContext:
allowPrivilegeEscalation: false # Ngăn sudo/setuid
readOnlyRootFilesystem: true # Filesystem read-only
capabilities:
drop: ["ALL"] # Bỏ tất cả Linux capabilities
runAsNonRoot: true
5.3 Network Policies
# Chỉ cho phép order-service nhận traffic từ api-gateway
# Và gọi payment-service, inventory-service, postgresql
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: order-service-netpol
namespace: services-prod
spec:
podSelector:
matchLabels:
app: order-service
policyTypes:
- Ingress
- Egress
ingress:
# Chỉ nhận từ api-gateway
- from:
- podSelector:
matchLabels:
app: api-gateway
ports:
- port: 8080
egress:
# Gọi payment-service
- to:
- podSelector:
matchLabels:
app: payment-service
ports:
- port: 8080
# Gọi PostgreSQL
- to:
- podSelector:
matchLabels:
app: postgresql
ports:
- port: 5432
# DNS resolution
- to:
- namespaceSelector: {}
ports:
- port: 53
protocol: UDP
6. Runtime Security — Falco
Falco detect anomalous behavior in runtime:
# Falco rules
- rule: Shell in container
desc: Shell được spawn trong container — có thể là attack
condition: >
container and
proc.name in (bash, sh, zsh) and
not user.name in (trusted-users)
output: >
Shell spawned in container
(user=%user.name container=%container.id image=%container.image.repository
shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)
priority: WARNING
- rule: Sensitive file read
desc: Đọc file sensitive trong container
condition: >
open_read and
container and
fd.name in (/etc/shadow, /etc/passwd, /root/.ssh/id_rsa)
output: >
Sensitive file opened for reading
(user=%user.name file=%fd.name container=%container.id)
priority: CRITICAL
- rule: Outbound connection to unexpected IP
desc: Container kết nối đến IP ngoài whitelist
condition: >
outbound and
container and
not net.dns.name contains "svc.cluster.local" and
not fd.rip in (allowed-external-ips)
priority: WARNING
7. Security Checklist
Image Security:
□ Non-root user trong Dockerfile
□ Read-only root filesystem
□ Multi-stage build (no build tools in runtime image)
□ Regular base image updates / automated rebuild
□ Vulnerability scanning trong CI pipeline (Trivy)
□ Image signing (Cosign)
Runtime Security:
□ Pod Security Standards: restricted
□ No privileged containers
□ Drop ALL capabilities
□ Network Policies per service
□ Falco runtime monitoring
□ Resource limits đặt (prevent resource exhaustion attacks)
Secrets Management:
□ No secrets in code/Git
□ Vault hoặc External Secrets Operator
□ Dynamic secrets với short TTL
□ Secret rotation automated
□ Audit log cho secret access
Authentication:
□ mTLS service-to-service (Service Mesh)
□ JWT validation tại API Gateway
□ Short-lived access tokens (< 15 phút)
□ RBAC cho service accounts (least privilege)
Summary
| Concept | Purpose |
|---|---|
| Vault Dynamic Secrets | Temporary DB credentials, auto-expire |
| Vault Agent Sidecar | Inject secrets into pods without code changes |
| Sealed Secrets | Encrypted secrets securely in Git |
| External Secrets Operator | Sync from Vault/AWS Secret Manager to K8s Secrets |
| Trivy | Container image vulnerability scanning |
| Cosign | Sign and verify container images (supply chain) |
| Pod Security Standards | Baseline rules for all pods |
| Network Policies | Whitelist traffic flow between services |
| Falcon | Runtime anomaly detection |
Next article: Production Readiness Checklist & Deployment Roadmap