Comprehensive production readiness review: infrastructure, security, observability, reliability, performance, compliance checklist, and go-live planning.
Deploy Microservices On-Premises with Kubernetes HA
Part 12: Production Operations & Capstone Project
xdev.asia
🎯 LESSON OBJECTIVE__HTMLTAG_66___
- ✅ Production readiness review framework__HTMLTAG_69___
- ✅ Infrastructure checklist
- ✅ Security hardening checklist
- ✅ Observability & reliability checklist
- ✅ Go-live planning and change management
PART 1: INFRASTRUCTURE CHECKLIST
| # | Category | Item | Status |
|---|
| 1 | K8s Cluster | 3+ control plane nodes (HA) | ☐ |
| 2 | K8s Cluster | 3+ worker nodes (anti-affinity) | ☐ |
| 3 | K8s Cluster | etcd backup scheduled (hourly) | ☐ |
| 4 | K8s Cluster | Kubernetes current version (N-1) | ☐ |
| 5 | Networking | CNI installed (Cilium) + NetworkPolicies | ☐ |
| 6 | Networking | MetalLB LoadBalancer configured | ☐ |
| 7 | Networking | Istio service mesh + mTLS | ☐ |
| 8 | Storage | Rook-Ceph cluster healthy (3+ OSDs) | ☐ |
| 9 | Storage | StorageClass default set | ☐ |
| 10_ | Storage | VolumeSnapshot class configured | ☐ |
| 11 | Database | PostgreSQL HA (3 replicas, sync replication) | ☐ |
| 12 | Database | Automated backup + PITR tested | ☐ |
| 13 | Database | Connection pooling (PgBouncer) | ☐ |
| 14 | MQ | RabbitMQ/Kafka cluster HA | ☐ |
| 15 | Cache | Redis Sentinel/Cluster HA | ☐ |
PART 2: SECURITY CHECKLIST
| # | Item | _Status |
|---|
| 1 | RBAC: No cluster-admin for applications_ | ☐ |
| 2 | Pod Security Standards: Restricted enforcement | ☐ |
| 3 | ServiceAccount: Auto-mount disabled | ☐ |
| 4 | Secrets: Stored in Vault (not plain K8s secrets) | ☐ |
| 5 | Network Policies: Default deny-all per namespace_ | ☐ |
| 6 | Kyverno: Validation policies enforced | ☐ |
| 7 | Falco: Runtime security monitoring active | ☐ |
| 8 | Harbor: Scanned images, no critical CVEs | ☐ |
| 9 | Image signing: cosign verification enabled | ☐ |
| 10_ | Audit logging: Enabled, forwarded to Loki | ☐ |
| 11 | etcd encryption at rest enabled | ☐ |
| 12 | TLS everywhere (Istio mTLS + ingress TLS) | ☐ |
PART 3: OBSERVABILITY CHECKLIST
| # | Item | Status |
|---|
| 1 | Prometheus: Metrics collection for all services_ | ☐ |
| 2 | Loki: Centralized logging with structured JSON | ☐ |
| 3 | Tempo: Distributed tracing with OTel_ | ☐ |
| 4 | Grafana: 3-level dashboards (platform → service → request) | ☐ |
| 5 | Correlation: Trace-Log-Metric linking configured | ☐ |
| 6 | SLOs defined: Availability + Latency per service_ | ☐ |
| 7 | Alerting: Multi-burn-rate SLO alerts | ☐ |
| 8 | Alert routing: Critical → PagerDuty, Warning → Slack | ☐ |
| 9 | On-call rotation configured | ☐ |
| 10 | Runbooks linked to alerts | ☐ |
PART 4: RELIABILITY CHECKLIST
| # | Item | Status |
|---|
| 1 | HPA configured for stateless services | ☐ |
| 2 | PodDisruptionBudget for all critical workloads_ | ☐ |
| 3 | Liveness + readiness probes on all containers | ☐ |
| 4 | Resource requests + limits set on all pods_ | ☐ |
| 5 | Pod anti-affinity: spread across nodes_ | ☐ |
| 6 | Circuit breaker configured (Istio DestinationRule) | ☐ |
| 7 | Retry + timeout policies in VirtualService_ | ☐ |
| 8 | Velero backup tested (restore verified) | ☐ |
| 9 | DR runbook documented + tested | ☐ |
| 10 | Chaos engineering: GameDay completed | ☐ |
PART 5: GO-LIVE PLANNING
Go-Live Timeline:
T-2 weeks: Feature freeze, final testing
T-1 week: Performance testing, DR drill, security scan
T-3 days: Staging deployment with production data clone
T-1 day: Final review meeting, rollback plan confirmed
T-0: Go-live (off-peak hours)
Go-Live Day:
08:00 Pre-checks (all systems green)
09:00 DNS cutover / traffic shift
09:30 Smoke tests
10:00 Gradual traffic ramp (10% → 25% → 50% → 100%)
12:00 Full traffic
18:00 Post-launch review
Rollback Plan:
- DNS revert to old infrastructure
- Estimated rollback time: 5 minutes
💡 KEY TAKEAWAYS
- Checklist: Systematic review prevents "forgot to configure X"
- Categories: Infrastructure, Security, Observability, Reliability
- Go-live: Gradual traffic ramp, always have rollback plan
- Review: Peer review checklist before production
- Living document: Update checklist after each incident
🎯 EXERCISE
Exercise 1: Readiness Review__HTMLTAG_595___
- Run through all checklists for your cluster
- Document gaps and create remediation plan
- Perform peer review with teammates
📚 NEXT POST
In Lesson 47: Day-2 Operations & Maintenance, we will learn daily production operations.