1. Performance Test Report — Cấu trúc
Performance Test Report Structure:
1. Cover Page
├── Project name, version
├── Test period
├── Author & reviewer
└── Classification (Confidential)
2. Executive Summary (1 page)
├── Test objective
├── Key findings (pass/fail)
├── SLO compliance status
└── Top recommendations
3. Test Configuration
├── Environment details
├── Test scenarios
├── Workload model
└── Tools used
4. Results Summary
├── Dashboard/scorecard
├── SLO compliance table
└── Trend comparison
5. Detailed Results
├── Per-scenario breakdown
├── Response time analysis
├── Throughput analysis
├── Error analysis
└── Resource utilization
6. Issues Found
├── Performance bottlenecks
├── Scalability limits
└── Stability concerns
7. Recommendations
├── Critical (before release)
├── High (within sprint)
└── Medium/Low (backlog)
8. Appendix
├── Raw data links
├── Grafana dashboard links
└── Test script references
2. Executive Summary Template
# Performance Test Report
## [Project Name] — Release v2.5.0
### Executive Summary
**Test Period:** April 1-7, 2026
**Environment:** Staging (equivalent to production)
**Overall Result:** ⚠️ CONDITIONAL PASS
#### Key Metrics
| Metric | SLO Target | Actual | Status |
|---------------------|---------------|-------------|--------|
| P95 Response Time | < 500ms | 420ms | ✅ PASS |
| P99 Response Time | < 1000ms | 1,250ms | ❌ FAIL |
| Error Rate | < 0.1% | 0.08% | ✅ PASS |
| Throughput | > 1,000 RPS | 1,150 RPS | ✅ PASS |
| CPU Utilization | < 70% | 65% | ✅ PASS |
| Memory Utilization | < 80% | 78% | ⚠️ WARN |
#### Summary
The system meets 4 of 6 SLO targets under expected load
(1,000 concurrent users). P99 response time exceeds target
by 25% during peak hours. Memory approaches threshold
during sustained load.
#### Top Recommendations
1. **[Critical]** Optimize database query in /api/orders
endpoint — causing P99 spike
2. **[High]** Increase memory limit from 4GB to 6GB
or investigate memory leak in session handler
3. **[Medium]** Enable CDN caching for static assets
to reduce origin load by ~30%
#### Release Recommendation
**Conditional Go** — Fix #1 before production release.
Items #2 and #3 can be addressed in next sprint.
3. Test Configuration Section
Test Environment:
Infrastructure:
├── Cloud: AWS ap-southeast-1
├── Compute: 3x c6i.xlarge (4 vCPU, 8GB RAM)
├── Database: RDS PostgreSQL 16 (db.r6g.large)
├── Cache: ElastiCache Redis (cache.r6g.large)
├── Load Balancer: ALB (Application)
└── CDN: CloudFront (disabled for testing)
Application:
├── Version: v2.5.0-rc1 (commit: abc123)
├── Runtime: Node.js 22.x
├── Framework: NestJS 11.x
└── Container: Docker (distroless)
Test Tools:
├── Load Generator: k6 v1.0 (distributed)
├── Monitoring: Grafana + Prometheus
├── APM: Jaeger (distributed tracing)
└── Profiling: Pyroscope (continuous profiling)
Workload Model:
User Journey Distribution:
┌──────────────────┬───────────┬──────────┐
│ Scenario │ % Traffic │ Think │
├──────────────────┼───────────┼──────────┤
│ Browse products │ 40% │ 5-10s │
│ Search │ 25% │ 3-8s │
│ View product │ 15% │ 5-15s │
│ Add to cart │ 10% │ 2-5s │
│ Checkout │ 7% │ 10-30s │
│ User profile │ 3% │ 5-10s │
└──────────────────┴───────────┴──────────┘
Test Scenarios Executed:
1. Baseline (100 VUs, 10 min) — establish baseline
2. Load Test (1,000 VUs, 30 min) — normal load
3. Stress Test (2,000 VUs, 20 min) — peak load
4. Spike Test (100→2,000→100 VUs) — sudden spike
5. Soak Test (500 VUs, 4 hours) — stability
4. Results Visualization
Load Test Results (1,000 VUs, 30 min):
Response Time Distribution:
P50 ████████████░░░░░░░░ 180ms
P75 ███████████████░░░░░ 290ms
P90 █████████████████░░░ 370ms
P95 ██████████████████░░ 420ms
P99 ████████████████████████ 1,250ms ← EXCEEDS SLO
Max ██████████████████████████████ 3,500ms
Throughput Over Time:
1,200 ┤ ╭──────╮
1,100 ┤ ╭─────╯ ╰──╮
1,000 ┤ ╭─────────╯ ╰──
900 ┤ ╭─╯
800 ┤╭─╯
└──────────────────────────────────
0 5 10 15 20 25 30 min
Error Rate by Endpoint:
/api/products 0.02% ████
/api/search 0.05% ████████
/api/cart 0.03% █████
/api/orders 0.15% █████████████████████ ← ISSUE
/api/users 0.01% ██
CPU Utilization per Pod:
pod-1 60% ████████████░░░░░░░░
pod-2 65% █████████████░░░░░░░
pod-3 71% ██████████████░░░░░░ ← Above threshold
Memory Usage Trend (Soak Test):
80% ┤ ╭──── ← Leak?
70% ┤ ╭────────╯
60% ┤ ╭────────────╯
50% ┤ ╭─────────╯
40% ┤────╯
└────────────────────────────────────────
0h 1h 2h 3h 4h
5. Bottleneck Analysis
Bottleneck #1: /api/orders P99 Response Time
Root Cause Analysis:
Request flow: Client → ALB → App → Database
Trace Analysis (Jaeger):
┤ HTTP /api/orders ─────────────────── 1,250ms
│ ├── Auth middleware ────────── 5ms
│ ├── Validation ─────────── 2ms
│ ├── DB: SELECT orders ──────────────── 980ms ← BOTTLENECK
│ │ └── Missing index on (user_id, created_at)
│ ├── DB: SELECT order_items ── 180ms
│ └── Serialization ────── 15ms
Database Analysis:
├── Query: SELECT * FROM orders WHERE user_id = $1
│ ORDER BY created_at DESC LIMIT 20
├── Execution Plan: Seq Scan (no index!)
├── Rows scanned: 2.3M (table scan)
└── Expected with index: < 50ms
Fix:
CREATE INDEX CONCURRENTLY idx_orders_user_created
ON orders (user_id, created_at DESC);
Expected Improvement:
P99: 1,250ms → ~200ms (84% reduction)
Bottleneck #2: Memory Growth (Soak Test)
Pattern: Linear memory growth 40% → 78% over 4 hours
Suspected cause: Session store not expiring entries
Investigation: heap dump analysis needed
Risk: OOM kill after ~6 hours under load
6. Recommendations Template
Recommendations:
Priority: CRITICAL — Must fix before release
#1: Add database index on orders table
Impact: P99 1,250ms → ~200ms
Effort: Low (1 hour)
Risk: None (CONCURRENTLY)
Priority: HIGH — Fix within current sprint
#2: Investigate memory leak in session handler
Impact: Prevent OOM under sustained load
Effort: Medium (2-3 days investigation)
#3: Enable CDN for static assets
Impact: Reduce origin load by 30%
Effort: Low (configuration change)
Priority: MEDIUM — Next sprint
#4: Implement connection pooling for DB
Impact: Reduce DB connection overhead
Effort: Medium
#5: Add response caching for /api/products
Impact: Reduce DB queries by 50%
Effort: Low-Medium
Priority: LOW — Backlog
#6: Evaluate horizontal auto-scaling
#7: Consider read replicas for reporting queries
7. Trend Comparison
Version Comparison (v2.4.0 vs v2.5.0):
Metric │ v2.4.0 │ v2.5.0 │ Change
────────────────┼──────────┼──────────┼──────────
P50 Response │ 160ms │ 180ms │ +12.5% ⚠️
P95 Response │ 380ms │ 420ms │ +10.5% ⚠️
P99 Response │ 850ms │ 1,250ms │ +47.1% ❌
Throughput │ 1,080 RPS│ 1,150 RPS│ +6.5% ✅
Error Rate │ 0.06% │ 0.08% │ +33.3% ⚠️
CPU Usage │ 58% │ 65% │ +12.1% ⚠️
Analysis:
Performance degradation in v2.5.0 is primarily due to
new order history feature (Bài 1.2.3) which introduced
unindexed queries. Throughput improvement is from optimized
product search in v2.5.0.
8. Tổng kết
- Executive Summary: 1-page overview — metrics, pass/fail, top recommendations
- Test Config: Environment, workload model, scenarios executed
- Results: Visualize response times, throughput, errors, resources
- Bottleneck Analysis: Trace-driven root cause with fix recommendations
- Recommendations: Prioritized by criticality with effort estimation
- Trend Comparison: Track performance across releases
Bài tiếp theo sẽ hướng dẫn viết Pentest Report — Technical và Executive.