🎯 LESSON OBJECTIVE__HTMLTAG_66___
- ✅ Grafana Loki Architecture (vs ELK Stack)
- ✅ Deploy Loki Distributed mode on K8s
- ✅ Promtail/Grafana Alloy log collection
- ✅ LogQL query language__HTMLTAG_75___
- ✅ Structured logging best practices
- ✅ Log-based alerting
PART 1: LOKI ARCHITECTURE
Loki Architecture (Read/Write Path):
Write Path:
App Logs → Promtail → Distributor → Ingester → Object Storage
↓
Index Store
Read Path:
Grafana → Query Frontend → Querier → Index + Chunks
(parallel query)
Components:
┌──────────┐ ┌──────────────┐ ┌──────────────┐
│ Promtail │ │ Distributor │ │ Ingester │
│(DaemonSet│─►│ (validates │─►│ (write-ahead│
│ per node)│ │ + routes) │ │ log + flush│
└──────────┘ └──────────────┘ └──────┬───────┘
│
┌──────▼───────┐
│Object Storage│
│(Ceph S3/MinIO│
│ /local disk)│
└──────────────┘
| Feature | Loki | ELK (Elasticsearch) |
|---|---|---|
| Indexing | Labels only (metadata) | Full-text index (all content) |
| Storage Cost | Low (compressed chunks) | High (inverted index) |
| Resource Usage | Low | High (RAM, CPU) |
| Query Speed | Slower for full-text_ | Fast for full-text_ |
| Best For | K8s logs, cost-effective | Complex log analytics_ |
| Query Language | LogQL | KQL/Lucene |
| Integration | Grafana native | Kibana |
PART 2: DEPLOY LOKI
# Install Loki:
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm install loki grafana/loki \
--namespace monitoring \
-f loki-values.yaml
# loki-values.yaml:
loki:
auth_enabled: false
commonConfig:
replication_factor: 1
schemaConfig:
configs:
- from: "2024-01-01"
store: tsdb
object_store: s3
schema: v13
index:
prefix: loki_index_
period: 24h
storage:
type: s3
bucketNames:
chunks: loki-chunks
ruler: loki-ruler
s3:
endpoint: ceph-rgw.storage:8080
accessKeyId: loki
secretAccessKey: loki-secret
s3ForcePathStyle: true
insecure: true
limits_config:
retention_period: 30d
max_query_length: 720h
max_entries_limit_per_query: 5000
ingestion_rate_mb: 10
ingestion_burst_size_mb: 20
deploymentMode: SimpleScalable
backend:
replicas: 2
persistence:
storageClass: ceph-block
size: 10Gi
read:
replicas: 2
write:
replicas: 2
persistence:
storageClass: ceph-block
size: 10Gi
# Enable Promtail as DaemonSet:
gateway:
replicas: 2
PART 3: LOG COLLECTION WITH PROMTAIL
# Install Promtail:
helm install promtail grafana/promtail \
--namespace monitoring \
-f promtail-values.yaml
# promtail-values.yaml:
config:
clients:
- url: http://loki-gateway.monitoring/loki/api/v1/push
scrape_configs:
# Kubernetes pod logs:
- job_name: kubernetes-pods
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_label_app]
target_label: app
- source_labels: [__meta_kubernetes_namespace]
target_label: namespace
- source_labels: [__meta_kubernetes_pod_name]
target_label: pod
- source_labels: [__meta_kubernetes_pod_container_name]
target_label: container
pipeline_stages:
# Parse JSON logs:
- json:
expressions:
level: level
message: msg
timestamp: timestamp
trace_id: trace_id
- labels:
level:
- timestamp:
source: timestamp
format: RFC3339Nano
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
cpu: 200m
memory: 256Mi
tolerations:
- effect: NoSchedule
operator: Exists
PART 4: LOGQL QUERIES
# Basic log queries:
# All logs from a service:
{app="order-service"}
# Filter by namespace:
{namespace="default", app="order-service"}
# Text filter:
{app="order-service"} |= "error"
{app="order-service"} != "healthcheck"
{app="order-service"} |~ "timeout|connection refused"
# JSON parsing:
{app="order-service"} | json | level="error"
{app="order-service"} | json | status_code >= 500
# Log rate (logs/sec):
rate({app="order-service"} |= "error" [5m])
# Count errors by service:
sum by(app) (count_over_time({namespace="default"} | json | level="error" [1h]))
# Top 10 error messages:
topk(10, sum by(message) (count_over_time({app="order-service"} | json | level="error" [1h])))
# P99 response time from access logs:
quantile_over_time(0.99, {app="nginx"} | json | unwrap response_time [5m]) by (path)
PART 5: STRUCTURED LOGGING BEST PRACTICES
// ✅ Good: Structured JSON log
{
"timestamp": "2024-06-15T10:30:00Z",
"level": "error",
"msg": "Failed to process order",
"service": "order-service",
"trace_id": "abc123def456",
"span_id": "span789",
"order_id": "ORD-12345",
"error": "connection refused",
"duration_ms": 1500
}
// ❌ Bad: Unstructured log
// 2024-06-15 10:30:00 ERROR Failed to process order ORD-12345: connection refused
// Go structured logging (slog):
import "log/slog"
logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
Level: slog.LevelInfo,
}))
logger.Error("Failed to process order",
"order_id", orderID,
"error", err,
"duration_ms", elapsed.Milliseconds(),
"trace_id", span.SpanContext().TraceID().String(),
)
PART 6: LOG-BASED ALERTING
# Loki ruler (alert on log patterns):
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: loki-log-alerts
namespace: monitoring
spec:
groups:
- name: log-alerts
rules:
- alert: HighErrorLogRate
expr: |
sum(rate({namespace="default"} | json | level="error" [5m])) by (app) > 1
for: 5m
labels:
severity: warning
annotations:
summary: "High error log rate on {{ $labels.app }}"
- alert: OOMKillDetected
expr: |
count_over_time({namespace="default"} |= "OOMKilled" [5m]) > 0
labels:
severity: critical
💡 KEY TAKEAWAYS
- Loki: Index labels only → cost-effective, Grafana-native
- Promtail: DaemonSet, auto-collect K8s pod logs
- LogQL: Powerful query language, log-to-metrics conversion
- Structured logging: JSON format, include trace_id/span_id
- Pipeline stages: Parse, label, timestamp extraction
- Retention: Configure per-tenant, 30-day default
🎯 EXERCISES__HTMLTAG_202___
Exercise 1: Loki Setup
- Deploy Loki + Promtail
- Add Loki datasource in Grafana
- Write LogQL queries to find errors__HTMLTAG_211___
Exercise 2: Structured Logging
- Implement JSON logging in sample app
- Configure Promtail pipeline to extract level, trace_id
- Create log-based alert rule
📚 NEXT POST
In Lesson 34: Tempo — Distributed Tracing, we will setup distributed tracing for microservices.