LESSON 38: FALCO RUNTIME SECURITY

Deploy Microservices On-Premises with Kubernetes HA

Part 9: Security Hardening

xdev.asia

🎯 LESSON OBJECTIVE__HTMLTAG_66___

• ✅ Runtime security concepts (shift-left vs runtime)
• ✅ Falco architecture (eBPF driver, rules engine)
• ✅ Deploy Falco on K8s
• ✅ Custom security rules__HTMLTAG_75___
• ✅ Falcosidekick event routing__HTMLTAG_77___
• ✅ Incident response automation

---

PART 1: FALCO ARCHITECTURE

Falco Runtime Detection:

┌─────────────────────────────────────────────┐
│                  Node (Host)                │
│                                             │
│  ┌─────────┐  syscalls  ┌────────────────┐  │
│  │Container│───────────►│ Falco (DaemonSet│  │
│  │  App    │            │  eBPF driver)   │  │
│  └─────────┘            │                 │  │
│                         │  Rules Engine:  │  │
│  ┌─────────┐            │  - Shell in     │  │
│  │Container│───────────►│    container?   │  │
│  │  App    │            │  - Write to bin?│  │
│  └─────────┘            │  - Read secrets?│  │
│                         └────────┬────────┘  │
└──────────────────────────────────┼───────────┘
                                   │
                          ┌────────▼────────┐
                          │  Falcosidekick  │
                          │  (event router) │
                          └───┬────┬────┬───┘
                              │    │    │
                           Slack  Loki  K8s
                                       Response

# Install Falco:
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

helm install falco falcosecurity/falco \
  --namespace falco \
  --create-namespace \
  -f falco-values.yaml

# falco-values.yaml:
driver:
  kind: modern_ebpf  # modern eBPF (no kernel module needed)

collectors:
  kubernetes:
    enabled: true

falco:
  grpc:
    enabled: true
  grpc_output:
    enabled: true
  json_output: true
  log_level: info

  rules_file:
    - /etc/falco/falco_rules.yaml
    - /etc/falco/falco_rules.local.yaml
    - /etc/falco/rules.d

falcosidekick:
  enabled: true
  config:
    slack:
      webhookurl: "https://hooks.slack.com/services/xxx"
      channel: "#security-alerts"
      minimumpriority: warning
    loki:
      hostport: "http://loki-gateway.monitoring"
    prometheus:
      extralabels: "source:falco"

resources:
  requests:
    cpu: 100m
    memory: 256Mi
  limits:
    cpu: 500m
    memory: 512Mi

tolerations:
  - effect: NoSchedule
    operator: Exists

---

PART 2: FALCO BUILT-IN RULES

Rule	Detects	Priority
Terminal shell in container	kubectl exec -it	Notice
Write below /etc	Config file modification_	Error
Read sensitive file_	/etc/shadow, /etc/passwd	Warning_
Launch privileged container	Privileged flag	Critical
Modify binary dirs	Write to /usr/bin, /sbin_	Error
Outbound connection	Unexpected network calls	Notice
Crypto mining detection	Known mining processes	Critical_
Container drift_	New executable not in image_	_Error

---

PART 3: CUSTOM FALCO RULES

# Custom rules ConfigMap:
apiVersion: v1
kind: ConfigMap
metadata:
  name: falco-custom-rules
  namespace: falco
data:
  custom-rules.yaml: |
    # Detect kubectl exec:
    - rule: Terminal shell in container
      desc: A shell was spawned in a container
      condition: >
        spawned_process and container
        and shell_procs
        and proc.tty != 0
      output: >
        Shell spawned in container
        (user=%user.name container=%container.name
        shell=%proc.name parent=%proc.pname
        namespace=%k8s.ns.name pod=%k8s.pod.name
        image=%container.image.repository)
      priority: WARNING
      tags: [container, shell, exec]

    # Detect writes to /etc in production:
    - rule: Write to /etc in production namespace
      desc: Detect config file changes in production containers
      condition: >
        open_write and container
        and fd.name startswith /etc
        and k8s.ns.name = "production"
        and not proc.name in (sed, tee)
      output: >
        File written to /etc in production
        (user=%user.name file=%fd.name
        container=%container.name
        namespace=%k8s.ns.name pod=%k8s.pod.name)
      priority: ERROR
      tags: [filesystem, production]

    # Detect outbound connections from DB pods:
    - rule: Unexpected outbound from database
      desc: Database pod making outbound internet connections
      condition: >
        outbound and container
        and k8s.pod.label.app in (postgresql, redis, rabbitmq)
        and not fd.sip.name in (private_ipv4)
      output: >
        Database pod making outbound connection
        (pod=%k8s.pod.name dest=%fd.sip:%fd.sport
        namespace=%k8s.ns.name)
      priority: CRITICAL
      tags: [network, database]

    # Detect package manager in running container:
    - rule: Package manager in container
      desc: Package manager executed in running container (drift)
      condition: >
        spawned_process and container
        and proc.name in (apt, apt-get, yum, dnf, apk, pip, npm)
      output: >
        Package manager executed in container
        (command=%proc.cmdline container=%container.name
         namespace=%k8s.ns.name pod=%k8s.pod.name)
      priority: ERROR
      tags: [drift, supply-chain]

---

PART 4: INCIDENT RESPONSE AUTOMATION

# Falcosidekick response: auto-label suspicious pods:
falcosidekick:
  config:
    kubernetesPolicyReport:
      enabled: true
      minimumpriority: warning

# Example: Webhook to auto-isolate pod:
# When critical alert → NetworkPolicy blocks all traffic

---
# Auto-response: isolate compromised pod
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: isolate-compromised
  namespace: production
spec:
  podSelector:
    matchLabels:
      security.falco/compromised: "true"
  policyTypes:
    - Ingress
    - Egress
  # Empty = deny all

# Test Falco rules:

# Trigger shell in container:
kubectl exec -it deploy/nginx -n production -- /bin/bash
# → Falco alert: "Terminal shell in container"

# Trigger file write:
kubectl exec deploy/nginx -n production -- touch /etc/test
# → Falco alert: "Write to /etc in production"

# View Falco alerts:
kubectl logs -n falco -l app.kubernetes.io/name=falco --tail=20

---

PART 5: FALCO MONITORING

# Falco metrics in Prometheus:
# falcosidekick exposes /metrics

# Grafana queries:
# Alert count by priority:
sum by (priority) (increase(falcosidekick_inputs_total[1h]))

# Alert count by rule:
topk(10, sum by (rule) (increase(falcosidekick_inputs_total[24h])))

# Critical alerts trend:
rate(falcosidekick_inputs_total{priority="critical"}[5m])

---

💡 KEY TAKEAWAYS

1. Falco: Runtime threat detection via eBPF syscall monitoring
2. Built-in rules: Shell in container, file writes, crypto mining
3. Custom rules: Condition + output format, priority levels
4. Falcosidekick: Route events to Slack, Loki, PagerDuty
5. Incident response: Auto-isolate compromised pods
6. Drift detection: Package manager execution = container drift

---

🎯 EXERCISE

Exercise 1: Falco Setup__HTMLTAG_209___

• Deploy Falco with eBPF driver__HTMLTAG_212___
• Trigger built-in rules (exec, file write)
• View alerts in Slack and Loki

Exercise 2: Custom Rules__HTMLTAG_219___

• Write rule to detect outbound connections from database pods
• Write rule to detect package manager execution__HTMLTAG_224___
• Configure auto-response: isolate flagged pods

---

📚 NEXT POST

In Lesson 39: Harbor Registry & Image Security, we will setup private container registry with vulnerability scanning.