Observability — Logs, Metrics, and Tracing

You can't debug what you can't see. In a monolith, you tail one log file and check one dashboard. In Kubernetes, your application is scattered across dozens of pods, on multiple nodes, with containers being created and destroyed constantly. A pod crashes and gets replaced — its logs vanish with it unless you collected them. A request is slow, but it touches six services — which one is the bottleneck? Observability is not optional in Kubernetes. It's the difference between understanding your system and guessing at it.

The Three Pillars

Logs: textual records of events — application output, error messages, access logs. Every container writes to stdout/stderr, and Kubernetes stores them temporarily on the node. But when a pod is evicted or a node dies, those logs are gone. You need a log aggregation pipeline.

Metrics: numerical measurements over time — CPU usage, request rate, error percentage, queue depth. Metrics tell you what's happening right now and how it trends over time. They're cheap to store and fast to query, making them your first line of defense for monitoring and alerting.

Traces: records of a single request as it flows through multiple services. A trace shows you that a request hit the API gateway, then the auth service (2ms), then the order service (50ms), then the database (200ms) — and the database was the bottleneck. Without traces, you're guessing which service is slow.

Metrics Pipeline: Prometheus + Grafana

Prometheus is the standard for Kubernetes metrics. It's a pull-based system — Prometheus scrapes HTTP endpoints (typically /metrics) on your pods at regular intervals. Applications expose metrics in Prometheus format using client libraries, and Kubernetes components (kubelet, API server, etcd) expose metrics natively.

# ServiceMonitor tells Prometheus what to scrape
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: web-app-metrics
spec:
  selector:
    matchLabels:
      app: web-app
  endpoints:
    - port: metrics
      interval: 15s
      path: /metrics

Grafana visualizes Prometheus data. It connects to Prometheus as a data source and renders dashboards with graphs, gauges, and tables. The community provides pre-built dashboards for Kubernetes cluster monitoring, node-level metrics, and popular applications.

Alerting: Prometheus includes Alertmanager for routing alerts to Slack, PagerDuty, or email. Define alert rules in PromQL:

# Alert when error rate exceeds 5%
groups:
  - name: web-app
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.05
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Error rate above 5% for {{ $labels.service }}"

Log Aggregation Pipeline

The standard pattern: run a DaemonSet on every node that tails container log files and forwards them to a central store.

Fluent Bit is the lightweight collector — it runs as a DaemonSet, reads container logs from /var/log/containers/, enriches them with Kubernetes metadata (pod name, namespace, labels), and forwards them. It uses minimal resources (~10MB RAM per node).

Loki (from Grafana Labs) is the modern log storage backend. Unlike Elasticsearch, Loki doesn't index log content — it indexes only metadata labels, which makes it dramatically cheaper to run. You query logs using LogQL, which feels like PromQL but for logs. If you're already running Grafana for metrics, Loki integrates natively.

ELK Stack (Elasticsearch + Logstash + Kibana) is the traditional choice. More powerful search capabilities but significantly more expensive to operate — Elasticsearch clusters need careful sizing, tuning, and maintenance.

Distributed Tracing Pipeline

OpenTelemetry (OTel) is the standard for instrumenting applications. It provides SDKs for every major language that automatically capture traces, metrics, and logs. OTel is vendor-neutral — you instrument once and can send data to any backend.

Jaeger is the most common open-source trace backend for Kubernetes. It stores traces and provides a UI for exploring them — visualizing request flows, identifying bottleneck services, and comparing trace durations.

The flow: your application code (instrumented with OTel SDK) → OTel Collector (runs as a sidecar or DaemonSet) → Jaeger/Tempo/Zipkin backend → visualization in Grafana or Jaeger UI.

Health Checks: Liveness, Readiness, and Startup Probes

Probes are how Kubernetes knows if your application is working correctly:

Liveness probe: is the container alive? If it fails, Kubernetes kills and restarts the container. Use this to recover from deadlocks or hung processes.

Readiness probe: can the container serve traffic? If it fails, the pod is removed from Service endpoints — no traffic is routed to it. Use this during startup (database connections loading) and for graceful degradation.

Startup probe: is the container finished starting? Runs only during startup, disabling liveness/readiness checks until it succeeds. Essential for slow-starting applications (JVM warmup, large model loading).

containers:
  - name: web
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 15
      periodSeconds: 10
      failureThreshold: 3
    readinessProbe:
      httpGet:
        path: /ready
        port: 8080
      periodSeconds: 5
      failureThreshold: 2
    startupProbe:
      httpGet:
        path: /healthz
        port: 8080
      failureThreshold: 30
      periodSeconds: 10

Debugging Commands

# Stream logs from a pod
kubectl logs -f deployment/web-app

# Previous container's logs (after a crash)
kubectl logs web-app-xyz --previous

# Resource usage per pod
kubectl top pods

# Resource usage per node
kubectl top nodes

# Debug a running pod with an ephemeral container
kubectl debug -it web-app-xyz --image=busybox

# Describe pod events (scheduling, probes, OOMKills)
kubectl describe pod web-app-xyz

The Practical Stack

For most teams, this is the right starting stack:

Layer	Tool	Why
Metrics	Prometheus + Grafana	Industry standard, free, massive community
Logs	Fluent Bit + Loki	Lightweight, cheap, integrates with Grafana
Traces	OpenTelemetry + Tempo/Jaeger	Vendor-neutral instrumentation, integrates with Grafana
Alerting	Alertmanager	Built into Prometheus, routes to Slack/PagerDuty

This entire stack can run inside your cluster. If you prefer managed services, every cloud provider offers integrated alternatives (CloudWatch, Cloud Monitoring, Azure Monitor), and vendors like Datadog and New Relic provide unified platforms.

Key Takeaways

• Logs, metrics, and traces are complementary — metrics tell you something is wrong, logs tell you what happened, traces tell you where the bottleneck is
• Prometheus + Grafana is the de facto metrics stack — learn PromQL, it's worth the investment
• Use Fluent Bit (not Fluentd) as your log collector — it uses 10x less memory
• Loki is dramatically cheaper than Elasticsearch for log storage if you don't need full-text search
• OpenTelemetry is the future of instrumentation — instrument once, send to any backend
• Set liveness, readiness, and startup probes on every container — they're how Kubernetes manages your app's lifecycle