eBPF Observability with Cilium Hubble in Production Kubernetes

Understanding eBPF Observability with Cilium Hubble

Network visibility in Kubernetes has traditionally meant scraping iptables counters or injecting a sidecar proxy into every pod. eBPF observability Cilium Hubble replaces both approaches by running observation programs directly in the Linux kernel, attached to the same datapath that already forwards your traffic. Because the instrumentation lives where the packets are, you see flows without adding a single proxy hop.

The result is a low-overhead window into L3 through L7 traffic that arrives essentially for free once Cilium is your CNI. Therefore you gain service maps, flow logs, and protocol-aware metrics without rewriting applications or doubling your pod count. This guide walks through what that buys you and where its limits sit.

What eBPF Gives You Over Sidecars and iptables

A sidecar mesh intercepts traffic by routing every connection through a userspace proxy, which adds latency, memory, and an extra failure domain per pod. In contrast, eBPF programs run in the kernel and observe traffic on the existing path, so there is no second network hop. The Cilium project uses this to deliver identity-aware networking and observability from one datapath.

Compared to iptables, eBPF avoids the linear rule-chain scaling problem that hurts large clusters. Moreover, because Cilium assigns a stable security identity to each workload rather than relying on ephemeral pod IPs, your flow data stays meaningful even as pods churn. For the policy side of this story, see the companion discussion of Kubernetes network policies for zero trust.

Installing Cilium and Enabling Hubble

Hubble is the observability layer that consumes the events Cilium’s eBPF programs emit. You enable it on an existing Cilium installation, then turn on the Relay component so flows aggregate across every node. The configuration below switches on Hubble, the metrics server, and the UI in a single Helm values fragment.

hubble:
  enabled: true
  relay:
    enabled: true
  ui:
    enabled: true
  metrics:
    enableOpenMetrics: true
    enabled:
      - dns:query;ignoreAAAA
      - drop
      - tcp
      - flow
      - port-distribution
      - icmp
      - httpV2:exemplars=true;labelsContext=source_namespace,destination_namespace
  tls:
    enabled: true
# Apply with:
#   helm upgrade cilium cilium/cilium --namespace kube-system \
#     --reuse-values -f hubble-values.yaml
#   cilium hubble enable --ui

Once Relay is healthy, the control plane exposes an aggregated flow API for the whole cluster. Additionally, enabling the L7 HTTP metrics line above instructs Cilium to parse HTTP requests so you get method, path, and status visibility, not just packet counts.

Flow Visibility from L3 to L7

The hubble observe CLI is the fastest way to confirm what is actually traversing your network. It streams live flows or queries recent history, with filters for namespace, pod, verdict, and protocol. For example, you can watch only dropped traffic to a sensitive namespace and immediately see which source is being denied.

# Stream live flows into the payments namespace
hubble observe --namespace payments --follow

# Show only dropped traffic with the deny reason
hubble observe --verdict DROPPED --last 200

# L7 HTTP visibility: method, path, and status code
hubble observe --protocol http --namespace api --follow

# DNS lookups that failed to resolve
hubble observe --protocol dns --verdict DROPPED

# Flows for one workload by Cilium identity
hubble observe --to-pod payments/checkout-7d9f --since 5m

Notably, when the HTTP visibility annotation is in place, each flow line includes the request method and response code, turning a raw packet trace into an application-level audit. Consequently, debugging a failing call often takes one command instead of a tcpdump session.

Service Maps and Verifying NetworkPolicy

The Hubble UI assembles observed flows into a live service dependency map, drawing edges between workloads as traffic actually moves between them. This is invaluable when an undocumented dependency surfaces only under load. For example, you may discover a batch job quietly calling a database it was never supposed to touch.

Flow data also closes the loop on policy. After you apply a CiliumNetworkPolicy, you verify it by watching for DROPPED verdicts that carry the policy-deny reason, confirming the rule blocks what you intended and nothing more. This pairs naturally with sidecarless lifecycle features; if you run init or telemetry containers, the patterns in native sidecar containers in Kubernetes still apply alongside an eBPF datapath.

Exporting Metrics to Prometheus and Grafana

Hubble exposes OpenMetrics endpoints that Prometheus scrapes directly, giving you time-series data for request rates, error rates, and dropped packets per service. From there, Grafana dashboards render golden-signal views keyed by Cilium identity rather than fragile pod IPs. The Hubble documentation lists each metric family and its label set.

Because these metrics originate in the kernel datapath, they capture traffic the application never logs, including connection refusals and policy drops. Furthermore, if you already run a tracing pipeline, the flow data complements span-level telemetry; teams often feed both into a shared backend as described in the guide on OpenTelemetry Collector pipelines.

Overhead and When eBPF Fits Versus a Full Mesh

Because there is no per-pod proxy, the steady-state cost is the kernel programs plus the Hubble Relay aggregating events, which is modest compared to a sidecar per workload. Enabling deep L7 parsing on high-throughput paths does add CPU, so a common pattern is to scope HTTP visibility to the namespaces that need it rather than the whole cluster. Under sustained load you can expect the observability overhead to stay well below that of an injected proxy mesh.

However, eBPF observability is not a full service mesh. If you need mutual TLS between services, fine-grained traffic shifting, retries, and circuit breaking, a mesh still earns its place, and you can layer Hubble beneath it for visibility. Choose eBPF observability when your primary need is seeing and securing traffic, and add a mesh only when you need to actively control it.

In conclusion, eBPF observability Cilium Hubble delivers L3 to L7 network insight in Kubernetes without sidecars, without iptables sprawl, and without touching application code. You get flow logs, service maps, policy verification, and Prometheus-ready metrics from the same datapath that already moves your packets. Adopt it when visibility and security are the goal, keep its L7 parsing scoped to where it pays off, and reserve a full mesh for when you must control traffic, not merely observe it.