The `bpftool perf` Command That Reveals Hidden Network Latency in Kubernetes

Network latency inside a Kubernetes cluster is notoriously difficult to trace using standard metrics. When packets traverse complex iptables rules or saturated veth interfaces, standard APM tools blindside you by attributing kernel delays to application processing time. You need a way to inspect the kernel’s network datapath directly.

TL;DR: The bpftool perf command attaches to kernel tracepoints to expose exact microsecond delays at the network interface layer. By tracing eBPF performance events, you can definitively isolate whether network latency originates from your application code or deep inside Kubernetes network filtering rules.

How Do You Expose Network Datapath Latency With bpftool?

You expose datapath latency by reading kernel tracepoints directly from the perf ring buffer using bpftool perf show. eBPF performance events are structured records emitted by the kernel during critical network transitions, allowing user-space tools to read high-resolution timestamps without halting packet processing. This pinpoints exactly where packet drops or queuing delays occur.

bpftool -j perf show

[{"pid":412,"fd":15,"cpu":0,"tracepoint":"net:netif_receive_skb"}]

Flag/Argument	Description	Best For
-j	Formats output as JSON for external parsing.	Automated scripting.
perf	Subcommand managing eBPF performance event interactions.	Accessing the ring buffer.
show	Lists active performance ring buffers mapped on the system.	Verifying tracepoints.

1. Find saturated CPUs handling network traffic:

   perf top -C 2
   

2. Inspect specific file descriptors for rogue filters:

   bpftool map show id 45
   

3. Dump verbose data including kernel probes:

   bpftool -p perf show
   

When NOT to use it: Do not use this for long-term historical metrics gathering, as reading directly from the perf ring buffer at high packet rates creates measurable CPU overhead.

Prefer parsing JSON output with jq over grepping raw stdout to prevent brittle scripts when kernel structures change.

Why Do Traditional APMs Miss Kubernetes Network Filtering Delays?

Traditional APMs miss filtering delays because they measure application-level execution time, completely ignoring the time a packet spends traversing kernel structures like iptables or eBPF datapath hooks. They only start the clock once the socket receives data. Consequently, a saturated veth interface looks identical to slow application logic.

If Aicademy runs a microservice experiencing 40ms spikes, standard telemetry blames the Go runtime. In reality, routing rules are choking the kernel. As noted in eBPF for the Impatient: Architecting Runtime Security at the Kernel Edge, observing the edge requires hooking into the kernel natively.

flowchart LR
    A["Network Interface"] -->|"Packet Arrives"| B["Kernel Filters"]
    B -->|"Socket Read"| C["Application"]

Under linux-kernel 6.6, the default perf ring buffer allocation used by bpftool 6.6 frequently defaults to 64KB per CPU. This limit is often insufficient for 10Gbps+ workloads, leading to dropped trace events.

Complete this audit checklist before profiling high-throughput nodes:

• Verify you are running linux-kernel 6.6 and bpftool 6.6.
• Disable hardware offloading features temporarily on the virtual interface.
• Run the trace during the specific window of the latency spike.

Always correlate kernel timestamps with your application logs using the exact CPU core ID to identify thread contention.

How Do You Isolate Specific Packet Drops Using eBPF Maps?

You isolate packet drops by dumping the eBPF maps that store network event counters linked to your tracepoints. By reading the specific map IDs exposed by the perf ring buffer, you pinpoint exactly which security rule or routing policy discarded the incoming packet.

To view the raw data held in an associated map, execute:

bpftool map dump id 120

View verbose map dump output

key: 00 00 00 00  value: 2a 00 00 00 00 00 00 00
key: 01 00 00 00  value: 00 00 00 00 00 00 00 00
Found 2 elements

This approach builds on the isolation techniques discussed in The bpftool prog show Command That Unmasks Hidden Kernel Backdoors, allowing you to inspect active kernel memory safely. For broader cluster-wide trends, mapping these drops perfectly complements eBPF Unleashed: The Future of Cloud-Native Observability.

Pin the eBPF map to the bpf virtual filesystem if you need to query it continuously from multiple user-space tools.

Bottom Line

Stop trusting application-layer telemetry for network-layer problems. The bpftool perf command grants you absolute visibility into kernel queuing delays without instrumenting your code. Run these traces immediately when you observe intermittent timeouts that your APM cannot explain.

FAQ

Q: What is the default perf ring buffer size in linux-kernel 6.6? A: The default allocation relies on system page sizes but frequently defaults to 64KB per CPU. For high-throughput environments, this must often be increased manually.

Q: Why does bpftool perf show return no output? A: This indicates no user-space applications are currently polling a perf ring buffer on the node. Ensure your tracing program is actively running in another terminal.

Q: Can eBPF tracepoints degrade network performance? A: Reading from the perf ring buffer introduces marginal overhead. However, tracing high-frequency events like individual packet arrivals on a 10Gbps interface can saturate a CPU core.

Q: How do I map a tracepoint ID to an actual kernel function? A: Inspect the /sys/kernel/debug/tracing/events directory. Each subsystem contains format files that define exactly what data the tracepoint captures.

In the next part of this series, we will dissect how to write custom XDP programs to drop malicious packets before they even reach the network stack.

Part of the series: advanced-ebpf-ops

1. The `bpftool perf` Command That Reveals Hidden Network Latency in Kubernetes (you are here)
2. Building Custom eBPF Programs to Catch Production Anomalies Before They Escalate
3. The 'Invisible Backdoor' Incident: How eBPF Caught a Kernel-Level Supply Chain Attack
4. Is Your eBPF Deployment Actually Secure? A 5-Minute Audit for Production Environments

Further Reading

• https://www.iovisor.org/
• https://www.cilium.io/blog/

---

🚀 Ready to get hands-on? Spin up an interactive AI or Kubernetes Sandbox at Aicademy Labs for free.