Cut Your Ingress Debugging Time by 90% After a Cascading Network Failure
Cut Your Ingress Debugging Time by 90% After a Cascading Network Failure: hands-on walkthrough with commands, expected outputs and a checklist. Learn how.
Everything looked perfectly normal on the Grafana dashboard until the entire us-east-1a availability zone flatlined for ingress traffic. Our API gateway pods were green, but client requests were timing out with zero logs at the application layer.
TL;DR: Undocumented kernel route changes can silently blackhole ingress traffic before it ever hits a pod namespace. By deploying the eBPF tool
pwru, we traced dropped packets to exact kernel functions and resolved a cascading network failure in minutes. This post gives you the exact tracing commands and deployment strategy to pinpoint internal packet drops.
What you’ll walk away with:
- How to deploy
pwruto trace packet traversal through the Linux kernel. - The exact CLI parameters to filter eBPF traces by destination IP and port.
- A diagnosis framework for determining if packets drop at the NIC, routing layer, or namespace boundary.
- A quick rollback diff for fixing route-based blackholes.
Why Do Ingress Packets Vanish Before Hitting the Application Layer?
Ingress packets vanish before the application layer primarily due to misconfigured routing rules, malformed packet headers, or strict kernel-level firewall drops executing before socket delivery. The Linux kernel processes packets through multiple subsystems, meaning a single incorrect routing table entry will silently drop traffic without application-level logs.
pwru (Packet, Where Are You?) is an eBPF-based network tracing tool that hooks into thousands of kernel functions to show you exactly where a packet is dropped. It attaches transparent tracepoints to the kernel’s networking stack, tracking the lifecycle of the fundamental sk_buff data structure.
We run a high-throughput video encoding API for Aicademy. Last Tuesday, incoming video streams to our primary cluster dropped to absolute zero. We checked our load balancers and even reviewed our recent 4 Ways to Migrate from iptables Logging to High-Fidelity eBPF Flow Tracking deployment, but saw no anomalies.
1
curl -v http://10.100.5.50:8080/health
1
2
3
4
* Trying 10.100.5.50:8080...
* connect to 10.100.5.50 port 8080 failed: Connection timed out
* Failed to connect to 10.100.5.50 port 8080: Connection timed out
* Closing connection 0
Traditional monitoring tools only watch the application or the very edge of the network. They fail completely when the kernel itself silently discards packets midway through the internal networking stack. You need deep, function-level visibility to identify exactly which subsystem is terminating the connection.
Run
ip route showimmediately when you suspect kernel-level packet drops before resorting to complex tracers.
How Do You Trace Packet Drops in the Linux Kernel?
You trace packet drops in the kernel by attaching eBPF programs to the kfree_skb tracepoint or by using pwru to track the exact kernel function where the sk_buff structure is freed. This approach bypasses standard userspace limitations and directly exposes network stack execution paths in real time.
Standard network monitoring confirmed packets were hitting the host’s physical network interface. The application namespace, however, remained entirely empty. We suspected a routing issue, similar to how The eBPF Tail Call Limit That Silently Truncates Your Packet Headers causes invisible network failures.
We needed granular visibility into the kernel’s routing decisions without installing heavy diagnostic suites. We immediately deployed pwru 1.0 on our Linux 7.3 nodes. The CLI allows specific filtering, which is strictly required to prevent server crashes from eBPF map overflows on busy interfaces.
1
pwru --filter-dst-ip 10.100.5.50 --filter-dst-port 8080 --output-tuple
1
2
3
1 / 1 [eth0] 10.100.5.50:8080 -> 10.100.5.50:8080 ip_rcv_core (1)
1 / 1 [eth0] 10.100.5.50:8080 -> 10.100.5.50:8080 ip_route_input_slow (1)
1 / 1 [eth0] 10.100.5.50:8080 -> 10.100.5.50:8080 kfree_skb_reason (1)
The output was unambiguous. The packet successfully entered ip_rcv_core but immediately died at ip_route_input_slow before triggering a kfree_skb_reason drop event. The kernel routing table was aggressively discarding the socket buffer instead of forwarding it.
flowchart LR
A["Host NIC"] -->|"ip_rcv_core"| B["Routing Layer"]
B -->|"ip_route_input_slow"| C["Drop (kfree_skb)"]
B -.->|"Normal flow"| D["Pod Namespace"]
Always filter your
pwrucommand by IP and port to avoid overwhelming the kernel and your terminal with trace data.
What Causes the Kernel to Drop Packets at ip_route_input_slow?
The kernel drops packets at ip_route_input_slow when the destination IP does not match a valid local route, or when reverse path filtering detects an asymmetric route. If the kernel cannot map the packet’s destination to a local interface or forwarding rule, it immediately frees the socket buffer.
Our Aicademy overlay network relies on explicit routing tables to direct traffic from the physical NIC into the correct container namespace. A recent infrastructure deployment had accidentally overwritten the local routing table. The node literally did not know where to send traffic destined for the 10.100.0.0/16 subnet.
We dumped the routing cache to confirm the missing entries. The verbose output proved our overlay network interface was completely ignored by the kernel.
View verbose routing debug output
1
2
3
4
IPv4 routing cache:
0.0.0.0/0 via 192.168.1.1 dev eth0
10.100.5.0/24 dev eth0 scope link
# Missing overlay route for 10.100.0.0/16 via cilium_host
We updated the routing configuration via our infrastructure-as-code repository. Pushing the fix took seconds once we isolated the exact failure domain.
1
2
3
4
- route_target: "0.0.0.0/0"
- interface: "eth0"
+ route_target: "10.100.0.0/16"
+ interface: "cilium_host"
Traffic immediately recovered and the API gateway pods resumed processing video streams. Using an eBPF tracing tool cut our standard incident response time down to just five minutes.
Use version control for all routing configurations to quickly identify and revert accidental overlay network drops.
How Does pwru Compare to Traditional Packet Capture Tools?
pwru is vastly superior to traditional tools like tcpdump or Wireshark for deep kernel debugging because it identifies the exact line of kernel code dropping a packet. While tcpdump only proves a packet arrived at an interface, eBPF tracers reveal internal routing, firewalling, and forwarding logic.
This distinction matters especially when diagnosing whether Why Your eBPF Network Tracers Are Probably Missing Cross-Node Traffic (And How to Audit Them). When debugging our production nodes at Aicademy, we abandoned tcpdump for internal drops years ago. We now follow a strict, standardized incident response checklist.
-
Verify physical arrival at the NIC using
tcpdump. -
Run
pwruwith--filter-dst-ipand--filter-dst-port. -
Check output for
kfree_skborkfree_skb_reason. - Map the function name immediately preceding the drop to kernel documentation.
Traditional tools force engineers to guess why a packet vanished. By tracking the sk_buff struct directly, you remove all ambiguity.
| Feature | tcpdump | iptables trace | pwru | Winner |
|---|---|---|---|---|
| Kernel Overhead | High (packet copying) | Medium | Low (eBPF hooks) | pwru |
| Function Tracing | None | Netfilter hooks only | Full kernel stack | pwru |
| Installation Speed | Instant (pre-installed) | Slow (requires rules) | Fast (single binary) | tcpdump |
| Best For | Edge perimeter checks | Legacy firewall rules | Deep internal drops | pwru |
Default to
pwruoveriptables-tracewhenever debugging unexplainable packet loss inside the Linux host.
Bottom Line
Network debugging without eBPF is like trying to fix a car engine by only looking at the exhaust pipe. By leveraging pwru 1.0 on Linux 7.3, you gain immediate, function-level visibility into exactly where and why the kernel discards traffic. Deploy this tooling to your incident response arsenal today, and stop wasting hours guessing about missing routing entries.
FAQ
What are the minimum requirements to run pwru?
Running pwru 1.0 requires a Linux kernel version of 5.15 or newer with BTF (BPF Type Format) enabled, though this guide targets Linux 7.3 for optimal compatibility. You also need root privileges to load the eBPF programs into the kernel.
How do you filter pwru output for a specific protocol like UDP?
You can pass the protocol flag directly via the CLI. Run pwru --filter-proto udp to drop all TCP or ICMP traffic from your trace, ensuring you only see targeted packets.
Why does the kernel drop packets with the kfree_skb function?
The kfree_skb function is the kernel’s universal method for freeing a packet buffer (sk_buff) when it is explicitly discarded. This happens due to missing routing table entries, failed checksums, or aggressive firewall drops.
Is pwru safe to run on production workloads?
Yes, because it relies on the eBPF verifier, pwru operates with guaranteed safety checks and will not crash the kernel. However, you must apply strict IP and port filters to prevent performance degradation on high-throughput servers.
In the next part of this series, we will dissect how to write custom XDP programs to drop malicious traffic before it even allocates an sk_buff.
Part of the series: ebpf-net-series
- 4 Ways to Migrate from iptables Logging to High-Fidelity eBPF Flow Tracking
- The eBPF Tail Call Limit That Silently Truncates Your Packet Headers
- Why Your eBPF Network Tracers Are Probably Missing Cross-Node Traffic (And How to Audit Them)
- Cut Your Ingress Debugging Time by 90% After a Cascading Network Failure (you are here)
Further Reading
🚀 Ready to get hands-on? Spin up an interactive AI or Kubernetes Sandbox at Aicademy Labs for free.
