Post

Why Your eBPF Network Tracers Are Probably Missing Cross-Node Traffic (And How to Audit Them)

Why Your eBPF Network Tracers Are Probably Missing Cross-Node Traffic (And How to Audit Them): hands-on walkthrough with commands, expected outputs and a

Why Your eBPF Network Tracers Are Probably Missing Cross-Node Traffic (And How to Audit Them)

Run this audit in 5 minutes. If your eBPF dashboards show zero cross-AZ packet loss or latency spikes, you are probably blind to them. BPF programs attached exclusively to container interfaces completely lose visibility once traffic is encapsulated for cluster routing.

TL;DR: Attaching eBPF tracers only to pod interfaces creates massive blind spots for cross-node traffic routed via overlays. This post explains why VXLAN and Geneve encapsulate your packets away from your current traces, and gives you a 5-minute checklist to audit exactly where your eBPF programs are hooked on Linux 7.3 nodes.

What you’ll walk away with:

  • A step-by-step audit to find orphaned or misaligned eBPF attachments.
  • A technical breakdown of how encapsulation breaks naive network tracing.
  • A comparison table to choose the correct attachment point for your workload.

The 5-Minute eBPF Attachment Audit

Run these commands on a single Aicademy worker node running Linux 7.3 to verify where your network tracers are actually sitting.

  1. List all active tc (Traffic Control) eBPF attachments across the node.
    1
    
    bpftool net show
    

    ```text xdp:

tc: veth1a2b3c(92) mactvlan(3) prog_id 401 link_type ingress name trace_rx prog_id 402 link_type egress name trace_tx

1
2
3
4
2. Verify if the physical interface (e.g., `eth0` or `ens192`) has any eBPF programs attached.
```bash
tc filter show dev eth0 ingress
1
filter parent ffff: protocol all pref 49152 bpf chain 0
  1. Check the encapsulation tunnel interfaces (like vxlan.calico or cilium_vxlan) to see if traffic bypasses your physical filters.
    1
    
    ip link show type vxlan
    
    1
    2
    
    5: vxlan.calico: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default
     link/ether 66:d5:ca:5e:32:04 brd ff:ff:ff:ff:ff:ff promiscuity 0
    

Scoring rubric:

  • 0 findings: Tracers attached to physical NICs and tunnels. Healthy visibility.
  • 1-3 findings: Missing physical NIC attachments, relying solely on veth. High risk of blind spots.

Why Do eBPF Programs Miss Cross-Node Encapsulated Traffic?

eBPF network tracers miss cross-node traffic because packets leaving a container are immediately wrapped by the cluster’s Overlay Network (a virtual network built on top of physical infrastructure using encapsulation protocols like VXLAN). If your program attaches inside the namespace, it never sees the final physical transmission.

When a pod at Aicademy sends an API request to a pod on another node, the packet hits the veth pair. Many tracing tools hook here. However, to cross the cluster, Kubernetes networking plugins route that packet through a tunnel interface. This operation completely alters the packet structure.

This encapsulation adds an outer UDP header containing physical node IPs, burying the original packet payload. If you are migrating from iptables logging to high-fidelity eBPF flow tracking, attaching only to veth means your trace terminates before the physical routing layer even touches the packet.

graph TD
    A["Pod App"] -->|"Raw Packet"| B["veth pair"]
    B --> C["eBPF tracer blind spot"]
    C --> D["Tunnel Interface VXLAN"]
    D -->|"Encapsulated Packet"| E["Physical eth0"]
    E --> F["eBPF tracer correct"]

Tracing at the veth boundary is sufficient for intra-node communication, but guarantees blind spots for anything leaving the host.

How Should You Attach eBPF Network Tracers for Full Visibility?

You must attach your eBPF programs to the physical node interfaces (like eth0) using tc filters or XDP, rather than relying exclusively on container namespaces. This ensures your trace captures the fully encapsulated packet immediately before it hits the physical wire.

Attaching to eth0 ingress and egress gives you the absolute truth of what enters and leaves the node. You will capture the VXLAN outer headers, which is critical for debugging cluster-level routing drops. If you suspect packet loss at this boundary, the one eBPF command that pinpoints exactly which container dropped your packet is crucial for cross-node correlation.

Parsing encapsulation headers requires specific care in your C code. Ensure your BPF program parses the outer UDP headers and VXLAN VNI before reading the inner IP headers. Otherwise, you risk hitting the eBPF tail call limit that silently truncates your packet headers if your parsing logic is too deeply nested.

Attachment Point Visibility Performance Overhead Best For
Container veth Pod-level, unencapsulated High (per-pod attachment) App-level L7 debugging
Physical tc Node-level, encapsulated Medium Winner: Full cross-node tracing
Physical XDP Node-level RX only Lowest DDoS mitigation

Default to tracing at physical NIC ingress/egress (eth0) using tc filters unless you strictly need pre-NAT container identities.

Verbose bpftool output for physical NIC attachments
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
[{
    "ifindex": 2,
    "ifname": "eth0",
    "xdp": {
        "ingress": []
    },
    "tc": {
        "ingress": [{
                "id": 412,
                "name": "trace_rx",
                "type": "tc",
                "flags": "cca"
            }
        ],
        "egress": [{
                "id": 413,
                "name": "trace_tx",
                "type": "tc",
                "flags": "cca"
            }
        ]
    }
}]

How Do You Fix Corrupted VXLAN Parsing in eBPF?

Fix corrupted VXLAN parsing by validating the outer UDP destination port (4789) before attempting to offset your pointer into the inner Ethernet frame. Failing to validate the offset bounds explicitly will cause the eBPF verifier to reject your program entirely.

On Linux 7.3, the verifier requires rigorous bounds checking for deeply encapsulated packets. You must calculate the exact byte offset of the VXLAN header and assert it against the data_end pointer. Attempting to parse the inner headers without this strict math guarantees a load failure.

1
2
3
4
5
- void *inner_mac = data + sizeof(struct ethhdr) + sizeof(struct iphdr) + sizeof(struct udphdr) + sizeof(struct vxlanhdr);
- if (inner_mac > data_end) return XDP_DROP;
+ int offset = sizeof(struct ethhdr) + sizeof(struct iphdr) + sizeof(struct udphdr) + 8; // VXLAN is 8 bytes
+ if (data + offset > data_end) return XDP_PASS;
+ void *inner_mac = data + offset;

Always validate both the outer UDP port and the data_end boundary before parsing inner encapsulated packets to satisfy the Linux 7.3 verifier.

Bottom Line

Relying on pod-level eBPF attachments creates a dangerous illusion of network visibility. Cross-node traffic is encapsulated and routed via physical interfaces, completely bypassing veth-bound tracing programs. Run the 5-minute audit today, map your attachment points to your physical NICs, and ensure your parsers can handle VXLAN headers natively.

Next time in the ebpf-net-series, we cover implementing zero-overhead eBPF packet deduplication.

FAQ

How do you check if eBPF programs are attached to a network interface?

Run bpftool net show on the worker node. This lists all attached XDP and tc programs across all active interfaces.

What is the difference between tracing at veth and eth0?

Tracing at veth captures raw, unencapsulated traffic leaving a specific container. Tracing at eth0 captures all traffic leaving the node, including packets wrapped in cluster overlay networks like VXLAN.

Why does my eBPF tracer drop packets when parsing VXLAN?

The eBPF verifier rejects programs that fail to perform strict boundary checks on inner packet headers. You must explicitly check your pointers against data_end after offsetting past the 8-byte VXLAN header.

Can XDP trace egress traffic leaving a worker node?

No. XDP operates strictly on the ingress (RX) path directly in the NIC driver. To trace egress (TX) traffic leaving the node, you must use tc filters.

What port does VXLAN use for cluster overlay networks?

VXLAN encapsulates traffic inside UDP and defaults to destination port 4789. Your eBPF programs must check for this exact port to identify inner payload packets.

Part of the series: ebpf-net-series

  1. 4 Ways to Migrate from iptables Logging to High-Fidelity eBPF Flow Tracking
  2. The eBPF Tail Call Limit That Silently Truncates Your Packet Headers
  3. Why Your eBPF Network Tracers Are Probably Missing Cross-Node Traffic (And How to Audit Them) (you are here)
  4. Cut Your Ingress Debugging Time by 90% After a Cascading Network Failure

Further Reading


🚀 Ready to get hands-on? Spin up an interactive AI or Kubernetes Sandbox at Aicademy Labs for free.

This post is licensed under CC BY 4.0 by the author.