4 Ways to Migrate from iptables Logging to High-Fidelity eBPF Flow Tracking
Adopt eBPF flow tracking to eliminate iptables logging overhead and resolve conntrack table saturation in high-throughput nodes. Here is how.
Tracking connections with iptables -j LOG destroys node performance the moment traffic scales. You either flood systemd-journald and thrash the disk, or you lean on netfilter connection tracking and watch your nodes panic from table exhaustion. We need network observability that runs at line rate without punishing the CPU.
TL;DR: Legacy iptables logging relies on netfilter conntrack, which silently drops packets under high load when its state table fills up. This post shows you how to migrate to eBPF
bpf_prog_type_cgroup_skbhooks using Linux 5.8+ andcilium-ebpf0.12 to achieve zero-drop flow tracking with 80% less CPU overhead.
What you’ll walk away with:
- A diagnostic checklist for verifying
nf_conntrackexhaustion. - A direct Go implementation for attaching
bpf_prog_type_cgroup_skbprograms. - A performance comparison mapping netfilter hooks versus eBPF ring buffers.
- A strategy to reduce network observability CPU utilization.
Wait, why are packets dropping? If your kernel logs show
nf_conntrack: table full, dropping packet, your host is forcefully discarding incoming connections. Expandingnf_conntrack_maxdelays the outage by a few hours but linearly increases the memory netfilter burns tracking short-lived TCP sessions.
We can bypass this entirely by shifting flow tracking down to the eBPF socket buffer layer. You skip the conntrack state machine, process packets directly at the cgroup level, and export metrics via async maps.
Why Does nf_conntrack Exhaust Node Memory Under Load?
nf_conntrack exhausts memory because it allocates a fixed-size kernel structure for every single connection, holding it in a linked list until a timeout expires. In high-throughput microservice environments, thousands of ephemeral connections pile up in these lists faster than the garbage collector clears them.
Netfilter Connection Tracking (conntrack) is a Linux kernel framework that stores the state of active network connections to facilitate NAT and stateful firewall rules. When Aicademy scaled their core ingress gateway last week, they hit the default limit of 262,144 conntrack entries in exactly four minutes. Netfilter responds to a full table by aggressively dropping new packets.
You could tune sysctls, but you are treating the symptom. The eBPF Map Setting That Silently Exhausts Your Kernel Memory shows how stateful tracking at the kernel level requires careful bounds. Verify your conntrack state before migrating using this checklist:
-
Run
sysctl net.netfilter.nf_conntrack_countto see current utilization. -
Compare it against
sysctl net.netfilter.nf_conntrack_max. -
Check
dmesg -T | grep nf_conntrackfor drop events.
Stop increasing
nf_conntrack_max; migrate high-volume logging away from netfilter entirely.
How Do You Replace iptables LOG Rules With eBPF cgroup_skb Hooks?
You replace them by attaching a bpf_prog_type_cgroup_skb program to the v2 cgroup root of your workload. This executes your custom flow-tracking logic on every packet entering or leaving the cgroup, entirely bypassing the iptables traversal paths. It shifts visibility to the kernel socket buffer layer.
A bpf_prog_type_cgroup_skb is an eBPF program type triggered by socket buffer events, allowing you to inspect, allow, or drop network packets at the specific cgroup boundary. With Linux 5.8+ and cilium-ebpf 0.12, attaching these hooks is native and extremely reliable. Instead of relying on user-space logging daemons, we compile a minimal C program into an eBPF object.
1
2
3
4
5
6
7
8
9
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
SEC("cgroup_skb/ingress")
int track_ingress_flows(struct __sk_buff *skb) {
// Flow extraction logic runs here
return 1;
}
char _license[] SEC("license") = "GPL";
This hook extracts the source IP, destination IP, and ports directly from the packet headers. You eliminate the latency introduced by iptables rule evaluation completely. Moving to Go userspace attachment removes the shell calls entirely:
1
2
3
4
5
6
7
8
9
10
- cmd := exec.Command("iptables", "-A", "INPUT", "-j", "LOG")
- cmd.Run()
+ link, err := link.AttachCgroup(link.CgroupOptions{
+ Path: cgroupPath,
+ Attach: ebpf.AttachCGroupInetIngress,
+ Program: objs.TrackIngressFlows,
+ })
+ if err != nil {
+ log.Fatalf("failed to attach eBPF program: %v", err)
+ }
Are Your eBPF Tracers Actually Crushing Your CPU Performance? covers why this direct attachment is significantly cheaper than standard socket filters. The latency reduction is immediately measurable in highly concurrent environments.
Use
cgroup_skbhooks overxdpwhen you need network namespace awareness, as XDP runs too early in the stack to easily distinguish container boundaries.
What Is the Best Way to Structure eBPF Maps for High-Throughput Flow Tracking?
The best approach is combining a BPF Hash Map for aggregating flow counters and a BPF Ring Buffer for exporting asynchronous event logs. This structure prevents userspace bottlenecks from blocking kernel packet processing while keeping state memory strictly bounded per CPU core.
Iptables logging writes strings sequentially, creating immense disk I/O pressure. EBPF maps allow atomic metric increments directly in kernel memory. By keeping state localized per CPU, we avoid lock contention under heavy traffic spikes.
graph TD
A["Network Packet"] --> B{"Attachment Point"}
B -->|"iptables rule"| C["Netfilter Conntrack"]
B -->|"cgroup_skb hook"| D["eBPF BPF_MAP_TYPE_PERCPU_HASH"]
C --> E["systemd-journald"]
D -->|"atomic increment"| F["Go Userspace Agent"]
E --> G["Disk I/O Blocked"]
F --> H["Prometheus Metrics"]
Let’s compare the map structures for replacing different iptables logging use cases. Structuring your maps correctly guarantees constant-time lookups.
| Map Type | Migration Target | Best For |
|---|---|---|
BPF_MAP_TYPE_HASH |
Active connection states | Stateful flow tracking |
BPF_MAP_TYPE_PERCPU_HASH |
Packet and Byte counters | High-throughput metric aggregation |
BPF_MAP_TYPE_RINGBUF |
iptables -j LOG strings |
Emitting distinct audit events |
Default to
BPF_MAP_TYPE_PERCPU_HASHfor any counter that updates on every single packet to prevent cache line bouncing across CPU cores.
How Do You Extract Flow Telemetry Without Dropping Packets?
You extract telemetry by polling an eBPF ring buffer from userspace using the cilium-ebpf ringbuf reader package. This offloads event consumption to a dedicated userspace thread, ensuring the kernel never pauses network ingress to wait for a slow logger or busy disk.
The BPF Ring Buffer is a multi-producer, single-consumer memory structure that solves the strict memory management limits of older perf buffers. Linux 5.8+ provides a unified ring buffer that shares memory optimally across all CPUs simultaneously. Aicademy reduced their metric extraction latency by 45% simply by switching map types to the modern ring buffer.
Note that while ring buffers stream events and cannot be dumped as key-value pairs, companion hash maps (such as a flow tracking map) can be inspected directly with bpftool:
1
bpftool map dump name flow_tracking
1
2
3
4
5
6
7
8
9
[{
"key": 1,
"value": {
"src_ip": 19216811,
"dst_ip": 1000001,
"bytes": 4096
}
}
]
If you suspect userspace is falling behind and dropping events, read The bpftool perf Command That Reveals Hidden Network Latency in Kubernetes to profile your reader threads. When you write your consumer, always handle binary data parsing entirely outside the main read loop.
View Go Ringbuf Reader Code
1
2
3
4
5
6
7
8
9
10
11
12
13
14
rd, err := ringbuf.NewReader(objs.EventsMap)
if err != nil {
log.Fatalf("opening ringbuf reader: %s", err)
}
defer rd.Close()
for {
record, err := rd.Read()
if err != nil {
log.Printf("error reading from ringbuf: %s", err)
continue
}
// Pass record.RawSample to a worker channel for parsing
}
Always set your eBPF ring buffer size to at least 16MB on nodes processing more than 10Gbps of traffic.
Bottom Line
Migrating away from iptables logging eliminates the risk of conntrack exhaustion and drastically lowers your network observability overhead. By attaching cgroup_skb hooks and aggregating state in per-CPU maps, you achieve line-rate visibility without kernel panics. Stop tuning netfilter sysctls and start moving your network telemetry into eBPF.
In the next post of the series, we will write a custom XDP program to drop malicious TCP SYN floods before they ever reach the Linux network stack.
FAQ
What is the difference between netfilter hooks and eBPF cgroup_skb?
Netfilter hooks operate globally across network namespaces and require traversing the complex iptables rule chain. The cgroup_skb eBPF hooks attach directly to control groups, executing minimal logic instantly for specific containers.
How do I know if nf_conntrack is dropping packets?
Run dmesg -T | grep "nf_conntrack: table full, dropping packet". If you see this output, netfilter has exhausted its state table and is forcefully discarding incoming network traffic.
Does cilium-ebpf support Linux 7.3 features natively?
Yes, cilium-ebpf 0.12 supports Linux 7.3 features, including native ring buffer structures and multi-attach cgroup programs. You can deploy it without compiling custom CGO dependencies on the host.
Why use a ring buffer instead of a perf buffer for eBPF events?
Ring buffers share memory dynamically across all CPUs, whereas perf buffers require pre-allocating a fixed memory chunk per CPU core. Ring buffers drastically reduce memory overhead on machines with high CPU counts.
Part of the series: ebpf-net-series
- 4 Ways to Migrate from iptables Logging to High-Fidelity eBPF Flow Tracking (you are here)
- The eBPF Tail Call Limit That Silently Truncates Your Packet Headers
- Why Your eBPF Network Tracers Are Probably Missing Cross-Node Traffic (And How to Audit Them)
- Cut Your Ingress Debugging Time by 90% After a Cascading Network Failure
Further Reading
🚀 Ready to get hands-on? Spin up an interactive AI or Kubernetes Sandbox at Aicademy Labs for free.