The eBPF Map Setting That Silently Exhausts Your Kernel Memory
The eBPF Map Setting That Silently Exhausts Your Kernel Memory: hands-on walkthrough with commands, expected outputs and a checklist. Learn how.
You are paged at 2 AM because your primary Aicademy database crashed. User-space memory metrics look perfectly fine, but the kernel Out-Of-Memory (OOM) killer just terminated your most critical Postgres process. You have just become a victim of invisible kernel memory pinning.
TL;DR: Unbounded eBPF hash maps pin kernel memory that bypasses standard user-space monitoring entirely. A single runaway observability agent tracking network connections can silently consume gigabytes of RAM until the kernel panic-kills critical processes. This post shows you how to detect invisible eBPF memory hogs and enforce strict BPF map limits on Linux 6.8.
What you’ll walk away with:
- A targeted
bpftoolworkflow to expose hidden kernel memory allocations. - The exact C configuration to convert standard hash maps into self-cleaning LRU structures.
- A scoring rubric to evaluate your tracing agents’ memory safety.
Why Does top Miss eBPF Memory Consumption?
top tracks user-space virtual and resident set sizes, completely ignoring memory pinned by the kernel. An eBPF map is a strictly typed key-value data structure residing entirely in kernel space that allows data sharing between eBPF programs and user-space applications. Because these structures allocate via vmalloc or the slab allocator, their footprint remains completely invisible to standard user-space observability tools.
When a privileged observability agent loads an eBPF program, its map memory is charged to the process’s memory cgroup (memcg). If that agent runs as root outside of a constrained systemd slice, the memory limit effectively equals the total host RAM. You can trace network packets and store every unique IP tuple, but without explicit eviction, the kernel locks that memory permanently.
According to the kernel BPF map documentation, standard hash maps pre-allocate their full size by default to guarantee predictable latency. On Linux 6.8, a 10-million entry flow map easily consumes gigabytes of unpageable kernel RAM the second you load the program. Standard monitoring alarms will remain green while the server starves.
Check
slabtoporgrep VmallocUsed /proc/meminfoto catch massive kernel-side memory allocations during a suspected leak.
How Do You Expose Hidden eBPF Memory Hogs?
You expose hidden eBPF memory hogs by querying the kernel directly using bpftool map show. This command forces the kernel to report every active map, its type, its maximum entry limit, and its exact memory footprint in bytes.
Run the following command on your Linux 6.8 host to inspect the currently loaded maps:
1
sudo bpftool map show
1
2
3
27: hash name flow_map flags 0x0
key 16B value 32B max_entries 10000000 memlock 671088640B
pids tracer_agent(1423)
Notice the memlock value. The map flow_map has pre-allocated roughly 640MB of RAM for 10 million entries. If multiple such maps exist, or if an agent dynamically generates them, you will rapidly exhaust physical memory. This is exactly Are Your eBPF Tracers Actually Crushing Your CPU Performance? and memory limits fail in production.
If you spot a suspicious map, you can dump its contents to understand what the tracer is hoarding. Be cautious, as dumping a massive map can briefly stall the kernel.
Click to view a safe map dump command and its raw output
1
sudo bpftool map dump id 27 limit 5
1
2
3
4
5
6
key:
c0 a8 01 0a 00 00 00 00 08 00 00 00 00 00 00 00
value:
01 00 00 00 00 00 00 00 a4 1b 00 00 00 00 00 00
...
Found 5 elements
Sort maps by memory usage immediately using
bpftool map show | grep memlockto identify the worst offenders.
How Do You Prevent eBPF Maps From Crashing Your Server?
The best way to prevent eBPF maps from crashing your server is by replacing BPF_MAP_TYPE_HASH with BPF_MAP_TYPE_LRU_HASH and lowering max_entries. This guarantees the kernel automatically prunes stale entries without requiring complex user-space garbage collection loops.
Consider a custom tracer tracking database connections. A standard hash map requires the user-space component to periodically iterate and delete inactive connections. If the user-space process stalls or encounters a sudden spike in unique IPs, the map fills up. The tracer drops new events, or worse, triggers out-of-memory kernel panics.
Here is how you fix the core C definition using libbpf:
1
2
3
4
5
6
7
8
struct {
- __uint(type, BPF_MAP_TYPE_HASH);
+ __uint(type, BPF_MAP_TYPE_LRU_HASH);
__type(key, struct flow_key);
__type(value, struct flow_metrics);
- __uint(max_entries, 10000000);
+ __uint(max_entries, 131072);
} flow_map SEC(".maps");
BPF_MAP_TYPE_LRU_HASH automatically evicts the least recently used entries. The memory usage hits a hard, predictable ceiling. This matches the exact diagnostic strategy discussed in The One eBPF Command That Pinpoints Exactly Which Container Dropped Your Packet.
graph TD
A["Network Traffic"] -->|"Packet Event"| B["eBPF Program"]
B -->|"Insert Flow"| C["LRU Hash Map"]
C -->|"Evicts Old"| C
C -->|"Pins Capped RAM"| D["Kernel Memory"]
E["System Monitor"] -.->|"Blind to"| D
Review your tracing architecture against this memory safety checklist:
- You exclusively use LRU maps for tracking transient data like network flows.
-
You calculate the absolute maximum memory footprint:
(key_size + value_size) * max_entries. - You strictly bind your tracing agent to a systemd memory slice.
For workloads requiring high concurrency, choose your map types carefully. The following table compares common map structures and their memory implications.
| Map Type | Pre-allocation | Eviction | Winner For |
|---|---|---|---|
BPF_MAP_TYPE_HASH |
Yes | Manual | Static lookup tables |
BPF_MAP_TYPE_LRU_HASH |
Yes | Automatic | Dynamic flow tracking |
BPF_MAP_TYPE_PERCPU_HASH |
Yes (per core) | Manual | High-throughput counters |
If you ignore these boundaries, Is Your eBPF Deployment Actually Secure? A 5-Minute Audit for Production Environments will fail your infrastructure on availability risks alone.
Default to
BPF_MAP_TYPE_LRU_HASHunless your logic absolutely dictates that failing to insert a new key is safer than dropping an old one.
Bottom Line
Do not trust user-space metrics when kernel instrumentation is involved. Audit your eBPF programs, swap basic hash maps for LRU maps, and drastically lower your max_entries limits to reflect reality. Capping your agent’s systemd memory slice provides the final backstop against kernel memory exhaustion.
FAQ
How do you calculate the memory size of an eBPF map?
You calculate it by multiplying the maximum number of entries by the sum of the key size, value size, and internal kernel overhead per element. Run bpftool map show to see the exact memlock byte count the kernel actually reserves.
Why does the Linux OOM killer target user-space apps instead of eBPF programs?
The OOM killer targets processes based on an oom_score, prioritizing user-space applications consuming large amounts of RAM. Because eBPF maps consume pinned kernel memory, the kernel considers them part of its own unkillable infrastructure.
Can you dynamically resize an eBPF map after it is loaded?
No. The max_entries value is strictly defined at load time and cannot be changed dynamically. You must unload the BPF program, update the configuration, and load a new map.
Does BPF_MAP_TYPE_ARRAY pre-allocate memory like hash maps?
Yes, array maps allocate their entire memory footprint immediately upon creation. They provide the fastest possible access times but require strict attention to the bounds you set in your C definitions.
How do you inspect the contents of an eBPF map?
Use the command sudo bpftool map dump id <map_id> to print the raw hex values of keys and values. For formatted output, you must write a custom user-space tool that parses the specific C structs your BPF program uses.
Stay tuned for the next part of the advanced-ebpf-observability series, where we expose the specific raw tracepoints needed to debug kernel scheduling latency.
Part of the series: advanced-ebpf-observability
- The One eBPF Command That Pinpoints Exactly Which Container Dropped Your Packet
- Are Your eBPF Tracers Actually Crushing Your CPU Performance?
- The eBPF Map Setting That Silently Exhausts Your Kernel Memory (you are here)
Further Reading
🚀 Ready to get hands-on? Spin up an interactive AI or Kubernetes Sandbox at Aicademy Labs for free.