The 'Invisible Backdoor' Incident: How eBPF Caught a Kernel-Level Supply Chain Attack
Our story begins with an unexplained exfiltration of sensitive data from a hardened production server, baffling our security teams.
We were bleeding customer PII from a hardened production database node, and our SIEM was completely silent. Standard security agents reported zero anomalous processes, while network monitors screamed about outbound gigabytes. The adversary wasn’t hiding in user space; they were executing instructions directly inside the kernel.
TL;DR: When attackers compromise the kernel, user-space audit tools become blind to data exfiltration and persistence mechanisms. This post demonstrates how to use Falco 0.45.0 and
bpftoolon linux-kernel 7.2 to hunt down malicious eBPF rootkits. You will learn how to extract suspicious kernel hooks and deploy rules to block unauthorized system calls immediately.
What you’ll walk away with:
- A methodology to bypass compromised user-space logs and query the kernel directly.
- The exact
bpftoolcommands to extract hidden bytecode from memory. - A custom Falco rule to alert on unauthorized
bpf()syscalls. - A post-incident hardening checklist for node-level security.
How Do You Detect a Kernel-Level Supply Chain Attack?
You detect a kernel-level supply chain attack by bypassing user-space telemetry and directly auditing kernel execution contexts using an independent runtime security engine like Falco. Because compromised kernel modules alter user-space audit logs, administrators must inspect loaded eBPF programs and raw syscalls to find the discrepancy.
The incident started on a Tuesday at Aicademy. We had deployed a locked-down microservice architecture, heavily influenced by our push for DevSecOps Shift-Left in Practice: Embedding Security from Inception. Despite strict image signing and immutability, a compromised third-party logging agent slipped a malicious shared object onto the host OS.
The attackers leveraged this foothold to load malicious eBPF code. eBPF is a kernel technology that allows custom, sandboxed programs to run directly within the operating system core without modifying kernel source code. They used this access to siphon off authentication tokens, a catastrophic failure since we were prepping for Preparing for Q-Day: The DevOps Guide to Post-Quantum Cryptography and strictly monitoring cryptographic asset access.
Always assume user-space telemetry is compromised if network egress metrics do not match host-level process logs.
Why Did Standard Security Agents Miss the Exfiltration?
Standard security agents missed the exfiltration because they rely on user-space APIs like netlink or auditd, which the kernel-level attacker had already hooked and manipulated. When a malicious kernel module intercepts these system calls, it simply drops the telemetry packets before the user-space agent ever sees them.
Our initial triage involved inspecting syslog and auditd. The logs were entirely pristine. We realized standard approaches for Securing Kubernetes in Production: Advanced Strategies for 2026 were insufficient because the underlying host kernel could no longer be trusted.
The malicious eBPF program intercepted the tcp_sendmsg kernel function. Before our security tooling could log the network socket activity, the hook filtered out the attacker’s target IP. The security agent essentially wore a blindfold fabricated by the kernel itself.
graph TD
A["Application Process"] -->|"Socket write"| B["Kernel: tcp_sendmsg"]
B --> C{"Is IP attacker?"}
C -->|"Yes"| D["Drop Audit Packet"]
C -->|"No"| E["Send to Audit/Syslog"]
D --> F["Send to Network Interface"]
E --> F
To visualize the architectural gap, review how data sources differ when an attacker owns the kernel:
| Feature | User-Space Agents (Auditd) | Kernel-Space Agents (eBPF/Falco) | Winner |
|---|---|---|---|
| Data Source | Application logs, /proc, standard syscalls |
Direct kernel execution context | Kernel-Space |
| Bypass Difficulty | Trivial (overwrite logs, drop packets) | High (requires altering raw kernel memory) | Kernel-Space |
| Performance Overhead | High context switching costs | Near-zero (JIT compiled in kernel) | Kernel-Space |
Never rely solely on in-band logging; forward network flow data from the hypervisor or physical switch level to maintain ground truth.
How Do You Inspect Malicious eBPF Programs in Production?
You inspect malicious eBPF programs in production by querying the kernel directly using bpftool prog show to list all loaded programs, followed by bpftool prog dump jited to extract their assembly instructions. This reveals hidden network filters or system call hooks that evade standard process monitors.
Knowing the kernel was lying to user space, we dropped to the lowest level available. Running linux-kernel 7.2 gave us access to advanced BTF (BPF Type Format) debug capabilities. We ran a blanket scan of all active eBPF programs on the Aicademy host.
1
bpftool prog show
1
2
3
4
5
6
21: cgroup_skb tag 6deef7357e7b4530 gpl
loaded_at 2026-04-12T10:14:02-0400 uid 0
xlated 296B jited 164B memlock 4096B
45: kprobe name hidden_hook tag a52a921d22 gpl
loaded_at 2026-04-12T14:22:10-0400 uid 0
xlated 824B jited 442B memlock 4096B
Program ID 45 immediately stood out. We did not deploy any kprobe programs named hidden_hook in this cluster. To understand exactly what it was doing, we dumped the Just-In-Time (JIT) compiled instructions directly from memory.
View the malicious JIT instruction dump
1
bpftool prog dump jited id 45
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
bpf_prog_a52a921d22_hidden_hook:
0: nopl 0x0(%rax,%rax,1)
5: xchg %ax,%ax
7: push %rbp
8: mov %rsp,%rbp
b: sub $0x28,%rsp
12: mov %rbx,0x0(%rsp)
16: mov %r13,0x8(%rsp)
1b: mov %r14,0x10(%rsp)
20: mov %r15,0x18(%rsp)
25: xor %eax,%eax
27: mov %rax,0x20(%rsp)
2c: mov 0x70(%rdi),%r13
30: cmp $0x5c12a8c0,%r13 # Attacker IP: 192.168.18.92
37: je 0x000000000000005a
39: xor %eax,%eax
3b: jmp 0x0000000000000062
# ... further instructions bypassing audit hooks ...
The hex value 0x5c12a8c0 translated to an IP address matching our mysterious outbound traffic. The rootkit was intercepting kernel execution, comparing the destination IP, and selectively bypassing the audit frameworks.
Audit the output of
bpftool prog showregularly and map every loaded program back to a known infrastructure tool.
How Do You Block Unauthorized bpf() System Calls?
You block unauthorized bpf() system calls by deploying Falco with custom rules that alert on or terminate any process attempting to load BPF programs outside of your approved infrastructure tooling. This locks down the kernel against post-exploitation persistence mechanisms.
We needed to detect and kill these unauthorized loads instantly. We upgraded to Falco 0.45.0, which includes native support for deep kernel system call inspection. By default, the Linux kernel sets kernel.unprivileged_bpf_disabled=2 in modern distributions, but attackers with root exploit the bpf() API to load raw eBPF bytecode.
We deployed a specific rule to catch the bpf system call originating from non-standard binaries. We targeted the exact bpf_prog_load command.
1
# falco_rules.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
- rule: Disallow shell in container
- desc: Alert on unexpected bash execution
+ rule: Detect Unauthorized eBPF Program Load
+ desc: Detect processes outside of the approved list loading eBPF programs
+ condition: >
+ evt.type = bpf and
+ evt.dir = < and
+ evt.arg.cmd = BPF_PROG_LOAD and
+ not proc.name in (cilium-agent, falco, datadog-agent)
+ output: >
+ Unauthorized eBPF program loaded (user=%user.name command=%proc.cmdline)
+ priority: CRITICAL
+ tags: [kernel, mitre_defense_evasion]
Implementing this immediately caught the compromised logging agent attempting to reload its backdoor after we rebooted the node. We built a strict checklist to prevent a recurrence across our fleets.
-
Restrict
CAP_SYS_ADMINandCAP_BPFacross all Kubernetes pod security policies. -
Pin
kernel.unprivileged_bpf_disabled=2in/etc/sysctl.conf. -
Deploy Falco rules targeting the
BPF_PROG_LOADsyscall command. - Verify image signatures for all third-party DaemonSets using eBPF natively.
Whitelist your authorized eBPF tools explicitly; any other binary attempting a
bpf()syscall is overwhelmingly likely to be hostile.
Bottom Line
Trusting user-space telemetry on a compromised kernel is a guaranteed path to failure. You must instrument your monitoring at the kernel level using independent tools like Falco and proactively hunt for malicious eBPF hooks. Deploy unauthorized bpf() syscall detection today before an attacker builds an invisible backdoor in your infrastructure.
Next up in the advanced-ebpf-ops series: building custom XDP programs to drop malicious packets before they hit the Linux network stack.
FAQ
How do you verify which processes are loading eBPF programs?
Use Falco with a rule targeting evt.type = bpf and evt.arg.cmd = BPF_PROG_LOAD. You can also trace the sys_bpf system call natively using bpftrace -e 'tracepoint:syscalls:sys_enter_bpf { printf("%s\n", comm); }'.
What is the difference between kprobes and tracepoints in eBPF?
Kprobes allow dynamic attachment to almost any kernel function, making them powerful for attackers hooking arbitrary execution paths. Tracepoints are static, pre-defined hooks compiled into the kernel by developers, offering better stability but less flexibility.
Can eBPF programs modify system call arguments?
Standard eBPF programs cannot directly modify syscall arguments to alter execution, but they can use bpf_override_return() if compiled with specific kernel configurations. Attackers often use bpf_probe_write_user() to overwrite user-space memory, effectively hijacking application data.
How do you remove a malicious eBPF program from the kernel?
You cannot forcefully unload an eBPF program directly via bpftool if an active process holds a file descriptor to it. You must identify the malicious process holding the handle using bpftool prog show combined with lsof, then kill the user-space process to release the program.
Part of the series: advanced-ebpf-ops
- The `bpftool perf` Command That Reveals Hidden Network Latency in Kubernetes
- Building Custom eBPF Programs to Catch Production Anomalies Before They Escalate
- The 'Invisible Backdoor' Incident: How eBPF Caught a Kernel-Level Supply Chain Attack (you are here)
- Is Your eBPF Deployment Actually Secure? A 5-Minute Audit for Production Environments
Further Reading
🚀 Ready to get hands-on? Spin up an interactive AI or Kubernetes Sandbox at Aicademy Labs for free.