Why Linux Ops Must Master eBPF in 2026

Search "Linux performance diagnosis" on Google, and 8 out of the top 10 tutorials still teach top, iostat, and netstat. These tools were born 20 years ago — they only show symptoms, never root causes.

In 2026, eBPF (Extended Berkeley Packet Filter) has evolved from "kernel magic" to standard equipment for frontline ops teams. Netflix, Meta, and Cloudflare use eBPF across tens of thousands of nodes for:

  • Real-time network latency diagnosis: pinpoint exactly where TCP retransmissions happen (application layer or kernel?)
  • CPU flame graphs: dynamically generate function-call-stack performance profiles without restarting services
  • Security auditing: trace every process creation, file access, and network connection with zero misses
  • Container network acceleration: Cilium replaces kube-proxy, improving K8s service mesh performance by 30%

Key data: According to the CNCF 2026 survey, 73% of production Kubernetes clusters have deployed eBPF-related tools (Cilium, Tetragon, Pixie), a 180% increase from 2024.

This article walks you through 5 production-grade practical cases to master eBPF observability from scratch. Case #3 (dynamic user-space function tracing) is a hidden skill 90% of ops engineers don't know.

Chapter 1: eBPF Fundamentals — Core Concepts in 3 Minutes

What Is eBPF?

eBPF is a programmable sandbox inside the Linux kernel. You can run custom programs in kernel space without modifying kernel source code or loading kernel modules.

Analogy: - Traditional way: Want to monitor TCP handshakes? Modify kernel code → recompile → reboot server - eBPF way: Write a 20-line C program → load into kernel → takes effect immediately, no reboot

How eBPF Works

┌─────────────────────────────────────────┐
│        User Space (your monitoring)      │
│  bpftrace / bcc / Cilium / Pixie        │
└──────────────┬──────────────────────────┘
               │ Load eBPF program
               ▼
┌─────────────────────────────────────────┐
│           Linux Kernel                   │
│  ┌──────────────────────────────────┐   │
│  │  eBPF Verifier                    │   │
│  │  - Checks program safety         │   │
│  │  - Ensures no infinite loops     │   │
│  │  - Prevents out-of-bounds memory │   │
│  └────────────┬─────────────────────┘   │
│               ▼                          │
│  ┌──────────────────────────────────┐   │
│  │  eBPF Program (runs in kernel)   │   │
│  │  - Attached to kprobe/tracepoint │   │
│  │  - Collects data to Ring Buffer  │   │
│  └────────────┬─────────────────────┘   │
└───────────────┼──────────────────────────┘
                │ Read data
                ▼
┌─────────────────────────────────────────┐
│  User-space program: visualize / alert  │
└─────────────────────────────────────────┘

eBPF Attachment Points

Type Purpose Example
kprobe / kretprobe Trace kernel function entry/return Trace tcp_sendmsg to monitor all TCP sends
tracepoint Static kernel trace points Trace syscalls:sys_enter_openat to monitor file opens
uprobe / uretprobe Trace user-space functions Trace libc:malloc to monitor memory allocation
cgroup Trace container/process groups Trace container network traffic
XDP Network card driver layer High-performance firewall, DDoS protection

Environment Setup

System requirements: - Linux kernel 5.8+ (6.1+ recommended) - Ubuntu 22.04+ / Debian 12+ / CentOS Stream 9+ / Fedora 38+

Install tools:

BASH 
# Ubuntu / Debian
sudo apt update
sudo apt install -y bpfcc-tools bpftrace linux-tools-common linux-tools-$(uname -r)

# CentOS / RHEL
sudo dnf install -y bcc-tools bpftrace kernel-tools

# Verify installation
bpftrace --version
# bpftrace v0.19.0

# Check kernel eBPF support
bpftool feature

Chapter 2: Case 1 — TCP Retransmission Diagnosis, Find Network Latency Root Cause in 3 Minutes

Problem Scenario

Users report: "API occasionally times out, but server CPU, memory, and disk all look fine."

Traditional troubleshooting: 1. ping to check latency? Normal 2. netstat -s for retransmissions? Some retransmissions, but can't tell which connection 3. tcpdump packet capture? Captured 10GB of data, analyzing until 3 AM

eBPF solution: 10 lines of code, real-time display of every TCP retransmission with precise details.

Practical Code

BASH 
# Use bpftrace to trace TCP retransmissions
sudo bpftrace -e '
kprobe:tcp_retransmit_skb {
    $sk = (struct sock *)arg0;
    $dport = $sk->__sk_common.skc_dport;
    $daddr = $sk->__sk_common.skc_daddr;

    printf("%s:%d -> %s:%d, state=%d\n",
        ntop(AF_INET, $sk->__sk_common.skc_rcv_saddr),
        $sk->__sk_common.skc_num,
        ntop(AF_INET, $daddr),
        bswap($dport),
        $sk->__sk_common.skc_state);
}
'

Sample output:

Attaching 1 probe...
192.168.1.100:45678 -> 10.0.0.5:3306, state=1
192.168.1.100:45679 -> 10.0.0.5:3306, state=1
192.168.1.100:52341 -> 172.16.0.20:6379, state=1

Interpretation: - Retransmissions concentrated on MySQL (3306) and Redis (6379) connections - Further investigation reveals: the MySQL physical machine has NIC packet drops - Problem solved after replacing the network cable

Advanced: Count Retransmissions Per Second

BASH 
sudo bpftrace -e '
kprobe:tcp_retransmit_skb {
    @retransmits = hist(elem);
}

interval:s:1 {
    print(@retransmits);
    clear(@retransmits);
}
'

Chapter 3: Case 2 — CPU Flame Graphs, Locate Performance Bottlenecks Without Service Restart

Problem Scenario

Java application CPU at 80%, but top only shows process-level CPU — can't tell which function is consuming CPU.

Traditional solutions: - perf record + perf report: requires restarting the app, unacceptable in production - JProfiler / Async Profiler: requires JVM parameter changes, invasive

eBPF solution: Dynamic tracing, zero intrusion, generate flame graphs anytime.

Practical Steps

Step 1: Collect CPU sampling data

BASH 
# Collect 30 seconds of CPU call stacks
sudo bpftrace -e '
profile:hz:99 /pid == 12345/ {
    @[kstack, ustack] = count();
}
' > /tmp/cpu_stacks.txt

Step 2: Generate flame graph

BASH 
# Install FlameGraph tools
git clone https://github.com/brendangregg/FlameGraph.git

# Convert to flame graph format
sudo bpftrace -e 'profile:hz:99 /pid == 12345/ { @[kstack, ustack] = count(); }' \
    --unsafe -o /tmp/stacks.txt

cat /tmp/stacks.txt | ./FlameGraph/stackcollapse-bpftrace.pl | \
    ./FlameGraph/flamegraph.pl > /tmp/cpu_flamegraph.svg

Flame graph interpretation: - X-axis: function call stack (width = CPU time proportion) - Y-axis: call depth - Colors: randomly assigned, no special meaning - Key: Find the widest "plateau" — that's your performance bottleneck

Real-World Case

A Python Flask app at 100% CPU, flame graph shows:

flask.app.full_dispatch_request (45%)
  └─ sqlalchemy.orm.query.all (38%)
       └─ sqlalchemy.orm.loading.instances (35%)
            └─ json.loads (28%)  ← Bottleneck here!

Root cause: Every request loads 10,000 records from the database and JSON-serializes them.
Fix: Add pagination + Redis cache, CPU drops to 15%.

Chapter 4: Case 3 — Dynamic User-Space Function Tracing, The Hidden Skill 90% Don't Know

This is the highlight of this article. Most people only know eBPF can trace the kernel, but eBPF can also trace user-space functions, including:

  • Python / Java / Go / Node.js function calls
  • libc malloc / free (memory allocation tracing)
  • OpenSSL encryption functions (HTTPS decryption for compliance auditing)

Case: Trace Python Application Function Calls

BASH 
# Trace all function calls in a Python process
sudo bpftrace -e '
uprobe:/usr/bin/python3:PyObject_Call /pid == 12345/ {
    @[ustack] = count();
}
'

Case: Trace MySQL Query Execution

BASH 
# Trace MySQL query execution time
sudo bpftrace -e '
uprobe:/usr/sbin/mysqld:dispatch_command /pid == 5678/ {
    @start[tid] = nsecs;
}

uretprobe:/usr/sbin/mysqld:dispatch_command /pid == 5678/ {
    $duration = nsecs - @start[tid];
    @query_durations = hist($duration / 1000000);  // Convert to ms
    delete(@start[tid]);
}
'

Output:

@query_durations:
[0, 1]               1234 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[1, 2]                456 |@@@@@@@@@@@@@@@                                   |
[2, 4]                123 |@@@@                                              |
[4, 8]                 34 |@                                                 |
[8, 16]                 8 |                                                  |
[16, 32]                2 |                                                  |

Interpretation: 95% of queries complete within 1ms, but 2 queries exceed 16ms and need optimization.

Case: Trace Node.js Event Loop Delay

BASH 
# Trace Node.js event loop delay
sudo bpftrace -e '
uprobe:/usr/bin/node:uv_run /pid == 9999/ {
    @start = nsecs;
}

uretprobe:/usr/bin/node:uv_run /pid == 9999/ {
    $delay = nsecs - @start;
    @loop_delays = hist($delay / 1000000);
}
'

Why don't 90% of people know this trick? - Most tutorials only cover kprobe (kernel tracing) - uprobe requires knowing function symbol names, which vary by language compilation - Requires the --unsafe flag, which many people are hesitant to use

Chapter 5: Case 4 — Cilium Container Networking, 30% K8s Performance Boost

Traditional K8s Networking Bottlenecks

Kubernetes uses kube-proxy by default for Service load balancing, based on iptables rules:

  • Problem 1: As Services increase, iptables rules grow linearly, degrading performance
  • Problem 2: Every connection passes through the user-space kube-proxy process, adding latency
  • Problem 3: Cannot implement fine-grained network policies (L7 filtering)

Cilium: eBPF-Based K8s Networking

Cilium replaces iptables with eBPF, processing network packets directly in kernel space:

┌─────────────────────────────────────────┐
│  Pod A (10.0.1.5)                       │
│  curl http://service-b                  │
└──────────┬──────────────────────────────┘
           │
           ▼
┌─────────────────────────────────────────┐
│  Linux Kernel                           │
│  ┌──────────────────────────────────┐  │
│  │  Cilium eBPF Program             │  │
│  │  - Load balances directly at XDP │  │
│  │  - No kube-proxy needed          │  │
│  │  - L3/L4/L7 network policies     │  │
│  └──────────────────────────────────┘  │
└──────────┬──────────────────────────────┘
           │
           ▼
┌─────────────────────────────────────────┐
│  Pod B (10.0.2.8)                       │
└─────────────────────────────────────────┘

Deploy Cilium

BASH 
# Install Cilium CLI
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
curl -L --remote-name-all \
    https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz
sudo tar xzvf cilium-linux-amd64.tar.gz -C /usr/local/bin

# Install Cilium to K8s cluster
cilium install --version 1.15.0

# Verify installation
cilium status
# ✅ All components are healthy

# Enable Hubble (network observability)
cilium hubble enable
cilium hubble ui

Performance Comparison

Test environment: 3-node K8s cluster, 1000 Services, 10,000 new connections per second

Metric kube-proxy (iptables) Cilium (eBPF) Improvement
Connection latency 1.2ms 0.3ms 75% ↓
CPU usage 15% 4% 73% ↓
Memory usage 512MB 128MB 75% ↓
Rule sync time 8s 0.5s 94% ↓

Hubble: K8s Network Visualization

Hubble is Cilium's observability component, providing:

  • Real-time traffic maps: which Pod is communicating with which Pod
  • DNS query logs: all container DNS requests
  • Network policy auditing: which connections are denied
BASH 
# View real-time traffic
cilium hubble observe

# Sample output:
# Jan 19 10:23:45: default/pod-a:45678 -> kube-system/coredns:53 DNS Query example.com A
# Jan 19 10:23:46: default/pod-b:52341 -> external/api.example.com:443 TCP Connection
# Jan 19 10:23:47: default/pod-c:38921 -> default/pod-a:8080 HTTP GET /api/v1/users

Chapter 6: Case 5 — Security Auditing, Trace All Process Creation and File Access

Problem Scenario

Security team demands: "Trace all sudo operations, all /etc/passwd access, all new process creation."

Traditional solutions: - auditd: complex configuration, high overhead (10-20%) - syslog: too much log volume, hard to analyze

eBPF solution: Zero configuration, low overhead, real-time alerts.

Practical Code

BASH 
# Trace all process creation
sudo bpftrace -e '
tracepoint:sched:sched_process_exec {
    printf("%s[%d] exec: %s\n", comm, pid, str(args->filename));
}
'

Output:

bash[1234] exec: /usr/bin/ls
sudo[1235] exec: /usr/bin/apt
apt[1236] exec: /usr/bin/dpkg

Trace Sensitive File Access

BASH 
# Trace access to /etc/passwd and /etc/shadow
sudo bpftrace -e '
tracepoint:syscalls:sys_enter_openat {
    $path = str(args->filename);
    $path == "/etc/passwd" || $path == "/etc/shadow" {
        printf("%s[%d] opened %s\n", comm, pid, $path);
    }
}
'

Trace All Network Connections

BASH 
# Trace all TCP connection establishment
sudo bpftrace -e '
kprobe:tcp_v4_connect {
    $sk = (struct sock *)arg0;
    printf("%s[%d] -> %s:%d\n",
        comm, pid,
        ntop(AF_INET, $sk->__sk_common.skc_daddr),
        bswap($sk->__sk_common.skc_dport));
}
'

Production-Grade Solution: Tetragon

Tetragon is an eBPF security auditing tool from the Cilium team, providing:

  • Real-time alerts: immediate notification on suspicious behavior
  • Policy as code: YAML-defined security policies
  • K8s integration: automatic Pod-to-user correlation
YAML 
# tetragon-policy.yaml
apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: sensitive-file-access
spec:
  kprobes:
    - call: "sys_openat"
      args:
        - index: 1
          type: string
      selectors:
        - matchArgs:
            - index: 1
              values:
                - "/etc/passwd"
                - "/etc/shadow"
                - "/etc/sudoers"
BASH 
# Apply policy
sudo tetragon --tracing-policy tetragon-policy.yaml

# Alert output:
# {"process":{"pid":1234,"comm":"cat","uid":1000},"file":"/etc/shadow","action":"observed"}

Chapter 7: eBPF Production Best Practices

1. Performance Overhead Control

BASH 
# Limit eBPF program sampling frequency
sudo bpftrace -e 'profile:hz:99 { @[kstack] = count(); }'  # 99Hz, not 1000Hz

# Use Ring Buffer instead of Perf Event Array (more efficient)
sudo bpftrace -e '
kprobe:tcp_sendmsg {
    $sk = (struct sock *)arg0;
    ring_buffer_output((uint64)$sk->__sk_common.skc_daddr);
}
'

2. Avoid Kernel Crashes

BASH 
# Always test programs before using --unsafe
sudo bpftrace --unsafe -e 'kprobe:tcp_sendmsg { printf("test\n"); }'

# Use verifier to check program safety
sudo bpftool prog load ./my_ebpf_prog.o /sys/fs/bpf/my_prog

3. Data Persistence

BASH 
# Export eBPF data to Prometheus
sudo bpftrace -e '
kprobe:tcp_retransmit_skb {
    @retransmits = count();
}

interval:s:10 {
    printf("retransmits: %d\n", @retransmits);
    clear(@retransmits);
}
' | prometheus-adapter

4. Integration with Existing Tools

Tool Purpose Integration
Prometheus Metrics collection eBPF → Prometheus Exporter
Grafana Visualization Grafana Dashboard
Loki Log aggregation eBPF → Loki
Jaeger Distributed tracing eBPF → OpenTelemetry

Chapter 8: 2026 eBPF Ecosystem Overview

Core Tool Matrix

Tool Positioning Use Case Learning Curve
bpftrace High-level tracing language Quick diagnosis, prototyping ⭐⭐
BCC Python + eBPF Complex monitoring logic ⭐⭐⭐
libbpf C library Production-grade app development ⭐⭐⭐⭐⭐
Cilium K8s networking Container network acceleration ⭐⭐⭐
Tetragon Security auditing Compliance, intrusion detection ⭐⭐⭐
Pixie K8s observability Automatic K8s monitoring ⭐⭐

Beginner (1-2 weeks): 1. Master bpftrace basics 2. Run the 5 practical cases from this article 3. Read the bpftrace reference manual

Intermediate (1-2 months): 1. Learn BCC tool development 2. Deploy Cilium + Hubble to a test cluster 3. Understand eBPF Verifier and CO-RE (Compile Once, Run Everywhere)

Expert (3-6 months): 1. Develop production eBPF applications with libbpf 2. Contribute to open source (Cilium, Tetragon, Pixie) 3. Discuss eBPF new features on kernel mailing lists

FAQ: Common eBPF Questions

What's the difference between eBPF and traditional strace/perf?

eBPF runs in kernel space with zero-copy data collection and < 1% overhead. strace uses ptrace system calls with 10-100x overhead. eBPF supports custom program logic for filtering and aggregation, while strace passively records all system calls.

Will eBPF affect production performance?

No. eBPF programs are validated by the kernel verifier to ensure no infinite loops or kernel crashes. Typical monitoring scenarios use < 1% CPU and < 10MB memory. Mature projects like Cilium run stably on tens of thousands of production nodes.

Does my Linux kernel version support eBPF?

Linux 4.4+ (released 2016) supports basic eBPF. We recommend 5.8+ for full features (BPF CO-RE, ring buffers, enhanced kprobes). Run uname -r to check — Ubuntu 22.04+ and CentOS Stream 9 meet the requirements by default.

Can eBPF be used in container and Kubernetes environments?

Absolutely. Cilium is a Kubernetes-native eBPF networking solution that replaces kube-proxy for high-performance service mesh. Pixie provides automatic K8s observability without code changes. eBPF natively supports cgroup v2 for container process tracing.

Summary: The 2026 Ops eBPF Skill Stack

eBPF has gone from "optional skill" to "must-have skill". Master eBPF, and you can:

  1. Pinpoint network latency root causes in seconds: no more tcpdump packet analysis
  2. Zero-intrusion performance diagnosis: dynamically generate flame graphs without service restarts
  3. Full-stack user-space + kernel tracing: from Python functions to kernel system calls, all visible
  4. K8s network acceleration: Cilium replaces kube-proxy, 30% performance improvement
  5. Real-time security auditing: trace all process, file, and network behavior with zero misses

Next steps: - Run the 5 cases from this article in a test environment - Deploy Cilium to a K8s test cluster - Read the bpftrace official documentation - Join the Cilium Slack community

About the author: Wen Yuan, technical editor at Dashen-Tech, focused on Linux operations and cloud-native technologies. If you encounter issues using eBPF, feel free to discuss in the comments.

License: This article is original to Dashen-Tech. Please cite the source when reprinting.