Modern Linux production systems move fast, and the old debugging playbook (restart with debug flags, attach heavy profilers, hope for the best) no longer scales. In 2026, the most practical way to understand live system behavior is eBPF: safe, kernel-verified programs you can load at runtime to observe syscalls, network activity, and latency without rebooting or recompiling. In this guide, you will build a lightweight observability workflow using bpftrace, export useful data to OpenTelemetry, and apply it to real troubleshooting scenarios.

Why eBPF is now a default Linux skill

eBPF became mainstream because it solves the hardest production visibility problem: getting high-signal data with low overhead. Instead of sampling everything, you can instrument exactly the kernel or userspace function you care about.

• No app restart: Attach probes to running processes.

• Low overhead: Filter and aggregate inside the kernel.

• Fine-grained insight: Track latency, errors, and hot paths by PID, cgroup, or container.

• Safer than kernel modules: Programs are verified before load.

Environment setup on Linux (Ubuntu/Debian example)

For a practical start, install bpftrace and essential tooling. Kernel 5.15+ works well for most modern examples, while 6.x gives better BTF and CO-RE compatibility.

sudo apt update
sudo apt install -y bpftrace linux-tools-common linux-tools-generic

# Optional but recommended for richer symbols
sudo apt install -y linux-headers-$(uname -r)

# Verify
bpftrace --version
uname -r

If you run inside Kubernetes nodes, execute these on the host (or via privileged debug DaemonSet). For local experiments, a VM is enough.

First useful probe: syscall latency histogram

This script measures how long read() syscalls take and prints a log2 histogram every 10 seconds.

#!/usr/bin/env bpftrace

begin
{
  printf("Tracing read() latency... Hit Ctrl+C to stop.\n");
}

kprobe:vfs_read
{
  @start[tid] = nsecs;
}

kretprobe:vfs_read
/@start[tid]/
{
  $delta_us = (nsecs - @start[tid]) / 1000;
  @read_latency_us = hist($delta_us);
  delete(@start[tid]);
}

interval:s:10
{
  print(@read_latency_us);
  clear(@read_latency_us);
}

Save as read_latency.bt and run:

sudo bpftrace read_latency.bt

When p99 drifts into high buckets, you likely have storage contention or noisy neighbors.

Container-aware tracing by cgroup

In 2026, most Linux workloads are containerized, so process-wide metrics are not enough. You can attribute activity to cgroups to isolate a single service.

# Show top write() callers by cgroup id
sudo bpftrace -e '
kprobe:vfs_write { @[cgroup, comm] = count(); }
interval:s:5 { print(@, 15); clear(@); }
'

Map cgroup IDs to container names using your runtime metadata (containerd, CRI-O, or systemd slice mapping), then alert on unusual write spikes.

Userspace tracing with uprobes

You are not limited to kernel functions. Use uprobes to instrument userspace binaries, such as a Go or Node.js service function with symbols available.

# Example: attach to OpenSSL function in a running process
sudo bpftrace -e '
uprobe:/usr/lib/x86_64-linux-gnu/libssl.so.3:SSL_write
{ @[comm] = count(); }
interval:s:10 { print(@); clear(@); }
'

This helps answer questions like, “Which process is flooding outbound TLS writes right now?” without editing code.

Exporting eBPF signals to OpenTelemetry

Raw terminal output is great for incidents, but teams need time-series data in a backend (Tempo/Prometheus-compatible pipelines, Grafana, Honeycomb, etc.). A simple pattern is:

1. Run bpftrace in JSON output mode (or parse predictable text output).

2. Convert records into OTLP metrics/spans.

3. Ship through an OpenTelemetry Collector.

Minimal Python bridge example:

import json
import subprocess
from opentelemetry import metrics
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader

reader = PeriodicExportingMetricReader(OTLPMetricExporter(endpoint="http://localhost:4318/v1/metrics"))
provider = MeterProvider(metric_readers=[reader])
metrics.set_meter_provider(provider)
meter = metrics.get_meter("ebpf.bridge")
latency = meter.create_histogram("linux.read.latency.us")

script = """
kprobe:vfs_read { @s[tid] = nsecs; }
kretprobe:vfs_read /@s[tid]/ {
  printf(\"{\\\"lat_us\\\":%llu}\\n\", (nsecs-@s[tid])/1000);
  delete(@s[tid]);
}
"""

proc = subprocess.Popen(["sudo", "bpftrace", "-e", script], stdout=subprocess.PIPE, text=True)
for line in proc.stdout:
    line = line.strip()
    if line.startswith("{"):
        event = json.loads(line)
        latency.record(event["lat_us"], {"host": "prod-node-1"})

In production, harden this with batching, retries, and cardinality control (avoid unbounded labels like raw PID in high-volume paths).

Real incident workflow you can copy

Scenario: API timeout spike after a deploy

• Check TCP retransmits and connect latency with eBPF network probes.

• Trace file and DNS syscalls for the service cgroup.

• Compare p50/p95 syscall latency before and after deploy.

• Push temporary eBPF metrics to OTel and correlate with app spans.

Typical outcome: you quickly separate app regressions from kernel or network bottlenecks, reducing MTTR from hours to minutes.

Performance and safety best practices

• Filter early: Restrict by PID/cgroup to reduce event volume.

• Avoid heavy string work in hot probes: Aggregate numeric data in-kernel when possible.

• Time-box incident probes: Run for short windows and remove.

• Version your scripts: Keep probe scripts in Git with runbooks.

• Control permissions: Use least-privilege and audited sudo access for tracing tools.

What to automate next

Once your first probes are useful, build a small internal library:

• cpu_hotspots.bt for scheduler and off-CPU time

• io_latency.bt for filesystem and block layer visibility

• tcp_retrans.bt for packet loss and retransmit spikes

• dns_slow.bt for resolver bottlenecks

Then wire these into an incident command that auto-runs a safe subset and streams to your observability stack.

Final thoughts

Linux eBPF is no longer niche kernel wizardry, it is a practical developer and SRE tool for 2026. If your team runs production APIs, queues, or data services on Linux, eBPF gives you immediate visibility where logs and APM sometimes go blind. Start with one latency probe, attach it to real incidents, and promote successful scripts into your standard runbooks. That single habit can dramatically improve reliability without expensive platform changes.