Skip to content
Skills Library
DevOps & cloud · Analysis

cloud-native-ebpf-observability

/cloud-native-ebpf-observability

Use this skill when the user says 'ebpf tracing', 'ebpf observability', 'bpftrace', 'trace syscalls', 'kprobe', 'tracepoint', 'uprobe', 'off-cpu profiling', 'continuous profiling ebpf', 'libbpf ringbuf', 'measure latency in kernel'. Builds kernel- and user-level observability with eBPF: bpftrace one-liners for investigation, libbpf CO-RE programs with ring buffers for production agents, and guidance on Kubernetes-ready tools (Pixie, Parca, Tetragon, Beyla/OBI). Do NOT use for: packet processing and Cilium datapath (cloud-native-ebpf-networking), OpenTelemetry SDK tracing (opentelemetry-tracing

Licence
MIT
Last repository change
10.11.2026

Assessment and security

95/ 100Class: good
Format20 / 25
Frontmatter with name and description, valid name, description up to 1,024 characters, name matching the folder.
Discoverability25 / 25
Description explains when to use the skill, has a reasonable length and uses third person.
Content35 / 35
At least 150 words and at most 500 lines, headings, step lists, examples or code.
Hygiene15 / 15
No placeholders such as TODO, no Windows paths, with a licence.
Riskno signals detected in the scan
Files in this package
1
Scripts
0
HTML comments
0
URLs in SKILL.md
0
Deprecation

No signals detected. Repository changed on 10.11.2026, without references to retired models or APIs.

Licence

MIT, open: use and modify while retaining author attribution.

What it does and when to use it

Use this skill when the user says 'ebpf tracing', 'ebpf observability', 'bpftrace', 'trace syscalls', 'kprobe', 'tracepoint', 'uprobe', 'off-cpu profiling', 'continuous profiling ebpf', 'libbpf ringbuf', 'measure latency in kernel'. Builds kernel- and user-level observability with eBPF: bpftrace one-liners for investigation, libbpf CO-RE programs with ring buffers for production agents, and guidance on Kubernetes-ready tools (Pixie, Parca, Tetragon, Beyla/OBI). Do NOT use for: packet processing and Cilium datapath (cloud-native-ebpf-networking), OpenTelemetry SDK tracing (opentelemetry-tracing

Triggers from the description

ebpf tracingebpf observabilitybpftracetrace syscallskprobetracepointuprobeoff-cpu profiling
Original author description

Use this skill when the user says 'ebpf tracing', 'ebpf observability', 'bpftrace', 'trace syscalls', 'kprobe', 'tracepoint', 'uprobe', 'off-cpu profiling', 'continuous profiling ebpf', 'libbpf ringbuf', 'measure latency in kernel'. Builds kernel- and user-level observability with eBPF: bpftrace one-liners for investigation, libbpf CO-RE programs with ring buffers for production agents, and guidance on Kubernetes-ready tools (Pixie, Parca, Tetragon, Beyla/OBI). Do NOT use for: packet processing and Cilium datapath (cloud-native-ebpf-networking), OpenTelemetry SDK tracing (opentelemetry-tracing

How it works

  1. Purpose
  2. Agent Protocol
  3. Probe Types
  4. bpftrace for Investigation
  5. libbpf CO-RE Agent (exec events via ring buffer)
  6. Kubernetes-Ready Tools
  7. Overhead Rules
  8. Anti-Patterns
  9. Checklist
  10. References
Original SKILL.md headings · 140 lines, 860 words
SKILL.md preview
---
name: cloud-native-ebpf-observability
description: >
  Use this skill when the user says 'ebpf tracing', 'ebpf observability', 'bpftrace', 'trace syscalls', 'kprobe', 'tracepoint', 'uprobe', 'off-cpu profiling', 'continuous profiling ebpf', 'libbpf ringbuf', 'measure latency in kernel'. Builds kernel- and user-level observability with eBPF: bpftrace one-liners for investigation, libbpf CO-RE programs with ring buffers for production agents, and guidance on Kubernetes-ready tools (Pixie, Parca, Tetragon, Beyla/OBI). Do NOT use for: packet processing and Cilium datapath (cloud-native-ebpf-networking), OpenTelemetry SDK tracing (opentelemetry-tracing), runtime threat detection rules (ebpf-threat-detection).
version: "2.0.0"
author: "j4flmao"
license: "MIT"
tags: [cloud-native, ebpf, observability, bpftrace, libbpf, profiling, tracing]
---

# eBPF Observability

## Purpose
Observe what the kernel and processes are actually doing — syscalls, scheduling, I/O latency, network connections, function calls in binaries — with low overhead and no code changes. Start with bpftrace to answer a question in minutes; build a libbpf CO-RE program only when you need a long-running agent.

## Agent Protocol

### Trigger
"ebpf tracing", "ebpf observability", "bpftrace", "trace syscalls", "kprobe", "tracepoint", "uprobe", "off-cpu", "continuous profiling", "ringbuf", "kernel latency".

### Input Context
- Question to answer (which process opens files, where is latency, who connects where).
- Kernel version and BTF availability (`ls /sys/kernel/btf/vmlinux`).
- One-off investigation vs permanent agent; container/Kubernetes context.

### Output Artifact
bpftrace script or libbpf program + user-space loader outline. No file unless requested.

### Response Format
```
Question: {...}
Probe type: {tracepoint|fentry/fexit|kprobe|uprobe|USDT|perf_event}
Tool: {bpftrace one-liner|libbpf CO-RE agent|existing tool}
Overhead note: {event rate, filtering in kernel}
Min kernel: {version}
```

### Completion Criteria
- [ ] Prefer stable tracepoints over kprobes where one exists.
- [ ] Aggregation/filtering happens in kernel (maps/histograms), not per-event to user space, when rates are high.
- [ ] Minimum kernel version stated.

## Probe Types
| Probe | Stability | Use |
|---|---|---|
| Tracepoints (`tracepoint:syscalls:sys_enter_openat`) | Stable ABI | First choice for syscalls, scheduler, block I/O, net events |
| fentry/fexit (BTF trampolines, kernel ≥ 5.5) | Function signatures may change between kernels | Low-overhead kernel function entry/exit with typed args |
| kprobe/kretprobe | Unstable (internal functions can be renamed/inlined) | When no tracepoint exists |
| uprobe/uretprobe | Binary-specific | User-space functions (TLS libraries, Go/Java runtimes with care) |
| USDT | Stable if the app defines them | Instrumented runtimes/databases |
| perf_event (CPU sampling) | Stable | CPU profiling (flame graphs) |

## bpftrace for Investigation
```bash
# Who opens which files (system-wide)
bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args.filename)); }'

# Syscall counts by process
bpftrace -e 'tracepoint:raw_syscalls:sys_enter { @[comm] = count(); }'

# Block I/O latency histogram (µs)
bpftrace -e 'tracepoint:block:block_rq_issue { @s[args.dev, args.sector] = nsecs; }
  tracepoint:block:block_rq_complete /@s[args.dev, args.sector]/ {
    @usecs = hist((nsecs - @s[args.dev, args.sector]) / 1000); delete(@s[args.dev, args.sector]); }'

# New process executions with arguments
bpftrace -e 'tracepoint:syscalls:sys_enter_execve { printf("%d %s\n", pid, str(args.filename)); }'

# CPU profile at 99 Hz, kernel+user stacks
bpftrace -e 'profile:hz:99 { @[kstack, ustack, comm] = count(); }'
```
Older bpftrace versions use `args->field` instead of `args.field`. The bcc-tools/libbpf-tools packages (`execsnoop`, `opensnoop`, `biolatency`, `tcplife`, `runqlat`) cover many common questions already — prefer the libbpf-tools builds.

## libbpf CO-RE Agent (exec events via ring buffer)
```c
// execsnoop.bpf.c — kernel >= 5.8 (BPF ring buffer), BTF enabled
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_core_read.h>

struct event {
    __u32 pid;
    __u32 ppid;
    char comm[16];
    char filename[256];
};

struct {
    __uint(type, BPF_MAP_TYPE_RINGBUF);
    __uint(max_entries, 256 * 1024);
} events SEC(".maps");

SEC("tracepoint/syscalls/sys_enter_execve")
int handle_execve(struct trace_event_raw_sys_enter *ctx)
{
    struct event *e = bpf_ringbuf_reserve(&events, sizeof(*e), 0);
    if (!e)
        return 0;                                   /* buffer full: drop, count if needed */

    struct task_struct *task = (struct task_struct *)bpf_get_current_task();
    e->pid  = bpf_get_current_pid_tgid() >> 32;     /* upper 32 bits = TGID (user-visible PID) */
    e->ppid = BPF_CORE_READ(task, real_parent, tgid);
    bpf_get_current_comm(&e->comm, sizeof(e->comm));
    bpf_probe_read_user_str(&e->filename, sizeof(e->filename), (const char *)ctx->args[0]);

    bpf_ringbuf_submit(e, 0);
    return 0;
}

char LICENSE[] SEC("license") = "GPL";
```
User space (C with skeleton): `bpftool gen skeleton execsnoop.bpf.o > execsnoop.skel.h`, then `execsnoop_bpf__open_and_load()`, `execsnoop_bpf__attach()`, `ring_buffer__new(bpf_map__fd(skel->maps.events), handle_event, NULL, NULL)`, and loop on `ring_buffer__poll(rb, 100)`. In Go use `cilium/ebpf` (`bpf2go` + `ringbuf.NewReader`).

Key details:
- `bpf_get_current_pid_tgid()` returns `tgid << 32 | pid`; the user-visible process ID is the upper half.
- `BPF_CORE_READ` relocates field offsets at load time using BTF → one binary works across kernels.
- Ring buffer (≥ 5.8) preserves event order across CPUs and uses one shared buffer; prefer it over `PERF_EVENT_ARRAY`.

## Kubernetes-Ready Tools
| Need | Tool |
|---|---|
| Auto-instrumented HTTP/gRPC metrics and traces | Grafana Beyla / OpenTelemetry eBPF Instrumentation (OBI), Pixie |
| Continuous CPU profiling | Parca, Pyroscope (eBPF agent) |
| Process/file/network security observability | Tetragon, Falco |
| Network flow visibility | Hubble (Cilium) |
| Ad-hoc tracing in pods | `kubectl debug` + bpftrace, or Inspektor Gadget |

Map host PIDs to containers via cgroup ID (`bpf_get_current_cgroup_id()`) and the container runtime.

## Overhead Rules
- Cost scales with event rate × work per event. Tracing every `sys_enter` on a busy host can be expensive; filter by PID/cgroup in-kernel and aggregate in maps.
- uprobes trap into the kernel per hit (microsecond-scale each) — avoid on very hot user functions.
- Stack collection and string copies are the expensive parts; cap sizes.

## Anti-Patterns
- BCC Python in production agents (runtime compilation, kernel headers required on nodes).
- kprobes on functions that have a tracepoint equivalent.
- Streaming every event to user space and aggregating there.
- Running privileged tracing DaemonSets without restricting who can deploy them (they can read any process memory via probes).

## Checklist
- [ ] BTF present; minimum kernel documented.
- [ ] Tracepoint/fentry chosen before kprobe.
- [ ] In-kernel filtering/aggregation; ring buffer sized; drops counted.
- [ ] Capabilities limited (`CAP_BPF`, `CAP_PERFMON` instead of full root where possible, kernel ≥ 5.8).

## References
- Related skills: `skills/cloud-native/ebpf-networking/SKILL.md`, `skills/low-level/ebpf-engineering/SKILL.md`, `skills/cybersecurity/ebpf-threat-detection/SKILL.md`.

Versions

The database has no other skills named cloud-native-ebpf-observability.

Database snapshot: 13.11.2026 · package version 327ef9bc623a (SHA-256 digest of the entire package)

Downloads in the last 30 days: 0 · ranking

Report a problem with this skill: harmful content, incorrect descriptions or an author’s removal request.

Similar skills

  • 100ebpf-engineering
    Use this skill when the user says 'eBPF', 'BPF program', 'libbpf', 'CO-RE', 'BTF', 'kprobe', 'fentry', 'tracepoint', 'uprobe', 'XDP', 'tc BPF', 'BPF map', 'ring buffer', 'verifier error', 'bpftrace', 'cilium/ebpf', 'aya'. Writes and debugs production eBPF: program types and…
  • 100ebpf
    eBPF skill for Linux observability and networking. Use when writing eBPF programs with libbpf or bpftrace, attaching kprobes/tracepoints/XDP hooks, debugging verifier errors, working with eBPF maps, or achieving CO-RE portability across kernel versions. Activates on queries…
  • 100ebpf
    Use when writing eBPF programs with libbpf, bpftrace, or Rust with aya-ebpf and aya-log, attaching kprobes, tracepoints, or XDP hooks, declaring maps, sharing data with a Tokio loader, triaging verifier or Aya load errors, or porting with CO-RE. Not for full kernel bypass: use dpdk.
  • 100implementing-ebpf-security-monitoring
    Implements eBPF-based security monitoring using Cilium Tetragon for real-time process execution tracking, network connection observability, file access auditing, and runtime enforcement. Covers TracingPolicy CRD authoring with kprobe/tracepoint hooks, in-kernel filtering via…
  • 100implementing-runtime-security-with-tetragon
    Implements eBPF-based runtime observability and in-kernel enforcement in Kubernetes with Cilium Tetragon, monitoring process execution, file access, network connections, and syscalls, and blocking dangerous calls at the kernel level. Use when deploying Tetragon to detect or…
  • 95cilium-ebpf
    Use this skill when the user says 'Cilium', 'eBPF', 'BPF', 'Hubble', 'service mesh', 'network policy', 'cluster mesh', 'eBPF observability', 'kube-proxy replacement', 'CiliumNetworkPolicy', 'CiliumClusterWideNetworkPolicy', 'CiliumEndpoint', 'Hubble UI', 'Tetragon', 'Cilium…

Feedback and reports

Your message goes to Bartosz Sokoliński. An email address is needed only for a reply. See the privacy policy.