Runtime / guide
Runtime performance analysis from zero
Start here: vocabulary, method, tool choice, and machine setup for perf and bpftrace.
On this page
- 1. The questions performance analysis answers
- 2. Three ways to observe a running program
- 3. A method: from symptom to evidence
- 4. Which tool answers which question
- 5. Preparing a Linux machine
- Install the tools
- Record versions first
- Permissions
- Build programs so that tools can explain them
- 6. Your first sixty seconds with a slow program
- 7. Common beginner mistakes
- Where to go next
- Sources
This guide is the starting point for readers who have never profiled or traced a program. It explains the vocabulary, a repeatable method, the role of each tool, and how to prepare a Linux machine for perf and bpftrace. No previous kernel or performance knowledge is assumed; you should be comfortable compiling a C program and running commands in a terminal.
By the end you should be able to state a performance question precisely, take a first baseline, decide whether a program is running or waiting, and choose the next tool instead of guessing.
1. The questions performance analysis answers#
"The program is slow" is a feeling, not a measurement. Turn it into a question that a tool can answer:
- How long does one operation take? That is latency: the time from a start event to an end event, for example one request or one function call.
- How many operations finish per second? That is throughput.
- How busy is a resource? That is utilization: the fraction of time a CPU, disk, or lock was in use.
- Is work queuing for a resource? That is saturation: runnable threads waiting for a CPU, I/O waiting for a device, threads waiting for a mutex.
- Are operations failing? Errors such as a failing
open()can make a program retry, wait, or take a slower path.
Two time measurements appear everywhere and must not be confused:
| Measurement | Meaning | Typical source |
|---|---|---|
| Wall-clock (elapsed) time | Real time that passed between two moments | time's real, clock_gettime(CLOCK_MONOTONIC) |
| CPU time | Time a CPU actually spent executing this process or thread | time's user + sys, /proc/<pid>/stat |
If elapsed time is much larger than CPU time, the program spent most of its life off-CPU: sleeping, waiting for I/O, waiting for a lock, or runnable but not scheduled. If CPU time is close to elapsed time (per busy thread), the program is on-CPU and the question becomes which code uses the CPU. This single comparison decides which half of the toolbox you need.
2. Three ways to observe a running program#
Every tool on this site uses one of three techniques. Knowing which one a tool uses tells you what it can and cannot see.
- Counting keeps a total and reports it later.
perf statcounts events such as instructions, cycles, context switches, and page faults. Counting is cheap, but it says how many, not where. - Sampling interrupts the program periodically and records where it was, for example 99 times per second.
perf recordand bpftrace'sprofileprobe sample. Samples show where CPU time goes statistically, but a short or rare function can be missed entirely and sampling sees nothing while a thread is asleep. - Tracing records every occurrence of a specific event, such as every
openatsystem call or every entry to a function.bpftrace,perf trace, andstracetrace. Tracing is exact for the chosen events, but each event costs time; tracing a very frequent event can slow the program noticeably.
The cost of observing is called overhead or the observer effect. Record it as part of the experiment: compare the elapsed time with and without the tool. The observer effects guide treats this in depth.
3. A method: from symptom to evidence#
Use the same loop for every investigation. It prevents the most common mistake: changing code based on an unverified guess.
- Describe the problem in one sentence with a number: "
./reporttakes 3 s for 200 jobs; last week it took 0.1 s." - Record the environment: kernel (
uname -r), CPU model, compiler and flags, tool versions, and the exact command line. Results without this context cannot be compared. - Take a baseline: run the program several times and note the spread. A 5 % change is meaningless if repeated runs already vary by 10 %.
- Decide on-CPU or off-CPU: compare elapsed time with CPU time (section 6).
- Drill down with one tool, chosen from the table in section 4.
- Form a hypothesis that predicts something you have not measured yet: "If the lookup is quadratic, doubling the input should roughly quadruple lookup time."
- Test the prediction with a second, independent observation. Then change the code and measure again under the same conditions.
A complementary checklist for system resources is Brendan Gregg's USE method: for every resource (CPU, memory, disk, network, locks) check Utilization, Saturation, and Errors. It is useful when you do not yet know which resource is involved.
4. Which tool answers which question#
| Question | First tool | Next step |
|---|---|---|
| How long did it take, and was it busy? | /usr/bin/time -v ./prog |
compare real with user + sys |
| Which threads are busy or switching? | pidstat -t -u -w 1 -p <pid> (sysstat) |
process and thread state |
| How many instructions, cycles, faults, and switches? | perf stat ./prog |
perf guide |
| Which functions consume CPU? | perf record -g ./prog, perf report |
flame graph, perf annotate |
| Which system calls happen, and which fail? | bpftrace syscall one-liners, perf trace -s |
bpftrace from zero |
| How long does one function or syscall take? | bpftrace entry/return probes with a histogram | function probe lab |
| Why is a thread not running? | off-CPU analysis with sched_switch |
scheduler timelines |
| What is the program doing right now, frozen? | gdb -p <pid> with thread apply all bt |
debugging playbook |
strace and GDB use ptrace, which stops the target for every event; they are excellent for correctness questions and costly for timing questions. perf and bpftrace collect data inside the kernel without stopping the process for each event, which usually makes them better suited for measuring time.
5. Preparing a Linux machine#
The laboratories assume a native Linux machine or a virtual machine you control. Containers and some cloud VMs hide hardware counters or block tracing; the tools then report <not supported> or permission errors. Treat those as unavailable measurements, never as zero.
Install the tools#
# Debian
sudo apt install linux-perf bpftrace sysstat gdb build-essential
# Ubuntu (perf comes from the kernel-specific linux-tools package)
sudo apt install linux-tools-common "linux-tools-$(uname -r)" bpftrace sysstat gdb build-essential
# Fedora / RHEL family
sudo dnf install perf bpftrace sysstat gdb gccPackage names vary between releases; search your distribution's package index if one is missing. Ubuntu's perf must match the running kernel, which is why the package name contains $(uname -r).
Record versions first#
uname -r
perf --version
bpftrace --version
sudo bpftrace --info 2>&1 | head -n 40
lscpu | head -n 20bpftrace --info lists kernel features, such as BTF and uprobe support. Save this output next to your results.
Permissions#
perf access for unprivileged users is controlled by /proc/sys/kernel/perf_event_paranoid; higher values allow less. bpftrace needs root or the BPF capabilities (CAP_BPF, CAP_PERFMON, and for some probes CAP_SYS_ADMIN). Read the values before deciding how to run:
cat /proc/sys/kernel/perf_event_paranoid
cat /proc/sys/kernel/kptr_restrictOn a disposable lab machine the simplest approach is to run the measuring tool with sudo. Lowering perf_event_paranoid system-wide changes security policy for every user; do not do it on a shared or production machine. The Linux perf security document explains each level. If kernel symbols appear as [unknown] or zero addresses, kptr_restrict or missing privileges are the usual reason.
Build programs so that tools can explain them#
Profilers report addresses; turning addresses into function names and stacks needs help from the build:
-gkeeps debug information (DWARF) so tools can name functions, source lines, and inlined code.-fno-omit-frame-pointerkeeps the frame-pointer chain, the cheapest and most reliable way forperfand bpftrace to walk user stacks.-O2keeps the build realistic. Profiling an-O0build measures code you will not ship. The compiler differences lab shows how much the code changes.__attribute__((noinline))or a non-staticfunction keeps a separate symbol that a uprobe can attach to. The laboratories use it on purpose; real programs may inline small functions, leaving no entry to probe.
System libraries may lack frame pointers or local debug information. Many distributions provide separate debug packages or a debuginfod server; see debug information for how tools find them.
6. Your first sixty seconds with a slow program#
Run the program once under time, preferably GNU time with -v for extra counters:
/usr/bin/time -v ./prog 2> time.txt
grep -E 'Elapsed|User time|System time|Percent of CPU|context switches|page faults' time.txtRead the result in this order:
- Elapsed versus User + System. Roughly equal for a single-threaded program: on-CPU, continue with
perf record. Much smaller: off-CPU, continue with system calls, locks, and scheduler events. - System time. High system time means the CPU was busy in the kernel on the program's behalf: many system calls, page faults, or copying data.
- Voluntary context switches count the times a thread gave up the CPU because it had to wait (sleep, I/O, lock). Involuntary switches count the times the scheduler took the CPU away, which suggests CPU competition.
- Page faults. Many major faults mean data was read from disk to satisfy memory accesses.
perf stat gives a similar first look with hardware counters, and works on a running process too:
perf stat ./prog
sudo perf stat -p <pid> -- sleep 10 # attach to a running process for 10 stask-clock with "CPUs utilized" near 1.0 means one CPU was busy for the whole run; near 0.05 means the program mostly waited. The CPU profiling lab walks through the remaining fields.
7. Common beginner mistakes#
- Profiling once. Repeat runs and report the spread.
perf stat -r 5repeats a command and prints the variation. - Profiling the wrong build. Debug builds without optimization have different hotspots.
- Reading a CPU profile for a waiting program. A sampling profiler sees nothing while threads sleep. If the program is mostly off-CPU, its CPU profile describes the small busy part, not the slowness.
- Trusting
[unknown]frames. Missing symbols or broken stacks are a measurement problem to fix, not a function to optimize. - Treating "no event" as "did not happen". A probe may have attached late, events may have been dropped, or sampling may have missed it. Check counts against something you know, such as the number of requests you sent.
- Changing several things at once. Change one variable, measure again, keep the old result.
Where to go next#
Follow the runtime route in order; each page reuses vocabulary from the previous one:
- Process, thread, and scheduler state: what a PID and TID are, and which counters the kernel keeps.
- Execution analysis with perf, then the hands-on CPU profiling and flame graph lab.
- bpftrace from zero, then the first bpftrace lab.
- Debugging slow, stuck, and failing programs, then the off-CPU diagnosis lab.
- The deeper timing material: runtime latency, timing and waiting lab, and scheduler timelines.