Sampling thread counters without scheduler tracing

Take two snapshots of one Linux thread's counters and subtract them. You will learn what the thread did during a short observation window, without attaching a tracer or changing its scheduling policy.

The main lesson is also about missing information: an unavailable waiting counter is not a measured zero.

Before you start#

Use Linux procfs, Python 3.7 or newer, and read access to your own lab process. Procfs is the information filesystem mounted at /proc.

Build timing_lab by completing the setup in timing and waiting. You need its executable, but do not need to repeat all its timing comparisons. The process and thread guide explains PID, TID, and scheduler state if those terms are new.

Download sample_thread.py before launching the target. It is already in examples/thread-observation if you have the checkout.

1. Select the reader and check the files#

Open a terminal in the timing lab's working directory—the one containing build/timing_lab and results/. In a checkout, this is examples/runtime-timing.

Choose the reader path that matches your setup. For the checkout layout:

bash

LAB_SAMPLER=../thread-observation/sample_thread.py

If you downloaded the reader into the current working directory instead, use:

bash

LAB_SAMPLER=./sample_thread.py

Then check both files:

bash

ls ./build/timing_lab "$LAB_SAMPLER"
mkdir -p results

Check before continuing: both paths should exist. LAB_SAMPLER is just a shell variable holding the reader's location; the next commands use that path.

2. Launch, sample, and wait in one block#

Run this whole block in the same terminal so the short-lived target remains alive during sampling:

bash

./build/timing_lab sleep 100 50 results/thread-sleep.csv &
LAB_PID=$!
LAB_TID=$LAB_PID
ps -L -p "$LAB_PID" -o pid,tid,stat,psr,cls,rtprio,pri,wchan
python3 "$LAB_SAMPLER" "$LAB_PID" "$LAB_TID" --interval 0.25 > results/thread-sample.json
wait "$LAB_PID"

The & starts the target in the background. $! gives the PID of the process you just launched. This particular program does its work in its main thread, so its TID equals its PID. That shortcut does not apply to an arbitrary worker in another application.

ps lists the task. The reader then takes a snapshot, waits about a quarter second, and takes another snapshot. Its JSON goes into results/thread-sample.json, so a quiet terminal is normal. wait lets the finite target finish; the whole block usually takes about five seconds.

3. Open the result and check its status#

bash

cat results/thread-sample.json

Start with status:

measured means the reader obtained two usable snapshots for the task it checked.
unavailable means the observation failed. Read reason; do not turn missing data into zero activity.

For a measured result, look at sample_interval_ns, runtime_ns, and runqueue_status before reading the full endpoint records. A nanosecond is one billionth of a second; 1,000,000 ns = 1 ms.

4. Interpret CPU activity and waiting separately#

runtime_ns is the change in the task's CPU execution counter. sample_interval_ns is the actual interval between the reader's sampling timestamps. It can be longer than the requested quarter second.

timeslices is the raw change in the third scheduler counter. The before and after objects preserve both original readings, so you can inspect what was subtracted.

For runqueue waiting, two outcomes are possible:

With runqueue_status: enabled-at-endpoints, runqueue_wait_ns contains the counter difference. It describes aggregate runnable waiting during the window.
With runqueue_status: disabled-or-unknown, runqueue_wait_ns is JSON null. The kernel did not provide enough collection-status information for the reader to interpret that wait counter.

Your checkpoint: explain the difference between a numeric zero and null. A successful CPU-counter sample can coexist with unavailable wait accounting.

The sampler reads /proc/sys/kernel/sched_schedstats where available; it never enables it. Even enabled readings at both ends cannot rule out someone changing the setting in between.

5. Optional: compare a busy target#

Keep LAB_SAMPLER set and run a new process, using different output files:

bash

./build/timing_lab busy 100 50 results/thread-busy.csv &
LAB_PID=$!
LAB_TID=$LAB_PID
python3 "$LAB_SAMPLER" "$LAB_PID" "$LAB_TID" --interval 0.25 > results/thread-busy-sample.json
wait "$LAB_PID"
cat results/thread-busy-sample.json

Busy mode will usually consume more CPU during comparable windows than sleep mode. Do not demand a fixed ratio: the sampler's window can overlap different parts of the periodic workload, and system load varies. Each run has a new PID/TID, which the block obtains afresh.

If sampling fails#

The executable or reader is missing: recheck pwd and both paths from step 1. Build the target first and select the correct reader location.

The process disappears: run the launch-and-sample block again. Do not reuse a PID after its process has ended, and do not pause between launching the finite target and sampling it.

Permission is denied or procfs data is absent: keep the unavailable result and its reason. This lab does not require changing system-wide permissions or enabling tracing.

The reader reports a changed identity or decreasing counter: do not subtract those samples. They are not a valid comparison window.

What the two snapshots cannot tell you#

The reader checks process and thread start times as well as numeric IDs. Start times have tick resolution and procfs reads are not atomic, so this is a diagnostic identity check rather than a perfect lifetime guarantee. Names containing spaces or parentheses are handled by the supplied reader; a simple whitespace split of the whole stat line would get its fields wrong.

Cumulative counter changes do not give the maximum delay of one function call. Likewise, subtracting CPU runtime from this sampling interval is not a precise off-CPU measurement: clock and counter reads happen at different instants.

For individual events, use the event correlation lab and, when justified, the runtime tracing guide. A task counter can narrow a question without proving that one message spent its entire delay waiting for a CPU.

Keep your evidence and repeat the check#

Save the raw JSON, source/executable identities, PID/TID, kernel, and collection flag with your notes. Explain whether runnable waiting was measured or unavailable.

From the repository root, python3 scripts/verify-debugging-labs.py --require-linux tests the reader against a finite live child. Synthetic fixtures check disabled accounting, identity changes, reversed counters, and parenthesized task names. The validation record distinguishes those checks from perf/BPF execution.

Sources#

Kernel scheduler statistics defines the three cumulative counters.
Linux proc_pid_task defines task paths.
Linux proc_pid_stat defines start-time fields and their resolution.