English for Systems Engineers: Performance Analysis and Kernel Vocabulary

Systems engineers work at the lowest levels of the software stack — operating systems, kernels, performance analysis tools, and hardware interfaces. The vocabulary in this domain is highly specialised, and communicating performance findings clearly in English requires precision. This guide covers the core terms used in performance analysis, kernel observability, and report writing.

Performance Analysis Vocabulary

Term	Definition
Flamegraph	A visualisation of call stacks, where the width of each frame represents the proportion of time spent there
p99 latency	The 99th percentile latency — the worst-case time experienced by the slowest 1% of requests
Throughput	The number of operations completed per unit of time
Saturation	The degree to which a resource is fully utilised, leading to queuing
Utilisation	The percentage of time a resource is busy
Bottleneck	The resource that limits overall system throughput
Working set	The set of pages or data actively used by a process at a given time
Context switch	When the CPU switches from executing one process to another, incurring overhead
Cache miss	When requested data is not in the CPU cache and must be retrieved from slower memory

The USE Method

Brendan Gregg’s USE method provides a systematic framework for performance analysis:

• U — Utilisation: Is the resource being used heavily?
• S — Saturation: Is there a queue forming?
• E — Errors: Are errors being reported?

“Applying the USE method to the storage subsystem, we found 95% utilisation, significant I/O queue depth (saturation), and no errors — indicating that the disk is the bottleneck rather than any fault condition.”

Flamegraph Vocabulary

Flamegraphs are one of the most powerful tools for performance analysis. When discussing flamegraphs in reports or team meetings, use precise language.

Term	Meaning
Stack frame	A single function call in the call stack
Hot path	The code path that consumes the most CPU time
Off-CPU flamegraph	A flamegraph showing time spent waiting (I/O, locks) rather than executing
On-CPU flamegraph	A flamegraph showing active CPU execution time
Wall-clock time	Total elapsed time, including both CPU and wait time
Wide frame	A function that appears wide in the flamegraph, indicating it accounts for a significant proportion of time

“The flamegraph reveals that approximately 40% of CPU time is spent in the JSON serialisation routine. This is the primary hot path and the target for our optimisation effort.”

eBPF and Kernel Tracing Vocabulary

eBPF (extended Berkeley Packet Filter) is a technology for running sandboxed programmes in the Linux kernel for observability, networking, and security.

Term	Definition
eBPF	A kernel technology for safe, programmable observability and tracing
kprobe	A kernel probe that attaches to a kernel function for tracing
uprobe	A user-space probe that attaches to a user-space function
tracepoint	A stable, defined kernel hook point used for tracing
BPF map	A key-value data structure shared between the kernel BPF programme and user space
perf events	The Linux kernel performance monitoring subsystem
ftrace	A built-in Linux kernel tracing framework
bpftrace	A high-level tracing language for eBPF programmes

eBPF in Practice

“We used bpftrace to attach a kprobe to the do_sys_open kernel function and trace all file open calls from the service process. This revealed that the service was opening and closing the configuration file on every request — a previously unknown performance issue.”

Writing Performance Analysis Reports

Performance reports must connect measurements to conclusions and conclusions to recommendations.

Report structure:

1. Methodology — What tools were used and how
2. Observations — What the data shows
3. Analysis — What the observations mean
4. Recommendations — What actions to take

Report language patterns:

• “CPU utilisation on the application server reached 94% under the test load, indicating saturation. The flamegraph identifies JSON parsing as the dominant consumer, accounting for 38% of CPU time.”
• “p99 latency was 850 ms under the test conditions, exceeding the SLO threshold of 500 ms. The off-CPU flamegraph reveals that the majority of this time is spent waiting on disk I/O.”
• “We recommend replacing the current JSON library with a zero-copy implementation. Based on our benchmarks, this should reduce serialisation time by approximately 60% and bring p99 latency within the SLO.”

Example Sentences

1. “The flamegraph from the production profiling session shows that the hot path is in the B-tree traversal code — this accounts for 55% of total on-CPU time.”
2. “p99 latency for database queries has increased from 8 ms to 47 ms over the past two weeks; the USE analysis suggests disk saturation caused by the new background analytics job.”
3. “We used bpftrace with a kprobe on tcp_sendmsg to measure network send latency per process and identified the logging daemon as an unexpected source of high-priority network traffic.”
4. “Context switch overhead accounts for approximately 12% of the total wall-clock time in this workload — reducing thread count should improve throughput.”
5. “The tracepoint data confirms that cache miss rate increased sharply after the memory allocator change, which explains the 25% regression in throughput we observed in the benchmarks.”