English for Systems Engineers: Performance Analysis and Kernel Vocabulary

Systems engineers work at the lowest levels of the software stack — operating systems, kernels, performance analysis tools, and hardware interfaces. The vocabulary in this domain is highly specialised, and communicating performance findings clearly in English requires precision. This guide covers the core terms used in performance analysis, kernel observability, and report writing.

Performance Analysis Vocabulary

Term	Definition
Flamegraph	A visualisation of call stacks, where the width of each frame represents the proportion of time spent there
p99 latency	The 99th percentile latency — the worst-case time experienced by the slowest 1% of requests
Throughput	The number of operations completed per unit of time
Saturation	The degree to which a resource is fully utilised, leading to queuing
Utilisation	The percentage of time a resource is busy
Bottleneck	The resource that limits overall system throughput
Working set	The set of pages or data actively used by a process at a given time
Context switch	When the CPU switches from executing one process to another, incurring overhead
Cache miss	When requested data is not in the CPU cache and must be retrieved from slower memory

The USE Method

Brendan Gregg’s USE method provides a systematic framework for performance analysis:

• U — Utilisation: Is the resource being used heavily?
• S — Saturation: Is there a queue forming?
• E — Errors: Are errors being reported?

“Applying the USE method to the storage subsystem, we found 95% utilisation, significant I/O queue depth (saturation), and no errors — indicating that the disk is the bottleneck rather than any fault condition.”

Flamegraph Vocabulary

Flamegraphs are one of the most powerful tools for performance analysis. When discussing flamegraphs in reports or team meetings, use precise language.

Term	Meaning
Stack frame	A single function call in the call stack
Hot path	The code path that consumes the most CPU time
Off-CPU flamegraph	A flamegraph showing time spent waiting (I/O, locks) rather than executing
On-CPU flamegraph	A flamegraph showing active CPU execution time
Wall-clock time	Total elapsed time, including both CPU and wait time
Wide frame	A function that appears wide in the flamegraph, indicating it accounts for a significant proportion of time

“The flamegraph reveals that approximately 40% of CPU time is spent in the JSON serialisation routine. This is the primary hot path and the target for our optimisation effort.”

eBPF and Kernel Tracing Vocabulary

eBPF (extended Berkeley Packet Filter) is a technology for running sandboxed programmes in the Linux kernel for observability, networking, and security.

Term	Definition
eBPF	A kernel technology for safe, programmable observability and tracing
kprobe	A kernel probe that attaches to a kernel function for tracing
uprobe	A user-space probe that attaches to a user-space function
tracepoint	A stable, defined kernel hook point used for tracing
BPF map	A key-value data structure shared between the kernel BPF programme and user space
perf events	The Linux kernel performance monitoring subsystem
ftrace	A built-in Linux kernel tracing framework
bpftrace	A high-level tracing language for eBPF programmes

eBPF in Practice

“We used bpftrace to attach a kprobe to the do_sys_open kernel function and trace all file open calls from the service process. This revealed that the service was opening and closing the configuration file on every request — a previously unknown performance issue.”

Writing Performance Analysis Reports

Performance reports must connect measurements to conclusions and conclusions to recommendations.

Report structure:

1. Methodology — What tools were used and how
2. Observations — What the data shows
3. Analysis — What the observations mean
4. Recommendations — What actions to take

Report language patterns:

• “CPU utilisation on the application server reached 94% under the test load, indicating saturation. The flamegraph identifies JSON parsing as the dominant consumer, accounting for 38% of CPU time.”
• “p99 latency was 850 ms under the test conditions, exceeding the SLO threshold of 500 ms. The off-CPU flamegraph reveals that the majority of this time is spent waiting on disk I/O.”
• “We recommend replacing the current JSON library with a zero-copy implementation. Based on our benchmarks, this should reduce serialisation time by approximately 60% and bring p99 latency within the SLO.”

Example Sentences

1. “The flamegraph from the production profiling session shows that the hot path is in the B-tree traversal code — this accounts for 55% of total on-CPU time.”
2. “p99 latency for database queries has increased from 8 ms to 47 ms over the past two weeks; the USE analysis suggests disk saturation caused by the new background analytics job.”
3. “We used bpftrace with a kprobe on tcp_sendmsg to measure network send latency per process and identified the logging daemon as an unexpected source of high-priority network traffic.”
4. “Context switch overhead accounts for approximately 12% of the total wall-clock time in this workload — reducing thread count should improve throughput.”
5. “The tracepoint data confirms that cache miss rate increased sharply after the memory allocator change, which explains the 25% regression in throughput we observed in the benchmarks.”

Navigating the Noise: Practical Phrases for Systems Engineers

As a non-native speaker of English, you’ll quickly realize that technical jargon can be particularly challenging. Beyond simply knowing the definitions of terms like “saturation” or “p99 latency,” it’s crucial to understand how they are used in real-world communication – especially when documenting performance issues and proposing solutions. The key is to move beyond a literal translation and grasp the nuanced phrasing that experienced systems engineers employ. It’s not just about saying “high latency”; it’s about conveying impact and initiating a discussion.

One area where this difference becomes apparent is in describing problems. Instead of simply stating “the system is slow,” you need to frame the issue within a performance context. Phrases like “increased p99 latency during peak hours” or “saturation observed in the network queue” immediately communicate severity and potential causes. Similarly, when discussing root causes, avoid vague statements like “there’s a problem with the kernel.” Instead, use precise language such as “a kprobe identified excessive CPU usage within the database driver.” This level of detail is expected – it’s about demonstrating your understanding of the system’s architecture and the tools used to analyze it.

Furthermore, learning how to request information effectively is critical. Don’t just say “Can you look at this?” Instead, formulate a specific query: “Could you investigate potential saturation issues in the message queue using eBPF tracing data?” This demonstrates your knowledge of relevant technologies and guides the investigation towards a productive outcome. Remember, clear communication minimizes ambiguity and accelerates problem resolution. A simple Slack message like “Looks like high latency on server X – investigating” is far less effective than a well-structured report highlighting metrics and potential causes.

Finally, consider how you frame recommendations. Rather than saying “we need to optimize the database,” try “We recommend implementing rate limiting based on eBPF trace data to mitigate saturation issues.” This demonstrates a deeper understanding of the problem’s root cause and proposes a targeted solution.

Here’s an example of using bpftrace to analyze CPU usage:

bpftrace -e 'cpu_total > 80 { printf("%ld\n", cpu_total); }' --pid 1234

This command, executed on a system with process ID 1234, uses bpftrace to monitor the total CPU usage and prints it to the console if it exceeds 80%. This is a tangible example of the type of data systems engineers would use in performance reports.