What gets measured, what gets modeled in performance engineering

Performance engineering is two activities: measure how fast a program actually runs, then use knowledge of the hardware to make it faster. The premise underneath both is easy to forget: computing feels abstract — things just get computed — but every algorithm executes on a machine, and the machine has opinions. If you want to compute on an abacus, you must represent your math the way an abacus represents numbers; a CPU is no different, it just hides its beads (caches, pipelines, branch predictors) better. The Understand the CPU pages do the measuring: cache-size detection, branch-misprediction experiments, memory-bandwidth tests. This page supplies the other half: the models that predict what those measurements should say — and what to change when a program comes in slower than its prediction.

What you measure

Four broad classes of measurement, distinguished by what instrument produces them and what they tell you about:

1. Time and throughput

Wall-clock time (with proper warm-up + statistical confidence intervals — naive single-run timings are nearly useless). Throughput in ops/sec, GB/s, or instructions per cycle (IPC). Latency distributions including the tails (p50, p99, p99.9 — averages hide the bad cases). Done with carefully-written benchmarks (Google Benchmark, Criterion.rs), Linux perf stat, or simple $\text{clock\_gettime}$ with repetition.

2. Hardware performance counters

The CPU has dedicated counters for everything that costs cycles. perf (Linux), Intel VTune, AMD uProf, Apple Instruments, and LIKWID read them: L1/L2/L3 cache misses, branch mispredictions, TLB misses, retired instructions, front-end stalls, back-end stalls, memory traffic, FLOPs issued, vector-unit utilization. The measurement methodology page covers the discipline; the cache-size detection and branch experiments are example readouts.

3. Scaling behavior

Strong scaling: keep the problem size fixed, vary the number of cores or nodes, measure speedup. Weak scaling: scale problem size with cores, measure how time per worker grows. The shape of the scaling curve diagnoses bottleneck type — flat from the start means serial-bound; curling over at 16 cores means contention or memory-bandwidth saturation; retrograde (slower at high core counts) means coherency traffic dominating.

4. Resource utilization

Memory footprint (RSS / VSS), heap allocation rate, file I/O rate, network bandwidth, GPU utilization, power draw. Each can be a hidden bottleneck even when CPU utilization looks fine.

What a model predicts

Performance modeling is built on a small set of foundational laws. Each predicts an observable that you compare to what your measurement actually says.

The roofline model (Williams-Waterman-Patterson 2009)

The single most useful framework. Plot performance (GFLOPS) on the y-axis against operational intensity (FLOPs/byte) on the x-axis. The achievable performance is bounded by two roofs:

where ${\pi }_{\mathrm{peak}}$ is the CPU's peak floating-point rate (clock × cores × SIMD lanes × FMA-factor-of-2) and ${\beta }_{\mathrm{mem}}$ is the achievable memory bandwidth (STREAM Triad gives this empirically). The ridge point where the two roofs intersect, at intensity ${\pi }_{\mathrm{peak}}/{\beta }_{\mathrm{mem}}$, separates memory-bound from compute-bound kernels. Knowing this single number per machine is worth more than any list of micro-benchmarks.

Amdahl's law (1967)

where $p$ is the parallelizable fraction. If 10% of your code is inherently serial, you can never go faster than 10× no matter how many cores. The strong-scaling ceiling. Brutal and universal.

Gustafson's law (1988)

The weak-scaling counterpart: if you scale the problem size with $N$, the parallel fraction stays effectively fixed and speedup is closer to linear. The argument against Amdahl's pessimism: in practice, bigger machines run bigger problems.

USL: Universal Scaling Law (Gunther)

Amdahl with two extra knobs: $\alpha$ is contention (serialization on a shared resource) and $\beta$ is coherency delay (cross-thread synchronization cost). Crucially, $\beta >0$ means scaling goes retrograde — adding cores past some optimum makes things slower. Almost every real distributed system has nonzero $\beta$. Fit your scaling curve to USL; if $\beta$ is large, coherency is your problem.

Little's law

Concurrency = throughput × latency. Universal — applies to a CPU pipeline, a network queue, a coffee shop. If you know any two, you know the third. For a CPU with 4-cycle L1 latency that can issue 4 loads per cycle, you need 16 outstanding loads to hide L1 latency. Same logic at every level of the stack.

Memory-hierarchy latencies

Typical modern x86, in cycles:

• Register: 1
• L1 cache: 4–5
• L2 cache: 12–15
• L3 cache: 30–50
• DRAM: 200–300
• NVMe SSD: ~50,000
• Network round-trip (datacenter): ~250,000

Each level is ~3-10× slower than the previous. The "memory wall" is the gap from L1 to DRAM — 50-75× — and it's been growing for decades while compute has been getting cheaper. This single table dictates most cache-optimization strategy.

The bridge

Predict from a model, measure from a profiler, compare. The gap is where the optimization lives:

Which technique for which quantity

Match the technique to the bottleneck, never the reverse. The table below is organized by what you're bottlenecked ON, not by technique alphabetically. The most common mistake in performance engineering is applying a clever technique to the wrong bottleneck — vectorizing memory-bound code, threading already-bandwidth-saturated code, prefetching when you're compute-bound. Match first.

One-sentence summary

Performance engineering is a measurement + modeling discipline, not a list of tricks. Profile your code (measurement), place it on the roofline and check Amdahl/USL (model), find the gap (bridge), pick a technique that matches the gap (methods table). The techniques in the methods table only help when applied to the bottleneck they target — match first, optimize second.

Related on this site

• Understand the CPU (overview) — the experiment side. All the "what your hardware actually does" measurements live there.
• Roofline model experiments — measure your machine's ${\pi }_{\mathrm{peak}}$ and ${\beta }_{\mathrm{mem}}$.
• Measurement methodology — how to time things correctly so the numbers mean what you think they mean.
• Cache-size detection and cache conflicts — concrete demonstrations of the memory-hierarchy latencies in the model section above.
• Vectorization and false sharing — concrete techniques and pitfalls.
• Capstone lab — where you put the whole pipeline together on a real kernel.
• Parallel quantities-of-interest pages: comp neuroscience, quantum chemistry, nuclear physics, condensed matter, statistical mechanics.