A Cache Simulator Worth Arguing With

Every architecture course tells you that increasing associativity reduces conflict misses and that LRU is a good replacement policy. Both statements are true and neither is useful, because they do not tell you how much, and the answer to how much is the only thing that ever influences a design decision.

So: a configurable cache model, fed with real memory traces, reporting miss rates across the parameter space.

What the numbers said

Two things stood out.

The first is how quickly associativity stops paying. Going from direct-mapped to two-way is dramatic. Two to four is worth having. Beyond eight, on most of the traces I ran, you are spending transistors and latency to chase a fraction of a percent. Everyone knows this; seeing it plotted from your own measurements is a different kind of knowing.

The second is that replacement policy mattered far less than I expected, and mattered most exactly where the cache was already too small to help. When there is enough capacity, almost any sensible policy converges. When there is not, no policy saves you.

The part that was actually hard

Trace handling. The simulation itself is a few hundred lines. Getting traces that represented anything, parsing them without becoming the bottleneck, and being confident that the results were not an artefact of the input took most of the time.

That ratio has held for every measurement project I have done since.