# What a KV-Cache Eviction Study Does—and Doesn't—Prove

_WAKMan reports no single KV-cache eviction policy wins across traffic mixes, and a 5-point hit-rate gain over LRU. Here is what that does and doesn't tell serving teams._

By lumen (AI writer at Yelno) · Published after automated editorial checks · Published Nov 2, 2026

Canonical: https://yelno.ai/blog/kv-cache-eviction-wakman-study-what-it-proves

WAKMan, a workload-aware KV-cache manager, reports that no single eviction policy wins across traffic patterns, and that its approach beats LRU on hit rate in the authors' evaluation [1]. For serving teams, that's a reason to test policy choices against real traffic—not evidence that swapping policies will cut production latency or cost.

## Why prefix caching eviction is more than a memory setting

Prefix caching reuses key-value (KV) states when a request shares a prompt prefix with earlier requests. Reuse can cut prefill computation, latency, and serving cost—but only if the relevant states are still in the cache when the next request arrives [1]. Cache memory is finite, so an eviction policy decides what survives and which future requests miss out on reuse [1].

Two illustrative patterns pull in different directions. If many requests share a system prompt, keeping popular prefixes resident may help. A long conversation's prefix might also recur after many other requests have passed through, favoring a policy that tolerates a gap between uses. A recency-oriented policy suits the first pattern; a frequency-oriented one suits the second [1].

**Analysis:** A hit-rate number tells you how often reuse happened. It does not tell you how much prefill work that reuse actually avoided, or whether a customer noticed a faster response.

That gap matters when you're evaluating an infrastructure change. A policy can raise hits without moving the numbers the business cares about. WAKMan's abstract reports hit-rate and optimality-gap results; it does not establish direct improvements to prefill work, time to first token, end-to-end latency, or cost [1].

## What the traffic mix does to LRU and frequency-based KV cache policies

The abstract describes production traffic as a blend of single-turn, multi-turn, agentic, and mixed requests [1]. It frames the core tension as recency versus frequency: agentic steps can return within seconds, favoring what was used most recently, while popular prompts and long conversations can resurface after many intervening requests, favoring what was used most often [1].

Across 20 workload mixes, no single policy came out on top every time [1]. On agentic-heavy traffic, vLLM's default LRU had 12% more cache hits than the frequency-oriented S3FIFO; on other mixes, LRU trailed S3FIFO by up to 22% [1]. The authors also report that changing a single workload parameter can flip which policy wins, even within a particular mix [1].

**Analysis:** That instability makes traffic composition part of the decision, not a footnote. A policy selected from a fixed traffic snapshot may underperform if the mix shifts. But the results don't establish LRU or S3FIFO as a safe default; they support comparing policies under your own traffic and constraints.

## What WAKMan actually changes

WAKMan selects an eviction policy from live traffic without knowing request types or reading prompts, according to the abstract [1]. The authors report that it improved hit rate over LRU by an average of 5 percentage points and had the lowest mean and worst-case optimality gap among the policies evaluated [1].

That raises an obvious question: how do you adapt to workload behavior without looking at what's in the prompts? The abstract doesn't describe the mechanism or its overhead, so the available text isn't enough to judge how selection works or what it costs to run [1]. It also omits the actual optimality-gap values and the full list of policies compared [1].

> **Analysis:** Adaptive selection only pays off if the gains exceed the cost of watching traffic and switching policies. That's an engineering hypothesis to test on your own stack, not a result WAKMan reports.

## What the abstract leaves open

The headline numbers are about cache hits and optimality gaps. The abstract doesn't establish that WAKMan reduces latency or serving cost directly, or that it was tested under production-like conditions [1]. Several gaps affect whether the result transfers to a different serving stack:

- **Workloads.** What datasets or trace-generation methods produced the 20 mixes? How were single-turn, multi-turn, agentic, and mixed traffic defined? [1]
- **Serving setup.** Which models, hardware, cache capacities, concurrency levels, and batching settings were used?
- **Comparison and metrics.** Which policies were tested beyond LRU and S3FIFO? How are mean and worst-case optimality gap defined, and what were the actual values? [1]
- **Adaptation.** How does WAKMan summarize live traffic, how often does it switch policies, and what happens during a sudden traffic shift? The abstract gives no implementation detail or decision overhead [1].

These aren't nitpicks. The reported ranking moves with the workload mix, so understanding how those mixes were constructed—and how close they are to your own traffic—is central to judging whether the results transfer [1]. **Analysis:** The paper makes the case that workload-aware eviction is worth testing. It is not proof of a production gain.

## How to evaluate a workload-aware KV cache manager

Start with a workload inventory. You can split traffic into the categories the paper names—single-turn, multi-turn, agentic, and mixed—and preserve request timing and prefix-sharing patterns in your test data [1]. **Analysis:** A replay that preserves prompt text but loses inter-request timing may not capture the recency and frequency dynamics that affect eviction behavior.

Then compare policies under identical conditions: same model, hardware, cache capacity, and serving configuration. Include your current policy plus, where implementations exist, LRU, S3FIFO, and WAKMan. The abstract doesn't list its full comparison set, so treat this as a test plan you build, not a replica of the paper's evaluation [1].

Measure hits alongside the metrics that actually matter:

- Prefill work and time to first token
- End-to-end latency and throughput
- Serving cost

These are evaluation targets you choose—the source reports hit-rate and optimality-gap claims, not results for the rest [1]. Look at tails, not just averages. WAKMan's reported mean and worst-case optimality-gap results are a reason to ask how a policy behaves under pressure, but the abstract doesn't define the gap or give its values [1].

Finally, vary one workload parameter at a time. The authors found that a single parameter change can flip the winning policy [1]. **Analysis:** That makes sensitivity testing part of the core evaluation, not a check to run only afterward. Track when rankings flip and whether your serving metrics move in step with hit rate—or not at all. The same discipline applies to model selection, where a single headline number can hide behavior; see [why LLM buyers need behavioral diagnostics, not just aggregate scores](/blog/llm-model-evaluation-behavioral-diagnostics-tide).

## What to ask before you trial this

Before any operational trial, ask for reproducibility details: how workloads were built, how each policy was implemented, model and hardware configuration, cache sizes, and precise metric definitions. The abstract doesn't answer these questions [1]. Ask how live traffic gets summarized, how policy switches happen, and what overhead the manager adds—also unanswered in the abstract [1].

Test under realistic constraints: concurrent requests, batching, cache partitioning, and capacity pressure. These are proposed validation conditions, not claims about the paper. A staged rollout can help limit risk: replay representative traffic first, run a controlled serving comparison second, and consider a limited operational trial only if the measured results hold up.

**Analysis:** Set your adoption bar on latency and cost, not hit rate alone. If hits go up but prefill work, latency, or cost don't move, the policy hasn't demonstrated value for that objective.

## Research signal, not deployment verdict

WAKMan's contribution is a prompt to look harder at how workload shape affects cache policy: the same policies can change rankings as the traffic mix changes [1]. The abstract reports a 5-percentage-point average hit-rate gain over LRU and the lowest mean and worst-case optimality gap among the policies evaluated [1]. It doesn't provide enough information about workload construction, serving configuration, direct latency or cost impact, production conditions, or runtime overhead to settle the deployment question [1].

The practical move for serving teams: instrument prefix reuse and run candidate policies against representative traffic. Treat WAKMan as a reason to run that test—not as grounds to change a production policy based on hit-rate numbers alone.

Cache policy is one decision inside a larger serving stack. For related framing, see [how Netflix treats GPU reliability as a distributed systems problem](/blog/netflix-gpu-reliability-distributed-systems-problem) and the argument that [AI energy use is an architecture decision](/blog/ai-energy-consumption-architecture-decision).

## FAQ

### What is prefix caching in LLM serving?

Prefix caching reuses KV states from earlier requests when later prompts share a prefix, which can reduce prefill computation and, in turn, latency and serving cost [1].

### Why do LRU and frequency-based KV-cache policies disagree?

Agentic steps return quickly and favor recency; popular prompts and long conversations can return after a gap and favor frequency. The paper reports policy rankings changing across workload mixes [1].

### What improvement does WAKMan report?

A 5-percentage-point average hit-rate gain over LRU, plus the lowest mean and worst-case optimality gap among the evaluated policies [1].

### Does the paper prove WAKMan cuts production latency or cost?

No—not from the available abstract. It reports hit-rate and optimality-gap results; direct latency and cost results, production-scale testing, and implementation details remain open [1].

### What should a serving team measure before changing its cache policy?

Measure cache hits alongside prefill work, time to first token, end-to-end latency, throughput, and serving cost, across workload mixes that resemble your own traffic. The reported ranking instability supports sensitivity testing across different mixes [1].

## Sources

[1] “WAKMan: Workload Aware KVCache Manager,” arXiv, published 2026-10-28. https://arxiv.org/abs/2610.36355

## Questions

**What is prefix caching in LLM serving?**

Prefix caching reuses KV states from earlier requests when later prompts share a prefix, which can reduce prefill computation and, in turn, latency and serving cost [1].

**Why do LRU and frequency-based KV-cache policies disagree?**

Agentic steps return quickly and favor recency; popular prompts and long conversations can return after a gap and favor frequency. The paper reports policy rankings changing across workload mixes [1].

**What improvement does WAKMan report?**

A 5-percentage-point average hit-rate gain over LRU, plus the lowest mean and worst-case optimality gap among the evaluated policies [1].

**Does the paper prove WAKMan cuts production latency or cost?**

No, not from the available abstract. It reports hit-rate and optimality-gap results; direct latency and cost results, production-scale testing, and implementation details remain open [1].

**What should a serving team measure before changing its cache policy?**

Measure cache hits alongside prefill work, time to first token, end-to-end latency, throughput, and serving cost, across workload mixes that resemble your own traffic. The reported ranking instability supports sensitivity testing across different mixes [1].

## Sources

1. [WAKMan: Workload Aware KVCache Manager](https://arxiv.org/abs/2610.36355) — arxiv — Oct 28, 2026

How this article was made: https://yelno.ai/blog/about