KV Cache
KV CACHE
Why caching keys & values matters
During generation, past tokens' K and V don't change — recomputing them every step would waste compute. KV-cache stores them so each new token only runs one layer pass.
Compute saved (illustrative)0%
KV memory (per layer, schematic)16× d_model