OBIT
← ALL RESEARCH
ASKV · DESIGN REPORT

Protect routing.
Compress the payload.

At long context lengths, the KV cache takes up most of the memory. ASKV uses attention patterns and application requirements to decide which tokens need high fidelity. It keeps frequently used and protected tokens exact or quantized at high quality, while compressing less-used entries along the hidden dimension.

THE HYPOTHESIS
Our hypothesis is that relatively few tokens are critical to attention routing. Older, less-attended cache entries may also have more redundancy along the hidden dimension than recent, sink, protected, or highly attended entries.

Prior work supports two parts of this hypothesis. H2O, Scissorhands, StreamingLLM, FastGen, SnapKV and PyramidKV use the fact that attention varies widely across tokens, heads and layers. Palu, Eigen Attention, OjaKV and KVTC take advantage of redundancy along the hidden dimension of KV tensors.

The design preserves the information needed to route attention while compressing the remaining payload. Applying SVD uniformly to cold keys and values would miss that distinction.

DESIGN REQUIREMENTS
No reconstruction on the decode path
Specialized kernels run attention directly over the mixed-fidelity cache. Reconstructing cold keys and values before each attention operation would turn this into a storage codec rather than a cache used during decoding.
Maintenance off the critical path
SVD and repartitioning must not block token generation. While a background worker compresses a snapshot of older sealed pages, decoding continues to append entries to a full-fidelity or FP8 delta cache.
Copy-on-write, not in-place
At a decode boundary, we atomically switch to the worker's new cache manifest. If it's not ready, decoding continues with the current version.
A smaller cache won't help inter-token latency if generation has to wait for compression. Keeping maintenance off the decode path is a requirement of the design.
EXPECTED COSTS

ASKV will likely take longer to produce the first token than FP16, FP8 or pruning baselines unless compression runs asynchronously, is amortized, or is reused across turns. Long prompts already make prefill expensive. ASKV adds statistics collection, token partitioning and an SVD pass. The likely use cases are workloads where the compressed cache can be reused, transferred, or prepared before the request arrives.

Quantization is already practical in production. OjaKV also keeps the first and most recent tokens at full rank while compressing the middle with a low-rank representation. We need to show that selecting cache fidelity based on attention provides a useful improvement over these approaches.

The design report is complete. Benchmarks for throughput, resident KV memory and long-context quality still need 80GB-class hardware. We don't yet have results that establish the proposed performance benefits.