AI Post Transformers

Latent Compaction with Test-Time Training: Learning Compact KV Caches

This episode examines "Latent Compaction with Test-Time Training," a preprint that asks whether a frozen model can shrink a document's KV cache by 16 to 32 times by running gradient descent on a small set of new key-value entries. The compaction happens before any questions are known. The discussion explains why the cache bottlenecks long-context memory and batching, and why text summaries lose…