Blog Posts and Articles

KV-Cache Refresh Methods for Long Generation

Authors: Yahya Emara, Woojeong Kim, Mohamed Abdelfattah

In this blog, we show how helpful KV cache refreshes can be for long generation from small models, along with efficient ways of finding when to refresh using inference algorithms.