Inference

In this blog, we show how helpful KV cache refreshes can be for long generation from small models, along with efficient ways of finding when to refresh using inference algorithms.

Overview of building an MCMC engine with practical implementation details.