Make it exact before making it fast
A bounded KV cache sped up Kronos inference 11.7× with bit-identical output, inside a boundary it has to respect.
- Topic
- Financial ML
- Analysis
- Jul 2026
- Status
- Working note
- Related
- Kronos evaluation
Kronos generates forecasts one bar at a time, and the original loop recomputed every layer over the whole context at each step. I added a pre-allocated rolling key–value cache. On CPU, with a 400-bar lookback and 100 greedy steps, it ran 11.7 times faster with bit-identical output, and the upstream regression tests still pass.
The cache is exact only while the lookback plus the forecast length fits inside the model's context window. Beyond that it falls back to the original path rather than returning a faster, subtly different answer.