0 / 5

Cold and warm: prefix caching on a self-hosted voice model

Our self-hosted model took nearly two seconds to start answering, and we were sure caching was broken. It wasn't. What we learned about the first call on a warm GPU.

Contents

We run an open model on a single GPU of our own for demos of Aika. On a phone call, what matters most is how quickly the model starts answering, because the caller is sitting in silence until it does. We set ourselves a bar of 800 ms to the first token.

Our readiness check kept failing it. The first answer took about 1.9 seconds, and the server's counters said the prefix cache had not been hit once. We were fairly sure caching was broken, and started looking for the bug.

There wasn't one, at least not the bug we were looking for. This post is about what was actually going on, because the same thing will bite anyone who self-hosts a model for a conversation.

Why the prompt is the problem

A voice assistant's prompt is long and mostly the same on every turn. Ours is around 22,000 tokens: instructions, tool descriptions, what we know about the caller, and the conversation so far. Only the last few hundred tokens are new on any turn, mostly the thing the caller just said.

Before a model can produce its first token, it has to read the whole prompt. That reading step, the prefill, is where most of the wait goes on a long prompt. Prefix caching is how you avoid paying for it again: the server keeps the work it already did on the start of the prompt, and when the next request begins with exactly the same tokens, it only has to read the part that changed.

Cold: nothing cachedread the whole prompt, then answer: about 1.9 sWarm: same start as last timereuse 21,952 tokens, read only the new tail: about 530 msCold: nothing cachedread the whole prompt, then answer: about 1.9 sWarm: same start as last timereuse 21,952 tokens, read only the new tail: about 530 ms
On a cold server the model reads the whole prompt before answering. On a warm one it reuses what it has already read, 21,952 of 22,647 tokens in our test, and reads only the new part.

So a warm turn should be fast and a cold one slow. Our numbers looked cold every time.

One host, so why miss?

The point that settled our approach was simple. It is one machine, with nothing routing requests between servers, so there is no reason for an identical prompt to miss the cache. Either the prompts weren't identical, or we were measuring something other than what we thought.

So we tested it directly. Six fresh sessions through our normal stack, one after another, followed by three byte-identical requests straight to the model as a control.

The six sessions came back with a median of 530 ms to the first token, and each one reused 21,952 of its 22,647 prompt tokens from the cache. The control told the same story on its own: zero cache hits on the first identical request, then 97.5% reuse on the two repeats. The control used its own prompt, so its share differs a little from the sessions' 96.9%.

Caching worked fine. Our readiness check had been measuring the one request that can never hit the cache: the first one after the server starts. An empty cache has nothing to reuse.

The first call is the one that matters

That sounds like a measurement mistake, and partly it was. It also points at a real problem, because on a phone line the first turn of a call is exactly the turn the caller judges you on. "Hello?" followed by two seconds of silence is how calls end early.

So we stopped treating warm-up as an optimisation and started treating it as part of bringing the model up. Three things mattered.

Warm the exact prefix the first turn will send. A warm-up only helps if its prompt starts exactly the way the real one does. Our first version of the warm-up left out the tool descriptions, which sit early in the prompt. Everything after that point differed, so the real first turn missed almost all of what the warm-up had prepared. A review caught it before it shipped.

Keep the prompt in a stable order. Anything that changes from turn to turn has to come after everything that doesn't. The instructions, then the tools, then the conversation so far, then the new material at the very end. One thing we chose to leave: some caller context that can change mid-call sits inside the instructions, so when it changes, the server re-reads everything from that point onward. We accepted that cost on the turns where it happens rather than restructure further.

Warm the real path, not a shortcut. Our readiness check now primes the same conversation path a real call takes before it measures anything. With that in place, readiness checks passed at between about 470 and 540 ms to the first token.

What we would tell someone starting out

Measure cold and warm separately. A single latency number for a self-hosted model mixes two very different things: the first request after start-up, and every request after that. If you average them you get a number that describes neither.

Don't trust a counter alone. The cache counters were accurate. We were reading them on the wrong request. A direct control, the same prompt sent twice to the model with nothing else in the way, is the quickest way to tell a caching problem from a measurement one.

And treat prompt layout as a performance decision. Where you put the thing that changes decides how much of the prompt the server can keep. On a 22,000-token prompt, that is the difference between reading a few hundred tokens and reading all of them.

References and further reading

Cite as

Aika Labs. 'Cold and warm: prefix caching on a self-hosted voice model.' aikalabs.ai/blog/cold-and-warm-prefix-caching, 2026-11-18.

Related