0 / 6

Let the voice model be a voice model

We used to give our conversation model a menu of tools and let it decide, extract and save. Now code decides and acts, and the model just speaks. Why we moved.

Contents

A caller said "tamatar kar do, kitna kar rahi ho?", which is roughly "add tomatoes, and how much is it coming to?". Aika searched the shop for that whole sentence. Tomatoes didn't come back. Ketchup did.

Nothing in that chain was broken in the usual sense. Speech recognition got roughly the right words. The conversation model called the search tool, the way we had designed it to. It just passed the caller's entire sentence along as the search term, because working out which part of a sentence is the item is a judgement call, and we had left that judgement inside a model whose main job was to talk.

This post is about how we stopped doing that, and why we think it was the right call for a voice product.

How we built it first

The standard recipe for an assistant that does things is to give a large language model a list of tools, describe each one, and let it decide what to call, when, and with which arguments. It is a good recipe for getting something working. You add a capability by writing a function and a paragraph describing it.

On live phone calls it had costs that kept growing.

The model became our extractor. A reminder's recipient, day and time were filled in by the model writing tool arguments from raw speech-to-text, so a schedule was only as good as the model's guess at a garbled sentence. "Kal das baje" (tomorrow at ten) is easy. "Har ekadashi" (every ekadashi) or "paanch minute baad" (in five minutes) is where guesses go wrong.

The model also made decisions it shouldn't own. Whether to save a reminder, whether the caller had agreed, whether to drop a setup halfway. In testing we saw it save on "is it done?", and on one real call it saved the wrong kind of thing twice for a single request. A conversation model is very good at sounding sure of itself, which is exactly the wrong property for the thing that decides whether to commit.

And the menu got heavy. On a typical production turn, the tool descriptions took up a large part of the prompt. The model was reading a manual on every turn before saying a word, and the caller was paying for that in latency.

How it works now

We split each turn into jobs and gave each job to the thing that is good at it.

BeforeSpeech-to-textConversation model: decides, extracts, saves through toolsReplyNowSpeech-to-textExtractor: the caller's factsOwners resolve places, items, timesDecision model: agreed? changing? leaving?Code acts and savesConversation model: speaksBeforeSpeech-to-textConversation model: decides, extracts, saves through toolsReplyNowSpeech-to-textExtractor: the caller's factsOwners resolve places, items, timesDecision model: agreed? changing? leaving?Code acts and savesConversation model: speaks
Before, one model heard garbled speech, chose a tool, filled its arguments and decided whether to save. Now each of those jobs belongs to a separate step, and the conversation model only speaks.

An extractor reads the caller's facts. After each turn, one model pulls out what the current task could use, in the caller's own words: items and quantities, a place, a time. "Doodh, ande aur bread chahiye" becomes three items, and words like "kar do" or "chahiye" stay out of them.

Code that owns each fact resolves it. A place is matched against places the caller has used before, an item is searched as an item, and a time becomes an actual time.

A decision model answers the small questions. Did the caller agree to what was just read back? Are they changing something, or dropping it? We already used a decision model like this for consent and payment on live calls, as we wrote in September. Now it covers the decisions inside each task too.

Code acts, and the model speaks. Nothing is saved unless the caller has just heard the full record and the decision model is confident they agreed to it. The conversation model is told what happened and what is still missing, and says that in its own words.

What callers notice

Mostly that Aika understood them. "Doodh chahiye, ghar pe" (I need milk, at home) searches for milk to deliver to their saved home, and "doodh chahiye" on its own asks where to send it. A correction like "sauce nahi, asli tamatar" (not sauce, real tomatoes) searches again for what they meant.

The less visible change is that, in the tasks we have moved over, Aika can no longer save, skip a save or fill in a detail on a whim. The model doesn't hold a tool for any of that. Code saves when the rule is met.

What changed for us

Every decision now gets logged with its outcome, without the caller's words, so we can count a wrong save or a missed one. When a decision lived inside a tool call inside a long generated reply, the decision and the wording were tangled together. Now we can test them separately.

The prompt got a lot lighter too, and a model that doesn't have to read a long list of tool descriptions starts talking sooner.

The part we find most interesting is what it does to the conversation model's job. It no longer has to be a good planner, a good extractor and a careful committer all at once. It has to be a good speaker that sticks to the facts it is given. That is a much smaller job, and one that smaller, cheaper models running on our own hardware might eventually do. We tested exactly that recently and we'll write about how it went soon.

What it cost

It is a lot more design work up front. Every task needs its facts listed and its decisions phrased as real questions we can check. Tool calling let us skip that and find out on live calls. We now think that was the expensive way to find out.

There is also an edge. When the extractor misses a turn, or a caller does something no task expected, the conversation still has to go somewhere sensible. We keep that path narrow on purpose: the model can talk, ask and explain, and the actions stay with code.

References and further reading

Cite as

Aika Labs. 'Let the voice model be a voice model.' aikalabs.ai/blog/let-the-voice-model-be-a-voice-model, 2026-11-16.

Related