0 / 7

The rig is our eval

Unit tests can't tell you whether a voice assistant handled a call well. We replay whole conversations, and every call that goes wrong becomes a new scenario.

Contents

On one of our test orders, every unit test would have passed. The assistant searched for each thing the caller asked for and found it, so the search tests were happy. It added each item to the basket, so the basket tests were happy. Then the order wouldn't go through. Items left over from earlier calls were still sitting in the basket beside the new ones, out of stock, and nobody had mentioned them.

Each part did its job. The conversation still went wrong. That is the gap a voice product lives in, and for us it is why the most important test we have is closer to an eval than a test. We call it the rig.

What the rig is

The rig replays a whole conversation end to end. A scripted caller speaks the turns of a real call we have seen. Everything on our side is real: the same turn loop that runs on live calls, the prompts, and the code that owns each task. The models can be the live ones, or scripted replies that cost nothing and test the plumbing around them. Only the outside world is simulated. The shop is a stand-in with its own catalogue, cart and checkout, so nothing is ever really ordered and nobody is ever really charged.

Each scenario then checks, in plain terms, the things that must be true. When the caller asks for something or takes something off, it is done by the next turn. An item the caller dropped stays dropped, and an offer they turned down isn't made again. A question gets an answer, never a stock line or silence. With live models, the assistant must not go round in circles on the same item. And talk that isn't about shopping never starts a shopping job.

A live call goes wrongWrite that call as a scenario, with the must-hold that would catch itFix until the rig passesLand the fixReal test calls on the real serviceNew failures go back to the topA live call goes wrongWrite that call as a scenario, with the must-hold that would catch itFix until the rig passesLand the fixReal test calls on the real serviceNew failures go back to the top
Every failure seen on a live call becomes a permanent scenario before it is fixed, and the fix has to pass the rig before it lands. Real calls still find what the stand-ins miss, and the loop starts again.

Why we call it an eval

A unit test expects the same answer every time. A conversation with a live model doesn't give you that. The wording changes from run to run, and sometimes a decision lands differently. So we judge a change to the shop the way we would judge a change of model: by replaying the calls and checking the outcomes, never the exact words.

That changes how we think about a failure. If a must-hold breaks one run in three, we treat it as a real problem and don't rerun until it goes green. It also means we keep each scenario honest about state. Every scenario, or chain of scenarios that build on each other, now runs in its own lane, with its own cart and its own test account, because we found out early that one scenario's leftover cart could quietly make the next one pass or fail.

Every bad call becomes a scenario

The rig started with a handful of real test shopping calls. It grew the only way an eval should: every time a test call went wrong, we wrote that call into the rig before fixing anything. A caller who asked for pork sausages and then a mango drink by its brand name, and heard about a sliced sausage. A caller who named a new item while still choosing the last one. Each of those is now a scenario, with the must-hold that would have caught it.

It has since grown well beyond shopping. Reminders, messages relayed to someone else, and setting up a horoscope reading all have their own scenarios now, more than two hundred and fifty in all. When someone on the team reports that a call went wrong, the first question we ask is which scenario reproduces it.

The rig is not the truth

The rig is only as good as its stand-ins. In late October our shopping scenarios were green, and the first real test orders still went wrong. The stand-in shop was simpler than the real one: it held none of the leftovers from earlier calls that a real account accumulates, so the rig had tested the conversation against a world that didn't exist.

So we never treat a green rig as permission to skip a real call. We still place real test orders ourselves. What the rig gives us is narrower and still very valuable: once we have seen a failure, it doesn't come back without us knowing. And each failure on a real call makes the stand-ins a little more like the real thing.

It has to run to count

The first version of the rig ran once a night. It never ran a single scenario. A missing credential made every run fail while fetching our code, before a scenario could start, and because it wasn't in anyone's way, nobody noticed.

Now it runs with every change. Our rule is that nothing touching shopping, reminders, relayed messages or horoscope set-up lands until the rig has been run on it, and every push to our main branch replays all the scenarios again with scripted replies. That run compares itself with the version we last deployed, and it flags a scenario only if it was green there, is red now, and stays red when run a second time. The scripted replays cost nothing in model calls, and the scenarios run in parallel, so a full run takes under half an hour. Saving money turned out to be the wrong thing to optimise for. An eval that doesn't run catches nothing, and one that runs on every change gets fixed the same day it breaks.

What we'd suggest

If you are building a voice assistant that does things for people, test the conversation as well as the parts. Replay real calls, check outcomes in plain language, and make every failure you see a permanent scenario. Then run it on every change, and keep making real calls anyway.

References and further reading

Cite as

Aika Labs. 'The rig is our eval.' aikalabs.ai/blog/the-rig-is-our-eval, 2026-11-20.

Related