0 / 9

How a phone call reaches a voice agent

Before a speech model hears a caller, their voice crosses a handset, two operators, a carrier trunk and a jitter buffer. What each hop does to the audio.

Contents

Most of what gets written about voice agents starts at the transcript. That has always bothered us a little, because by the time there is a transcript, a lot has already happened to the caller's voice, and most of it is damage.

At Aika, people reach us on an ordinary phone number. A lot of them call from basic phones, from a market or a bus stop, on whatever network they have. So we have spent a fair amount of time over the last few weeks following a call from the caller's mouth to our speech model, hop by hop, measuring what each one does. This is that trip, written down the way we wish someone had explained it to us.

HandsetAMR, 8 or 16 kHzCaller's operator8 or 16 kHzInterconnectusually 8 kHzNumber's operator8 kHzTrunkSIP + RTP, G.711, 20 ms packetsGatewayjitter buffer, 8 to 16 kHzSpeech modelexpects 16 kHzEach hop can lose or add something.HandsetAMR, 8 or 16 kHzCaller's operator8 or 16 kHzInterconnectusually 8 kHzNumber's operator8 kHzTrunkSIP + RTP, G.711, 20 ms packetsGatewayjitter buffer, 8 to 16 kHzSpeech modelexpects 16 kHz
The trip a caller's voice makes before any model hears it. Wideband usually ends at the interconnect between operators; from there the audio is 8 kHz until a gateway resamples it.

It starts in the handset

The phone compresses speech before it ever leaves the device. On a 2G call, and on most feature phones, that is AMR narrowband or an older narrowband codec: the audio is sampled 8,000 times a second, which keeps sound up to about 3.4 kHz. On a 4G phone with VoLTE and HD voice on, it can be AMR wideband: 16,000 samples a second and sound up to about 7 kHz.

You can hear the difference. A lot of what separates "s" from "f", or "sh" from "s", sits above 4 kHz. Narrowband keeps the vowels and throws away much of what makes consonants distinct, which is exactly the part a speech recognizer leans on to tell two similar words apart.

Then it crosses operators

The caller's operator carries the call to whichever operator owns the number they dialled. If those are different networks, the call crosses an interconnect between them, and in practice that is usually where wideband ends. Interconnects are commonly narrowband, so even a caller on a new phone tends to arrive as 8 kHz audio.

Neither the caller nor we can see any of this. We only get whatever audio comes out the other end.

Then the trunk

A business number sits on a trunk, which is a connection between the operator and a telephony provider, or between the operator and your own equipment. Calls on it are set up with SIP, and the audio travels as RTP packets.

The codec on that last hop is almost always G.711. It is from 1972 and it still carries most of the world's phone audio. 8-bit samples, 8 kHz, 64 kilobits a second, on a logarithmic scale (A-law in India and Europe, µ-law in North America) so quiet speech keeps more detail than loud speech. Each packet usually holds 20 ms of audio, which is 160 samples.

From the trunk there are two common ways for audio to reach your code. A telephony provider can take the call and stream the audio to you over a WebSocket in small frames. Or the trunk can send SIP and RTP straight to a gateway you run yourself. The first is much easier to start with and it is what we started with. The second means every conversion is yours, which starts to matter the moment you measure them.

The jitter buffer, which is harder than it looks

RTP runs over UDP, so packets show up late, out of order, twice, or never. A stream that should arrive every 20 ms arrives as a ragged burst. Something has to collect the packets, put them back in order, wait a bit for stragglers and then play out a steady 20 ms clock.

That is the jitter buffer, and the hard part is deciding how long to wait. Wait too little and late packets get thrown away as loss. Wait too long and every turn of the conversation gets slower. On top of that the two ends' clocks drift, and some senders simply stop sending during silence, which looks exactly like packet loss unless you handle it.

We learned this the expensive way. The first gateway we built had a jitter buffer we wrote ourselves, and its timing logic kept producing small but real audio loss bugs, round after round of review. In the end we swapped it for SpeexDSP's adaptive jitter buffer, which has been carrying VoIP audio for years, and only kept the bits that are actually specific to us: parsing packets, pulling keypad presses out of the stream, filling silence on the output clock and counting loss and lateness on every call. Writing your own jitter buffer is a great way to learn how one works. We would not ship one again.

8 kHz to 16 kHz

Most speech recognition models are trained on 16 kHz audio and phone audio is 8 kHz, so somewhere the samples have to be resampled to double the rate.

Resampling cannot bring back frequencies that were never there. Done badly, though, it adds something. On one route we measured, there was a sharp jump between samples at almost every 20 ms boundary, roughly twice the size of the normal step between neighbouring samples. On a clean route the same measurement sat close to 1. You hear it as a faint, regular crackle. It is what you get when each packet is converted on its own, with no memory of how the previous packet ended.

That measurement is why, when we built our own gateway for a test route, we did the conversion ourselves with a resampler that keeps its state from one packet to the next.

Keypad presses travel separately

When a caller presses a key, the tone could in principle travel inside the audio. It usually does not. Most trunks send key presses as separate RTP events (RFC 4733), so a codec cannot smear them and a speech model never hears a beep. Your gateway has to pull those events out and pass them on as a key press, not as sound.

Knowing when your audio finished playing

The return path has its own problem. We send synthesised speech back down the line, and we need to know when it finished playing, both to tell whether the caller interrupted and to know whether an important sentence actually went out.

Providers usually give you a marker for this. You put a named mark after your audio and they echo it back once everything before it has played, or has been cleared because the caller interrupted. It is worth being precise about what that echo proves, because it is easy to read too much into it. At most it tells you the audio left our side of the line. It does not tell you a person heard it. We treat it as evidence about our side of the line and design anything that really matters, like a consent notice, so it does not depend on more than that.

What we take from all this

Every hop takes something away or adds something. By the time audio reaches a model it has usually been squeezed to 8 kHz at least once, crossed a network boundary, been reordered and padded, and been resampled. A model trained on clean 16 kHz recordings is listening to a different signal from the one it learned on.

So we test on real phone audio, we measure every conversion instead of trusting it, and where a conversion is ours to own, we own it. The obvious next question is why we don't just get wideband calls. We tried. In India that turns out to be a much longer story, and it is the next post.

References and further reading

Cite as

Aika Labs. 'How a phone call reaches a voice agent.' aikalabs.ai/blog/how-a-phone-call-reaches-a-voice-agent, 2026-11-12.

Related