7 seconds of silence after a caller read out a phone number — and it wasn't the model

We build a voice agent that answers calls for service businesses and books appointments. It was fast in testing. On real calls, some turns took far longer than others — the caller would finish speaking and get nothing back for an uncomfortably long time. Not every turn. Only some.

Eventually we measured it properly. Same agent, same ten-digit number, two readings:

Read straight through, no pauses: reply in about 1 second.

Read the way people actually read numbers, "oh-two, one-two-three... four-five-six-seven": about 7 seconds.

Nothing else changed. Same model, same TTS, same network.

The tempting place to look is the model, because that's the part that "thinks." It wasn't the model. It was the speech-to-text layer, one step earlier, doing something entirely reasonable.

Transcription engines have a formatting feature that turns spoken digits into a tidy (555) 123-4567, adds punctuation, formats dates and currency. To format a number, the engine has to know where the number ends. While you're still mid-number, it doesn't — so it holds the text back. Deepgram documents this plainly: with smart formatting on, the transcript is only finalised once you move on to other words, or after about three seconds of silence.

Now think about how anyone reads out a phone number. Digits, pause, digits, pause. Every one of those pauses looks to the engine like a number that might not be finished. Our agent wasn't slow to answer — it hadn't been told the caller had stopped talking.

The fix was disabling that formatting on the streaming path and reformatting downstream. Messier text, and worth it twice over: you can clean up a number after the fact, you can't give the caller back six seconds.

The generalisable part, and why I'm writing it up: defaults in a speech pipeline are tuned for transcript quality, not conversation. Every knob on that path is quietly choosing one over the other, and the docs present them as features rather than tradeoffs. If you're building anything real-time on ASR, walk the defaults and ask what each one buys and what it pays with. Ours cost six seconds on the single most important sentence a caller says.

For scale: the gap between two people in conversation is 200-300ms. Past a second, callers start repeating themselves. Past two, they reach for the zero key. Seven seconds isn't a slow bot, it's a hung call.