My local voice call took 105s per reply
A tester sent me a log where one voice reply took 105.6 seconds. First audio landed at 40.7 seconds. The reply was 20 characters long.
The machine was not weak. RTX 3080, Ryzen 9 5900X, Windows 11 Pro. My app runs the whole voice loop on that card: speech to text, the language model, and the text to speech. Nothing goes to a server. That is the entire point of the product, and it was also the reason every failure was mine to find.
Four separate bugs were stacked on top of each other. Here they are.
1. The app was not slow, it was dying
Before the latency work, the report was worse: calls just ended. The window vanished. No dialog, no error, and nothing in any log.
The logs were never going to have the answer. My crash handler installs a Rust panic hook, and a panic hook catches Rust panics and nothing else. A native fault inside a C dependency never unwinds and never reaches it.
The real cause: I ship whisper.cpp, and I compile it in CI. Its build probes the build machine for AVX-512 support and bakes the answer into the binary. GitHub runners have AVX-512. Zen 3 does not. The first real compute kernel is the sentence you just spoke, so the process was killed by an illegal instruction at the exact moment the tester started talking.
The same commit produced a working or a crashing installer depending on which runner picked it up. That is how it passed CI for weeks.
Speech to text on that build was taking 137 seconds. It now takes low hundreds of milliseconds.
2. Two CUDA processes fighting over one card
This one I found by reading two sets of logs side by side.
The language model was running at 115 tokens per second on an idle GPU. While the voice engine was synthesizing, it fell to 0.63 tokens per second. The voice engine ran at 26.5 iterations per second alone, and 1.5 while the model was generating.
Both collapsed together. Both recovered the instant the other stopped. There were 5 GiB free on a 12 GiB card, so this was never memory pressure. It was Windows context switching between two separate processes that each held their own CUDA context on the same device.
The painful part: my architecture caused it. I was deliberately overlapping generation of sentence N+1 with synthesis of sentence N. That is a real win when the voice engine lives in your process or on the CPU. Point it at a separate process on the same GPU and the overlap turns into an 8x latency multiplier.
The fix was to stop being clever. When the model is local and the natural voice is on, generation finishes first, then synthesis runs back to back at full speed.
3. She was re-learning her own voice every sentence
To speak in a cloned voice, the engine studies a short reference clip. Mine was doing that study again for every single sentence. Same clip, same result, roughly three quarters of a second of silence each time.
It learns it once now and keeps it. On a three sentence answer that is about two seconds of dead air removed.
4. The end of turn timer was shorter than a human pause
She decided you had finished speaking after 0.7 seconds of silence. That is shorter than the pause most people leave between two sentences.
So a two sentence answer lost its second sentence. Not delayed. Discarded, never transcribed. Moved to 0.9 seconds, and the slider still respects a user who wants it faster.
Where it is now
Calls are stable on that same 3080 and they move at the pace I expect from a cloud assistant on my phone, with every part of it running on the local machine.
One thing is still wrong and I am not going to pretend otherwise: she can still clip a word at the end of a spoken segment. The current build adds the per sentence timing fields I need to find out which sentences do it. The fix comes after the data does.
What I would take from this
Your crash reporter probably cannot see your worst crash. Anything that dies below your language runtime writes nothing. I burned several round trips asking a tester for a log that could not exist.
A build that probes the build machine is not reproducible. It is a coin flip you run once per release.
Overlapping work is only a win if the two halves are not queued behind the same resource. One GPU, two processes, no shared scheduler.
Instrument before you optimize. I split turn timing into generation, join and synthesis before shipping the fix, so the next report is a number instead of a story.
Local Waifu is the app, if anyone wants to compare notes on running an LLM and a TTS engine on one consumer card.
What is the worst failure you have shipped that your logs were structurally incapable of recording?
