Building a "Highlight-to-Speak" Engine: Running Local Whisper in Rust to Score English Fluency
Part 2 of our journey building Listen & Learn's dual-engine Offline AI. How we turned any EPUB - PDF or text into an interactive speaking test using Whisper.cpp and Longest Common Subsequence
In [Part 1 of this series], I shared how we built a lightning-fast offline pronunciation scorer using a Quantized Acoustic Model (Charsiu) and DTW alignment to catch micro-mistakes at the phoneme level (like confusing /p/ and /b/).
But phonemes are only half the battle. Real language isn't just isolated sounds; it's about Flow, Rhythm, and Sentences.
If you read a 15-word sentence, hesitate, stutter, or skip a word entirely, a strict phoneme-level checker will break down. The alignment fails. We needed a "Brain" that understands natural speech fluency, not just a "Microscope" that looks at syllables.
So, for sentence-level scoring in Listen & Learn by WynAI, we built a completely independent, secondary AI pipeline.
Here is how we implemented a 100% local, zero-latency sentence scoring system using Whisper.cpp, Rust, and a "humane" mathematical formula to power our killer UX feature: The SelectionSpeakPopup.
The UX Challenge: Reading meets Speaking
Most apps separate reading and speaking. You read in one tab, you take a speaking test in another.
We wanted a seamless workflow for serious learners (engineers, professionals) reading long-form content like The Great Gatsby (EPUB) or BBC News.
The goal: Highlight any text -> Click "Try Speak" -> Read it out loud -> Get instant red/green feedback on exactly which words you missed or mispronounced.
To do this without cloud latency, we brought OpenAI's Whisper model locally to the user's desktop.
Step 1: Local Whisper Inference (No Cloud Needed)
When the user finishes speaking, our Rust backend (Tauri) grabs the 16kHz float32 audio array and feeds it directly into a locally loaded ggml-base.en.bin (148MB) model using whisper-rs.
Why not Quantize Whisper to INT8 like we did in Part 1?
Because we need the exact, uncompressed probability distribution (Confidence scores) of the AI. At 148MB, the base model runs in under 100ms on a modern CPU. It's incredibly fast and gives us the exact transcript and the logprob of every single token.
Step 2: The Alignment Magic (LCS Algorithm)
Now we have two strings:
Expected: The text the user highlighted in the book.
Transcript: What Whisper actually heard.
How do we match them if the user skipped a word or mumbled?
Instead of basic Sequence Matchers, we implemented the LCS (Longest Common Subsequence) algorithm using Rust's similar crate.
LCS is brilliant here. If a user tries to say "conversations" but completely fails, Whisper won't transcribe it. LCS doesn't panic; it just marks that specific word as missing/wrong and correctly aligns the rest of the sentence without shifting all the subsequent words into the red zone.
Step 3: The "Humane" Scoring Formula
We don't want a robotic, punishing grading system. If a sentence has 24 words and you miss 6, pure math says you get 75%. That's discouraging.
Instead, we designed a weighted formula:
Overall Score = (0.6 Accuracy) + (0.4 Token_Score)
Accuracy (60%): Did you say the right words in the right order? (Calculated via LCS).
Token_Score (40%): How confident was the AI in what it heard? We dive into Whisper's token-level data, extract the probability (exp(logprob)), and average it.
If you stumble on one word but pronounce the other 18 words with loud, clear, native-like confidence, the Token_Score pulls your overall grade up. It behaves exactly like a human teacher: "You stumbled a bit at the end, but your accent and flow were great. 76%!"
The Result: An OS for Language Immersion
By combining the Phoneme-level engine (Part 1) for strict pronunciation, and the Whisper-level engine (Part 2) for sentence fluency, we’ve created an offline AI architecture that completely outclasses mobile apps.
You aren't just doing flashcards anymore. You are reading classic literature, highlighting complex sentences, and having a local AI instantly grade your fluency. No loading spinners. No API costs.
If you want to see what a desktop-first, AI-native English learning environment looks like, check out Listen & Learn by WynAI or download it directly from the Microsoft Store.
I’d love to hear how other devs are handling audio diffing and scoring. Are you using Whisper locally for anything similar?
