4 local AI engines, 0 cloud API calls
My desktop AI companion app runs four separate local engines, and it never touches a cloud API unless the user turns that on themselves.
That was not the plan on day one. I started building assuming I would just call OpenAI or Anthropic for everything and ship a thin wrapper. Then I actually priced out what a chat app with daily active users looks like on pay-per-token billing, and the math didn't work for a one-time-purchase product. So the app runs entirely on the user's own machine instead.
What "local" actually means here
Four engines, each doing one job:
Ollama for the chat model itself. Bundled, not a separate install the user has to figure out.
sd.cpp for image generation (selfies, in my case). CPU, CUDA, and ROCm all work.
Chatterbox and Supertonic for text-to-speech. Two engines, not one, because they cover different hardware.
whisper.cpp for speech-to-text.
None of these are exotic choices. The hard part was never picking them. The hard part was making all four behave the same way across wildly different machines.
The real problem: GPU vendors don't agree on anything
A user with an NVIDIA card gets CUDA. AMD gets ROCm, except Chatterbox has no ROCm support at all, so AMD users fall back to CPU for voice specifically. Windows users without a discrete GPU get DirectML through Supertonic. Everyone else gets a CPU path that has to actually work, not just exist as a theoretical fallback.
I ended up writing a resilience pass whose only job was verifying that every one of those paths degrades safely instead of crashing. Five separate model download URLs for sd.cpp alone, checked against the vendor combinations they need to run on.
The part that surprised me: hardware detection is the real product
The engines were the easy 20%. The 80% was figuring out, automatically, which model a given machine should even run. I ended up with tiered defaults based on available RAM: a small model under roughly 35 GB, a mid-size one up to about 63 GB, and the full lineup above that. The boundary numbers themselves came from actually measuring where models started swapping to disk, not from a spec sheet.
The model catalog has 11 Ollama models to choose from, ranging from a 9.6 GB vision-capable model for lightweight machines up to a mixture-of-experts model with 17B active parameters out of 109B total for the high end. Picking wrong in either direction means either garbage output or a frozen app, and the user has no idea which one just happened.
What I'd tell someone starting this today
Don't design for "the GPU." Design for four separate fallback ladders (LLM, image, voice, transcription) that all have to fail down to CPU independently, because a single machine can be strong on one and weak on another. My biggest early bug was assuming GPU availability was one flag instead of four.
Also: benchmark the RAM boundaries yourself. Every spec sheet estimate I found for model memory footprint was wrong once I measured it on real hardware.
Where this landed
The app is called Local Waifu, an AI companion that runs on the user's own machine. Cloud is there as an optional bring-your-own-key fallback, not the default path. If the local-first approach is useful as a reference, the site is localwaifu.com.
Anyone else built a GPU-vendor-agnostic fallback chain for local inference? What broke first for you, CUDA, ROCm, or DirectML?
