We tried building real-time AI “video therapists”… and hit a wall

I wanted to share something candid from the trenches.

We’re building therappai — an interactive AI video experience where users talk to a “real” human-like avatar (not text, not voice… actual face-to-face conversation).

The vision is simple:
A user opens the app → taps a therapist → and has a natural, flowing, human conversation in real time.

The reality (right now) is very different.

The app is live: https://apps.apple.com/nz/app/therappai/id6753192961

I’m also vibe coding some variations here too: https://calm-ai-care.base44.app/

The core problem

We’re trying to combine 4 extremely heavy systems at once:

1. Speech recognition (listening in real time)

2. LLM reasoning (thinking + generating response)

3. Text-to-speech (natural voice output)

4. Real-time avatar rendering (lip sync + facial movement)

Each of these works individually.

Together? It breaks.

Where it falls apart

• Latency kills the experience
  Even a 1–2 second delay makes it feel unnatural. 3–5 seconds is unusable.

• Avatars aren’t truly “live”
  They feel reactive, not present. You can tell it’s stitched together.

• Compute requirements are insane
  To do this properly at scale, the cost per session is not viable yet.

• Conversation flow is fragile
  Interruptions, overlaps, and timing issues break immersion instantly.

• Long sessions degrade quality
  The longer the interaction, the more obvious the cracks become.

The uncomfortable truth

We’re asking for something that is just barely possible today.

Not impossible.
But not product-ready at a level users will pay for.

What we’re testing now

• Hybrid models (video + voice fallback)

• Shorter, structured sessions instead of open-ended conversations

• Pre-buffering / prediction tricks to reduce perceived latency

• Web-based versions to experiment faster outside app store constraints

The big question

Is this:

1. A “wait 12–24 months” problem as models + compute improve?
   or

2. A product design problem where we need to rethink the experience entirely?

Would love to hear from anyone working on:

• Real-time avatars

• Voice AI

• Low-latency AI systems

• Or anyone who has actually cracked this

Because right now, it feels like we’re building on the edge of what’s technically possible — and falling just short.