How to implement realtime translation for WebRTC Video meeting?

Hi there!

Usually it goes this way:

For the local audio listener
Browser =(webrtc)=> Mediaserver =(webrtc)=> Browser

The streamer transmits the stream to a media server, it distributes the stream to the listeners

For the listener of the translated audio
Browser =(webrtc)=> Mediaserver
=(rtp)=> ffmpeg =(flac)=> google speech to text => google translate => google text to speech => ffmpeg =(rtp)=> Mediaserver =(webrtc)=> Browser

The streamer transmits the stream to a media server

From the media server we convert the stream to a RTP-stream for FFmpeg and then to FLAC for Google Speech to Text to support
Once we get a response, we convert the text into audio
Convert the audio into FFmpeg
FFmpeg delivers the stream to the media server
The users get the translated stream from the media server via HTTP

In the algorithm you’ve suggested it’s not quite clear why use MediaRecorder and transmit the data via Data Channel since you can enable MediaRecorder for the recipient as well. Plus, if several people listen to the translated audio, each listener will translate the audio separately instead of just receiving the audio translated once for every listener.

And you also haven’t mentioned whether you use a media server. If you plan on 3-10 people participating in the stream, we recommend you use one :)