How are online video editing tools like Kapwing/Spark made?
Is the video manipulation happening on the frontend using Canvas API that renders the final video on client-side(browser) or specs are passed back to the server and FFMPEG takes in all overlay/transform specs and renders the final video? I would much appreciate if anyone here can guide me. Thanks in advance.
Hey, a great question. I am the founder and engineer of https://storycreatorapp.com/.
I am excited you asked this question. It's two things, building a UI that can generate a communication protocol that can be interpreted and parsed into a command for FFMPEG. Then the actual parsing the command nicely to get the overlays and transitions to work.
Let me know if you want me to go deeper. First checkout the free product, let me know what you think and if you see room for improvement. If you have specific questions. I would love to address them and maybe form new ideas.
Cheers
Thanks, Michael for sharing this.
I checked StoryCreator, amazing product! Very impressive!
I am curious to know which frontend framework you used to build the editor? Looks slick.
Also, is the editor just a view that tries to mimic the end result(video) and you pass all the required arguments(x,y,text/transform ID, etc) back to the server which then runs relevant FFMPEG commands with arguments?
Sorry for bugging with more questions but I couldn't find a lot of resources on how browser-based video editors work. Thanks again!
Great questions. That's exactly how it works! The UI acts as a layer that receives information from the user. The point of a UI is to make it easier for the user to communicate low-level details.
So the idea is if you can build a beautiful and intuitive interface you can make it 10x easier to achieve a video result. Technically you can get really hardcore and live in your CLI and get a lot of really complex things done but just imagine writing everything out by hand lol.
This is pretty much a universal principle though, nothing new. I haven't actually looked at the implementation details of a program like After Effects but it's a similar concept except they have the rendering engine locally on the disk that handles the information set within the UI.
The beauty of it being online though is you can shut off your computer and still have it render. Then walk to your car, and get a notification on your phone that it is done. A feature I plan to add soon. If you shut off your computer with AE the render process will die.
More specifically the artboard does collect a map of all objects on the board and their respective x,y coords. That information is sent to the server and decoded into a command.
I could talk about this for hours :) It's fun and something I plan on working on for a few years. I will eventually have to delegate as I grow though.
This is a great post. I tried out StoryCreator, a great app. Especially I liked the waveforms. Do you mind sharing what library is behind waveform?
Thanks a lot, Micheal for such a detailed response. Appreciate it a ton!
I personally believe it's the latter. In Kapwing's case they have what looks to be cmd output on screen when rendering out the video. Also, I don't believe the canvas API works with sound well. If you find out otherwise in the second part please let me know.
Thanks for your response. I agree it must be the latter. But I did see them hiring WebAssembly folks so I assumed they are using bleeding-edge tech to make it all happen on the frontend. :)