We stopped asking AI to render video and made it direct the edit instead
We have been building WM AI Editor, a desktop video editor that turns raw clips and a plain-language request into an editable first cut.
The original idea sounded almost too simple:
Add the footage, explain what the finished video should be, and let AI assemble it.
The difficult part was deciding what “let AI edit” should actually mean.
Uploading every source video to a cloud model would make the workflow slower, more expensive and less private. Asking a model to generate a final flattened video would also remove the most important feature of an editor: the ability to correct individual decisions.
So we split the job in two.
The AI acts as a director. The computer acts as the editing room.
What the AI sees
The application analyzes the footage locally and prepares a compact package containing:
- the user’s brief;
- a local transcript and word-level timecodes;
- detected scenes and source boundaries;
- technical media information;
- the selected format, duration and editing template.
The AI returns a structured first-cut plan. It does not receive the full original videos and does not render the output. A limited set of reduced key frames can be included only when the user explicitly allows visual analysis.
The local application validates the plan and converts it into regular timeline operations.
This keeps API usage relatively small: the model reasons about the structure once instead of processing every video frame or handling the final render.
What stays local
The heavy work happens on the user’s Windows PC or Mac:
- speech recognition;
- media inspection and normalization;
- face detection and tracking;
- timeline assembly;
- reframing for 9:16 or 16:9;
- subtitles, transitions, music and sound effects;
- H.264/AAC MP4 export.
The user’s hardware provides the rendering power. Compatible GPU encoding is used when available, with a CPU fallback.
The first version taught us an important lesson
We initially focused on getting the AI to produce a valid sequence. That was only half the product.
A generated cut becomes frustrating as soon as the model chooses the wrong take, removes a useful pause or frames the wrong part of the shot. If the output is flattened, the user has to regenerate everything.
We therefore built the automatic result on the same data model as manual editing. Every AI decision becomes a normal clip that can be split, trimmed, moved, deleted or reframed. Subtitles can be corrected. B-roll and music can be replaced. The user can switch between the original and AI versions and export again without paying for another AI request.
That changed the product from a generation demo into an editor.
What is in the current release
WM AI Editor now supports:
- Windows and macOS;
- vertical 9:16 and horizontal 16:9 projects;
- Auto mode plus 10 editing templates;
- local transcription in English, Russian and Spanish;
- an editable timeline, transcript and subtitles;
- B-roll, music, transitions and local sound effects;
- face tracking, autocrop, colour profiles and speed control;
- local MP4 export.
We decided on a $50 perpetual licence for the purchased version instead of a subscription. Users provide their own AI access, while the expensive media work stays on their own system.
What we are still learning
The hardest part is not adding more effects. It is measuring whether an automatic first cut is actually useful.
We now score plans locally for problems such as repeated material, weak openings, missing endings, micro-cuts and poor subtitle coverage. A weak plan can receive one bounded correction attempt. We are still improving the artistic side: choosing between two technically valid scenes is much harder than validating a timecode.
Our current product principle is:
AI should prepare the first cut. The creator should keep the final decision.
For people building AI creative tools: where do you draw the boundary between model decisions and deterministic local execution?
Product case and screens: https://wemaide.com/work/ai-video-editor
Great breakdown. What feedback have you had from early users?
Makes sense. Are you planning to charge for it, or keep it free for now?
Nice work shipping it. What has been the biggest challenge since launch?
Helpful post. How did you get your first bit of traction?
Appreciate the honesty here, most people only share the wins.
Appreciate the honesty here, most people only share the wins.
Appreciate the honesty here, most people only share the wins.
Appreciate the honesty here, most people only share the wins.
Appreciate the honesty here, most people only share the wins.
Appreciate the honesty here, most people only share the wins.
Hi
Treating the first cut as a structured, validated plan seems like the key design choice here. I’d make every generated clip reference a source ID plus start/end time, then run a local pass that rejects overlaps, gaps, and out-of-range timestamps before touching the timeline. For the weak-opening score, logging which suggested changes creators keep or undo could give you a useful signal without sending footage off-device.
At this stage, templates have been implemented that allow the AI director to understand what type of content he is working on (the results of the generations are used in the future to improve quality)
That is a great (if somewhat uncomfortable) lesson that has far reaching implications. Where else might we be having AI do the wrong task?
Appreciate the honesty here, most people only share the wins.
Appreciate the honesty here, most people only share the wins.
Appreciate the honesty here, most people only share the wins.
Appreciate the honesty here, most people only share the wins.
Helpful post. How did you get your first bit of traction?
What made you pick this stack over the alternatives?
Solid lesson. Which channel has worked best for you so far?
Appreciate the honesty here, most people only share the wins.
Appreciate the honesty here, most people only share the wins.
Appreciate the honesty here, most people only share the wins.
Appreciate the honesty here, most people only share the wins.
Appreciate the honesty here, most people only share the wins.
Makes sense. Are you planning to charge for it, or keep it free for now?
Currently, there is free access for testers, while the paid version—including all future updates—is available for customers.
Interesting. How are you measuring whether it is working?
Clear and practical, thanks. Did anything surprise you along the way?
Love this framing — "AI directs, computer edits" is such a clean way to describe that split, and I think it's the right call. Flattening everything into one generated output removes the thing that makes an editor actually useful: the ability to fix one decision without redoing the whole thing.
The local-first approach is a great trust-builder too, especially for something touching people's raw footage. Really well thought through.
Thanks)
Really good writeup, thanks for sharing it. What's the next thing you're planning to try here?
We combine a full-fledged manual editor (professional-grade) and an AI mode in a single piece of software.
Same split here, I render route videos on the user's device. Though for me the
line wasn't AI vs local, it was deterministic vs probabilistic: coordinates
come from a geocoding API, never from the model. A wrong timecode is annoying,
a wrong location kills trust in the whole output.
How do you score a weak opening without a human in the loop?
Good point. Did you test that with users before committing to it?
Good point. Did you test that with users before committing to it?
Good point. Did you test that with users before committing to it?
Good point. Did you test that with users before committing to it?
Certainly! Overall, we are at the 70% mark of our goal (full AI control and obtaining a correct result in 95% of cases).
This is the right boundary for AI creative tools. Let the model decide structure from a compact representation, then keep the timeline editable so a wrong choice costs a click instead of a rerun. The real proof will be time to a satisfying first cut and time to fix it when the model gets one key moment wrong.
Interesting take. Would you still recommend this approach to someone starting today?
Curious how long it took before you saw the first real results?
It took about four weeks from the initial idea to the first test, and another six weeks or so to reach a finished solution. However, updates are ongoing.
What made you pick this stack over the alternatives?
Appreciate the honesty here, most people only share the wins.
Good point. Did you test that with users before committing to it?
Nice progress. What is the next thing you are focusing on?
Solid lesson. Which channel has worked best for you so far?
When users choose WM AI Editor, what outcome most consistently drives that decision—faster first cuts, easier correction, or avoiding repeated AI generation costs?
Work is accelerated and simplified, and AI costs are very low; a single video costs between $0.01 and $0.10, even with top-tier AI models.
How are you measuring whether the first cut genuinely saves creators time—fewer manual edits, faster completion, or users choosing it over their existing editing workflow?
It significantly speeds up the process. It automatically finds the relevant moments, adds subtitles, stitches the clips together, and so on. This makes the task much easier for the user when dealing with a large volume of source footage.
That makes sense, but “significantly speeds up” is still hard to separate from perception. Are you measuring actual time-to-finished-clip against the old workflow yet?
That’s the part I’d be interested in digging into a bit more. Could be easier to continue over email sometime, if you’re open to it.