A friend of mine makes narration-first videos, voiceover carries the whole piece, the visuals are just there to keep eyes on the screen. He was burning hours every video doing the same manual job: transcribe the audio in his head, decide what should be on screen at every moment, then dig through a folder of stock clips and stills trying to match each beat. The narration was the actual content. The editing was just plumbing.
I'd been wanting an excuse to actually build something in Rust instead of reading about it, so I picked up the plumbing problem. The result is ai-vedit, you give it an MP3 and a folder of categorized assets, and it hands back a rendered video.
What it actually does
Two subcommands, plan and render, deliberately split so the expensive, non-deterministic part (calling out to AI) is separate from the deterministic part (ffmpeg does what the plan says, every time, no API key required).
plan takes the audio and:
- Transcribes it with Whisper.
- Sends the transcript to an LLM and asks it to cut the narration into timed "beats," each with a description and an asset category (city-broll, charts, general, whatever categories exist in your asset folder).
- Writes the result to a plan JSON file.
render takes that plan plus your asset folder and:
- Matches each beat to an asset from the right category folder.
- Applies a Ken Burns zoom to stills, loops/trims clips to fit the beat duration.
- Concatenates everything and muxes the narration back in with ffmpeg.
No manifest file for categories, folder names are the categories. Drop assets into city-broll/, charts/, general/, and the tool discovers them at render time.
That's the pitch. The interesting part is everything that had to go wrong first to get there.
Keeping the video locked to the audio
The LLM's beat timestamps don't line up with the real audio duration, they drift. My first fix was naive: clamp every beat to a --min-beat-duration floor. That made it worse, because clamping first meant the drift compounded on top of an already-wrong timeline.
The fix was ordering: pull the true duration out of the Whisper response, rescale every beat's timing against that first, then apply the minimum-duration floor and merge any beat that's still too short into its neighbor. Rescale, then clamp, never the other way around. A one-line reordering, but it's the kind of bug that only shows up once you're staring at a rendered video where the last three beats are all crammed into two seconds.
The asset-fitting saga
This one took four iterations, and each iteration failed in a way that told me what the real constraint was.
- v1 stretch to frame. Works, looks bad. Distorted aspect ratios everywhere.
- v2 scale-based cover-crop. Scaled everything to cover the frame, then cropped overflow. Fixed the distortion, but softened assets that were already pixel-perfect for the frame, and over-cropped anything with a mismatched aspect ratio.
- v3 native-resolution crop/pad. Stopped rescaling entirely. Crop overflow from the center, pad shortfall with black bars. Sharp again but now a 4K asset gets center-cropped down and wastes all that detail, and a slightly-off aspect ratio gets ugly black bars instead of a clean fit.
- v4 conditional downscale. ffprobe reads the asset's real dimensions at render time. If it exceeds the frame in both dimensions and its aspect ratio is within 1% of the target, scale it down to fit. If the aspect is a near-miss, center-crop just the small remainder instead of adding bars. Everything else falls back to v3's crop/pad. If ffprobe itself is missing or gives back something unparseable, the tool degrades to v3 rather than crashing.
Four rewrites to converge on "look at the actual asset, don't assume."
The zoom that flickered
Ken Burns effect on stills uses ffmpeg's zoompan. On a slow zoom, it visibly shimmers, a 1px snap every so often. Turns out zoompan quantizes its crop window to whole source pixels on every frame, so a slow, sub-pixel zoom gets rounded and the rounding error is what you see as flicker.
Fix: upscale the source 4x with before zoompan, so the quantization happens on a much finer grid, then let zoompan's own output scale bring it back down to the target resolution. The rounding error shrinks by roughly the same factor as the upscale. Cheap trick, not obvious until you know zoompan's crop math is pixel-quantized in the first place.
Shipping a static binary
There's a statically linked Linux binary built off every tagged release, no need to have Rust installed to try it, though you'll still need ffmpeg on your system (see the install section in the repo for that). Grab the binary from the GitHub releases page, verify it against the included .sha256, and run it.
Where it stands
~3,400 lines, 93 tests, built over about 10 days. It's a CLI, it's rough around a few edges, and it only does one narrow thing, but it does the one thing my friend was bored of doing by hand.
Next up: burned-in subtitles (the transcript's already sitting right there, unused for that), and support for AI providers beyond OpenAI so the pipeline isn't locked to one API key. If you're narrating over stock footage and dreading the matching-assets-to-beats part, it's on GitHub: github.com/Jancera/ai-vedit.