As many of you know, I'm always trying to share knowledge — whether through videos, talks, or interviews. And from every session, I like to extract the most powerful moments and turn them into Shorts. But there’s a problem. But there’s a problem. Most tools that can help me do this easily are paid. And they don't cost just a few bucks; they're quite expensive. Now, I'm not being cheap. But I thought: An engineer built this tool. I'm an engineer too. So I can build my own." An engineer built this tool. I'm an engineer too. So I can build my own." And then I reflected: Well... not just me. **Me + Claude.
\ And that's how Video Wizard was born — an open-source, AI-powered tool for turning long-form video into short-form content with subtitles, smart cropping, and professional templates. Well... not just me. **Me + Claude.
\ And that's how Video Wizard was born — an open-source, AI-powered tool for turning long-form video into short-form content with subtitles, smart cropping, and professional templates. Video Wizard In this post, I'll walk you through the architecture, the tech decisions, and what I learned building a production-grade video processing pipeline as a side project. What Does Video Wizard Do? Here's the full user flow: Here's the full user flow: Upload a video (or paste a YouTube URL)
AI transcribes the audio using OpenAI Whisper
GPT analyzes the transcript and detects the most viral moments (scored 0-100)
Smart cropping with face detection (MediaPipe + OpenCV) reframes for vertical
Edit subtitles in a visual editor — merge, split, clean filler words
Pick a template — 9 professional caption styles built as React components
Render the final video with Remotion
Download— ready for TikTok, Reels, or YouTube Shorts Upload a video (or paste a YouTube URL) Upload a video AI transcribes the audio using OpenAI Whisper AI transcribes GPT analyzes the transcript and detects the most viral moments (scored 0-100) GPT analyzes Smart cropping with face detection (MediaPipe + OpenCV) reframes for vertical Smart cropping Edit subtitles in a visual editor — merge, split, clean filler words Edit subtitles Pick a template — 9 professional caption styles built as React components Pick a template Render the final video with Remotion Render the final Download— ready for TikTok, Reels, or YouTube Shorts Download But it's not just a clip extractor. There's also a standalone Subtitle Generator that skips the AI analysis and goes straight from upload to rendered subtitles, as well as a Content Intelligencetool for analyzing transcripts without video. Subtitle Generator Content Intelligence The Tool’s Architecture: Three Services, One Goal I chose a microservices monorepo approach with Turborepo. Three independent services, each doing what it does best: Why three services? Why three services? Python has the best video/ML libraries (FFmpeg, MediaPipe, Whisper). There's no good alternative in the JS ecosystem for face detection + smart cropping.
Remotion needs its own server because it bundles React components into video frames — it's resource-intensive and benefits from isolated rendering.
Next.js handles the UI, API routing, and orchestration. It's the glue. Python has the best video/ML libraries (FFmpeg, MediaPipe, Whisper). There's no good alternative in the JS ecosystem for face detection + smart cropping. Python Remotion needs its own server because it bundles React components into video frames — it's resource-intensive and benefits from isolated rendering. Remotion Next.js handles the UI, API routing, and orchestration. It's the glue. Next.js Key Insight: Video as React Components
This was the "aha" moment for me.I choseRemotion because it lets you treat video like a UI. Components, props, composition — but applied to audiovisual content. Key Insight: Video as React Components Key Insight: Video as React Components This was the "aha" moment for me.I choseRemotion because it lets you treat video like a UI. Components, props, composition — but applied to audiovisual content. Remotion Smart Face Tracking: The Python Side The most interesting algorithmic challenge was the smart cropping. When you convert a 16:9 video to 9:16, you need to decide  where to crop — and ideally, you follow the speaker's face. The processing engine uses MediaPipe's BlazeFace model for detection, then applies aweighted scoring algorithmwhen multiple faces appear weighted scoring algorithm The result? Cinematic-quality camera movement that tracks the speaker without the "security camera" feel. And it's smart enough to skip face detection entirely when the source video already matches the target aspect ratio. Subtitle Synchronization: The Hard Part Nobody Talks About Getting subtitles to sync properly across three services with different time formats was the trickiest part. Stage: Whisper outputFormat: SecondsExample: { start: 0.5, end: 2.3 } Stage: Frontend editorFormat: MillisecondsExample: { start: 500, end: 2300 } Stage: Remotion rendererFormat: SecondsExample: { start: 0.5, end: 2.3 } One early bug had 60-second videos rendering as 0.06 seconds because the times were getting divided by 1000 twice. Fun times. I also added a configurable 200ms subtitle offset to account for the perceptual delay between hearing a word and reading it: const SUBTITLE_OFFSET = 0.2; // secondsconst adjustedTime = currentTime - SUBTITLE_OFFSET; It’s a small detail, but it makes the subtitles feel perfectly synced. The Subtitle Cleanup Toolkit Beyond basic editing, I built automated detection for common subtitle issues: Silence detection — gaps greater than 1 second between segments
Filler word detection — “um”, “uh”, “like”, “you know” (13 defaults)
Short segment detection — segments under 300ms (usually noise) Silence detection — gaps greater than 1 second between segments Filler word detection — “um”, “uh”, “like”, “you know” (13 defaults) Short segment detection — segments under 300ms (usually noise) These are pure functions — no side effects, easily testable: const result = detectIssues(subtitles, config);const cleaned = removeDetectedIssues(subtitles, result.issues); With one click, subtitles go from raw Whisper output to clean, professional captions. Architecture Decisions I’m Proud Of Screaming Architecture Screaming Architecture The folder structure tells you what the app does before you read a single line of code: features/video/components/   // Renders video UI
hooks/                       // Manages video state
containers/                  // Orchestrates video workflows
types/                       // Defines video data shapes
lib/                         // Provides video utilities features/video/components/   // Renders video UI
hooks/                       // Manages video state
containers/                  // Orchestrates video workflows
types/                       // Defines video data shapes
lib/                         // Provides video utilities Strict Separation of Concerns Strict Separation of Concerns API routes are thin. They only handle HTTP and delegate to services: export async function POST(request: NextRequest) { 
  const body = await request.json(); 
  const data = await subtitleGenerationService.generateSubtitles(body); 
  return NextResponse.json({ success: true, data }); 
} export async function POST(request: NextRequest) { 
  const body = await request.json(); 
  const data = await subtitleGenerationService.generateSubtitles(body); 
  return NextResponse.json({ success: true, data }); 
} All business logic lives in service classes — reusable, testable, and independent of HTTP: export class SubtitleGenerationService { 
  async generateSubtitles(input) { // 1. Call Python transcription // 2. Convert time formats // 3. Structure response }
  async renderWithSubtitles(input) { // 1. Send job to Remotion // 2. Poll until complete // 3. Return video URL } 
} export class SubtitleGenerationService { 
  async generateSubtitles(input) { // 1. Call Python transcription // 2. Convert time formats // 3. Structure response }
  async renderWithSubtitles(input) { // 1. Send job to Remotion // 2. Poll until complete // 3. Return video URL } 
} Zod Everywhere Every external boundary is validated with Zod. Types are inferred, not duplicated: const BrandKitSchema = z.object({ 
  logoUrl: z.string().optional(), 
  logoPosition: z.enum(['top-left', 'top-right', 'bottom-left', 'bottom-right']), 
  logoScale: z.number().min(0.1).max(2), 
  primaryColor: z.string().regex(/^#[0-9A-Fa-f]{6}$/).optional(), 
});

type BrandKit = z.infer<typeof BrandKitSchema>; const BrandKitSchema = z.object({ 
  logoUrl: z.string().optional(), 
  logoPosition: z.enum(['top-left', 'top-right', 'bottom-left', 'bottom-right']), 
  logoScale: z.number().min(0.1).max(2), 
  primaryColor: z.string().regex(/^#[0-9A-Fa-f]{6}$/).optional(), 
});

type BrandKit = z.infer<typeof BrandKitSchema>; Types come from schemas — not the other way around. The Tech Stack Frontend: Next.js 16 + React 19 \  App Router, server components, API routes
Styling: Tailwind + shadcn/ui \  Fast, accessible, consistent
AI Analysis: Vercel AI SDK + GPT-4o \  Structured output for viral clip detection
Transcription: OpenAI Whisper \  Best multilingual accuracy
Face Detection: MediaPipe (BlazeFace) \  Lightweight, real-time, no GPU required
Video Processing: FFmpeg + OpenCV \  Industry standard, battle-tested
Video Rendering: Remotion \  React-based, programmatic, template-friendly
Validation: Zod \  Runtime + TypeScript safety
Monorepo: Turborepo + pnpm \  Fast builds, shared packages Frontend: Next.js 16 + React 19 \  App Router, server components, API routes Styling: Tailwind + shadcn/ui \  Fast, accessible, consistent AI Analysis: Vercel AI SDK + GPT-4o \  Structured output for viral clip detection Transcription: OpenAI Whisper \  Best multilingual accuracy Face Detection: MediaPipe (BlazeFace) \  Lightweight, real-time, no GPU required Video Processing: FFmpeg + OpenCV \  Industry standard, battle-tested Video Rendering: Remotion \  React-based, programmatic, template-friendly Validation: Zod \  Runtime + TypeScript safety Monorepo: Turborepo + pnpm \  Fast builds, shared packages What I Learned Engineer + AI = Multiplied Output Engineer + AI = Multiplied Output Engineer + AI = Multiplied Output This project would have taken months working solo. With Claude as a pair programmer, I went from idea to working prototype significantly faster. Not because the AI wrote everything, but because it accelerated the tedious parts — boilerplate, FFmpeg flags, Remotion configuration — so I could focus on architecture and product decisions. Build Your Own Tools Build Your Own Tools Build Your Own Tools We increasingly depend on external SaaS for everything. Sometimes the best way to learn is to build the tool yourself. You’ll understand video processing, ML pipelines, and rendering engines at a depth that no tutorial can give you. Time Formats Will Haunt You Time Formats Will Haunt You Time Formats Will Haunt You If you’re building anything with subtitles, pick one time format (seconds or milliseconds) and stick with it. Document your conversions. Test edge cases. Your future self will thank you. What’s Next? This is still a personal project, but it already includes: 9 professional caption templates
Multi-aspect ratio support (9:16, 1:1, 4:5, 16:9)
Brand kit customization (logo, colors, fonts)
Silence and filler word auto-cleanup
SRT/VTT subtitle export
YouTube URL support
Multi-language transcription 9 professional caption templates Multi-aspect ratio support (9:16, 1:1, 4:5, 16:9) Brand kit customization (logo, colors, fonts) Silence and filler word auto-cleanup SRT/VTT subtitle export YouTube URL support Multi-language transcription Roadmap ideas: Batch processing for multiple clips
Custom template builder (visual editor)
Cloud deployment with render queue scaling
Speaker diarization (who said what) Batch processing for multiple clips Custom template builder (visual editor) Cloud deployment with render queue scaling Speaker diarization (who said what) Try It Yourself The repo is open source. You can run the entire stack locally I’d love your feedback: I’d love your feedback: What feature would you want to see?
Would you use this for your own content?
What would you improve? What feature would you want to see? Would you use this for your own content? What would you improve? PRs, issues, and stars are all welcome. GitHub: https://github.com/el-frontend/video-wizard https://github.com/el-frontend/video-wizard Built with Next.js, Python, Remotion, and a lot of help from Claude.

This story contains new, firsthand information uncovered by the writer.

This story contains AI-generated text. The author has used AI either for research, to generate outlines, or write the text itself. 

Walkthroughs, tutorials, guides, and tips. This story will teach you how to do something new or how to do something better.

Render

Vercel

I Built My Own AI Video Clipping Tool Because the Alternatives Were Too Expensive

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Debugging Life From Banking To QA Leadership

How I Earned $12,000 on Crypto Exchange Affiliate: Idea for Income Without Investment

Forget CRUD: Workflow APIs Are How Modern Platforms Actually Work

Multimodal AI for High-Fidelity Video Creation and Editing

The Future of Video Editing: Integrating Generative AI for Enhanced Creativity

Debugging Life From Banking To QA Leadership

How I Earned $12,000 on Crypto Exchange Affiliate: Idea for Income Without Investment

Forget CRUD: Workflow APIs Are How Modern Platforms Actually Work

Multimodal AI for High-Fidelity Video Creation and Editing

The Future of Video Editing: Integrating Generative AI for Enhanced Creativity

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps