The problem with video generation today
For years, video generation and audio generation have been strangers in separate labs. Current video models have become genuinely impressive, capable of synthesizing photorealistic scenes with complex motion and rich detail. Yet they operate in a vacuum, treating audio as optional decoration or ignoring it entirely.
This creates a concrete problem: temporal misalignment. When you generate a video of rain hitting a metal roof, the audio (if present at all) was created independently. A door slam in the video doesn't sync with a door slam in the audio. A character's dialogue doesn't match their lip movements. The result feels uncanny, like a dubbed film where something is always slightly off.
The deeper issue is architectural. Most multimodal models treat text as the sole conductor, with everything else serving it. But in real film production, video and audio inform each other constantly. A tight shot of rain isn't just about pixels, it's about acoustics. A crowded market scene needs audio that tells you which conversations matter. The cinematographer and sound engineer need to collaborate, not work sequentially.
Why sound needs to be born with vision, not added later
Imagine two musicians in a darkened room, unable to see each other but listening intently. One plays strings, one plays percussion. They share a conductor (the text prompt) and reference recording (the scene description). They can't see each other, but they hear themselves making music and they stay in time. That's the architectural insight of SkyReels-V4.
Audio doesn't get generated after video here. Instead, both branches generate in parallel, conditioning each other. The video branch learns that an audio reference contains a dog barking, so it synthesizes motion matching that bark's timing and energy. The audio branch hears that the video contains a dog, so it generates sounds consistent with that animal's presence. This is fundamentally different from other approaches that bolt audio onto video as an afterthought.
When two generative processes share the same input understanding, they can be orchestrated. They're not independent models handed off sequentially, they're two parts of one unified thought.
Architecture: dual streams with a shared mind
SkyReels-V4 uses a Dual-stream Multimodal Diffusion Transformer (MMDiT) where one branch synthesizes video and another generates audio, while both draw from a shared conceptual foundation. Here's how the pieces fit together.
The video branch synthesizes frames in a learned latent space using diffusion, accepting rich visual conditioning: text descriptions, reference images, masks for inpainting, even full video clips. The audio branch generates sound spectrograms via the same diffusion process, conditioned on text and audio references. Both branches are grounded in a Multimodal Large Language Model (MMLM) based text encoder that understands visual concepts as well as language. When you describe a "thunderstorm over a wheat field," this encoder captures both the visual richness and the sonic expectations embedded in that description.
Overview of the SkyReels-V4 architecture showing dual-stream video and audio generation branches sharing a multimodal encoder.
The dual-stream architecture with shared multimodal encoder, where video and audio branches generate simultaneously while conditioned by the same text understanding.
Information flows from the text prompt into the shared encoder, gets decomposed into understanding, and that understanding flows into both branches. They don't wait for each other, but they're orchestrated by the same conceptual input.
Diffusion models are ideal for this joint generation because both video and audio benefit from step-by-step refinement. At each diffusion step, the video branch can be gently nudged by the audio branch's current estimate, and vice versa. It's like two musicians refining their performance in real time, each listening and adjusting to the other.
One interface for generation, editing, and inpainting
Here's where architectural elegance becomes practical power. Most video models require separate code paths for "generate from scratch," "edit this video," and "extend this clip." SkyReels-V4 unifies all of these under a single mechanism using channel concatenation.
The trick is deceptively simple. Different input channels can be filled with different content, or left masked:
- Text-to-video generation: All input channels are empty (masked), so the model generates everything from scratch.
- Image-to-video: A starting image is embedded into certain channels, others remain empty, and the model generates the video that follows.
- Video extension: Existing video frames fill some channels, others are masked, and the model generates what comes next.
- Inpainting: A video with masked regions is provided, those regions' channels are empty, and the model fills the gaps coherently.
- Vision-referenced editing: Both a video to edit and a reference image showing the desired style get embedded as conditioning, and the model edits accordingly.
Traditional approaches require different models or training procedures for each task. SkyReels-V4 learns one unified diffusion process. During training, it sees random combinations of filled and empty channels and learns to inpaint intelligently. This unified treatment extends naturally to complex scenarios where multiple references guide the generation, something crucial for cinema-level production.
Making cinema resolution computationally feasible
Generating 1080p video at 32 frames per second for 15 seconds is computationally expensive. You can't simply make the diffusion process bigger and hope for feasible inference times. Instead, SkyReels-V4 uses a three-stage strategy that maintains quality where it matters most while reducing computational cost elsewhere.
The first stage generates the entire video at lower resolution using the dual-stream MMDiT. This is computationally efficient and captures full temporal coherence, overall composition, and audio-video synchronization. The model already solves the hard problem: what the scene should look like and how sound and vision should align.
The second stage identifies critical frames, points of maximum visual or audio change, key narrative moments, and regenerates only those frames at full 1080p resolution. This is where detail and fidelity matter most.
The third stage applies intelligent upscaling and interpolation. Low-resolution frames feed through a Super-Resolution model to upscale to 1080p while preserving content. Keyframes and their upscaled neighbors feed through a Frame Interpolation model to generate frames in between, maintaining smooth motion and temporal coherence.
Pipeline showing low-resolution generation followed by keyframe upscaling and frame interpolation.
The three-stage pipeline: low-resolution full sequence (F) generation, keyframe (KF) selection and upscaling, and frame interpolation to maintain smooth motion.
This approach works because the low-resolution model has already learned the hard constraints. Upscaling and interpolation are much simpler, learnable problems. You're not asking the model to invent detail from scratch, you're asking it to plausibly complete a pattern it already understands. The memory footprint and inference time shrink dramatically compared to generating everything at full resolution, making cinema-quality output feasible.
How it actually performs
Architecture matters only if it delivers results. The paper positions SkyReels-V4 within a competitive landscape rather than claiming obvious dominance, which paradoxically strengthens credibility.
Artificial Analysis Text-to-Video with Audio Arena Leaderboard showing SkyReels-V4 ranking.
SkyReels-V4 ranks third on the Artificial Analysis Text-to-Video with Audio Arena Leaderboard, competing alongside Veo 3.1, Sora-2, and other state-of-the-art models.
On the Artificial Analysis leaderboard, SkyReels-V4 ranks third overall, competing against models like Veo 3.1, Sora-2, and Wan 2.6. This positions it in a tier of genuinely competitive models rather than claiming the obvious winner.
Absolute quality metrics across multiple dimensions (visual quality, temporal coherence, audio-video sync) establish baselines for what good performance looks like:
Absolute scoring results on a 5-point Likert scale comparing SkyReels V4 against baselines.
Absolute quality metrics show SkyReels-V4 performing consistently well across dimensions including visual quality, motion coherence, and audio-video synchronization.
Head-to-head comparisons reveal nuance. Against Kling 2.6, performance is comparable with some edge cases favoring each. Against Veo 3.1, competition is close, suggesting these models operate in a different tier. Against Seedance 1.5 Pro and Wan 2.6, the quality comparison shows SkyReels-V4 consistently in the "Good" range.
Overall quality comparison showing Good, Same, and Bad ratings across all baselines.
Overall quality comparison (Good/Same/Bad ratings) shows SkyReels-V4 competitive across the field.
SkyReels V4 vs. Kling 2.6 comparison.
Comparison with Kling 2.6 shows competitive performance with complementary strengths.
SkyReels V4 vs. Seedance 1.5 Pro comparison.
Comparison with Seedance 1.5 Pro demonstrates consistent quality across evaluation criteria.
SkyReels V4 vs. Veo 3.1 comparison.
Comparison with Veo 3.1 shows close competition between two models operating at the frontier.
SkyReels V4 vs. Wan 2.6 comparison.
Comparison with Wan 2.6 indicates SkyReels-V4's consistent performance advantage.
The architecture truly shines where it was designed to: scenarios with rich conditioning. Cases with multiple image references, audio references, and complex masks benefit hugely from the unified multimodal interface. Simpler prompts might not show as much advantage, but cinema-level production, which by nature involves rich references and complex guidance, is where this model excels.
Example demonstrating multiple images and audio references guiding generation.
Complex conditioning with multiple image and audio references demonstrates the model's flexibility in handling rich multimodal input.
This competitive positioning is honest. SkyReels-V4 isn't presented as the obvious winner, which makes the actual strengths more credible. It excels in specific niches (multimodal conditioning, audio-video synchronization, unified editing) while being part of a landscape where multiple models push the frontier forward.
What makes this approach matter
The true innovation in SkyReels-V4 isn't any single component. It's the vision that video and audio generation should be unified, that rich multimodal conditioning should feel natural, and that efficiency shouldn't require sacrificing cinema-level quality.
By building a dual-stream architecture with a shared conceptual foundation, by unifying generation and editing under one interface, and by strategically applying superresolution and interpolation, the model demonstrates something that feels inevitable in hindsight but required genuine insight to execute.
Previous work like SkyReels-V3 established the video generation foundation, while SkyReels Audio showed the promise of audio-conditioned visual synthesis. SkyReels-V2 tackled long-duration generation. V4 brings these threads together into a unified system that thinks in video and sound simultaneously, a rare architectural choice in a field where specialization usually wins.
The result is a foundation model that generates cinema-level video with perfectly synchronized audio while maintaining the flexibility to edit, extend, or fill in gaps. For production, that changes what becomes possible.
This is a Plain English Papers summary of a research paper called SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model. If you like these kinds of analysis, join AIModels.fyi or follow us on Twitter.