FlowVid: Taming Imperfect Optical Flows: Training With Joint Spatial-temporal Conditions

Table of Links

• Abstract and Introduction
• 2. Related Work
• 3. Preliminary
• 4. FlowVid
• 4.1. Inflating image U-Net to accommodate video
• 4.2. Training with joint spatial-temporal conditions
• 4.3. Generation: edit the first frame then propagate
• 5. Experiments
• 5.1. Settings
• 5.2. Qualitative results
• 5.3. Quantitative results
• 5.4. Ablation study and 5.5. Limitations
• Conclusion, Acknowledgments and References
• A. Webpage Demo and B. Quantitative comparisons

4.2. Training with joint spatial-temporal conditions

Upon expanding the image model, a straightforward method might be to train the video model using paired depth-video data. Yet, our empirical analysis indicates that this leads to sub-optimal results, as detailed in the ablation study in Section 5.4. We hypothesize that this method neglects the temporal clue within the video, making the frame consistency hard to maintain. While some studies, such as Rerender [49] and CoDeF [32], incorporate optical flow in video synthesis, they typically apply it as a rigid constraint. In contrast, our approach uses flow as a soft condition, allowing us to manage the imperfections commonly found in flow estimation.