Mastering Motion Dynamics in Animation with Temporal Transformers
by Model TuningNovember 18th, 2024
Too Long; Didn't Read
AnimateDiff uses a temporal Transformer-based motion module to model motion dynamics in animation, reshaping 2D diffusion models for video generation while maintaining frame consistency.
Authors:
(1) Yuwei Guo, The Chinese University of Hong Kong;
(2) Ceyuan Yang, Shanghai Artificial Intelligence Laboratory with Corresponding Author;
(3) Anyi Rao, Stanford University;
(4) Zhengyang Liang, Shanghai Artificial Intelligence Laboratory;
(5) Yaohui Wang, Shanghai Artificial Intelligence Laboratory;
(6) Yu Qiao, Shanghai Artificial Intelligence Laboratory;
(7) Maneesh Agrawala, Stanford University;
(8) Dahua Lin, Shanghai Artificial Intelligence Laboratory;
(9) Bo Dai, The Chinese University of Hong Kong and The Chinese University of Hong Kong.
Table of Links
- AnimateDiff
4.1 Alleviate Negative Effects from Training Data with Domain Adapter
4.2 Learn Motion Priors with Motion Module
4.3 Adapt to New Motion Patterns with MotionLora
5 Experiments and 5.1 Qualitative Results
8 Reproducibility Statement, Acknowledgement and References
4.2 LEARN MOTION PRIORS WITH MOTION MODULE
To model motion dynamics along the temporal dimension on top of a pre-trained T2I, we must 1) inflate the 2-dimensional diffusion model to deal with 3-dimensional video data and 2) design a sub-module to enable efficient information exchange along the temporal axis.
This paper is available on arxiv under CC BY 4.0 DEED license.
L O A D I N G
. . . comments & more!
. . . comments & more!
