Boston police say they received a call about an individual walking around with what looked like a rifle or long gun, but officers have yet to find someone matching that description.

East Boston schools put into safe mode after report of armed person in area

Bitterness over the contentious legal action was evident from both sides this week.

Google to purge 'private' Incognito mode user records, but will keep snooping

Google has agreed to settle a lawsuit from consumer who alleged the company tracked user data even in private browsing <strong>mode</strong>.

Google agrees to delete billions of Incognito mode data records | Mashable

MODE

mode

Understanding Measures of Centrality: A Deep Dive into Mean, Median, and Mode

Authors:
(1) Yuwei Guo, The Chinese University of Hong Kong;
(2) Ceyuan Yang, Shanghai Artificial Intelligence Laboratory with Corresponding Author;
(3) Anyi Rao, Stanford University;
(4) Zhengyang Liang, Shanghai Artificial Intelligence Laboratory;
(5) Yaohui Wang, Shanghai Artificial Intelligence Laboratory;
(6) Yu Qiao, Shanghai Artificial Intelligence Laboratory;
(7) Maneesh Agrawala, Stanford University;
(8) Dahua Lin, Shanghai Artificial Intelligence Laboratory;
(9) Bo Dai, The Chinese University of Hong Kong and The Chinese University of Hong Kong. Table of Links Abstract and 1 Introduction 2 Work Related 3 Preliminary AnimateDiff 4.1 Alleviate Negative Effects from Training Data with Domain Adapter 4.2 Learn Motion Priors with Motion Module 4.3 Adapt to New Motion Patterns with MotionLora 4.4 AnimateDiff in Practice 5 Experiments and 5.1 Qualitative Results 5.2 Qualitative Comparison 5.3 Ablative Study 5.4 Controllable Generation 6 Conclusion 7 Ethics Statement 8 Reproducibility Statement, Acknowledgement and References 6 CONCLUSION In this paper, we present AnimateDiff, a practical pipeline directly turning personalized text-toimage (T2I) models for animation generation once and for all, without compromising quality or losing pre-learned domain knowledge. To accomplish this, we design three component modules in AnimateDiff to learn meaningful motion priors while alleviating visual quality degradation and enabling motion personalization with a lightweight fine-tuning technique named MotionLoRA. Once trained, our motion module can be integrated into other personalized T2Is to generate animated images with natural and coherent motions while remaining faithful to the personalized domain. Extensive evaluation with various personalized T2I models also validates the effectiveness and generalizability of our AnimateDiff and MotionLoRA. Furthermore, we demonstrate the compatibility of our method with existing content-controlling approaches, enabling controllable generation without incurring additional training costs. Overall, AnimateDiff provides an effective baseline for personalized animation and holds significant potential for a wide range of applications. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Yuwei Guo, The Chinese University of Hong Kong; (2) Ceyuan Yang, Shanghai Artificial Intelligence Laboratory with Corresponding Author; (3) Anyi Rao, Stanford University; (4) Zhengyang Liang, Shanghai Artificial Intelligence Laboratory; (5) Yaohui Wang, Shanghai Artificial Intelligence Laboratory; (6) Yu Qiao, Shanghai Artificial Intelligence Laboratory; (7) Maneesh Agrawala, Stanford University; (8) Dahua Lin, Shanghai Artificial Intelligence Laboratory; (9) Bo Dai, The Chinese University of Hong Kong and The Chinese University of Hong Kong. Authors: Authors: (1) Yuwei Guo, The Chinese University of Hong Kong; (2) Ceyuan Yang, Shanghai Artificial Intelligence Laboratory with Corresponding Author; (3) Anyi Rao, Stanford University; (4) Zhengyang Liang, Shanghai Artificial Intelligence Laboratory; (5) Yaohui Wang, Shanghai Artificial Intelligence Laboratory; (6) Yu Qiao, Shanghai Artificial Intelligence Laboratory; (7) Maneesh Agrawala, Stanford University; (8) Dahua Lin, Shanghai Artificial Intelligence Laboratory; (9) Bo Dai, The Chinese University of Hong Kong and The Chinese University of Hong Kong. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Work Related 2 Work Related 3 Preliminary 3 Preliminary AnimateDiff AnimateDiff 4.1 Alleviate Negative Effects from Training Data with Domain Adapter 4.1 Alleviate Negative Effects from Training Data with Domain Adapter 4.2 Learn Motion Priors with Motion Module 4.2 Learn Motion Priors with Motion Module 4.3 Adapt to New Motion Patterns with MotionLora 4.3 Adapt to New Motion Patterns with MotionLora 4.4 AnimateDiff in Practice 4.4 AnimateDiff in Practice 5 Experiments and 5.1 Qualitative Results 5 Experiments and 5.1 Qualitative Results 5.2 Qualitative Comparison 5.2 Qualitative Comparison 5.3 Ablative Study 5.3 Ablative Study 5.4 Controllable Generation 5.4 Controllable Generation 6 Conclusion 6 Conclusion 7 Ethics Statement 7 Ethics Statement 8 Reproducibility Statement, Acknowledgement and References 8 Reproducibility Statement, Acknowledgement and References 6 CONCLUSION In this paper, we present AnimateDiff, a practical pipeline directly turning personalized text-toimage (T2I) models for animation generation once and for all, without compromising quality or losing pre-learned domain knowledge. To accomplish this, we design three component modules in AnimateDiff to learn meaningful motion priors while alleviating visual quality degradation and enabling motion personalization with a lightweight fine-tuning technique named MotionLoRA. Once trained, our motion module can be integrated into other personalized T2Is to generate animated images with natural and coherent motions while remaining faithful to the personalized domain. Extensive evaluation with various personalized T2I models also validates the effectiveness and generalizability of our AnimateDiff and MotionLoRA. Furthermore, we demonstrate the compatibility of our method with existing content-controlling approaches, enabling controllable generation without incurring additional training costs. Overall, AnimateDiff provides an effective baseline for personalized animation and holds significant potential for a wide range of applications. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

How AnimateDiff Transforms T2I Models into High-Quality Animation Generators with MotionLORA

AnyModal: A Flexible Multimodal Language Model Framework for PyTorch The promise of multimodal AI is everywhere, from advanced healthcare diagnostics to creating richer, more dynamic customer experiences. But for those of us in the trenches, building multimodal systems—capable of processing text, images, audio, and beyond—often feels like an endless tangle of custom integrations, boilerplate code, and compatibility issues. This was my frustration, and it ultimately led to the creation of AnyModal. Why Multimodal AI? Let’s face it: human interactions with the world aren’t limited to one type of data. We interpret words, visuals, sounds, and physical sensations simultaneously. The concept of multimodal AI stems from this very idea. By bringing multiple types of data into the same processing pipeline, multimodal AI enables models to tackle tasks that were previously too complex for single-modality systems. Imagine healthcare applications that analyze X-rays and medical notes together, or customer service systems that consider both text and audio cues to gauge customer sentiment accurately. But here’s the challenge: while single-modality models for text (like GPT) or images (like ViT) are well-established, combining them to interact fluidly is not straightforward. The technical complexities have prevented many researchers and developers from effectively exploring multimodal AI. Enter AnyModal. The Problem with Existing Multimodal Solutions In my own work with machine learning, I noticed that while tools like GPT, ViT, and audio processors are powerful in isolation, creating multimodal systems by combining these tools often means stitching them together with clunky, project-specific code. This approach doesn’t scale. Current solutions for integrating modalities are either highly specialized, only designed for specific tasks (like image captioning or visual question answering), or they require a frustrating amount of boilerplate code just to get the data types working together. Existing frameworks focus narrowly on specific combinations of modalities, making it difficult to expand into new data types or to adapt the same setup to different tasks. This “siloed” structure of AI models meant I was constantly reinventing the wheel. That’s when I decided to build AnyModal—a flexible, modular framework that brings all types of data together without the hassle. What is AnyModal? AnyModal is a framework designed to simplify and streamline multimodal AI development. It’s built to reduce the complexity of combining diverse input types by handling the tokenization, encoding, and generation for non-text inputs, making it easier to add new data types to large language models (LLMs). The concept revolves around a modular approach to the input pipeline. With AnyModal, you can swap out feature encoders (like a Vision Transformer for images or a spectrogram processor for audio) and seamlessly connect them to an LLM. The framework abstracts much of the complexity, meaning you don’t have to spend weeks writing code to make these systems compatible with each other. The Fundamentals of AnyModal: Input Tokenization A crucial component of AnyModal is the input tokenizer, which bridges the gap between non-textual data and the LLM’s text-based input processing. Here’s how it works: Feature Encoding: For each modality (like images or audio), a specialized encoder is used to extract essential features. For example, when working with images, AnyModal can use a Vision Transformer (ViT) that processes the image and outputs a series of feature vectors. These vectors capture key aspects, such as objects, spatial relations, and textures, essential for applications like image captioning or visual question answering.
Projection Layer: After encoding, the feature vectors often don’t match the LLM’s token space. To ensure smooth integration, AnyModal uses a projection layer that transforms these vectors to align with the LLM’s input tokens. For instance, the encoded vectors from ViT are mapped into the LLM’s embedding space, allowing for a coherent flow of multimodal data within the LLM’s architecture. This dual-layer approach enables the model to treat multimodal data as a single sequence, allowing it to generate responses that account for all input types. Essentially, AnyModal transforms disparate data sources into a unified format that LLMs can understand. How It Works: An Example with Image Inputs To give you a sense of how AnyModal operates, let’s look at an example of using image data with LLMs. from transformers import ViTImageProcessor, ViTForImageClassification
from anymodal import MultiModalModel
from vision import VisionEncoder, Projector

# Step 1: Initialize Vision Components
processor = ViTImageProcessor.from_pretrained('google/vit-base-patch16-224')
vision_model = ViTForImageClassification.from_pretrained('google/vit-base-patch16-224')
vision_encoder = VisionEncoder(vision_model)

# Step 2: Define Projection Layer for Compatibility
vision_tokenizer = Projector(in_features=vision_model.config.hidden_size, out_features=768)

# Step 3: Initialize LLM and Tokenizer
from transformers import AutoTokenizer, AutoModelForCausalLM
llm_tokenizer = AutoTokenizer.from_pretrained("gpt2")
llm_model = AutoModelForCausalLM.from_pretrained("gpt2")

# Step 4: Build the AnyModal Multimodal Model
multimodal_model = MultiModalModel(
    input_processor=None,
    input_encoder=vision_encoder,
    input_tokenizer=vision_tokenizer,
    language_tokenizer=llm_tokenizer,
    language_model=llm_model,
    input_start_token='<|imstart|>',
    input_end_token='<|imend|>',
    prompt_text="Describe this image: "
) This modular setup enables developers to plug and play with different encoders and LLMs, adapting the model to various multimodal tasks, from image captioning to question answering. Current Applications of AnyModal AnyModal has already been applied to several use cases, with exciting results: LaTeX OCR: Translating complex mathematical equations into readable text.
Chest X-Ray Captioning: Generating medical descriptions for diagnostic support in healthcare.
Image Captioning: Automatically generating captions for visual content, which is helpful for accessibility and media applications. By abstracting the complexities of handling different data types, AnyModal empowers developers to quickly build prototypes or refine advanced systems without the bottlenecks that typically come with multimodal integration. Why Use AnyModal? If you’re trying to build a multimodal system, you’ve probably encountered these challenges: High complexity in aligning different data types with LLMs.
Redundant and tedious boilerplate code for each modality.
Limited scalability when adding new data types. AnyModal addresses these pain points by reducing boilerplate, offering flexible modules, and allowing quick customization. Instead of fighting with compatibility issues, developers can focus on building smart systems faster and more efficiently. What’s Next for AnyModal? The journey of AnyModal is just beginning. I’m currently working on adding support for additional modalities like audio captioning and expanding the framework to make it even more adaptable for niche use cases. Community feedback and contributions are crucial to its development—if you’re interested in multimodal AI, I’d love to hear your ideas or collaborate. Where to Find AnyModal GitHub: https://github.com/ritabratamaiti/AnyModal?embedable=true Reddit Community https://www.reddit.com/r/AnyModal/?embedable=true Hugging Face: https://huggingface.co/AnyModal?embedable=true If you’re excited about multimodal AI or looking to streamline your development process, give AnyModal a try. Let’s work together to unlock the next frontier of AI innovation. AnyModal: A Flexible Multimodal Language Model Framework for PyTorch The promise of multimodal AI is everywhere, from advanced healthcare diagnostics to creating richer, more dynamic customer experiences. But for those of us in the trenches, building multimodal systems—capable of processing text, images, audio, and beyond—often feels like an endless tangle of custom integrations, boilerplate code, and compatibility issues. This was my frustration, and it ultimately led to the creation of AnyModal . AnyModal AnyModal Why Multimodal AI? Let’s face it: human interactions with the world aren’t limited to one type of data. We interpret words, visuals, sounds, and physical sensations simultaneously. The concept of multimodal AI stems from this very idea. By bringing multiple types of data into the same processing pipeline, multimodal AI enables models to tackle tasks that were previously too complex for single-modality systems. Imagine healthcare applications that analyze X-rays and medical notes together, or customer service systems that consider both text and audio cues to gauge customer sentiment accurately. But here’s the challenge: while single-modality models for text (like GPT) or images (like ViT) are well-established, combining them to interact fluidly is not straightforward. The technical complexities have prevented many researchers and developers from effectively exploring multimodal AI. Enter AnyModal . AnyModal The Problem with Existing Multimodal Solutions In my own work with machine learning, I noticed that while tools like GPT, ViT, and audio processors are powerful in isolation, creating multimodal systems by combining these tools often means stitching them together with clunky, project-specific code. This approach doesn’t scale. Current solutions for integrating modalities are either highly specialized, only designed for specific tasks (like image captioning or visual question answering), or they require a frustrating amount of boilerplate code just to get the data types working together. Existing frameworks focus narrowly on specific combinations of modalities, making it difficult to expand into new data types or to adapt the same setup to different tasks. This “siloed” structure of AI models meant I was constantly reinventing the wheel. That’s when I decided to build AnyModal —a flexible, modular framework that brings all types of data together without the hassle. AnyModal What is AnyModal? AnyModal is a framework designed to simplify and streamline multimodal AI development. It’s built to reduce the complexity of combining diverse input types by handling the tokenization, encoding, and generation for non-text inputs, making it easier to add new data types to large language models (LLMs). AnyModal The concept revolves around a modular approach to the input pipeline. With AnyModal, you can swap out feature encoders (like a Vision Transformer for images or a spectrogram processor for audio) and seamlessly connect them to an LLM. The framework abstracts much of the complexity, meaning you don’t have to spend weeks writing code to make these systems compatible with each other. The Fundamentals of AnyModal: Input Tokenization A crucial component of AnyModal is the input tokenizer , which bridges the gap between non-textual data and the LLM’s text-based input processing. Here’s how it works: input tokenizer Feature Encoding: For each modality (like images or audio), a specialized encoder is used to extract essential features. For example, when working with images, AnyModal can use a Vision Transformer (ViT) that processes the image and outputs a series of feature vectors. These vectors capture key aspects, such as objects, spatial relations, and textures, essential for applications like image captioning or visual question answering. Projection Layer: After encoding, the feature vectors often don’t match the LLM’s token space. To ensure smooth integration, AnyModal uses a projection layer that transforms these vectors to align with the LLM’s input tokens. For instance, the encoded vectors from ViT are mapped into the LLM’s embedding space, allowing for a coherent flow of multimodal data within the LLM’s architecture. Feature Encoding : For each modality (like images or audio), a specialized encoder is used to extract essential features. For example, when working with images, AnyModal can use a Vision Transformer (ViT) that processes the image and outputs a series of feature vectors. These vectors capture key aspects, such as objects, spatial relations, and textures, essential for applications like image captioning or visual question answering. Feature Encoding Projection Layer : After encoding, the feature vectors often don’t match the LLM’s token space. To ensure smooth integration, AnyModal uses a projection layer that transforms these vectors to align with the LLM’s input tokens. For instance, the encoded vectors from ViT are mapped into the LLM’s embedding space, allowing for a coherent flow of multimodal data within the LLM’s architecture. Projection Layer This dual-layer approach enables the model to treat multimodal data as a single sequence, allowing it to generate responses that account for all input types. Essentially, AnyModal transforms disparate data sources into a unified format that LLMs can understand. How It Works: An Example with Image Inputs To give you a sense of how AnyModal operates, let’s look at an example of using image data with LLMs. from transformers import ViTImageProcessor, ViTForImageClassification
from anymodal import MultiModalModel
from vision import VisionEncoder, Projector

# Step 1: Initialize Vision Components
processor = ViTImageProcessor.from_pretrained('google/vit-base-patch16-224')
vision_model = ViTForImageClassification.from_pretrained('google/vit-base-patch16-224')
vision_encoder = VisionEncoder(vision_model)

# Step 2: Define Projection Layer for Compatibility
vision_tokenizer = Projector(in_features=vision_model.config.hidden_size, out_features=768)

# Step 3: Initialize LLM and Tokenizer
from transformers import AutoTokenizer, AutoModelForCausalLM
llm_tokenizer = AutoTokenizer.from_pretrained("gpt2")
llm_model = AutoModelForCausalLM.from_pretrained("gpt2")

# Step 4: Build the AnyModal Multimodal Model
multimodal_model = MultiModalModel(
    input_processor=None,
    input_encoder=vision_encoder,
    input_tokenizer=vision_tokenizer,
    language_tokenizer=llm_tokenizer,
    language_model=llm_model,
    input_start_token='<|imstart|>',
    input_end_token='<|imend|>',
    prompt_text="Describe this image: "
) from transformers import ViTImageProcessor, ViTForImageClassification
from anymodal import MultiModalModel
from vision import VisionEncoder, Projector

# Step 1: Initialize Vision Components
processor = ViTImageProcessor.from_pretrained('google/vit-base-patch16-224')
vision_model = ViTForImageClassification.from_pretrained('google/vit-base-patch16-224')
vision_encoder = VisionEncoder(vision_model)

# Step 2: Define Projection Layer for Compatibility
vision_tokenizer = Projector(in_features=vision_model.config.hidden_size, out_features=768)

# Step 3: Initialize LLM and Tokenizer
from transformers import AutoTokenizer, AutoModelForCausalLM
llm_tokenizer = AutoTokenizer.from_pretrained("gpt2")
llm_model = AutoModelForCausalLM.from_pretrained("gpt2")

# Step 4: Build the AnyModal Multimodal Model
multimodal_model = MultiModalModel(
    input_processor=None,
    input_encoder=vision_encoder,
    input_tokenizer=vision_tokenizer,
    language_tokenizer=llm_tokenizer,
    language_model=llm_model,
    input_start_token='<|imstart|>',
    input_end_token='<|imend|>',
    prompt_text="Describe this image: "
) This modular setup enables developers to plug and play with different encoders and LLMs, adapting the model to various multimodal tasks, from image captioning to question answering. Current Applications of AnyModal AnyModal has already been applied to several use cases, with exciting results: LaTeX OCR: Translating complex mathematical equations into readable text. Chest X-Ray Captioning: Generating medical descriptions for diagnostic support in healthcare. Image Captioning: Automatically generating captions for visual content, which is helpful for accessibility and media applications. LaTeX OCR : Translating complex mathematical equations into readable text. LaTeX OCR Chest X-Ray Captioning : Generating medical descriptions for diagnostic support in healthcare. Chest X-Ray Captioning Image Captioning : Automatically generating captions for visual content, which is helpful for accessibility and media applications. Image Captioning By abstracting the complexities of handling different data types, AnyModal empowers developers to quickly build prototypes or refine advanced systems without the bottlenecks that typically come with multimodal integration. Why Use AnyModal? If you’re trying to build a multimodal system, you’ve probably encountered these challenges: High complexity in aligning different data types with LLMs. Redundant and tedious boilerplate code for each modality. Limited scalability when adding new data types. High complexity in aligning different data types with LLMs. Redundant and tedious boilerplate code for each modality. Limited scalability when adding new data types. AnyModal addresses these pain points by reducing boilerplate, offering flexible modules, and allowing quick customization. Instead of fighting with compatibility issues, developers can focus on building smart systems faster and more efficiently. What’s Next for AnyModal? The journey of AnyModal is just beginning. I’m currently working on adding support for additional modalities like audio captioning and expanding the framework to make it even more adaptable for niche use cases. Community feedback and contributions are crucial to its development—if you’re interested in multimodal AI, I’d love to hear your ideas or collaborate. Where to Find AnyModal GitHub: GitHub : GitHub GitHub https://github.com/ritabratamaiti/AnyModal?embedable=true https://github.com/ritabratamaiti/AnyModal?embedable=true Reddit Community Reddit Community Reddit Community Reddit Community https://www.reddit.com/r/AnyModal/?embedable=true https://www.reddit.com/r/AnyModal/?embedable=true Hugging Face: Hugging Face : Hugging Face Hugging Face https://huggingface.co/AnyModal?embedable=true https://huggingface.co/AnyModal?embedable=true If you’re excited about multimodal AI or looking to streamline your development process, give AnyModal a try. Let’s work together to unlock the next frontier of AI innovation.

Building a Flexible Framework for Multimodal Data Input in Large Language Models

Construcción de un marco flexible para la entrada de datos multimodales en modelos de lenguaje de gran tamaño

Joustavan kehyksen luominen multimodaalista tiedonsyöttöä varten suuriin kielimalleihin

Création d'un cadre flexible pour la saisie de données multimodales dans de grands modèles linguistiques

בניית מסגרת גמישה לקלט נתונים רב-מודאלי במודלים של שפות גדולות

Rugalmas keretrendszer kialakítása a multimodális adatbevitelhez nagy nyelvi modellekben

大規模言語モデルにおけるマルチモーダルデータ入力のための柔軟なフレームワークの構築

대규모 언어 모델에서 다중 모달 데이터 입력을 위한 유연한 프레임워크 구축

ການສ້າງກອບແບບຍືດຫຍຸ່ນສໍາລັບການປ້ອນຂໍ້ມູນ Multimodal ໃນຮູບແບບພາສາຂະຫນາດໃຫຍ່

Membina Rangka Kerja Fleksibel untuk Input Data Multimodal dalam Model Bahasa Besar

Kuvaka Flexible Framework yeMultimodal Data Input muMakuru Mutauro Models

Сохтани чаҳорчӯбаи чандир барои воридкунии маълумотҳои мултимодалӣ дар моделҳои забонҳои калон

Pagbuo ng Flexible Framework para sa Multimodal Data Input sa Malaking Modelo ng Wika

From Centralized to Federated: Evolving Data Governance Operating Model

Authors:
(1) Yuwei Guo, The Chinese University of Hong Kong;
(2) Ceyuan Yang, Shanghai Artificial Intelligence Laboratory with Corresponding Author;
(3) Anyi Rao, Stanford University;
(4) Zhengyang Liang, Shanghai Artificial Intelligence Laboratory;
(5) Yaohui Wang, Shanghai Artificial Intelligence Laboratory;
(6) Yu Qiao, Shanghai Artificial Intelligence Laboratory;
(7) Maneesh Agrawala, Stanford University;
(8) Dahua Lin, Shanghai Artificial Intelligence Laboratory;
(9) Bo Dai, The Chinese University of Hong Kong and The Chinese University of Hong Kong. Table of Links Abstract and 1 Introduction 2 Work Related 3 Preliminary AnimateDiff 4.1 Alleviate Negative Effects from Training Data with Domain Adapter 4.2 Learn Motion Priors with Motion Module 4.3 Adapt to New Motion Patterns with MotionLora 4.4 AnimateDiff in Practice 5 Experiments and 5.1 Qualitative Results 5.2 Qualitative Comparison 5.3 Ablative Study 5.4 Controllable Generation 6 Conclusion 7 Ethics Statement 8 Reproducibility Statement, Acknowledgement and References 5 EXPERIMENTS We implement AnimateDiff upon Stable Diffusion V1.5 and train motion module using the WebVid10M (Bain et al., 2021) dataset. Detailed configurations can be found in supplementary materials. 5.1 QUALITATIVE RESULTS Evaluate on community models. We evaluated the AnimateDiff with a diverse set of representative personalized T2Is collected from Civitai (2022). These personalized T2Is encompass a wide range of domains, thus serving as a comprehensive benchmark. Since personalized domains in these T2Is only respond to certain “trigger words”, we abstain from using common text prompts but refer to the model homepage to construct the evaluation prompts. In Fig. 4, we show eight qualitative results of AnimateDiff. Each sample corresponds to a distinct personalized T2I. In the second row of Figure 1, we present the outcomes obtained by integrating AnimateDiff with MotionLoRA to achieve shot type controls. The last two samples exhibit the composition capability of MotionLoRA, achieved by linearly combining the individually trained weights. Compare with baselines. In the absence of existing methods specifically designed for animating personalized T2Is, we compare our method with two recent works in video generation that can be adapted for this task: 1) Text2Video-Zero (Khachatryan et al., 2023) and 2) Tune-a-Video (Wu 7 et al., 2023). We also compare AnimateDiff with two commercial tools: 3) Gen-2 (2023) for textto-video generation, and 4) Pika Labs (2023) for image animation. The results are shown in Fig. 5. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Yuwei Guo, The Chinese University of Hong Kong; (2) Ceyuan Yang, Shanghai Artificial Intelligence Laboratory with Corresponding Author; (3) Anyi Rao, Stanford University; (4) Zhengyang Liang, Shanghai Artificial Intelligence Laboratory; (5) Yaohui Wang, Shanghai Artificial Intelligence Laboratory; (6) Yu Qiao, Shanghai Artificial Intelligence Laboratory; (7) Maneesh Agrawala, Stanford University; (8) Dahua Lin, Shanghai Artificial Intelligence Laboratory; (9) Bo Dai, The Chinese University of Hong Kong and The Chinese University of Hong Kong. Authors: Authors: (1) Yuwei Guo, The Chinese University of Hong Kong; (2) Ceyuan Yang, Shanghai Artificial Intelligence Laboratory with Corresponding Author; (3) Anyi Rao, Stanford University; (4) Zhengyang Liang, Shanghai Artificial Intelligence Laboratory; (5) Yaohui Wang, Shanghai Artificial Intelligence Laboratory; (6) Yu Qiao, Shanghai Artificial Intelligence Laboratory; (7) Maneesh Agrawala, Stanford University; (8) Dahua Lin, Shanghai Artificial Intelligence Laboratory; (9) Bo Dai, The Chinese University of Hong Kong and The Chinese University of Hong Kong. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Work Related 2 Work Related 3 Preliminary 3 Preliminary AnimateDiff AnimateDiff 4.1 Alleviate Negative Effects from Training Data with Domain Adapter 4.1 Alleviate Negative Effects from Training Data with Domain Adapter 4.2 Learn Motion Priors with Motion Module 4.2 Learn Motion Priors with Motion Module 4.3 Adapt to New Motion Patterns with MotionLora 4.3 Adapt to New Motion Patterns with MotionLora 4.4 AnimateDiff in Practice 4.4 AnimateDiff in Practice 5 Experiments and 5.1 Qualitative Results 5 Experiments and 5.1 Qualitative Results 5.2 Qualitative Comparison 5.2 Qualitative Comparison 5.3 Ablative Study 5.3 Ablative Study 5.4 Controllable Generation 5.4 Controllable Generation 6 Conclusion 6 Conclusion 7 Ethics Statement 7 Ethics Statement 8 Reproducibility Statement, Acknowledgement and References 8 Reproducibility Statement, Acknowledgement and References 5 EXPERIMENTS We implement AnimateDiff upon Stable Diffusion V1.5 and train motion module using the WebVid10M (Bain et al., 2021) dataset. Detailed configurations can be found in supplementary materials. 5.1 QUALITATIVE RESULTS Evaluate on community models. We evaluated the AnimateDiff with a diverse set of representative personalized T2Is collected from Civitai (2022). These personalized T2Is encompass a wide range of domains, thus serving as a comprehensive benchmark. Since personalized domains in these T2Is only respond to certain “trigger words”, we abstain from using common text prompts but refer to the model homepage to construct the evaluation prompts. In Fig. 4, we show eight qualitative results of AnimateDiff. Each sample corresponds to a distinct personalized T2I. In the second row of Figure 1, we present the outcomes obtained by integrating AnimateDiff with MotionLoRA to achieve shot type controls. The last two samples exhibit the composition capability of MotionLoRA, achieved by linearly combining the individually trained weights. Compare with baselines. In the absence of existing methods specifically designed for animating personalized T2Is, we compare our method with two recent works in video generation that can be adapted for this task: 1) Text2Video-Zero (Khachatryan et al., 2023) and 2) Tune-a-Video (Wu 7 et al., 2023). We also compare AnimateDiff with two commercial tools: 3) Gen-2 (2023) for textto-video generation, and 4) Pika Labs (2023) for image animation. The results are shown in Fig. 5. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

How AnimateDiff Brings Personalized T2Is to Life with Efficient Motion Modeling

Authors:
(1) Yuwei Guo, The Chinese University of Hong Kong;
(2) Ceyuan Yang, Shanghai Artificial Intelligence Laboratory with Corresponding Author;
(3) Anyi Rao, Stanford University;
(4) Zhengyang Liang, Shanghai Artificial Intelligence Laboratory;
(5) Yaohui Wang, Shanghai Artificial Intelligence Laboratory;
(6) Yu Qiao, Shanghai Artificial Intelligence Laboratory;
(7) Maneesh Agrawala, Stanford University;
(8) Dahua Lin, Shanghai Artificial Intelligence Laboratory;
(9) Bo Dai, The Chinese University of Hong Kong and The Chinese University of Hong Kong. Table of Links Abstract and 1 Introduction 2 Work Related 3 Preliminary AnimateDiff 4.1 Alleviate Negative Effects from Training Data with Domain Adapter 4.2 Learn Motion Priors with Motion Module 4.3 Adapt to New Motion Patterns with MotionLora 4.4 AnimateDiff in Practice 5 Experiments and 5.1 Qualitative Results 5.2 Qualitative Comparison 5.3 Ablative Study 5.4 Controllable Generation 6 Conclusion 7 Ethics Statement 8 Reproducibility Statement, Acknowledgement and References8 Reproducibility Statement, Acknowledgement and References 2 RELATED WORK Text-to-image diffusion models. Diffusion models (Ho et al., 2020; Dhariwal & Nichol, 2021; Song et al., 2020) for text-to-image (T2I) generation (Gu et al., 2022; Mokady et al., 2023; Podell et al., 2023; Ding et al., 2021; Zhou et al., 2022b; Ramesh et al., 2021; Li et al., 2022) have gained significant attention in both academic and non-academic communities recently. GLIDE (Nichol et al., 2021) introduced text conditions and demonstrated that incorporating classifier guidance leads to more pleasing results. DALL-E2 (Ramesh et al., 2022) improves text-image alignment by leveraging the CLIP (Radford et al., 2021) joint feature space. Imagen (Saharia et al., 2022) incorporates a large language model (Raffel et al., 2020) and a cascade architecture to achieve photorealistic results. Latent Diffusion Model (Rombach et al., 2022), also known as Stable Diffusion, moves the diffusion process to the latent space of an auto-encoder to enhance efficiency. eDiff-I (Balaji et al., 2022) employs an ensemble of diffusion models specialized for different generation stages. Personalizing T2I models. To facilitate the creation with pre-trained T2Is, many works focus on efficient model personalization (Shi et al., 2023; Lu et al., 2023; Dong et al., 2022; Kumari et al., 2023), i.e., introducing concepts or styles to the base T2I using reference images. The most straightforward approach to achieve this is complete fine-tuning of the model. Despite its potential to significantly enhance overall quality, this practice can lead to catastrophic forgetting (Kirkpatrick et al., 2017; French, 1999) when the reference image set is small. Instead, DreamBooth (Ruiz et al., 2023) fine-tunes the entire network with preservation loss and uses only a few images. Textual Inversion (Gal et al., 2022) optimize a token embedding for each new concept. Low-Rank Adaptation (LoRA) (Hu et al., 2021) facilitates the above fine-tuning process by introducing additional LoRA layers to the base T2I and optimizing only the weight residuals. There are also encoder-based approaches that address the personalization problem (Gal et al., 2023; Jia et al., 2023). In our work, we focus on tuning-based methods, including overall fine-tuning, DreamBooth (Ruiz et al., 2023), and LoRA (Hu et al., 2021), as they preserve the original feature space of the base T2I. Animating personalized T2Is. There are not many existing works regarding animating personalized T2Is. Text2Cinemagraph (Mahapatra et al., 2023) proposed to generate cinematography via flow prediction. In the field of video generation, it is common to extend a pre-trained T2I with temporal structures. Existing works (Esser et al., 2023; Zhou et al., 2022a; Singer et al., 2022; Ho et al., 2022b,a; Ruan et al., 2023; Luo et al., 2023; Yin et al., 2023b,a; Wang et al., 2023b; Hong et al., 2022; Luo et al., 2023) mostly update all parameters and modify the feature space of the original T2I and is not compatible with personalized ones. Align-Your-Latents (Blattmann et al., 2023) shows that the frozen image layers in a general video generator can be personalized. Recently, some video generation approaches have shown promising results in animating a personalized T2I model. Tune-a-Video (Wu et al., 2023) fine-tune a small number of parameters on a single video. Text2Video-Zero (Khachatryan et al., 2023) introduces a training-free method to animate a pre-trained T2I via latent wrapping based on a pre-defined affine matrix. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Yuwei Guo, The Chinese University of Hong Kong; (2) Ceyuan Yang, Shanghai Artificial Intelligence Laboratory with Corresponding Author; (3) Anyi Rao, Stanford University; (4) Zhengyang Liang, Shanghai Artificial Intelligence Laboratory; (5) Yaohui Wang, Shanghai Artificial Intelligence Laboratory; (6) Yu Qiao, Shanghai Artificial Intelligence Laboratory; (7) Maneesh Agrawala, Stanford University; (8) Dahua Lin, Shanghai Artificial Intelligence Laboratory; (9) Bo Dai, The Chinese University of Hong Kong and The Chinese University of Hong Kong. Authors: Authors: (1) Yuwei Guo, The Chinese University of Hong Kong; (2) Ceyuan Yang, Shanghai Artificial Intelligence Laboratory with Corresponding Author; (3) Anyi Rao, Stanford University; (4) Zhengyang Liang, Shanghai Artificial Intelligence Laboratory; (5) Yaohui Wang, Shanghai Artificial Intelligence Laboratory; (6) Yu Qiao, Shanghai Artificial Intelligence Laboratory; (7) Maneesh Agrawala, Stanford University; (8) Dahua Lin, Shanghai Artificial Intelligence Laboratory; (9) Bo Dai, The Chinese University of Hong Kong and The Chinese University of Hong Kong. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Work Related 2 Work Related 3 Preliminary 3 Preliminary AnimateDiff AnimateDiff 4.1 Alleviate Negative Effects from Training Data with Domain Adapter 4.1 Alleviate Negative Effects from Training Data with Domain Adapter 4.2 Learn Motion Priors with Motion Module 4.2 Learn Motion Priors with Motion Module 4.3 Adapt to New Motion Patterns with MotionLora 4.3 Adapt to New Motion Patterns with MotionLora 4.4 AnimateDiff in Practice 4.4 AnimateDiff in Practice 5 Experiments and 5.1 Qualitative Results 5 Experiments and 5.1 Qualitative Results 5.2 Qualitative Comparison 5.2 Qualitative Comparison 5.3 Ablative Study 5.3 Ablative Study 5.4 Controllable Generation 5.4 Controllable Generation 6 Conclusion 6 Conclusion 7 Ethics Statement 7 Ethics Statement 8 Reproducibility Statement, Acknowledgement and References8 Reproducibility Statement, Acknowledgement and References 8 Reproducibility Statement, Acknowledgement and References8 Reproducibility Statement, Acknowledgement and References 2 RELATED WORK Text-to-image diffusion models. Diffusion models (Ho et al., 2020; Dhariwal & Nichol, 2021; Song et al., 2020) for text-to-image (T2I) generation (Gu et al., 2022; Mokady et al., 2023; Podell et al., 2023; Ding et al., 2021; Zhou et al., 2022b; Ramesh et al., 2021; Li et al., 2022) have gained significant attention in both academic and non-academic communities recently. GLIDE (Nichol et al., 2021) introduced text conditions and demonstrated that incorporating classifier guidance leads to more pleasing results. DALL-E2 (Ramesh et al., 2022) improves text-image alignment by leveraging the CLIP (Radford et al., 2021) joint feature space. Imagen (Saharia et al., 2022) incorporates a large language model (Raffel et al., 2020) and a cascade architecture to achieve photorealistic results. Latent Diffusion Model (Rombach et al., 2022), also known as Stable Diffusion, moves the diffusion process to the latent space of an auto-encoder to enhance efficiency. eDiff-I (Balaji et al., 2022) employs an ensemble of diffusion models specialized for different generation stages. Text-to-image diffusion models. Personalizing T2I models. To facilitate the creation with pre-trained T2Is, many works focus on efficient model personalization (Shi et al., 2023; Lu et al., 2023; Dong et al., 2022; Kumari et al., 2023), i.e., introducing concepts or styles to the base T2I using reference images. The most straightforward approach to achieve this is complete fine-tuning of the model. Despite its potential to significantly enhance overall quality, this practice can lead to catastrophic forgetting (Kirkpatrick et al., 2017; French, 1999) when the reference image set is small. Instead, DreamBooth (Ruiz et al., 2023) fine-tunes the entire network with preservation loss and uses only a few images. Textual Inversion (Gal et al., 2022) optimize a token embedding for each new concept. Low-Rank Adaptation (LoRA) (Hu et al., 2021) facilitates the above fine-tuning process by introducing additional LoRA layers to the base T2I and optimizing only the weight residuals. There are also encoder-based approaches that address the personalization problem (Gal et al., 2023; Jia et al., 2023). In our work, we focus on tuning-based methods, including overall fine-tuning, DreamBooth (Ruiz et al., 2023), and LoRA (Hu et al., 2021), as they preserve the original feature space of the base T2I. Personalizing T2I models. Animating personalized T2Is. There are not many existing works regarding animating personalized T2Is. Text2Cinemagraph (Mahapatra et al., 2023) proposed to generate cinematography via flow prediction. In the field of video generation, it is common to extend a pre-trained T2I with temporal structures. Existing works (Esser et al., 2023; Zhou et al., 2022a; Singer et al., 2022; Ho et al., 2022b,a; Ruan et al., 2023; Luo et al., 2023; Yin et al., 2023b,a; Wang et al., 2023b; Hong et al., 2022; Luo et al., 2023) mostly update all parameters and modify the feature space of the original T2I and is not compatible with personalized ones. Align-Your-Latents (Blattmann et al., 2023) shows that the frozen image layers in a general video generator can be personalized. Recently, some video generation approaches have shown promising results in animating a personalized T2I model. Tune-a-Video (Wu et al., 2023) fine-tune a small number of parameters on a single video. Text2Video-Zero (Khachatryan et al., 2023) introduces a training-free method to animate a pre-trained T2I via latent wrapping based on a pre-defined affine matrix. Animating personalized T2Is. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Text-to-Image Diffusion Models and Personalized Animation Techniques

Authors:
(1) Yuwei Guo, The Chinese University of Hong Kong;
(2) Ceyuan Yang, Shanghai Artificial Intelligence Laboratory with Corresponding Author;
(3) Anyi Rao, Stanford University;
(4) Zhengyang Liang, Shanghai Artificial Intelligence Laboratory;
(5) Yaohui Wang, Shanghai Artificial Intelligence Laboratory;
(6) Yu Qiao, Shanghai Artificial Intelligence Laboratory;
(7) Maneesh Agrawala, Stanford University;
(8) Dahua Lin, Shanghai Artificial Intelligence Laboratory;
(9) Bo Dai, The Chinese University of Hong Kong and The Chinese University of Hong Kong. Table of Links Abstract and 1 Introduction 2 Work Related 3 Preliminary AnimateDiff 4.1 Alleviate Negative Effects from Training Data with Domain Adapter 4.2 Learn Motion Priors with Motion Module 4.3 Adapt to New Motion Patterns with MotionLora 4.4 AnimateDiff in Practice 5 Experiments and 5.1 Qualitative Results 5.2 Qualitative Comparison 5.3 Ablative Study 5.4 Controllable Generation 6 Conclusion 7 Ethics Statement 8 Reproducibility Statement, Acknowledgement and References ABSTRACT With the advance of text-to-image (T2I) diffusion models (e.g., Stable Diffusion) and corresponding personalization techniques such as DreamBooth and LoRA, everyone can manifest their imagination into high-quality images at an affordable cost. However, adding motion dynamics to existing high-quality personalized T2Is and enabling them to generate animations remains an open challenge. In this paper, we present AnimateDiff, a practical framework for animating personalized T2I models without requiring model-specific tuning. At the core of our framework is a plug-and-play motion module that can be trained once and seamlessly integrated into any personalized T2Is originating from the same base T2I. Through our proposed training strategy, the motion module effectively learns transferable motion priors from real-world videos. Once trained, the motion module can be inserted into a personalized T2I model to form a personalized animation generator. We further propose MotionLoRA, a lightweight fine-tuning technique for AnimateDiff that enables a pre-trained motion module to adapt to new motion patterns, such as different shot types, at a low training and data collection cost. We evaluate AnimateDiff and MotionLoRA on several public representative personalized T2I models collected from the community. The results demonstrate that our approaches help these models generate temporally smooth animation clips while preserving the visual quality and motion diversity. Codes and pre-trained weights are available at https://github.com/guoyww/AnimateDiff. 1 INTRODUCTION Text-to-image (T2I) diffusion models (Nichol et al., 2021; Ramesh et al., 2022; Saharia et al., 2022; Rombach et al., 2022) have greatly empowered artists and amateurs to create visual content using text prompts. To further stimulate the creativity of existing T2I models, lightweight personalization methods, such as DreamBooth (Ruiz et al., 2023) and LoRA (Hu et al., 2021) have been proposed. These methods enable customized fine-tuning on small datasets using consumer-grade hardware such as a laptop with an RTX3080, thereby allowing users to adapt a base T2I model to new domains and improve visual quality at a relatively low cost. Consequently, a large community of AI artists and amateurs has contributed numerous personalized models on model-sharing platforms such as Civitai (2022) and Hugging Face (2022). While these personalized T2I models can generate remarkable visual quality, their outputs are limited to static images. On the other hand, the ability to generate animations is more desirable in real-world production, such as in the movie and cartoon industries. In this work, we aim to directly transform existing high-quality personalized T2I models into animation generators without requiring model-specific fine-tuning, which is often impractical in terms of computation and data collection costs for amateur users. We present AnimateDiff, an effective pipeline for addressing the problem of animating personalized T2Is while preserving their visual quality and domain knowledge. The core of AnimateDiff is an approach for training a plug-and-play motion module that learns reasonable motion priors from video datasets, such as WebVid-10M (Bain et al., 2021). At inference time, the trained motion module can be directly integrated into personalized T2Is and produce smooth and visually appealing animations without requiring specific tuning. The training of the motion module in AnimateDiff consists of three stages. Firstly, we fine-tune a domain adapter on the base T2I to align with the visual distribution of the target video dataset. This preliminary step guarantees the motion module concentrates on learning the motion priors rather than pixel-level details from the training videos. Secondly, we inflate the base T2I together with the domain adapter and introduce a newly initialized motion module for motion modeling. We then optimize this module on videos while keeping the domain adapter and base T2I weights fixed. By doing so, the motion module learns generalized motion priors and can, via module insertion, enable other personalized T2Is to generate smooth and appealing animations aligned with their personalized domains. The third stage of AnimateDiff, also dubbed as MotionLoRA, aims to adapt the pre-trained motion module to specific motion patterns with a small number of reference videos and training iterations. We achieve this by fine-tuning the motion module with the aid of Low-Rank Adaptation (LoRA) (Hu et al., 2021). Remarkably, adapting to a new motion pattern can be achieved with as few as 50 reference videos. Moreover, a MotionLoRA model requires only approximately 30M of additional storage space, further enhancing the efficiency of model sharing. This efficiency is particularly valuable for users who are unable to bear the expensive costs of pre-training but desire to fine-tune the motion module for specific effects. We evaluate the performance of AnimateDiff and MotionLoRA on a diverse set of personalized T2I models collected from model-sharing platforms (Civitai, 2022; Hugging Face, 2022). These models encompass a wide spectrum of domains, ranging from 2D cartoons to realistic photographs, thereby forming a comprehensive benchmark for our evaluation. The results of our experiments demonstrate promising outcomes. In practice, we also found that a Transformer (Vaswani et al., 2017) architecture along the temporal axis is adequate for capturing appropriate motion priors. We also demonstrate that our motion module can be seamlessly integrated with existing content-controlling approaches (Zhang et al., 2023; Mou et al., 2023) such as ControlNet without requiring additional training, enabling AnimateDiff for controllable animation generation. In summary, (1) we present AnimateDiff, a practical pipeline that enables the animation generation ability of any personalized T2Is without specific fine-tuning; (2) we verify that a Transformer architecture is adequate for modeling motion priors, which provides valuable insights for video generation; (3) we propose MotionLoRA, a lightweight fine-tuning technique to adapt pre-trained motion modules to new motion patterns; (4) we comprehensively evaluate our approach with representative community models and compare it with both academic baselines and commercial tools such as Gen2 (2023) and Pika Labs (2023). Furthermore, we showcase its compatibility with existing works for controllable generation. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Yuwei Guo, The Chinese University of Hong Kong; (2) Ceyuan Yang, Shanghai Artificial Intelligence Laboratory with Corresponding Author; (3) Anyi Rao, Stanford University; (4) Zhengyang Liang, Shanghai Artificial Intelligence Laboratory; (5) Yaohui Wang, Shanghai Artificial Intelligence Laboratory; (6) Yu Qiao, Shanghai Artificial Intelligence Laboratory; (7) Maneesh Agrawala, Stanford University; (8) Dahua Lin, Shanghai Artificial Intelligence Laboratory; (9) Bo Dai, The Chinese University of Hong Kong and The Chinese University of Hong Kong. Authors: Authors: (1) Yuwei Guo, The Chinese University of Hong Kong; (2) Ceyuan Yang, Shanghai Artificial Intelligence Laboratory with Corresponding Author; (3) Anyi Rao, Stanford University; (4) Zhengyang Liang, Shanghai Artificial Intelligence Laboratory; (5) Yaohui Wang, Shanghai Artificial Intelligence Laboratory; (6) Yu Qiao, Shanghai Artificial Intelligence Laboratory; (7) Maneesh Agrawala, Stanford University; (8) Dahua Lin, Shanghai Artificial Intelligence Laboratory; (9) Bo Dai, The Chinese University of Hong Kong and The Chinese University of Hong Kong. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Work Related 2 Work Related 3 Preliminary 3 Preliminary AnimateDiff AnimateDiff 4.1 Alleviate Negative Effects from Training Data with Domain Adapter 4.1 Alleviate Negative Effects from Training Data with Domain Adapter 4.2 Learn Motion Priors with Motion Module 4.2 Learn Motion Priors with Motion Module 4.3 Adapt to New Motion Patterns with MotionLora 4.3 Adapt to New Motion Patterns with MotionLora 4.4 AnimateDiff in Practice 4.4 AnimateDiff in Practice 5 Experiments and 5.1 Qualitative Results 5 Experiments and 5.1 Qualitative Results 5.2 Qualitative Comparison 5.2 Qualitative Comparison 5.3 Ablative Study 5.3 Ablative Study 5.4 Controllable Generation 5.4 Controllable Generation 6 Conclusion 6 Conclusion 7 Ethics Statement 7 Ethics Statement 8 Reproducibility Statement, Acknowledgement and References 8 Reproducibility Statement, Acknowledgement and References ABSTRACT With the advance of text-to-image (T2I) diffusion models (e.g., Stable Diffusion) and corresponding personalization techniques such as DreamBooth and LoRA, everyone can manifest their imagination into high-quality images at an affordable cost. However, adding motion dynamics to existing high-quality personalized T2Is and enabling them to generate animations remains an open challenge. In this paper, we present AnimateDiff, a practical framework for animating personalized T2I models without requiring model-specific tuning. At the core of our framework is a plug-and-play motion module that can be trained once and seamlessly integrated into any personalized T2Is originating from the same base T2I. Through our proposed training strategy, the motion module effectively learns transferable motion priors from real-world videos. Once trained, the motion module can be inserted into a personalized T2I model to form a personalized animation generator. We further propose MotionLoRA, a lightweight fine-tuning technique for AnimateDiff that enables a pre-trained motion module to adapt to new motion patterns, such as different shot types, at a low training and data collection cost. We evaluate AnimateDiff and MotionLoRA on several public representative personalized T2I models collected from the community. The results demonstrate that our approaches help these models generate temporally smooth animation clips while preserving the visual quality and motion diversity. Codes and pre-trained weights are available at https://github.com/guoyww/AnimateDiff. 1 INTRODUCTION Text-to-image (T2I) diffusion models (Nichol et al., 2021; Ramesh et al., 2022; Saharia et al., 2022; Rombach et al., 2022) have greatly empowered artists and amateurs to create visual content using text prompts. To further stimulate the creativity of existing T2I models, lightweight personalization methods, such as DreamBooth (Ruiz et al., 2023) and LoRA (Hu et al., 2021) have been proposed. These methods enable customized fine-tuning on small datasets using consumer-grade hardware such as a laptop with an RTX3080, thereby allowing users to adapt a base T2I model to new domains and improve visual quality at a relatively low cost. Consequently, a large community of AI artists and amateurs has contributed numerous personalized models on model-sharing platforms such as Civitai (2022) and Hugging Face (2022). While these personalized T2I models can generate remarkable visual quality, their outputs are limited to static images. On the other hand, the ability to generate animations is more desirable in real-world production, such as in the movie and cartoon industries. In this work, we aim to directly transform existing high-quality personalized T2I models into animation generators without requiring model-specific fine-tuning, which is often impractical in terms of computation and data collection costs for amateur users. We present AnimateDiff, an effective pipeline for addressing the problem of animating personalized T2Is while preserving their visual quality and domain knowledge. The core of AnimateDiff is an approach for training a plug-and-play motion module that learns reasonable motion priors from video datasets, such as WebVid-10M (Bain et al., 2021). At inference time, the trained motion module can be directly integrated into personalized T2Is and produce smooth and visually appealing animations without requiring specific tuning. The training of the motion module in AnimateDiff consists of three stages. Firstly, we fine-tune a domain adapter on the base T2I to align with the visual distribution of the target video dataset. This preliminary step guarantees the motion module concentrates on learning the motion priors rather than pixel-level details from the training videos. Secondly, we inflate the base T2I together with the domain adapter and introduce a newly initialized motion module for motion modeling. We then optimize this module on videos while keeping the domain adapter and base T2I weights fixed. By doing so, the motion module learns generalized motion priors and can, via module insertion, enable other personalized T2Is to generate smooth and appealing animations aligned with their personalized domains. The third stage of AnimateDiff, also dubbed as MotionLoRA, aims to adapt the pre-trained motion module to specific motion patterns with a small number of reference videos and training iterations. We achieve this by fine-tuning the motion module with the aid of Low-Rank Adaptation (LoRA) (Hu et al., 2021). Remarkably, adapting to a new motion pattern can be achieved with as few as 50 reference videos. Moreover, a MotionLoRA model requires only approximately 30M of additional storage space, further enhancing the efficiency of model sharing. This efficiency is particularly valuable for users who are unable to bear the expensive costs of pre-training but desire to fine-tune the motion module for specific effects. We evaluate the performance of AnimateDiff and MotionLoRA on a diverse set of personalized T2I models collected from model-sharing platforms (Civitai, 2022; Hugging Face, 2022). These models encompass a wide spectrum of domains, ranging from 2D cartoons to realistic photographs, thereby forming a comprehensive benchmark for our evaluation. The results of our experiments demonstrate promising outcomes. In practice, we also found that a Transformer (Vaswani et al., 2017) architecture along the temporal axis is adequate for capturing appropriate motion priors. We also demonstrate that our motion module can be seamlessly integrated with existing content-controlling approaches (Zhang et al., 2023; Mou et al., 2023) such as ControlNet without requiring additional training, enabling AnimateDiff for controllable animation generation. In summary, (1) we present AnimateDiff, a practical pipeline that enables the animation generation ability of any personalized T2Is without specific fine-tuning; (2) we verify that a Transformer architecture is adequate for modeling motion priors, which provides valuable insights for video generation; (3) we propose MotionLoRA, a lightweight fine-tuning technique to adapt pre-trained motion modules to new motion patterns; (4) we comprehensively evaluate our approach with representative community models and compare it with both academic baselines and commercial tools such as Gen2 (2023) and Pika Labs (2023). Furthermore, we showcase its compatibility with existing works for controllable generation. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

How to Transform Static Text-to-Image Models into Dynamic Animation Generators

So wandeln Sie statische Text-zu-Bild-Modelle in dynamische Animationsgeneratoren um

Cómo transformar modelos estáticos de texto a imagen en generadores de animación dinámicos

Nola eraldatu testu-irudi eredu estatikoak animazio-sorgailu dinamikoetan

静的なテキストから画像へのモデルを動的なアニメーション ジェネレータに変換する方法

ວິທີການປ່ຽນຕົວແບບຂໍ້ຄວາມເປັນຮູບພາບຄົງທີ່ໄປສູ່ເຄື່ອງສ້າງພາບເຄື່ອນໄຫວແບບໄດນາມິກ

गतिशील एनिमेसन जेनरेटरहरूमा स्थिर पाठ-देखि-छवि मोडेलहरू कसरी रूपान्तरण गर्ने

Как преобразовать статические модели «текст-изображение» в динамические генераторы анимации

Sida Loo Beddelo Qaabka Qoraal-Ka-Sawirka Jooga Ah Loona Bedeli Karaa Animation Generators

Jinsi ya Kubadilisha Miundo Tuli ya Maandishi-hadi-Picha kuwa Jenereta za Uhuishaji Zinazobadilika

Statiki tekstden-şekil modellerini dinamiki animasiýa generatorlaryna nädip öwürmeli

Statik Metin-Görüntü Modellerini Dinamik Animasyon Oluşturucularına Nasıl Dönüştürebilirsiniz

جامد متن سے تصویری ماڈلز کو متحرک اینیمیشن جنریٹرز میں کیسے تبدیل کیا جائے

Steal My Subscription, Revenue Forecasting Model (P.S.: It Tracks Churn!)

Primer on Large Language Model (LLM) Inference Optimizations: 3. Model Architecture Optimizations

Authors:
(1) Nora Schneider, Computer Science Department, ETH Zurich, Zurich, Switzerland (nschneide@student.ethz.ch);
(2) Shirin Goshtasbpour, Computer Science Department, ETH Zurich, Zurich, Switzerland and Swiss Data Science Center, Zurich, Switzerland (shirin.goshtasbpour@inf.ethz.ch);
(3) Fernando Perez-Cruz, Computer Science Department, ETH Zurich, Zurich, Switzerland and Swiss Data Science Center, Zurich, Switzerland (fernando.perezcruz@sdsc.ethz.ch). Table of Links Abstract and 1 Introduction 2 Background 2.1 Data Augmentation 2.2 Anchor Regression 3 Anchor Data Augmentation 3.1 Comparison to C-Mixup and 3.2 Preserving nonlinear data structure 3.3 Algorithm 4 Experiments and 4.1 Linear synthetic data 4.2 Housing nonlinear regression 4.3 In-distribution Generalization 4.4 Out-of-distribution Robustness 5 Conclusion, Broader Impact, and References A Additional information for Anchor Data Augmentation B Experiments 3.3 Algorithm Finally, in this section, we present the ADA algorithm step by step (Algorithm 1) to generate minibatches of data that can be used to train neural networks (or any other nonlinear regressor) by any stochastic gradient descent method. As discussed previously, we propose to repeat the augmentation with different parameter combinations for each minibatch. This paper is available on arxiv under CC0 1.0 DEED license. Authors: (1) Nora Schneider, Computer Science Department, ETH Zurich, Zurich, Switzerland (nschneide@student.ethz.ch); (2) Shirin Goshtasbpour, Computer Science Department, ETH Zurich, Zurich, Switzerland and Swiss Data Science Center, Zurich, Switzerland (shirin.goshtasbpour@inf.ethz.ch); (3) Fernando Perez-Cruz, Computer Science Department, ETH Zurich, Zurich, Switzerland and Swiss Data Science Center, Zurich, Switzerland (fernando.perezcruz@sdsc.ethz.ch). Authors: Authors: (1) Nora Schneider, Computer Science Department, ETH Zurich, Zurich, Switzerland (nschneide@student.ethz.ch); (2) Shirin Goshtasbpour, Computer Science Department, ETH Zurich, Zurich, Switzerland and Swiss Data Science Center, Zurich, Switzerland (shirin.goshtasbpour@inf.ethz.ch); (3) Fernando Perez-Cruz, Computer Science Department, ETH Zurich, Zurich, Switzerland and Swiss Data Science Center, Zurich, Switzerland (fernando.perezcruz@sdsc.ethz.ch). Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Background 2.1 Data Augmentation 2.1 Data Augmentation 2.2 Anchor Regression 2.2 Anchor Regression 3 Anchor Data Augmentation 3 Anchor Data Augmentation 3.1 Comparison to C-Mixup and 3.2 Preserving nonlinear data structure 3.1 Comparison to C-Mixup and 3.2 Preserving nonlinear data structure 3.3 Algorithm 3.3 Algorithm 4 Experiments and 4.1 Linear synthetic data 4 Experiments and 4.1 Linear synthetic data 4.2 Housing nonlinear regression 4.2 Housing nonlinear regression 4.3 In-distribution Generalization 4.3 In-distribution Generalization 4.4 Out-of-distribution Robustness 4.4 Out-of-distribution Robustness 5 Conclusion, Broader Impact, and References 5 Conclusion, Broader Impact, and References A Additional information for Anchor Data Augmentation A Additional information for Anchor Data Augmentation B Experiments B Experiments 3.3 Algorithm Finally, in this section, we present the ADA algorithm step by step (Algorithm 1) to generate minibatches of data that can be used to train neural networks (or any other nonlinear regressor) by any stochastic gradient descent method. As discussed previously, we propose to repeat the augmentation with different parameter combinations for each minibatch. This paper is available on arxiv under CC0 1.0 DEED license. This paper is available on arxiv under CC0 1.0 DEED license. available on arxiv

How to Implement ADA for Data Augmentation in Nonlinear Regression Models

Stories, News & Mentions about MODE

Mode’s Stories on HackerNoon

Mode's News & Mentions