Moka

moka

🏆 Startups of The Year: 2 Months Left to Cast Your Vote!

🏆 Maran qalltawinakapa: 2 Phaxsiw Voto churañatakix utji!

🏆 Årets startups: 2 måneder tilbage til at afgive din stemme!

🏆 Startups del año: ¡Quedan 2 meses para emitir tu voto!

🏆 استارت‌آپ‌های سال: ۲ ماه دیگر برای رای دادن باقی مانده است!

🏆 今年のスタートアップ: 投票期限まであと 2 か月!

🏆 Оны шилдэг гарааны бизнесүүд: Санал өгөхөд 2 сар үлдлээ!

🏆 Startups do Ano: Restam 2 meses para você votar!

🏆 Kutanga Kwegore: Mwedzi miviri Yasara Kuti Uvhote Yako!

🏆 Årets startups: 2 månader kvar att rösta!

🏆 Yilning eng yaxshi startaplari: Ovoz berishga 2 oy qoldi!

🏆 Khởi nghiệp của năm: Còn 2 tháng nữa để bình chọn!

🏆 年度最佳初创企业：还剩 2 个月可投票！

This Open-Source Image Generator From Russian Scientists Uses A Novel Approach to Diffusion

A Reference List to Learn More About Image Editing, Video Editing, and Diffusion Models

DreamLLM: What We Can Conclude From This Comprehensive Framework?

Ensuring Reproducibility in AI Research: Code and Pre-trained Weights Open-Sourced

Authors:
(1) Yuwei Guo, The Chinese University of Hong Kong;
(2) Ceyuan Yang, Shanghai Artificial Intelligence Laboratory with Corresponding Author;
(3) Anyi Rao, Stanford University;
(4) Zhengyang Liang, Shanghai Artificial Intelligence Laboratory;
(5) Yaohui Wang, Shanghai Artificial Intelligence Laboratory;
(6) Yu Qiao, Shanghai Artificial Intelligence Laboratory;
(7) Maneesh Agrawala, Stanford University;
(8) Dahua Lin, Shanghai Artificial Intelligence Laboratory;
(9) Bo Dai, The Chinese University of Hong Kong and The Chinese University of Hong Kong. Table of Links Abstract and 1 Introduction 2 Work Related 3 Preliminary AnimateDiff 4.1 Alleviate Negative Effects from Training Data with Domain Adapter 4.2 Learn Motion Priors with Motion Module 4.3 Adapt to New Motion Patterns with MotionLora 4.4 AnimateDiff in Practice 5 Experiments and 5.1 Qualitative Results 5.2 Qualitative Comparison 5.3 Ablative Study 5.4 Controllable Generation 6 Conclusion 7 Ethics Statement 8 Reproducibility Statement, Acknowledgement and References8 Reproducibility Statement, Acknowledgement and References 2 RELATED WORK Text-to-image diffusion models. Diffusion models (Ho et al., 2020; Dhariwal & Nichol, 2021; Song et al., 2020) for text-to-image (T2I) generation (Gu et al., 2022; Mokady et al., 2023; Podell et al., 2023; Ding et al., 2021; Zhou et al., 2022b; Ramesh et al., 2021; Li et al., 2022) have gained significant attention in both academic and non-academic communities recently. GLIDE (Nichol et al., 2021) introduced text conditions and demonstrated that incorporating classifier guidance leads to more pleasing results. DALL-E2 (Ramesh et al., 2022) improves text-image alignment by leveraging the CLIP (Radford et al., 2021) joint feature space. Imagen (Saharia et al., 2022) incorporates a large language model (Raffel et al., 2020) and a cascade architecture to achieve photorealistic results. Latent Diffusion Model (Rombach et al., 2022), also known as Stable Diffusion, moves the diffusion process to the latent space of an auto-encoder to enhance efficiency. eDiff-I (Balaji et al., 2022) employs an ensemble of diffusion models specialized for different generation stages. Personalizing T2I models. To facilitate the creation with pre-trained T2Is, many works focus on efficient model personalization (Shi et al., 2023; Lu et al., 2023; Dong et al., 2022; Kumari et al., 2023), i.e., introducing concepts or styles to the base T2I using reference images. The most straightforward approach to achieve this is complete fine-tuning of the model. Despite its potential to significantly enhance overall quality, this practice can lead to catastrophic forgetting (Kirkpatrick et al., 2017; French, 1999) when the reference image set is small. Instead, DreamBooth (Ruiz et al., 2023) fine-tunes the entire network with preservation loss and uses only a few images. Textual Inversion (Gal et al., 2022) optimize a token embedding for each new concept. Low-Rank Adaptation (LoRA) (Hu et al., 2021) facilitates the above fine-tuning process by introducing additional LoRA layers to the base T2I and optimizing only the weight residuals. There are also encoder-based approaches that address the personalization problem (Gal et al., 2023; Jia et al., 2023). In our work, we focus on tuning-based methods, including overall fine-tuning, DreamBooth (Ruiz et al., 2023), and LoRA (Hu et al., 2021), as they preserve the original feature space of the base T2I. Animating personalized T2Is. There are not many existing works regarding animating personalized T2Is. Text2Cinemagraph (Mahapatra et al., 2023) proposed to generate cinematography via flow prediction. In the field of video generation, it is common to extend a pre-trained T2I with temporal structures. Existing works (Esser et al., 2023; Zhou et al., 2022a; Singer et al., 2022; Ho et al., 2022b,a; Ruan et al., 2023; Luo et al., 2023; Yin et al., 2023b,a; Wang et al., 2023b; Hong et al., 2022; Luo et al., 2023) mostly update all parameters and modify the feature space of the original T2I and is not compatible with personalized ones. Align-Your-Latents (Blattmann et al., 2023) shows that the frozen image layers in a general video generator can be personalized. Recently, some video generation approaches have shown promising results in animating a personalized T2I model. Tune-a-Video (Wu et al., 2023) fine-tune a small number of parameters on a single video. Text2Video-Zero (Khachatryan et al., 2023) introduces a training-free method to animate a pre-trained T2I via latent wrapping based on a pre-defined affine matrix. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Yuwei Guo, The Chinese University of Hong Kong; (2) Ceyuan Yang, Shanghai Artificial Intelligence Laboratory with Corresponding Author; (3) Anyi Rao, Stanford University; (4) Zhengyang Liang, Shanghai Artificial Intelligence Laboratory; (5) Yaohui Wang, Shanghai Artificial Intelligence Laboratory; (6) Yu Qiao, Shanghai Artificial Intelligence Laboratory; (7) Maneesh Agrawala, Stanford University; (8) Dahua Lin, Shanghai Artificial Intelligence Laboratory; (9) Bo Dai, The Chinese University of Hong Kong and The Chinese University of Hong Kong. Authors: Authors: (1) Yuwei Guo, The Chinese University of Hong Kong; (2) Ceyuan Yang, Shanghai Artificial Intelligence Laboratory with Corresponding Author; (3) Anyi Rao, Stanford University; (4) Zhengyang Liang, Shanghai Artificial Intelligence Laboratory; (5) Yaohui Wang, Shanghai Artificial Intelligence Laboratory; (6) Yu Qiao, Shanghai Artificial Intelligence Laboratory; (7) Maneesh Agrawala, Stanford University; (8) Dahua Lin, Shanghai Artificial Intelligence Laboratory; (9) Bo Dai, The Chinese University of Hong Kong and The Chinese University of Hong Kong. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2 Work Related 2 Work Related 3 Preliminary 3 Preliminary AnimateDiff AnimateDiff 4.1 Alleviate Negative Effects from Training Data with Domain Adapter 4.1 Alleviate Negative Effects from Training Data with Domain Adapter 4.2 Learn Motion Priors with Motion Module 4.2 Learn Motion Priors with Motion Module 4.3 Adapt to New Motion Patterns with MotionLora 4.3 Adapt to New Motion Patterns with MotionLora 4.4 AnimateDiff in Practice 4.4 AnimateDiff in Practice 5 Experiments and 5.1 Qualitative Results 5 Experiments and 5.1 Qualitative Results 5.2 Qualitative Comparison 5.2 Qualitative Comparison 5.3 Ablative Study 5.3 Ablative Study 5.4 Controllable Generation 5.4 Controllable Generation 6 Conclusion 6 Conclusion 7 Ethics Statement 7 Ethics Statement 8 Reproducibility Statement, Acknowledgement and References8 Reproducibility Statement, Acknowledgement and References 8 Reproducibility Statement, Acknowledgement and References8 Reproducibility Statement, Acknowledgement and References 2 RELATED WORK Text-to-image diffusion models. Diffusion models (Ho et al., 2020; Dhariwal & Nichol, 2021; Song et al., 2020) for text-to-image (T2I) generation (Gu et al., 2022; Mokady et al., 2023; Podell et al., 2023; Ding et al., 2021; Zhou et al., 2022b; Ramesh et al., 2021; Li et al., 2022) have gained significant attention in both academic and non-academic communities recently. GLIDE (Nichol et al., 2021) introduced text conditions and demonstrated that incorporating classifier guidance leads to more pleasing results. DALL-E2 (Ramesh et al., 2022) improves text-image alignment by leveraging the CLIP (Radford et al., 2021) joint feature space. Imagen (Saharia et al., 2022) incorporates a large language model (Raffel et al., 2020) and a cascade architecture to achieve photorealistic results. Latent Diffusion Model (Rombach et al., 2022), also known as Stable Diffusion, moves the diffusion process to the latent space of an auto-encoder to enhance efficiency. eDiff-I (Balaji et al., 2022) employs an ensemble of diffusion models specialized for different generation stages. Text-to-image diffusion models. Personalizing T2I models. To facilitate the creation with pre-trained T2Is, many works focus on efficient model personalization (Shi et al., 2023; Lu et al., 2023; Dong et al., 2022; Kumari et al., 2023), i.e., introducing concepts or styles to the base T2I using reference images. The most straightforward approach to achieve this is complete fine-tuning of the model. Despite its potential to significantly enhance overall quality, this practice can lead to catastrophic forgetting (Kirkpatrick et al., 2017; French, 1999) when the reference image set is small. Instead, DreamBooth (Ruiz et al., 2023) fine-tunes the entire network with preservation loss and uses only a few images. Textual Inversion (Gal et al., 2022) optimize a token embedding for each new concept. Low-Rank Adaptation (LoRA) (Hu et al., 2021) facilitates the above fine-tuning process by introducing additional LoRA layers to the base T2I and optimizing only the weight residuals. There are also encoder-based approaches that address the personalization problem (Gal et al., 2023; Jia et al., 2023). In our work, we focus on tuning-based methods, including overall fine-tuning, DreamBooth (Ruiz et al., 2023), and LoRA (Hu et al., 2021), as they preserve the original feature space of the base T2I. Personalizing T2I models. Animating personalized T2Is. There are not many existing works regarding animating personalized T2Is. Text2Cinemagraph (Mahapatra et al., 2023) proposed to generate cinematography via flow prediction. In the field of video generation, it is common to extend a pre-trained T2I with temporal structures. Existing works (Esser et al., 2023; Zhou et al., 2022a; Singer et al., 2022; Ho et al., 2022b,a; Ruan et al., 2023; Luo et al., 2023; Yin et al., 2023b,a; Wang et al., 2023b; Hong et al., 2022; Luo et al., 2023) mostly update all parameters and modify the feature space of the original T2I and is not compatible with personalized ones. Align-Your-Latents (Blattmann et al., 2023) shows that the frozen image layers in a general video generator can be personalized. Recently, some video generation approaches have shown promising results in animating a personalized T2I model. Tune-a-Video (Wu et al., 2023) fine-tune a small number of parameters on a single video. Text2Video-Zero (Khachatryan et al., 2023) introduces a training-free method to animate a pre-trained T2I via latent wrapping based on a pre-defined affine matrix. Animating personalized T2Is. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Text-to-Image Diffusion Models and Personalized Animation Techniques

Avoid Jumbled Acronyms for Clarity TL;DR: Treat acronyms like normal words to improve human readability. Problems Reduced readability
Breaking naming style
Words confusion
Harder to pronounce Solutions Treat acronyms as Capitalized words


Use camelCase or snake_case Context Acronyms in uppercase (like JSON, XML, REST) in camel case break the natural reading flow. You may think sendJSONRequestOnHTTPREST is a clear name, but it makes your code harder to read, especially when you string multiple acronyms together. Treating acronyms like normal words in camelCase (sendJsonRequest) keeps your code more readable and easier to understand. Sample Code Wrong class NetworkConnector {
    func validateXMLFile() { /*...*/ }
    func sendJSONRequest() { /*...*/ }
    func parseURLResponse() { /*...*/ }
    func setRESTAPIURL() { /*...*/ }
    func retrieveHTTPStatusCode() { /*...*/ }
    func updateDBConnection() { /*...*/ }
    func configureSSLCertificate() { /*...*/ }
    func setHTMLTemplate() { /*...*/ }
    func generateUUID() { /*...*/ }
    func connectViaFTP() { /*...*/ }
} Right class NetworkConnector {
    func validateXmlFile() { /*...*/ }
    func sendJsonRequest() { /*...*/ }
    func parseUrlResponse() { /*...*/ }
    func setRestApiUrl() { /*...*/ }
    func retrieveHttpStatusCode() { /*...*/ }
    func updateDbConnection() { /*...*/ }
    func configureSslCertificate() { /*...*/ }
    func setHtmlTemplate() { /*...*/ }
    func generateUuid() { /*...*/ }
    func connectViaFtp() { /*...*/ }
} Detection [x]Semi-Automatic It would help if you had a smart thesaurus. To detect this smell, look for methods or variable names with uppercase acronyms that disrupt readability. Code reviewers or linters can also flag camelCase inconsistencies. Tags Code Standards Level [x]Beginner AI Generation Modern AI code generators may produce inconsistent acronym casing. Always review and adjust their output to match your conventions. AI Detection With prompts, AI can fix these naming issues and suggest improvements based on the camelCase style. Try Them! Remember: AI Assistants make lots of mistakes Without Proper Instructions

With Specific Instructions



ChatGPT

ChatGPT



Claude

Claude



Perplexity

Perplexity



Copilot

Copilot



Gemini

Gemini Conclusion Naming conventions are key to readable code. Treat acronyms like normal words, and avoid uppercase blocks to keep your code easy to understand. Relations https://hackernoon.com/how-to-find-the-stinky-parts-of-your-code-part-xxxii https://hackernoon.com/how-to-find-the-stinky-parts-of-your-code-part-x-i7r34uj More Info https://www.linkedin.com/posts/danielmoka_clean-code-tip-name-acronyms-as-normal-words-activity-7251472396438687744-C4kx/?embedable=true Disclaimer Code Smells are my opinion. Credits Photo by Csabi Elter on Unsplash Thank you @Daniel Moka for this tip. Simple can be harder than complex Steve Jobs https://hackernoon.com/400-thought-provoking-software-engineering-quotes?embedable=true This article is part of the CodeSmell Series. How to Find the Stinky Parts of your Code Avoid Jumbled Acronyms for Clarity Avoid Jumbled Acronyms for Clarity TL;DR: Treat acronyms like normal words to improve human readability. TL;DR: Treat acronyms like normal words to improve human readability. Problems Reduced readability Breaking naming style Words confusion Harder to pronounce Reduced readability Breaking naming style Words confusion Harder to pronounce Solutions Treat acronyms as Capitalized words Use camelCase or snake_case Treat acronyms as Capitalized words Treat acronyms as Capitalized words Capitalized Use camelCase or snake_case Use camelCase or snake_case camelCase camelCase snake_case snake_case Context Acronyms in uppercase (like JSON, XML, REST ) in camel case break the natural reading flow. JSON, XML, REST camel case You may think sendJSONRequestOnHTTPREST is a clear name, but it makes your code harder to read, especially when you string multiple acronyms together. sendJSONRequestOnHTTPREST Treating acronyms like normal words in camelCase ( sendJsonRequest ) keeps your code more readable and easier to understand. sendJsonRequest Sample Code Wrong class NetworkConnector {
    func validateXMLFile() { /*...*/ }
    func sendJSONRequest() { /*...*/ }
    func parseURLResponse() { /*...*/ }
    func setRESTAPIURL() { /*...*/ }
    func retrieveHTTPStatusCode() { /*...*/ }
    func updateDBConnection() { /*...*/ }
    func configureSSLCertificate() { /*...*/ }
    func setHTMLTemplate() { /*...*/ }
    func generateUUID() { /*...*/ }
    func connectViaFTP() { /*...*/ }
} class NetworkConnector {
    func validateXMLFile() { /*...*/ }
    func sendJSONRequest() { /*...*/ }
    func parseURLResponse() { /*...*/ }
    func setRESTAPIURL() { /*...*/ }
    func retrieveHTTPStatusCode() { /*...*/ }
    func updateDBConnection() { /*...*/ }
    func configureSSLCertificate() { /*...*/ }
    func setHTMLTemplate() { /*...*/ }
    func generateUUID() { /*...*/ }
    func connectViaFTP() { /*...*/ }
} Right class NetworkConnector {
    func validateXmlFile() { /*...*/ }
    func sendJsonRequest() { /*...*/ }
    func parseUrlResponse() { /*...*/ }
    func setRestApiUrl() { /*...*/ }
    func retrieveHttpStatusCode() { /*...*/ }
    func updateDbConnection() { /*...*/ }
    func configureSslCertificate() { /*...*/ }
    func setHtmlTemplate() { /*...*/ }
    func generateUuid() { /*...*/ }
    func connectViaFtp() { /*...*/ }
} class NetworkConnector {
    func validateXmlFile() { /*...*/ }
    func sendJsonRequest() { /*...*/ }
    func parseUrlResponse() { /*...*/ }
    func setRestApiUrl() { /*...*/ }
    func retrieveHttpStatusCode() { /*...*/ }
    func updateDbConnection() { /*...*/ }
    func configureSslCertificate() { /*...*/ }
    func setHtmlTemplate() { /*...*/ }
    func generateUuid() { /*...*/ }
    func connectViaFtp() { /*...*/ }
} Detection [x]Semi-Automatic [x] Semi-Automatic [x] It would help if you had a smart thesaurus. To detect this smell, look for methods or variable names with uppercase acronyms that disrupt readability. Code reviewers or linters can also flag camelCase inconsistencies. Tags Code Standards Code Standards Level [x]Beginner [x] Beginner [x] AI Generation Modern AI code generators may produce inconsistent acronym casing. Always review and adjust their output to match your conventions. AI Detection With prompts, AI can fix these naming issues and suggest improvements based on the camelCase style. camelCase camelCase Try Them! Remember: AI Assistants make lots of mistakes Remember: AI Assistants make lots of mistakes Without Proper Instructions

With Specific Instructions



ChatGPT

ChatGPT



Claude

Claude



Perplexity

Perplexity



Copilot

Copilot



Gemini

Gemini Without Proper Instructions

With Specific Instructions ChatGPT

ChatGPT Claude

Claude Perplexity

Perplexity Copilot

Copilot Gemini

Gemini Without Proper Instructions With Specific Instructions Without Proper Instructions Without Proper Instructions With Specific Instructions With Specific Instructions ChatGPT ChatGPT ChatGPT ChatGPT ChatGPT ChatGPT ChatGPT ChatGPT Claude Claude Claude Claude Claude Claude Claude Claude Perplexity Perplexity Perplexity Perplexity Perplexity Perplexity Perplexity Perplexity Copilot Copilot Copilot Copilot Copilot Copilot Copilot Copilot Gemini Gemini Gemini Gemini Gemini Gemini Gemini Gemini Conclusion Naming conventions are key to readable code. Treat acronyms like normal words, and avoid uppercase blocks to keep your code easy to understand. Relations https://hackernoon.com/how-to-find-the-stinky-parts-of-your-code-part-xxxii https://hackernoon.com/how-to-find-the-stinky-parts-of-your-code-part-xxxii https://hackernoon.com/how-to-find-the-stinky-parts-of-your-code-part-x-i7r34uj https://hackernoon.com/how-to-find-the-stinky-parts-of-your-code-part-x-i7r34uj More Info https://www.linkedin.com/posts/danielmoka_clean-code-tip-name-acronyms-as-normal-words-activity-7251472396438687744-C4kx/?embedable=true https://www.linkedin.com/posts/danielmoka_clean-code-tip-name-acronyms-as-normal-words-activity-7251472396438687744-C4kx/?embedable=true Disclaimer Code Smells are my opinion . opinion Credits Photo by Csabi Elter on Unsplash Csabi Elter Unsplash Thank you @Daniel Moka for this tip. Simple can be harder than complex Simple can be harder than complex Steve Jobs Steve Jobs https://hackernoon.com/400-thought-provoking-software-engineering-quotes?embedable=true https://hackernoon.com/400-thought-provoking-software-engineering-quotes?embedable=true This article is part of the CodeSmell Series. How to Find the Stinky Parts of your Code How to Find the Stinky Parts of your Code

Code Smell 277 - UPPERCASE Acronyms 

FlowVid: Taming Imperfect Optical Flows: Conclusion, Acknowledgments and References

Authors:
(1) Yuxuan Yan,Tencent with Equal contributions and yuxuanyan@tencent.com;
(2) Chi Zhang, Tencent with Equal contributions and Corresponding Author, johnczhang@tencent.com;
(3) Rui Wang, Tencent and raywwang@tencent.com;
(4) Yichao Zhou, Tencent and yichaozhou@tencent.com;
(5) Gege Zhang, Tencent and gretazhang@tencent.com;
(6) Pei Cheng, Tencent and peicheng@tencent.com;
(7) Bin Fu, Tencent and brianfu@tencent.com;
(8) Gang Yu, Tencentm and skicyyu@tencent.com. Table of Links Abstract and 1 Introduction 2. Related Work 3. Method and 3.1. Hybrid Guidance Strategy 3.2. Handling Multiple Identities 3.3. Training 4. Experiments 4.1. Implementation details. 4.2. Results 5. Conclusion and References 2. Related Work Text-to-image diffusion models. Diffusion models have recently come to the forefront of generative model research. Their exceptional capability to generate high-quality and diverse images has placed them at a distinct advantage over their predecessors, namely the GAN-based and autoregressive image generation models. This new generation of diffusion models, having the capacity to produce stateof-the-art synthesis results, owes much of its success to being trained on image-text pairs data at a billion scale. The integration of textual prompts into the diffusion model serves as a vital ingredient in the development of various text-to-image diffusion models. Distinguished models in this domain include GLIDE [38], DALL·E 2 [41], Imagen [45], and StableDiffusion [42]. These models leverage text as guidance during the image generation process. Consequently, they have shown considerable proficiency in synthesizing high-quality images that closely align with the provided textual description. Compared to previous efforts [29, 39, 51, 69] on face image synthesis based on Generative Adversarial Networks (GANs) [25], diffusion models exhibit greater stability during training and demonstrate enhanced capability in effectively integrating diverse forms of guidance, such as texts and stylized images. However, textual descriptions for guidance, while immensely useful, often fall short when it comes to the generation of complex and nuanced details that are frequently associated with certain subjects. For instance, generating images of human faces proves challenging with this approach. While generating images of celebrities might be feasible due to the substantial presence of their photographs in the training data that can be linked to their names, it becomes an uphill task when it comes to generating images of ordinary people using these text-to-diffusion models. Subject-driven image generation. Subject-driven image generation seeks to overcome the limitations posed by text-to-image synthesis models. Central to this novel research area is the inclusion of subject-specific reference images, which supplement the textual description to yield more precise and personalized image synthesis. To this end, several optimization-based methods have been employed, with popular ones being Low-Rank Adaptation (LoRA), DreamBooth [43], Textual Inversion [14], and Hypernetwork [3]. These methods typically involve fine-tuning a pre-trained text-to-image framework or textual embeddings to align the existing model with user-defined concepts, as indicated by a set of example images. There are several other methods derived from these works such as Unitune [49], HyperDreamBooth [44], EasyPhoto [53], FaceChain [33], etc. [4, 23, 60]. However, these methods pose some challenges. For one, they are time-consuming due to the need for model fine-tuning, which also requires multiple reference images to achieve accurate fitting. Overfitting also becomes a potential issue with these optimization-based methods, given their heavy reliance on example images. In response to these challenges, recent studies have proposed various improved methods. DreamArtist [13], for instance, mitigates the problem of overfitting by incorporating both positive and negative embeddings and jointly training them. Similarly, E4T [15] introduced an encoder-based domain-tuning method to accelerate the personalization of text-to-image models, offering a faster solution to model fitting for new subjects. There are large numbers of similar methods which are encoder-based including Composer [21], ProFusion [68] MagiCapture [22], IP-Adapter [59], ELITE [52], DisenBooth [8], Face0 [50], PhotoVerse [9], AnyDoor [11], SingleInsert [54], etc. [24, 35, 56]. Alongside optimization-based methods, a number of tuning-free methods have been concurrently proposed, such as InstantBooth [47]. This method converts input images into a textual token for general concept learning and introduces adapter layers to obtain rich image representation for generating fine-grained identity details. However, it comes with the drawback of having to train the model separately for each category. Bansal et al.[6] put forward a universal guidance algorithm that enables diffusion models to be controlled by arbitrary guidance modalities without the need to retrain the network. Similarly, Yang et al.[57] propose an inpainting-like diffusion model that learns to inpaint a masked image under the guidance of example images of a given subject. Similar methods leveraged by inversion-based personalization including Cao et al. [7], Han et al. [17], Gu et al. [16], Mokady et al. [37]. Besides, Controlnet [67] is also an effective way to personalize. In addition, there are some solutions that use editing to achieve the purpose of maintaining identity, such as SDEdit [36], Imagic [26], etc. [18, 48]. Based on all the solutions mentioned above, the method of combining multiple objects while maintaining identity is also widely used in Break-a-scene [5], FastComposer [55], Cones [34] and MultiConceptCustomDiffusion [27]. Within the text-to-image community, one specific area of interest that has garnered significant attention is the generation of human images. Human-centric image generation, owing to its vast applicability and widespread popularity, constitutes a substantial proportion of posts in the community, such as Civitai [1]. In light of these considerations, our work primarily focuses on the preservation of human identity during the image generation process. Few-shot Learning. It is a common practice in the field of few-shot learning literature to convert optimizationbased tasks into a feed-forward approach, which can significantly reduce computation time. Numerous previous studies in the realm of few-shot learning have adopted this approach to address computer vision tasks, including but not limited to classification [63–66], detection [58], and segmentation [10, 28, 30–32, 61, 62]. In this research, our objective aligns with this approach as we aim to integrate identity information into the generative process through a feed-forward module. This paper is available on arxiv under CC0 1.0 DEED license. Authors: (1) Yuxuan Yan,Tencent with Equal contributions and yuxuanyan@tencent.com; (2) Chi Zhang, Tencent with Equal contributions and Corresponding Author, johnczhang@tencent.com; (3) Rui Wang, Tencent and raywwang@tencent.com; (4) Yichao Zhou, Tencent and yichaozhou@tencent.com; (5) Gege Zhang, Tencent and gretazhang@tencent.com; (6) Pei Cheng, Tencent and peicheng@tencent.com; (7) Bin Fu, Tencent and brianfu@tencent.com; (8) Gang Yu, Tencentm and skicyyu@tencent.com. Authors: Authors: (1) Yuxuan Yan,Tencent with Equal contributions and yuxuanyan@tencent.com; (2) Chi Zhang, Tencent with Equal contributions and Corresponding Author, johnczhang@tencent.com; (3) Rui Wang, Tencent and raywwang@tencent.com; (4) Yichao Zhou, Tencent and yichaozhou@tencent.com; (5) Gege Zhang, Tencent and gretazhang@tencent.com; (6) Pei Cheng, Tencent and peicheng@tencent.com; (7) Bin Fu, Tencent and brianfu@tencent.com; (8) Gang Yu, Tencentm and skicyyu@tencent.com. Table of Links Abstract and 1 Introduction Abstract and 1 Introduction 2. Related Work 2. Related Work 3. Method and 3.1. Hybrid Guidance Strategy 3. Method and 3.1. Hybrid Guidance Strategy 3.2. Handling Multiple Identities 3.2. Handling Multiple Identities 3.3. Training 3.3. Training 4. Experiments 4.1. Implementation details. 4.1. Implementation details. 4.2. Results 4.2. Results 5. Conclusion and References 5. Conclusion and References 2. Related Work Text-to-image diffusion models. Diffusion models have recently come to the forefront of generative model research. Their exceptional capability to generate high-quality and diverse images has placed them at a distinct advantage over their predecessors, namely the GAN-based and autoregressive image generation models. This new generation of diffusion models, having the capacity to produce stateof-the-art synthesis results, owes much of its success to being trained on image-text pairs data at a billion scale. The integration of textual prompts into the diffusion model serves as a vital ingredient in the development of various text-to-image diffusion models. Distinguished models in this domain include GLIDE [38], DALL·E 2 [41], Imagen [45], and StableDiffusion [42]. These models leverage text as guidance during the image generation process. Consequently, they have shown considerable proficiency in synthesizing high-quality images that closely align with the provided textual description. Compared to previous efforts [29, 39, 51, 69] on face image synthesis based on Generative Adversarial Networks (GANs) [25], diffusion models exhibit greater stability during training and demonstrate enhanced capability in effectively integrating diverse forms of guidance, such as texts and stylized images. Text-to-image diffusion models. However, textual descriptions for guidance, while immensely useful, often fall short when it comes to the generation of complex and nuanced details that are frequently associated with certain subjects. For instance, generating images of human faces proves challenging with this approach. While generating images of celebrities might be feasible due to the substantial presence of their photographs in the training data that can be linked to their names, it becomes an uphill task when it comes to generating images of ordinary people using these text-to-diffusion models. Subject-driven image generation . Subject-driven image generation seeks to overcome the limitations posed by text-to-image synthesis models. Central to this novel research area is the inclusion of subject-specific reference images, which supplement the textual description to yield more precise and personalized image synthesis. To this end, several optimization-based methods have been employed, with popular ones being Low-Rank Adaptation (LoRA), DreamBooth [43], Textual Inversion [14], and Hypernetwork [3]. These methods typically involve fine-tuning a pre-trained text-to-image framework or textual embeddings to align the existing model with user-defined concepts, as indicated by a set of example images. There are several other methods derived from these works such as Unitune [49], HyperDreamBooth [44], EasyPhoto [53], FaceChain [33], etc. [4, 23, 60]. However, these methods pose some challenges. For one, they are time-consuming due to the need for model fine-tuning, which also requires multiple reference images to achieve accurate fitting. Overfitting also becomes a potential issue with these optimization-based methods, given their heavy reliance on example images. In response to these challenges, recent studies have proposed various improved methods. DreamArtist [13], for instance, mitigates the problem of overfitting by incorporating both positive and negative embeddings and jointly training them. Similarly, E4T [15] introduced an encoder-based domain-tuning method to accelerate the personalization of text-to-image models, offering a faster solution to model fitting for new subjects. There are large numbers of similar methods which are encoder-based including Composer [21], ProFusion [68] MagiCapture [22], IP-Adapter [59], ELITE [52], DisenBooth [8], Face0 [50], PhotoVerse [9], AnyDoor [11], SingleInsert [54], etc. [24, 35, 56]. Subject-driven image generation Alongside optimization-based methods, a number of tuning-free methods have been concurrently proposed, such as InstantBooth [47]. This method converts input images into a textual token for general concept learning and introduces adapter layers to obtain rich image representation for generating fine-grained identity details. However, it comes with the drawback of having to train the model separately for each category. Bansal et al.[6] put forward a universal guidance algorithm that enables diffusion models to be controlled by arbitrary guidance modalities without the need to retrain the network. Similarly, Yang et al.[57] propose an inpainting-like diffusion model that learns to inpaint a masked image under the guidance of example images of a given subject. Similar methods leveraged by inversion-based personalization including Cao et al. [7], Han et al. [17], Gu et al. [16], Mokady et al. [37]. Besides, Controlnet [67] is also an effective way to personalize. In addition, there are some solutions that use editing to achieve the purpose of maintaining identity, such as SDEdit [36], Imagic [26], etc. [18, 48]. Based on all the solutions mentioned above, the method of combining multiple objects while maintaining identity is also widely used in Break-a-scene [5], FastComposer [55], Cones [34] and MultiConceptCustomDiffusion [27]. Within the text-to-image community, one specific area of interest that has garnered significant attention is the generation of human images. Human-centric image generation, owing to its vast applicability and widespread popularity, constitutes a substantial proportion of posts in the community, such as Civitai [1]. In light of these considerations, our work primarily focuses on the preservation of human identity during the image generation process. Few-shot Learning. It is a common practice in the field of few-shot learning literature to convert optimizationbased tasks into a feed-forward approach, which can significantly reduce computation time. Numerous previous studies in the realm of few-shot learning have adopted this approach to address computer vision tasks, including but not limited to classification [63–66], detection [58], and segmentation [10, 28, 30–32, 61, 62]. In this research, our objective aligns with this approach as we aim to integrate identity information into the generative process through a feed-forward module. Few-shot Learning. This paper is available on arxiv under CC0 1.0 DEED license. This paper is available on arxiv under CC0 1.0 DEED license. available on arxiv

FaceStudio: Put Your Face Everywhere in Seconds: Related Work

FaceStudio: Put Your Face Everywhere in Seconds: Conclusion and References

Stories, News & Mentions about MOKA

Categories

Trending Topics

Stories, News & Mentions about MOKA

Moka’s Stories on HackerNoon

Categories

Trending Topics