Driffis in the Enterprise Collaboration Industry

Driff

driff

Anthony Chen’s quiet character study follows a traumatized Liberian woman (Cynthia Erivo) on a Greek island who befriends an American tour guide (Alia Shawkat).

‘Drift’ Review: Cynthia Erivo Keeps a Breakdown at Bay - The New York Times

POS Financing vs. BNPL: Understanding the Differences and Evolution of Solutions

Authors:
(1) Omri Avrahami, Google Research and The Hebrew University of Jerusalem;
(2) Amir Hertz, Google Research;
(3) Yael Vinker, Google Research and Tel Aviv University;
(4) Moab Arar, Google Research and Tel Aviv University;
(5) Shlomi Fruchter, Google Research;
(6) Ohad Fried, Reichman University;
(7) Daniel Cohen-Or, Google Research and Tel Aviv University;
(8) Dani Lischinski, Google Research and The Hebrew University of Jerusalem. Table of Links Abstract and Introduction
Related Work
Method
Experiments
Limitations and Conclusions
A. Additional Experiments
B. Implementation Details
C. Societal Impact
References 2. Related Work Text-to-image generation. Text conditioned image generative models (T2I) [63, 69, 95] show unprecedented capabilities of generating high quality images from mere natural language text descriptions. They are quickly becoming a fundamental tool for any creative vision task. In particular, text-to-image diffusion models [9, 30, 52, 77–79] are employed for guided image synthesis [8, 15, 18, 22, 27, 51, 87, 97] and image editing tasks [5, 7, 10, 13, 28, 38, 47, 49, 55, 75, 84]. Using image editing methods, one can edit an image of a given character, and change its pose, etc., however, these methods cannot ensure consistency of the character in novel contexts, as our problem dictates. In addition, diffusion models were used in other tasks [56, 96], such as: video editing [23, 44, 45, 50, 59, 92], 3D synthesis [19, 31, 48, 58], editing [11, 25, 74, 98] and texturing [67], typography generation [35], motion generation [60, 81], and solving inverse problems [32]. Text-to-image personalization. Text-conditioned models cannot generate an image of a specific object or character. To overcome this limitation, a line of works utilizes several images of the same instance to encapsulate new priors in the generative model. Existing solutions range from optimization of text tokens [20, 85, 88] to fine-tuning the parameters of the entire model [6, 70], where in the middle, recent works suggest fine-tuning a small subset of parameters [1, 17, 26, 33, 41, 71, 82]. Models trained in this manner can generate consistent images of the same subject. However, they typically require a collection of images depicting the subject, which naturally narrows their ability to generate any imaginary character. Moreover, when training on a single input image [6], these methods tend to overfit and produce similar images with minimal diversity during inference. Unlike previous works, our method does not require an input image; instead, it can generate consistent and diverse images of the same character based only on a text description. Additional works are aimed to bypass the personalization training by introducing a dedicated personalization encoder [3, 16, 21, 37, 42, 76, 90, 93]. Given an image and a prompt, these works can produce images with a character similar to the input. However, as shown in Section 4.1, they lack consistency when generating multiple images from the same input. Concurrently, ConceptLab [66] is able to generate new members of a broad category (e.g., a new pet); in contrast, we seek a consistent instance of a character described by the input text prompt. Story visualization. Consistent character generation is well studied in the field of story visualization. Early GAN works [43, 80] employ a story discriminator for the imagetext alignment. Recent works, such as StoryDALL-E [46] and Make-A-Story [62] utilize pre-trained T2I models for the image generation, while an adapter model is trained to embed story captions and previous images into the T2I model. However, those methods cannot generalize to novel characters, as they are trained over specific datasets. More closely related, Jeong et al. [36] generate consistent storybooks by combining textual inversion with a face-swapping mechanism; therefore, their work relies on images of existing human-like characters. TaleCrafter [24] presents a comprehensive pipeline for storybook visualization. However, their consistent character module is based on an existing personalization method that requires fine-tuning on several images of the same character. Manual methods. Other attempts for achieving consistent character generation using a generative model rely on ad hoc and manually-intensive tricks such as using text tokens of a celebrity, or a combination of celebrities [64] in order to create a consistent human; however, the generated characters resemble the original celebrities, and this approach does not generalize to other character types (e.g., animals). Users have also proposed to ensure consistency by manually crafting very long and elaborate text prompts [65], or by using image variations [63] and filtering them manually by similarity [65]. Other users suggested generating a full design sheet of a character, then manually filter the best results and use them for further generation [94]. All these methods are manual, labor-intensive, and ad hoc for specific domains (e.g., humans). In contrast, our method is fully automated and domain-agnostic. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Omri Avrahami, Google Research and The Hebrew University of Jerusalem; (2) Amir Hertz, Google Research; (3) Yael Vinker, Google Research and Tel Aviv University; (4) Moab Arar, Google Research and Tel Aviv University; (5) Shlomi Fruchter, Google Research; (6) Ohad Fried, Reichman University; (7) Daniel Cohen-Or, Google Research and Tel Aviv University; (8) Dani Lischinski, Google Research and The Hebrew University of Jerusalem. Authors: Authors: (1) Omri Avrahami, Google Research and The Hebrew University of Jerusalem; (2) Amir Hertz, Google Research; (3) Yael Vinker, Google Research and Tel Aviv University; (4) Moab Arar, Google Research and Tel Aviv University; (5) Shlomi Fruchter, Google Research; (6) Ohad Fried, Reichman University; (7) Daniel Cohen-Or, Google Research and Tel Aviv University; (8) Dani Lischinski, Google Research and The Hebrew University of Jerusalem. Table of Links Abstract and Introduction Related Work Method Experiments Limitations and Conclusions A. Additional Experiments B. Implementation Details C. Societal Impact References Abstract and Introduction Abstract and Introduction Related Work Related Work Method Method Experiments Experiments Limitations and Conclusions Limitations and Conclusions A. Additional Experiments A. Additional Experiments B. Implementation Details B. Implementation Details C. Societal Impact C. Societal Impact References References 2. Related Work Text-to-image generation. Text conditioned image generative models (T2I) [63, 69, 95] show unprecedented capabilities of generating high quality images from mere natural language text descriptions. They are quickly becoming a fundamental tool for any creative vision task. In particular, text-to-image diffusion models [9, 30, 52, 77–79] are employed for guided image synthesis [8, 15, 18, 22, 27, 51, 87, 97] and image editing tasks [5, 7, 10, 13, 28, 38, 47, 49, 55, 75, 84]. Using image editing methods, one can edit an image of a given character, and change its pose, etc., however, these methods cannot ensure consistency of the character in novel contexts, as our problem dictates. Text-to-image generation. In addition, diffusion models were used in other tasks [56, 96], such as: video editing [23, 44, 45, 50, 59, 92], 3D synthesis [19, 31, 48, 58], editing [11, 25, 74, 98] and texturing [67], typography generation [35], motion generation [60, 81], and solving inverse problems [32]. Text-to-image personalization. Text-conditioned models cannot generate an image of a specific object or character. To overcome this limitation, a line of works utilizes several images of the same instance to encapsulate new priors in the generative model. Existing solutions range from optimization of text tokens [20, 85, 88] to fine-tuning the parameters of the entire model [6, 70], where in the middle, recent works suggest fine-tuning a small subset of parameters [1, 17, 26, 33, 41, 71, 82]. Models trained in this manner can generate consistent images of the same subject. However, they typically require a collection of images depicting the subject, which naturally narrows their ability to generate any imaginary character. Moreover, when training on a single input image [6], these methods tend to overfit and produce similar images with minimal diversity during inference. Text-to-image personalization. Unlike previous works, our method does not require an input image; instead, it can generate consistent and diverse images of the same character based only on a text description. Additional works are aimed to bypass the personalization training by introducing a dedicated personalization encoder [3, 16, 21, 37, 42, 76, 90, 93]. Given an image and a prompt, these works can produce images with a character similar to the input. However, as shown in Section 4.1, they lack consistency when generating multiple images from the same input. Concurrently, ConceptLab [66] is able to generate new members of a broad category (e.g., a new pet); in contrast, we seek a consistent instance of a character described by the input text prompt. Story visualization. Consistent character generation is well studied in the field of story visualization. Early GAN works [43, 80] employ a story discriminator for the imagetext alignment. Recent works, such as StoryDALL-E [46] and Make-A-Story [62] utilize pre-trained T2I models for the image generation, while an adapter model is trained to embed story captions and previous images into the T2I model. However, those methods cannot generalize to novel characters, as they are trained over specific datasets. More closely related, Jeong et al. [36] generate consistent storybooks by combining textual inversion with a face-swapping mechanism; therefore, their work relies on images of existing human-like characters. TaleCrafter [24] presents a comprehensive pipeline for storybook visualization. However, their consistent character module is based on an existing personalization method that requires fine-tuning on several images of the same character. Story visualization. Manual methods. Other attempts for achieving consistent character generation using a generative model rely on Manual methods. ad hoc and manually-intensive tricks such as using text tokens of a celebrity, or a combination of celebrities [64] in order to create a consistent human; however, the generated characters resemble the original celebrities, and this approach does not generalize to other character types (e.g., animals). Users have also proposed to ensure consistency by manually crafting very long and elaborate text prompts [65], or by using image variations [63] and filtering them manually by similarity [65]. Other users suggested generating a full design sheet of a character, then manually filter the best results and use them for further generation [94]. All these methods are manual, labor-intensive, and ad hoc for specific domains (e.g., humans). In contrast, our method is fully automated and domain-agnostic. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

The Chosen One: Consistent Characters in Text-to-Image Diffusion Models: Related Work

Authors:
(1) Omri Avrahami, Google Research and The Hebrew University of Jerusalem;
(2) Amir Hertz, Google Research;
(3) Yael Vinker, Google Research and Tel Aviv University;
(4) Moab Arar, Google Research and Tel Aviv University;
(5) Shlomi Fruchter, Google Research;
(6) Ohad Fried, Reichman University;
(7) Daniel Cohen-Or, Google Research and Tel Aviv University;
(8) Dani Lischinski, Google Research and The Hebrew University of Jerusalem. Table of Links Abstract and Introduction
Related Work
Method
Experiments
Limitations and Conclusions
A. Additional Experiments
B. Implementation Details
C. Societal Impact
References Abstract Recent advances in text-to-image generation models have unlocked vast potential for visual creativity. However, these models struggle with generation of consistent characters, a crucial aspect for numerous real-world applications such as story visualization, game development asset design, advertising, and more. Current methods typically rely on multiple pre-existing images of the target character or involve labor-intensive manual processes. In this work, we propose a fully automated solution for consistent character generation, with the sole input being a text prompt. We introduce an iterative procedure that, at each stage, identifies a coherent set of images sharing a similar identity and extracts a more consistent identity from this set. Our quantitative analysis demonstrates that our method strikes a better balance between prompt alignment and identity consistency compared to the baseline methods, and these findings are reinforced by a user study. To conclude, we showcase several practical applications of our approach. 1. Introduction The ability to maintain consistency of generated visual content across various contexts, as shown in Figure 1, plays a central role in numerous creative endeavors. These include illustrating a book, crafting a brand, creating comics, developing presentations, designing webpages, and more. Such consistency serves as the foundation for establishing brand identity, facilitating storytelling, enhancing communication, and nurturing emotional engagement. Despite the increasingly impressive abilities of text-to-image generative models, these models struggle with such consistent generation, a shortcoming that we aim to rectify in this work. Specifically, we introduce the task of consistent character generation, where given an input text prompt describing a character, we derive a representation that enables generating consistent depictions of the same character in novel contexts. Although we refer to characters throughout this paper, our work is in fact applicable to visual subjects in general. Consider, for example, an illustrator working on a Plasticine cat character. As demonstrated in Figure 2, providing a state-of-the-art text-to-image model with a prompt describing the character, results in a variety of outcomes, which may lack consistency (top row). In contrast, in this work we show how to distill a consistent representation of the cat (2nd row), which can then be used to depict the same character in a multitude of different contexts. The widespread popularity of text-to-image generative models [57, 63, 69, 72], combined with the need for consistent character generation, has already spawned a variety of ad hoc solutions. These include, for example, using celebrity names in prompts [64] for creating consistent humans, or using image variations [63] and filtering them manually by similarity [65]. In contrast to these ad hoc, manually intensive solutions, we propose a fully-automatic principled approach to consistent character generation. The academic works most closely related to our setting are ones dealing with personalization [20, 70] and story generation [24, 36, 62]. Some of these methods derive a representation for a given character from several user-provided images [20, 24, 70]. Others cannot generalize to novel characters that are not in the training data [62], or rely on textual inversion of an existing depiction of a human face [36]. In this work, we argue that in many applications the goal is to generate some consistent character, rather than visually matching a specific appearance. Thus, we address a new setting, where we aim to automatically distill a consistent representation of a character that is only required to comply with a single natural language description. Our method does not require any images of the target character as input; thus, it enables creating a novel consistent character that does not necessarily resemble any existing visual depiction. Our fully-automated solution to the task of consistent character generation is based on the assumption that a sufficiently large set of generated images, for a certain prompt, will contain groups of images with shared characteristics. Given such a cluster, one can extract a representation that captures the “common ground” among its images. Repeating the process with this representation, we can increase the consistency among the generated images, while still remaining faithful to the original input prompt. We start by generating a gallery of images based on the provided text prompt, and embed them in a Euclidean space using a pre-trained feature extractor. Next, we cluster these embeddings, and choose the most cohesive cluster to serve as the input for a personalization method that attempts to extract a consistent identity. We then use the resulting model to generate the next gallery of images, which should exhibit more consistency, while still depicting the input prompt. This process is repeated iteratively until convergence. We evaluate our method quantitatively and qualitatively against several baselines, as well as conducting a user study. Finally, we present several applications of our method. In summary, our contributions are: (1) we formalize the task of consistent character generation, (2) propose a novel solution to this task, and (3) we evaluate our method quantitatively and qualitatively, in addition to a user study, to demonstrate its effectiveness. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Omri, Yael, Moab, Daniel and Dani performed this work while working at Google. Authors: (1) Omri Avrahami, Google Research and The Hebrew University of Jerusalem; (2) Amir Hertz, Google Research; (3) Yael Vinker, Google Research and Tel Aviv University; (4) Moab Arar, Google Research and Tel Aviv University; (5) Shlomi Fruchter, Google Research; (6) Ohad Fried, Reichman University; (7) Daniel Cohen-Or, Google Research and Tel Aviv University; (8) Dani Lischinski, Google Research and The Hebrew University of Jerusalem. Authors: Authors: (1) Omri Avrahami, Google Research and The Hebrew University of Jerusalem; (2) Amir Hertz, Google Research; (3) Yael Vinker, Google Research and Tel Aviv University; (4) Moab Arar, Google Research and Tel Aviv University; (5) Shlomi Fruchter, Google Research; (6) Ohad Fried, Reichman University; (7) Daniel Cohen-Or, Google Research and Tel Aviv University; (8) Dani Lischinski, Google Research and The Hebrew University of Jerusalem. Table of Links Abstract and Introduction Related Work Method Experiments Limitations and Conclusions A. Additional Experiments B. Implementation Details C. Societal Impact References Abstract and Introduction Abstract and Introduction Related Work Related Work Method Method Experiments Experiments Limitations and Conclusions Limitations and Conclusions A. Additional Experiments A. Additional Experiments B. Implementation Details B. Implementation Details C. Societal Impact C. Societal Impact References References Abstract Recent advances in text-to-image generation models have unlocked vast potential for visual creativity. However, these models struggle with generation of consistent characters, a crucial aspect for numerous real-world applications such as story visualization, game development asset design, advertising, and more. Current methods typically rely on multiple pre-existing images of the target character or involve labor-intensive manual processes. In this work, we propose a fully automated solution for consistent character generation, with the sole input being a text prompt. We introduce an iterative procedure that, at each stage, identifies a coherent set of images sharing a similar identity and extracts a more consistent identity from this set. Our quantitative analysis demonstrates that our method strikes a better balance between prompt alignment and identity consistency compared to the baseline methods, and these findings are reinforced by a user study. To conclude, we showcase several practical applications of our approach. Recent advances in text-to-image generation models have unlocked vast potential for visual creativity. However, these models struggle with generation of consistent characters, a crucial aspect for numerous real-world applications such as story visualization, game development asset design, advertising, and more. Current methods typically rely on multiple pre-existing images of the target character or involve labor-intensive manual processes. In this work, we propose a fully automated solution for consistent character generation, with the sole input being a text prompt. We introduce an iterative procedure that, at each stage, identifies a coherent set of images sharing a similar identity and extracts a more consistent identity from this set. Our quantitative analysis demonstrates that our method strikes a better balance between prompt alignment and identity consistency compared to the baseline methods, and these findings are reinforced by a user study. To conclude, we showcase several practical applications of our approach. 1. Introduction The ability to maintain consistency of generated visual content across various contexts, as shown in Figure 1, plays a central role in numerous creative endeavors. These include illustrating a book, crafting a brand, creating comics, developing presentations, designing webpages, and more. Such consistency serves as the foundation for establishing brand identity, facilitating storytelling, enhancing communication, and nurturing emotional engagement. Despite the increasingly impressive abilities of text-to-image generative models, these models struggle with such consistent generation, a shortcoming that we aim to rectify in this work. Specifically, we introduce the task of consistent character generation, where given an input text prompt describing a character, we derive a representation that enables generating consistent depictions of the same character in novel contexts. Although we refer to characters throughout this paper, our work is in fact applicable to visual subjects in general. Consider, for example, an illustrator working on a Plasticine cat character. As demonstrated in Figure 2, providing a state-of-the-art text-to-image model with a prompt describing the character, results in a variety of outcomes, which may lack consistency (top row). In contrast, in this work we show how to distill a consistent representation of the cat (2nd row), which can then be used to depict the same character in a multitude of different contexts. The widespread popularity of text-to-image generative models [57, 63, 69, 72], combined with the need for consistent character generation, has already spawned a variety of ad hoc solutions. These include, for example, using celebrity names in prompts [64] for creating consistent humans, or using image variations [63] and filtering them manually by similarity [65]. In contrast to these ad hoc, manually intensive solutions, we propose a fully-automatic principled approach to consistent character generation. The academic works most closely related to our setting are ones dealing with personalization [20, 70] and story generation [24, 36, 62]. Some of these methods derive a representation for a given character from several user-provided images [20, 24, 70]. Others cannot generalize to novel characters that are not in the training data [62], or rely on textual inversion of an existing depiction of a human face [36]. In this work, we argue that in many applications the goal is to generate some consistent character, rather than visually matching a specific appearance. Thus, we address a new setting, where we aim to automatically distill a consistent representation of a character that is only required to comply with a single natural language description. Our method does not require any images of the target character as input; thus, it enables creating a novel consistent character that does not necessarily resemble any existing visual depiction. Our fully-automated solution to the task of consistent character generation is based on the assumption that a sufficiently large set of generated images, for a certain prompt, will contain groups of images with shared characteristics. Given such a cluster, one can extract a representation that captures the “common ground” among its images. Repeating the process with this representation, we can increase the consistency among the generated images, while still remaining faithful to the original input prompt. We start by generating a gallery of images based on the provided text prompt, and embed them in a Euclidean space using a pre-trained feature extractor. Next, we cluster these embeddings, and choose the most cohesive cluster to serve as the input for a personalization method that attempts to extract a consistent identity. We then use the resulting model to generate the next gallery of images, which should exhibit more consistency, while still depicting the input prompt. This process is repeated iteratively until convergence. We evaluate our method quantitatively and qualitatively against several baselines, as well as conducting a user study. Finally, we present several applications of our method. In summary, our contributions are: (1) we formalize the task of consistent character generation, (2) propose a novel solution to this task, and (3) we evaluate our method quantitatively and qualitatively, in addition to a user study, to demonstrate its effectiveness. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv Omri, Yael, Moab, Daniel and Dani performed this work while working at Google.

The Chosen One: Consistent Characters in Text-to-Image Diffusion Models: Abstract and Introduction

Authors:
(1) Omri Avrahami, Google Research and The Hebrew University of Jerusalem;
(2) Amir Hertz, Google Research;
(3) Yael Vinker, Google Research and Tel Aviv University;
(4) Moab Arar, Google Research and Tel Aviv University;
(5) Shlomi Fruchter, Google Research;
(6) Ohad Fried, Reichman University;
(7) Daniel Cohen-Or, Google Research and Tel Aviv University;
(8) Dani Lischinski, Google Research and The Hebrew University of Jerusalem. Table of Links Abstract and Introduction
Related Work
Method
Experiments
Limitations and Conclusions
A. Additional Experiments
B. Implementation Details
C. Societal Impact
References C. Societal Impact We believe that the emergence of technology that facilitates the effortless creation of consistent characters holds exciting promise in a variety of creative and practical applications. It can empower storytellers and content creators to bring their narratives to life with vivid and unique characters, enhancing the immersive quality of their work. In addition, it may offer accessibility to those who may not possess traditional artistic skills, democratizing character design in the creative industry. Furthermore, it can reduce the cost of advertising, and open up new opportunities for small and underprivileged entrepreneurs, enabling them to reach a wider audience and compete in the market more effectively. On the other hand, as any other generative AI technology, it can be misused by creating false and misleading visual content for deceptive purposes. Creating fake characters or personas can be used for online scams, disinformation campaigns, etc., making it challenging to discern genuine information from fabricated content. Such technologies underscore the vital importance of developing generated content detection systems, making it a compelling research direction to address. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Omri Avrahami, Google Research and The Hebrew University of Jerusalem; (2) Amir Hertz, Google Research; (3) Yael Vinker, Google Research and Tel Aviv University; (4) Moab Arar, Google Research and Tel Aviv University; (5) Shlomi Fruchter, Google Research; (6) Ohad Fried, Reichman University; (7) Daniel Cohen-Or, Google Research and Tel Aviv University; (8) Dani Lischinski, Google Research and The Hebrew University of Jerusalem. Authors: Authors: (1) Omri Avrahami, Google Research and The Hebrew University of Jerusalem; (2) Amir Hertz, Google Research; (3) Yael Vinker, Google Research and Tel Aviv University; (4) Moab Arar, Google Research and Tel Aviv University; (5) Shlomi Fruchter, Google Research; (6) Ohad Fried, Reichman University; (7) Daniel Cohen-Or, Google Research and Tel Aviv University; (8) Dani Lischinski, Google Research and The Hebrew University of Jerusalem. Table of Links Abstract and Introduction Related Work Method Experiments Limitations and Conclusions A. Additional Experiments B. Implementation Details C. Societal Impact References Abstract and Introduction Abstract and Introduction Related Work Related Work Method Method Experiments Experiments Limitations and Conclusions Limitations and Conclusions A. Additional Experiments A. Additional Experiments B. Implementation Details B. Implementation Details C. Societal Impact C. Societal Impact References References C. Societal Impact We believe that the emergence of technology that facilitates the effortless creation of consistent characters holds exciting promise in a variety of creative and practical applications. It can empower storytellers and content creators to bring their narratives to life with vivid and unique characters, enhancing the immersive quality of their work. In addition, it may offer accessibility to those who may not possess traditional artistic skills, democratizing character design in the creative industry. Furthermore, it can reduce the cost of advertising, and open up new opportunities for small and underprivileged entrepreneurs, enabling them to reach a wider audience and compete in the market more effectively. On the other hand, as any other generative AI technology, it can be misused by creating false and misleading visual content for deceptive purposes. Creating fake characters or personas can be used for online scams, disinformation campaigns, etc., making it challenging to discern genuine information from fabricated content. Such technologies underscore the vital importance of developing generated content detection systems, making it a compelling research direction to address. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

The Chosen One: Consistent Characters in Text-to-Image Diffusion Models: Societal Impact

Authors:
(1) Omri Avrahami, Google Research and The Hebrew University of Jerusalem;
(2) Amir Hertz, Google Research;
(3) Yael Vinker, Google Research and Tel Aviv University;
(4) Moab Arar, Google Research and Tel Aviv University;
(5) Shlomi Fruchter, Google Research;
(6) Ohad Fried, Reichman University;
(7) Daniel Cohen-Or, Google Research and Tel Aviv University;
(8) Dani Lischinski, Google Research and The Hebrew University of Jerusalem. Table of Links Abstract and Introduction
Related Work
Method
Experiments
Limitations and Conclusions
A. Additional Experiments
B. Implementation Details
C. Societal Impact
References A. Additional Experiments Below, we provide additional experiments that were omitted from the main paper. In Appendix A.1 we provide additional comparisons and results of our method, and demonstrate its non-deterministic nature in Appendix A.2. In Appendix A.3 we compare our method against two na¨ıve baselines. Appendix A.4 presents the results of our method using different feature extractors. Lastly, in Appendix A.6 we provide results that reduce the concerns of dataset memorization by our method. A.1. Additional Comparisons and Results In Figure 9 we provide a qualitative comparison on the automatically generated prompts, and in Figure 10 we provide an additional qualitative comparison. Concurrently to our work, the DALL·E 3 model [12] was commercially released as part of the paid ChatGPT Plus [53] subscription, enabling generating images in a conversational setting. We tried, using a conversation, to create a consistent character of a Plasticine cat, as demonstrated in Figure 11. As can be seen, the generated characters share only some of the characteristics (e.g., big eyes) but not all of them (e.g., colors, textures and shapes). In Figure 12 we provide a qualitative comparison of the ablated cases. In addition, as demonstrated in Figure 13, our approach is applicable to consistent generation of a wide range of subjects, without the requirement for them to necessarily depict human characters or creatures. Figure 14 shows additional results of our method, demonstrating a variety of character styles. Lastly, in Figure 15 we demonstrate the ability of creating a fully consistent “life story” of a character using our method. A.2. Non-determinism of Our Method In Figures 16 and 17 we demonstrate the non-deterministic nature of our method. Using the same text prompt, we run our method multiple times with different initial seeds, thereby generating a different set of images for the identity clustering stage (Section 3.1). Consequently, the most cohesive cluster ccohesive is different in each run, yielding different consistent identities. This behavior of our method is aligned with the one-to-many nature of our task — a single text prompt may correspond to many identities. A.3. Naıve Baselines As explained in Section 4.1, we compared our method against a version of TI [20] and LoRA DB [71] that were trained on a single image (with a single identity). Instead, we could generate a small set of five images for the given prompt (that are not guaranteed to be of the same identity), and use this small dataset for TI and LoRA DB baselines, referred to as TI multi and LoRA DB multi, respectively. As can be seen in Figures 18 and 19, these baselines fail to achieve satisfactory identity consistency. A.4. Additional Feature Extractors Instead of using DINOv2 [54] features for the identity clustering stage (Section 3.1), we also experimented with two alternative feature extractors: DINOv1 [14] and CLIP [61] image encoder. We quantitatively evaluate our method with each of these feature extractors in terms of identity consistency and prompt similarity, as explained in Section 4.1. As can be seen in Figure 20, DINOv1 produces higher identity consistency, while sacrificing prompt similarity, whereas CLIP achieves higher prompt similarity at the expense of identity consistency. Qualitatively, as demonstrated in Figure 21, we found the DINOv1 extractor to perform similarly to DINOv2, whereas CLIP produces results with a slightly lower identity consistency. A.5. Additional Clustering Visualization In Figure 22 we provide a visualization of the clustering algorithm described in Section 3.1. As can be seen, given the input text prompt “a purple astronaut, digital art, smooth, sharp focus, vector art”, in the first iteration (top three rows), our algorithm divides the generated image set into three clusters: (1) focusing on the astronaut’s head, (2) an astronaut with no face, and (3) a full body astronaut. In the second iteration (bottom three rows), all the clusters share the same identity, that was extracted in the first iteration, as described in Section 3.2, and our algorithm divides them into clusters by their pose. A.6. Dataset Non-Memorization Our method is able to produce consistent characters, which raises the question of whether these characters already exist in the training data of the generative model. We employed SDXL [57] as our text-to-image model, whose training dataset is, unfortunately, undisclosed in the paper [57]. Consequently, we relied on the most likely overlapping dataset, LAION-5B [73], which was also utilized by Stable Diffusion V2. To probe for dataset memorization, we found the top 5 nearest neighbors in the dataset in terms of CLIP [61] image similarity, for a few representative characters from our paper, using an open-source solution [68]. As demonstrated in Figure 23, our method does not simply memorize images from the LAION-5B dataset. A.7. Stable Diffusion 2 Results We experimented with a version of our method that uses the Stable Diffusion 2 [69] model. The implementation is the same as explained in Appendix B.1, with the following changes: (1) The set of custom text embeddings τ in the character representation Θ (as explained in Section 2 in the main paper ), contains only one text embedding. (2) We used a higher learning rate of 5e-4. The rest of the implementation details are the same. More specifically, we used Stable Diffusion v2.1 implementation from Diffusers [86] library. As can be seen in Figure 24, when using the Stable Diffusion 2 backbone, our method can extract a consistent character, however, as expected, the results are of a lower quality than when using the SDXL [57] backbone that we use in the rest of this paper. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Omri Avrahami, Google Research and The Hebrew University of Jerusalem; (2) Amir Hertz, Google Research; (3) Yael Vinker, Google Research and Tel Aviv University; (4) Moab Arar, Google Research and Tel Aviv University; (5) Shlomi Fruchter, Google Research; (6) Ohad Fried, Reichman University; (7) Daniel Cohen-Or, Google Research and Tel Aviv University; (8) Dani Lischinski, Google Research and The Hebrew University of Jerusalem. Authors: Authors: (1) Omri Avrahami, Google Research and The Hebrew University of Jerusalem; (2) Amir Hertz, Google Research; (3) Yael Vinker, Google Research and Tel Aviv University; (4) Moab Arar, Google Research and Tel Aviv University; (5) Shlomi Fruchter, Google Research; (6) Ohad Fried, Reichman University; (7) Daniel Cohen-Or, Google Research and Tel Aviv University; (8) Dani Lischinski, Google Research and The Hebrew University of Jerusalem. Table of Links Abstract and Introduction Related Work Method Experiments Limitations and Conclusions A. Additional Experiments B. Implementation Details C. Societal Impact References Abstract and Introduction Abstract and Introduction Related Work Related Work Method Method Experiments Experiments Limitations and Conclusions Limitations and Conclusions A. Additional Experiments A. Additional Experiments B. Implementation Details B. Implementation Details C. Societal Impact C. Societal Impact References References A. Additional Experiments Below, we provide additional experiments that were omitted from the main paper. In Appendix A.1 we provide additional comparisons and results of our method, and demonstrate its non-deterministic nature in Appendix A.2. In Appendix A.3 we compare our method against two na¨ıve baselines. Appendix A.4 presents the results of our method using different feature extractors. Lastly, in Appendix A.6 we provide results that reduce the concerns of dataset memorization by our method. A.1. Additional Comparisons and Results In Figure 9 we provide a qualitative comparison on the automatically generated prompts, and in Figure 10 we provide an additional qualitative comparison. Concurrently to our work, the DALL·E 3 model [12] was commercially released as part of the paid ChatGPT Plus [53] subscription, enabling generating images in a conversational setting. We tried, using a conversation, to create a consistent character of a Plasticine cat, as demonstrated in Figure 11. As can be seen, the generated characters share only some of the characteristics (e.g., big eyes) but not all of them (e.g., colors, textures and shapes). In Figure 12 we provide a qualitative comparison of the ablated cases. In addition, as demonstrated in Figure 13, our approach is applicable to consistent generation of a wide range of subjects, without the requirement for them to necessarily depict human characters or creatures. Figure 14 shows additional results of our method, demonstrating a variety of character styles. Lastly, in Figure 15 we demonstrate the ability of creating a fully consistent “life story” of a character using our method. A.2. Non-determinism of Our Method In Figures 16 and 17 we demonstrate the non-deterministic nature of our method. Using the same text prompt, we run our method multiple times with different initial seeds, thereby generating a different set of images for the identity clustering stage (Section 3.1). Consequently, the most cohesive cluster c cohesive is different in each run, yielding different consistent identities. This behavior of our method is aligned with the one-to-many nature of our task — a single text prompt may correspond to many identities. cohesive A.3. Naıve Baselines As explained in Section 4.1, we compared our method against a version of TI [20] and LoRA DB [71] that were trained on a single image (with a single identity). Instead, we could generate a small set of five images for the given prompt (that are not guaranteed to be of the same identity), and use this small dataset for TI and LoRA DB baselines, referred to as TI multi and LoRA DB multi, respectively. As can be seen in Figures 18 and 19, these baselines fail to achieve satisfactory identity consistency. A.4. Additional Feature Extractors Instead of using DINOv2 [54] features for the identity clustering stage (Section 3.1), we also experimented with two alternative feature extractors: DINOv1 [14] and CLIP [61] image encoder. We quantitatively evaluate our method with each of these feature extractors in terms of identity consistency and prompt similarity, as explained in Section 4.1. As can be seen in Figure 20, DINOv1 produces higher identity consistency, while sacrificing prompt similarity, whereas CLIP achieves higher prompt similarity at the expense of identity consistency. Qualitatively, as demonstrated in Figure 21, we found the DINOv1 extractor to perform similarly to DINOv2, whereas CLIP produces results with a slightly lower identity consistency. A.5. Additional Clustering Visualization In Figure 22 we provide a visualization of the clustering algorithm described in Section 3.1. As can be seen, given the input text prompt “a purple astronaut, digital art, smooth, sharp focus, vector art”, in the first iteration (top three rows), our algorithm divides the generated image set into three clusters: (1) focusing on the astronaut’s head, (2) an astronaut with no face, and (3) a full body astronaut. In the second iteration (bottom three rows), all the clusters share the same identity, that was extracted in the first iteration, as described in Section 3.2, and our algorithm divides them into clusters by their pose. A.6. Dataset Non-Memorization Our method is able to produce consistent characters, which raises the question of whether these characters already exist in the training data of the generative model. We employed SDXL [57] as our text-to-image model, whose training dataset is, unfortunately, undisclosed in the paper [57]. Consequently, we relied on the most likely overlapping dataset, LAION-5B [73], which was also utilized by Stable Diffusion V2. To probe for dataset memorization, we found the top 5 nearest neighbors in the dataset in terms of CLIP [61] image similarity, for a few representative characters from our paper, using an open-source solution [68]. As demonstrated in Figure 23, our method does not simply memorize images from the LAION-5B dataset. A.7. Stable Diffusion 2 Results We experimented with a version of our method that uses the Stable Diffusion 2 [69] model. The implementation is the same as explained in Appendix B.1, with the following changes: (1) The set of custom text embeddings τ in the character representation Θ (as explained in Section 2 in the main paper ), contains only one text embedding. (2) We used a higher learning rate of 5e-4. The rest of the implementation details are the same. More specifically, we used Stable Diffusion v2.1 implementation from Diffusers [86] library. As can be seen in Figure 24, when using the Stable Diffusion 2 backbone, our method can extract a consistent character, however, as expected, the results are of a lower quality than when using the SDXL [57] backbone that we use in the rest of this paper. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

The Chosen One: Consistent Characters in Text-to-Image Diffusion Models: Additional Experiments

Authors:
(1) Omri Avrahami, Google Research and The Hebrew University of Jerusalem;
(2) Amir Hertz, Google Research;
(3) Yael Vinker, Google Research and Tel Aviv University;
(4) Moab Arar, Google Research and Tel Aviv University;
(5) Shlomi Fruchter, Google Research;
(6) Ohad Fried, Reichman University;
(7) Daniel Cohen-Or, Google Research and Tel Aviv University;
(8) Dani Lischinski, Google Research and The Hebrew University of Jerusalem. Table of Links Abstract and Introduction
Related Work
Method
Experiments
Limitations and Conclusions
A. Additional Experiments
B. Implementation Details
C. Societal Impact
References 3. Method As stated earlier, our goal in this work is to enable generation of consistent images of a character (or another kind of visual subject) based on a textual description. We achieve this by iteratively customizing a pre-trained text-to-image model, using sets of images generated by the model itself as training data. Intuitively, we refine the representation of the target character by repeatedly funneling the model’s output into a consistent identity. Once the process has converged, the resulting model can be used to generate consistent images of the target character in novel contexts. In this section, we describe our method in detail. Formally, we are given a text-to-image model MΘ, parameterized by Θ, and a text prompt p that describes a target character. The parameters Θ consist of a set of model weights θ and a set of custom text embeddings τ . We seek a representation Θ(p), s.t., the parameterized model MΘ(p) is able to generate consistent images of the character described by p in novel contexts. Our approach, described in Algorithm 1 and depicted in Figure 3, is based on the premise that a sufficiently large set of images generated by M for the same text prompt, but with different seeds, will reflect the non-uniform density of the manifold of generated images. Specifically, we expect to find some groups of images with shared characteristics. The “common ground” among the images in one of these groups can be used to refine the representation Θ(p) so as to better capture and fit the target. We therefore propose to iteratively cluster the generated images, and use the most cohesive cluster to refine Θ(p). This process is repeated, with the refined representation Θ(p), until convergence. Below, we describe the clustering and the representation refinement components of our method in detail. 3.1. Identity Clustering In Figure 4 we show a visualization of the DINOv2 embedding space, where the high-dimensional embeddings S are projected into 2D using t-SNE [29] and colored according to their K-MEANS++ [4] clusters. Some of the embeddings are clustered together more tightly than others, and the black cluster is chosen as the most cohesive one. 3.2. Identity Extraction We base our solution on a pre-trained Stable Diffusion XL (SDXL) [57] model, which utilizes two text encoders: CLIP [61] and OpenCLIP [34]. We perform textual inversion [20] to add a new pair of textual tokens τ , one for each of the two text encoders. However, we found that this parameter space is not expressive enough, as demonstrated in Section 4.3, hence we also update the model weights θ via a low-rank adaptation (LoRA) [33, 71] of the self- and cross-attention layers of the model. We use the standard denoising loss: 3.3. Convergence As explained earlier (Algorithm 1 and Figure 3), the above process is performed iteratively. Note that the representation Θ extracted in each iteration is the one used to generate the set of N images for the next iteration. The generated images are thus funneled into a consistent identity. Finally, it should be noticed that our method is nondeterministic, i.e., when running our method multiple times, on the same input prompt p, different consistent characters will be generated. This is aligned with the one-to-many nature of our task. For more details and examples, please refer to the supplementary material. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Omri Avrahami, Google Research and The Hebrew University of Jerusalem; (2) Amir Hertz, Google Research; (3) Yael Vinker, Google Research and Tel Aviv University; (4) Moab Arar, Google Research and Tel Aviv University; (5) Shlomi Fruchter, Google Research; (6) Ohad Fried, Reichman University; (7) Daniel Cohen-Or, Google Research and Tel Aviv University; (8) Dani Lischinski, Google Research and The Hebrew University of Jerusalem. Authors: Authors: (1) Omri Avrahami, Google Research and The Hebrew University of Jerusalem; (2) Amir Hertz, Google Research; (3) Yael Vinker, Google Research and Tel Aviv University; (4) Moab Arar, Google Research and Tel Aviv University; (5) Shlomi Fruchter, Google Research; (6) Ohad Fried, Reichman University; (7) Daniel Cohen-Or, Google Research and Tel Aviv University; (8) Dani Lischinski, Google Research and The Hebrew University of Jerusalem. Table of Links Abstract and Introduction Related Work Method Experiments Limitations and Conclusions A. Additional Experiments B. Implementation Details C. Societal Impact References Abstract and Introduction Abstract and Introduction Related Work Related Work Method Method Experiments Experiments Limitations and Conclusions Limitations and Conclusions A. Additional Experiments A. Additional Experiments B. Implementation Details B. Implementation Details C. Societal Impact C. Societal Impact References References 3. Method As stated earlier, our goal in this work is to enable generation of consistent images of a character (or another kind of visual subject) based on a textual description. We achieve this by iteratively customizing a pre-trained text-to-image model, using sets of images generated by the model itself as training data. Intuitively, we refine the representation of the target character by repeatedly funneling the model’s output into a consistent identity. Once the process has converged, the resulting model can be used to generate consistent images of the target character in novel contexts. In this section, we describe our method in detail. Formally, we are given a text-to-image model MΘ, parameterized by Θ, and a text prompt p that describes a target character. The parameters Θ consist of a set of model weights θ and a set of custom text embeddings τ . We seek a representation Θ(p), s.t., the parameterized model MΘ(p) is able to generate consistent images of the character described by p in novel contexts. Our approach, described in Algorithm 1 and depicted in Figure 3, is based on the premise that a sufficiently large set of images generated by M for the same text prompt, but with different seeds, will reflect the non-uniform density of the manifold of generated images. Specifically, we expect to find some groups of images with shared characteristics. The “common ground” among the images in one of these groups can be used to refine the representation Θ(p) so as to better capture and fit the target. We therefore propose to iteratively cluster the generated images, and use the most cohesive cluster to refine Θ(p). This process is repeated, with the refined representation Θ(p), until convergence. Below, we describe the clustering and the representation refinement components of our method in detail. 3.1. Identity Clustering In Figure 4 we show a visualization of the DINOv2 embedding space, where the high-dimensional embeddings S are projected into 2D using t-SNE [29] and colored according to their K-MEANS++ [4] clusters. Some of the embeddings are clustered together more tightly than others, and the black cluster is chosen as the most cohesive one. 3.2. Identity Extraction We base our solution on a pre-trained Stable Diffusion XL (SDXL) [57] model, which utilizes two text encoders: CLIP [61] and OpenCLIP [34]. We perform textual inversion [20] to add a new pair of textual tokens τ , one for each of the two text encoders. However, we found that this parameter space is not expressive enough, as demonstrated in Section 4.3, hence we also update the model weights θ via a low-rank adaptation (LoRA) [33, 71] of the self- and cross-attention layers of the model. We use the standard denoising loss: 3.3. Convergence As explained earlier (Algorithm 1 and Figure 3), the above process is performed iteratively. Note that the representation Θ extracted in each iteration is the one used to generate the set of N images for the next iteration. The generated images are thus funneled into a consistent identity. Finally, it should be noticed that our method is nondeterministic, i.e., when running our method multiple times, on the same input prompt p, different consistent characters will be generated. This is aligned with the one-to-many nature of our task. For more details and examples, please refer to the supplementary material. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

The Chosen One: Consistent Characters in Text-to-Image Diffusion Models: Method

Authors:
(1) Omri Avrahami, Google Research and The Hebrew University of Jerusalem;
(2) Amir Hertz, Google Research;
(3) Yael Vinker, Google Research and Tel Aviv University;
(4) Moab Arar, Google Research and Tel Aviv University;
(5) Shlomi Fruchter, Google Research;
(6) Ohad Fried, Reichman University;
(7) Daniel Cohen-Or, Google Research and Tel Aviv University;
(8) Dani Lischinski, Google Research and The Hebrew University of Jerusalem. Table of Links Abstract and Introduction
Related Work
Method
Experiments
Limitations and Conclusions
A. Additional Experiments
B. Implementation Details
C. Societal Impact
References 5. Limitations and Conclusions We found our method to suffer from the following limitations: (a) Inconsistent identity — in some cases, our method is not able to converge to a fully consistent identity (without overfitting). As demonstrated in Figure 8(a), when trying to generate a portrait of a robot, our method generated robots with slightly different colors and shapes (e.g., different arms). This may result from a prompt that is too general, for which identity clustering (Section 3.1) is not able to find a sufficiently cohesive set. (b) Inconsistent supporting characters/elements— although our method is able to find a consistent identity for the character described by the input prompt, the identities of other characters, related to the input character (e.g., their pet), might be inconsistent. For example, in Figure 8(b) the input prompt p to our method described only the girl, and when asked to generate the girl with her cat, different cats were generated. In addition, our framework does not support finding multiple concepts concurrently [6]. (c) Spurious attributes — we found that in some cases, our method binds additional attributes, which are not part of the input text prompt, with the final identity of the character. For example, in Figure 8(c), the input text prompt was “a sticker of a ginger cat”, however, our method added green leaves to the generated sticker, even though it was not asked to do so. This stems from the stochastic nature of the text-to-image model — the model added these leaves in some of the stickers generated during the identity clustering stage (Section 3.1), and the stickers containing the leaves happened to form the most cohesive set ccohesive. (d) Significant computational cost — each iteration of our method involves generating a large number of images, and learning the identity of the most cohesive cluster. It takes about 20 minutes to converge into a consistent identity. Reducing the computational costs is an appealing direction for further research. In conclusion, in this paper we offered the first fully-automated solution to the problem of consistent character generation. We hope that our work will pave the way for future advancements, as we believe this technology of consistent character generation may have a disruptive effect on numerous sectors, including education, storytelling, entertainment, fashion, brand design, advertising, and more. Acknowledgments. We thank Yael Pitch, Matan Cohen, Neal Wadhwa and Yaron Brodsky for their valuable help and feedback. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Omri Avrahami, Google Research and The Hebrew University of Jerusalem; (2) Amir Hertz, Google Research; (3) Yael Vinker, Google Research and Tel Aviv University; (4) Moab Arar, Google Research and Tel Aviv University; (5) Shlomi Fruchter, Google Research; (6) Ohad Fried, Reichman University; (7) Daniel Cohen-Or, Google Research and Tel Aviv University; (8) Dani Lischinski, Google Research and The Hebrew University of Jerusalem. Authors: Authors: (1) Omri Avrahami, Google Research and The Hebrew University of Jerusalem; (2) Amir Hertz, Google Research; (3) Yael Vinker, Google Research and Tel Aviv University; (4) Moab Arar, Google Research and Tel Aviv University; (5) Shlomi Fruchter, Google Research; (6) Ohad Fried, Reichman University; (7) Daniel Cohen-Or, Google Research and Tel Aviv University; (8) Dani Lischinski, Google Research and The Hebrew University of Jerusalem. Table of Links Abstract and Introduction Related Work Method Experiments Limitations and Conclusions A. Additional Experiments B. Implementation Details C. Societal Impact References Abstract and Introduction Abstract and Introduction Related Work Related Work Method Method Experiments Experiments Limitations and Conclusions Limitations and Conclusions A. Additional Experiments A. Additional Experiments B. Implementation Details B. Implementation Details C. Societal Impact C. Societal Impact References References 5. Limitations and Conclusions We found our method to suffer from the following limitations: (a) Inconsistent identity — in some cases, our method is not able to converge to a fully consistent identity (without overfitting). As demonstrated in Figure 8(a), when trying to generate a portrait of a robot, our method generated robots with slightly different colors and shapes (e.g., different arms). This may result from a prompt that is too general, for which identity clustering (Section 3.1) is not able to find a sufficiently cohesive set. (b) Inconsistent supporting characters/elements— although our method is able to find a consistent identity for the character described by the input prompt, the identities of other characters, related to the input character (e.g., their pet), might be inconsistent. For example, in Figure 8(b) the input prompt p to our method described only the girl, and when asked to generate the girl with her cat, different cats were generated. In addition, our framework does not support finding multiple concepts concurrently [6]. (c) Spurious attributes — we found that in some cases, our method binds additional attributes, which are not part of the input text prompt, with the final identity of the character. For example, in Figure 8(c), the input text prompt was “a sticker of a ginger cat”, however, our method added green leaves to the generated sticker, even though it was not asked to do so. This stems from the stochastic nature of the text-to-image model — the model added these leaves in some of the stickers generated during the identity clustering stage (Section 3.1), and the stickers containing the leaves happened to form the most cohesive set c cohesive . (d) Significant computational cost — each iteration of our method involves generating a large number of images, and learning the identity of the most cohesive cluster. It takes about 20 minutes to converge into a consistent identity. Reducing the computational costs is an appealing direction for further research. cohesive In conclusion, in this paper we offered the first fully-automated solution to the problem of consistent character generation. We hope that our work will pave the way for future advancements, as we believe this technology of consistent character generation may have a disruptive effect on numerous sectors, including education, storytelling, entertainment, fashion, brand design, advertising, and more. Acknowledgments . We thank Yael Pitch, Matan Cohen, Neal Wadhwa and Yaron Brodsky for their valuable help and feedback. Acknowledgments This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

The Chosen One: Consistent Characters in Text-to-Image Diffusion Models: Limitations and Conclusions

The Chosen One: Consistent Characters in Text-to-Image Diffusion Models: References

Authors:
(1) Omri Avrahami, Google Research and The Hebrew University of Jerusalem;
(2) Amir Hertz, Google Research;
(3) Yael Vinker, Google Research and Tel Aviv University;
(4) Moab Arar, Google Research and Tel Aviv University;
(5) Shlomi Fruchter, Google Research;
(6) Ohad Fried, Reichman University;
(7) Daniel Cohen-Or, Google Research and Tel Aviv University;
(8) Dani Lischinski, Google Research and The Hebrew University of Jerusalem. Table of Links Abstract and Introduction
Related Work
Method
Experiments
Limitations and Conclusions
A. Additional Experiments
B. Implementation Details
C. Societal Impact
References 4. Experiments In Section 4.1 we compare our method against several baselines, both qualitatively and quantitatively. Next, in Section 4.2 we describe the user study we conducted and present its results. The results of an ablation study are reported in Section 4.3. Finally, in Section 4.4 we demonstrate several applications of our method. 4.1. Qualitative and Quantitative Comparison We compared our method against the most related personalization techniques [20, 42, 71, 89, 93]. In each experiment, each of these techniques is used to extract a character from a single image, generated by SDXL [57] from an input prompt p. The same prompt p is also provided as input to our method. Textual Inversion (TI) [20] optimizes a textual token using several images of the same concept, and we converted it to support SDXL by learning two text tokens, one for each of its text encoders, as we did in our method. In addition, we used LoRA DreamBooth [71] (LoRA DB), which we found less prone to overfitting than standard DB. Furthermore, we compared against all available image encoder techniques that encode a single image into the textual space of the diffusion model for later generation in novel contexts: BLIP-Diffusion [42], ELITE [89], and IP-adapter [93]. For all the baselines, we used the same prompt p to generate a single image, and used it to extract the identity via optimization (TI and LoRA DB) or encoding (ELITE, BLIP-diffusion and IP-adapter). In Figure 5 we qualitatively compare our method against the above baselines. While TI [20], BLIP-diffusion [42] and IP-adapter [93] are able to follow the specified prompt, they fail to produce a consistent character. LoRA DB [71] succeeds in consistent generation, but it does not always respond to the prompt. Furthermore, the resulting character is generated in the same fixed pose. ELITE [90] struggles with prompt following and the generated characters tend to be deformed. In comparison, our method is able to follow the prompt and maintain consistency, while generating appealing characters in different poses and viewing angles. In order to automatically evaluate our method and the baselines quantitatively, we instructed ChatGPT [53] to generate prompts for characters of different types (e.g., animals, creatures, objects, etc.) in different styles (e.g., stickers, animations, photorealistic images, etc.). Each of these prompts was then used to extract a consistent character by our method and by each of the baselines. Next, we generated these characters in a predefined collection of novel contexts. For a visual comparison, please refer to the supplementary material. We employ two standard evaluation metrics: prompt similarity and identity consistency, which are commonly used in the personalization literature [6, 20, 70]. Prompt similarity measures the correspondence between the generated images and the input text prompt. We use the standard CLIP [61] similarity, i.e., the normalized cosine similarity between the CLIP image embedding of the generated images and the CLIP text embedding of the source prompts. For measuring identity consistency, we calculate the similarity between the CLIP image embeddings of generated images of the same concept across different contexts. As can be seen in Figure 6 (left), there is an inherent trade-off between prompt similarity and identity consistency: LoRA DB and ELITE exhibit high identity consistency, while sacrificing prompt similarity. TI and BLIPdiffusion achieve high prompt similarity but low identity consistency. Our method and IP-adapter both lie on the Pareto front. However, our method achieves better identity consistency than IP-adapter, which is significant from the user’s perspective, as supported by our user study. 4.2. User Study We conducted a user study to evaluate our method, using the Amazon Mechanical Turk (AMT) platform [2]. We used the same generated prompts and samples that were used in Section 4.1 and asked the evaluators to rate the prompt similarity and identity consistency of each result on a Likert scale of 1–5. For ranking the prompt similarity, the evaluators were presented with the target text prompt and the result of our method and the baselines on the same page, and were asked to rate each of the images. For identity consistency, for each of the generated concepts, we compared our method and the baselines by randomly choosing pairs of generated images with different target prompts, and the evaluators were asked to rate on a scale of 1–5 whether the images contain the same main character. Again, all the pairs of the same character for the different baselines were shown on the same page. As can be seen in Figure 6 (right), our method again exhibits a good balance between identity consistency and prompt similarity, with a wider gap separating it from the baselines. For more details and statistical significance analbaselines. For more details and statistical significance analysis, read the supplementary material. 4.3. Ablation Study We conducted an ablation study for the following cases: (1) Without clustering — we omit the clustering step described in Section 3.1, and instead simply generate 5 images according to the input prompt. (2) Without LoRA — we reduce the optimizable representation Θ in the identity extraction stage, as described in Section 3.2, to consist of only the newly-added text tokens without the additional LoRA weights. (3) With re-initialization — instead of using the latest representation Θ in each of the optimization iterations, as described in Section 3.3, we re-initialize it in each iteration. (4) Single iteration — rather than iterating until convergence (Section 3.3), we stop after a single iteration. As can be seen in Figure 6 (left), all of the above key components are crucial for achieving a consistent identity in the final result: (1) removing the clustering harms the identity extraction stage because the training set is too diverse, (2) reducing the representation causes underfitting, as the model does not have enough parameters to properly capture the identity, (3) re-initializing the representation in each iteration, or (4) performing a single iteration, does not allow the model to converge into a single identity. For a visual comparison of the ablation study, as well as comparison of alternative feature extractors (DINOv1 [14] and CLIP [61]), please refer to the supplementary material. 4.4. Applications As demonstrated in Figure 7, our method can be used for various down-stream tasks, such as (a) Illustrating a story by breaking it into a different scenes and using the same consistent character for all of them. (b) Local text-driven image editing by integrating Blended Latent Diffusion [5, 7] — a consistent character can be injected into a specified location of a provided background image, in a novel pose specified by a text prompt. (c) Generating a consistent character with an additional pose control using ControlNet [97]. For more details, please refer to the supplementary material. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Omri Avrahami, Google Research and The Hebrew University of Jerusalem; (2) Amir Hertz, Google Research; (3) Yael Vinker, Google Research and Tel Aviv University; (4) Moab Arar, Google Research and Tel Aviv University; (5) Shlomi Fruchter, Google Research; (6) Ohad Fried, Reichman University; (7) Daniel Cohen-Or, Google Research and Tel Aviv University; (8) Dani Lischinski, Google Research and The Hebrew University of Jerusalem. Authors: Authors: (1) Omri Avrahami, Google Research and The Hebrew University of Jerusalem; (2) Amir Hertz, Google Research; (3) Yael Vinker, Google Research and Tel Aviv University; (4) Moab Arar, Google Research and Tel Aviv University; (5) Shlomi Fruchter, Google Research; (6) Ohad Fried, Reichman University; (7) Daniel Cohen-Or, Google Research and Tel Aviv University; (8) Dani Lischinski, Google Research and The Hebrew University of Jerusalem. Table of Links Abstract and Introduction Related Work Method Experiments Limitations and Conclusions A. Additional Experiments B. Implementation Details C. Societal Impact References Abstract and Introduction Abstract and Introduction Related Work Related Work Method Method Experiments Experiments Limitations and Conclusions Limitations and Conclusions A. Additional Experiments A. Additional Experiments B. Implementation Details B. Implementation Details C. Societal Impact C. Societal Impact References References 4. Experiments In Section 4.1 we compare our method against several baselines, both qualitatively and quantitatively. Next, in Section 4.2 we describe the user study we conducted and present its results. The results of an ablation study are reported in Section 4.3. Finally, in Section 4.4 we demonstrate several applications of our method. 4.1. Qualitative and Quantitative Comparison We compared our method against the most related personalization techniques [20, 42, 71, 89, 93]. In each experiment, each of these techniques is used to extract a character from a single image, generated by SDXL [57] from an input prompt p. The same prompt p is also provided as input to our method. Textual Inversion (TI) [20] optimizes a textual token using several images of the same concept, and we converted it to support SDXL by learning two text tokens, one for each of its text encoders, as we did in our method. In addition, we used LoRA DreamBooth [71] (LoRA DB), which we found less prone to overfitting than standard DB. Furthermore, we compared against all available image encoder techniques that encode a single image into the textual space of the diffusion model for later generation in novel contexts: BLIP-Diffusion [42], ELITE [89], and IP-adapter [93]. For all the baselines, we used the same prompt p to generate a single image, and used it to extract the identity via optimization (TI and LoRA DB) or encoding (ELITE, BLIP-diffusion and IP-adapter). In Figure 5 we qualitatively compare our method against the above baselines. While TI [20], BLIP-diffusion [42] and IP-adapter [93] are able to follow the specified prompt, they fail to produce a consistent character. LoRA DB [71] succeeds in consistent generation, but it does not always respond to the prompt. Furthermore, the resulting character is generated in the same fixed pose. ELITE [90] struggles with prompt following and the generated characters tend to be deformed. In comparison, our method is able to follow the prompt and maintain consistency, while generating appealing characters in different poses and viewing angles. In order to automatically evaluate our method and the baselines quantitatively, we instructed ChatGPT [53] to generate prompts for characters of different types (e.g., animals, creatures, objects, etc.) in different styles (e.g., stickers, animations, photorealistic images, etc.). Each of these prompts was then used to extract a consistent character by our method and by each of the baselines. Next, we generated these characters in a predefined collection of novel contexts. For a visual comparison, please refer to the supplementary material. We employ two standard evaluation metrics: prompt similarity and identity consistency, which are commonly used in the personalization literature [6, 20, 70]. Prompt similarity measures the correspondence between the generated images and the input text prompt. We use the standard CLIP [61] similarity, i.e., the normalized cosine similarity between the CLIP image embedding of the generated images and the CLIP text embedding of the source prompts. For measuring identity consistency, we calculate the similarity between the CLIP image embeddings of generated images of the same concept across different contexts. As can be seen in Figure 6 (left), there is an inherent trade-off between prompt similarity and identity consistency: LoRA DB and ELITE exhibit high identity consistency, while sacrificing prompt similarity. TI and BLIPdiffusion achieve high prompt similarity but low identity consistency. Our method and IP-adapter both lie on the Pareto front. However, our method achieves better identity consistency than IP-adapter, which is significant from the user’s perspective, as supported by our user study. 4.2. User Study We conducted a user study to evaluate our method, using the Amazon Mechanical Turk (AMT) platform [2]. We used the same generated prompts and samples that were used in Section 4.1 and asked the evaluators to rate the prompt similarity and identity consistency of each result on a Likert scale of 1–5. For ranking the prompt similarity, the evaluators were presented with the target text prompt and the result of our method and the baselines on the same page, and were asked to rate each of the images. For identity consistency, for each of the generated concepts, we compared our method and the baselines by randomly choosing pairs of generated images with different target prompts, and the evaluators were asked to rate on a scale of 1–5 whether the images contain the same main character. Again, all the pairs of the same character for the different baselines were shown on the same page. As can be seen in Figure 6 (right), our method again exhibits a good balance between identity consistency and prompt similarity, with a wider gap separating it from the baselines. For more details and statistical significance analbaselines. For more details and statistical significance analysis, read the supplementary material. 4.3. Ablation Study We conducted an ablation study for the following cases: (1) Without clustering — we omit the clustering step described in Section 3.1, and instead simply generate 5 images according to the input prompt. (2) Without LoRA — we reduce the optimizable representation Θ in the identity extraction stage, as described in Section 3.2, to consist of only the newly-added text tokens without the additional LoRA weights. (3) With re-initialization — instead of using the latest representation Θ in each of the optimization iterations, as described in Section 3.3, we re-initialize it in each iteration. (4) Single iteration — rather than iterating until convergence (Section 3.3), we stop after a single iteration. As can be seen in Figure 6 (left), all of the above key components are crucial for achieving a consistent identity in the final result: (1) removing the clustering harms the identity extraction stage because the training set is too diverse, (2) reducing the representation causes underfitting, as the model does not have enough parameters to properly capture the identity, (3) re-initializing the representation in each iteration, or (4) performing a single iteration, does not allow the model to converge into a single identity. For a visual comparison of the ablation study, as well as comparison of alternative feature extractors (DINOv1 [14] and CLIP [61]), please refer to the supplementary material. 4.4. Applications As demonstrated in Figure 7, our method can be used for various down-stream tasks, such as (a) Illustrating a story by breaking it into a different scenes and using the same consistent character for all of them. (b) Local text-driven image editing by integrating Blended Latent Diffusion [5, 7] — a consistent character can be injected into a specified location of a provided background image, in a novel pose specified by a text prompt. (c) Generating a consistent character with an additional pose control using ControlNet [97]. For more details, please refer to the supplementary material. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

The Chosen One: Consistent Characters in Text-to-Image Diffusion Models: Experiments

Authors:
(1) Omri Avrahami, Google Research and The Hebrew University of Jerusalem;
(2) Amir Hertz, Google Research;
(3) Yael Vinker, Google Research and Tel Aviv University;
(4) Moab Arar, Google Research and Tel Aviv University;
(5) Shlomi Fruchter, Google Research;
(6) Ohad Fried, Reichman University;
(7) Daniel Cohen-Or, Google Research and Tel Aviv University;
(8) Dani Lischinski, Google Research and The Hebrew University of Jerusalem. Table of Links Abstract and Introduction
Related Work
Method
Experiments
Limitations and Conclusions
A. Additional Experiments
B. Implementation Details
C. Societal Impact
References B. Implementation Details In this section, we provide the implementation details that were omitted from the main paper. In Appendix B.1 we provide the implementation details of our method and the baselines. Then, in Appendix B.2 we provide the implementation details of the automatic metrics that we used to evaluate our method against the baselines. In Appendix B.3 we provide the implementation details and the statistical analysis for the user study we conducted. Lastly, in Appendix B.4 we provide the implementation details for the applications we presented. B.1. Method Implementation Details We based our method, and all the baselines (except ELITE [90] and BLIP-diffusion [42]) on Stable Diffusion XL (SDXL) [57], which is the state-of-the-art open source text-to-image model, at the writing of this paper. We used the official ELITE implementation, that uses Stable Diffusion V1.4, and the official implementation of BLIPdiffusion, that uses Stable Diffusion V1.5. We could not replace these two baselines to SDXL backbone, as the encoders were trained on these specific models. As for the rest of the baselines, we used the same SDXL architecture and weights. List of the third-party packages that we used: • Official SDXL [57] implementation by HuggingFace Diffusers [86] at https : / / github . com / huggingface/diffusers • Official SDXL LoRA DB implementation by HuggingFace Diffusers [86] at https : / / github . com / huggingface/diffusers. • Official ELITE [90] implementation at https : / / github.com/csyxwei/ELITE • Official BLIP-diffusion [42] implementation at https: / / github . com / salesforce / LAVIS / tree / main/projects/blip-diffusion • Official IP-adapter [93] implementation at https:// github.com/tencent-ailab/IP-Adapter • DINOv2 [54] ViT-g/14, DINOv1 [14] ViT-B/16 and CLIP [61] ViT-L/14 implementation by HuggingFace Transformers [91] at https : / / github . com / huggingface/transformers B.2. Automatic Metrics Implementation Details In order to automatically evaluate our method and the baselines quantitatively, we instructed ChatGPT [53] to generate prompts for characters of different types (e.g., animals, creatures, objects, etc.) in different styles (e.g., stickers, animations, photorealistic images, etc.). These prompts were then used to generate a set of consistent characters by our method and by each of the baselines. Next, these prompts were used to generate these characters in a predefined collection of novel contexts from the following list: • “a photo of [v] at the beach” • “a photo of [v] in the jungle” • “a photo of [v] in the snow” • “a photo of [v] in the street” • “a photo of [v] with a city in the background” • “a photo of [v] with a mountain in the background” • “a photo of [v] with the Eiffel Tower in the background” • “a photo of [v] near the Statue of Liberty” • “a photo of [v] near the Sydney Opera House” • “a photo of [v] floating on top of water” • “a photo of [v] eating a burger” • “a photo of [v] drinking a beer” • “a photo of [v] wearing a blue hat” • “a photo of [v] wearing sunglasses” • “a photo of [v] playing with a ball” • “a photo of [v] as a police officer” where [v] is the newly-added token that represents the consistent character. B.3. User Study Details As explained in Section 4.2, we conducted a user study to evaluate our method, using the Amazon Mechanical Turk (AMT) platform [2]. We used the same generated prompts and samples that were used in Section 4.1, and asked the evaluators to rate the prompt similarity and identity consistency of each result on a Likert scale of 1–5. For ranking the prompt similarity, the evaluators were instructed the following: “For each of the following images, please rank on a scale of 1 to 5 its correspondence to this text description: {PROMPT}. The character in the image can be anything (e.g., a person, an animal, a toy etc.” where {PROMPT} is the target text prompt (in which we replaced the special token with the word “character”). All the baselines, as well as our method, were presented in the same page, and the evaluators were asked to rate each one of the results using a slider from 1 (“Do not match at all”) to 5 (“Match perfectly”). Next, to assess identity consistency, we took for each one of the characters two generated images that correspond to different target text prompts, put them next to each other, and instructed the evaluators the following: “For each of the following image pairs, please rank on a scale of 1 to 5 if they contain the same character (1 means that they contain totally different characters and 5 means that they contain exactly the same character). The images can have different backgrounds”. We put all the compared images on the same page, and the evaluators were asked to rate each one of the pairs using a slider from 1 (“Totally different characters”) to 5 (“Exactly the same character”). We collected three ratings per question, resulting in 1104 ratings per task (prompt similarity and identity consistency). The time allotted per task was one hour, to allow the raters to properly evaluate the results without time pressure. The means and variances of the user study responses are reported in Table 1. In addition, we conducted a statistical analysis of our user study by validating that the difference between all the conditions is statistically significant using Kruskal-Wallis [40] test (p < 1e−28 for the text similarity test and p < 1e−76 for the identity consistency text). Lastly, we used Tukey’s honestly significant difference procedure [83] to show that the comparison of our method against all the baselines is statistically significant, as detailed in Table 2. B.4. Applications Implementation Details In Section 4.4, we presented three downstream applications of our method. Story illustration. Given a long story, e.g., “This is a story about Jasper, a cute mink with a brown jacket and red pants. Jasper started his day by jogging on the beach, and afterwards, he enjoyed a coffee meetup with a friend in the heart of New York City. As the day drew to a close, he settled into his cozy apartment to review a paper”, one can create a consistent character from the main character description (“a cute mink with a brown jacket and red pants”), then they can generate the various scenes by simply rephrasing the sentence: “[v] jogging on the beach”


“[v] drinking coffee with his friend in the heart of New York City”


“[v] reviewing a paper in his cozy apartment” Local image editing. Our method can be simply integrated with Blended Latent Diffusion [5, 7] for editing images locally: given a text prompt, we start by running our method to extract a consistent identity, then, given an input image and mask, we can plant the character in the image within the mask boundaries. In addition, we can provide a local text description for the character. Additional pose control. Our method can be integrated with ControlNet [97]: given a text prompt, we first apply our method to extract a consistent identity Θ = (θ, τ ), where θ are the LoRA weights and τ is a set of custom text embeddings. Then, we can take an off-the-shelf pre-trained ControlNet model, plug-in our representation Θ, and use it to generate the consistent character in different poses given by the user. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. Authors: (1) Omri Avrahami, Google Research and The Hebrew University of Jerusalem; (2) Amir Hertz, Google Research; (3) Yael Vinker, Google Research and Tel Aviv University; (4) Moab Arar, Google Research and Tel Aviv University; (5) Shlomi Fruchter, Google Research; (6) Ohad Fried, Reichman University; (7) Daniel Cohen-Or, Google Research and Tel Aviv University; (8) Dani Lischinski, Google Research and The Hebrew University of Jerusalem. Authors: Authors: (1) Omri Avrahami, Google Research and The Hebrew University of Jerusalem; (2) Amir Hertz, Google Research; (3) Yael Vinker, Google Research and Tel Aviv University; (4) Moab Arar, Google Research and Tel Aviv University; (5) Shlomi Fruchter, Google Research; (6) Ohad Fried, Reichman University; (7) Daniel Cohen-Or, Google Research and Tel Aviv University; (8) Dani Lischinski, Google Research and The Hebrew University of Jerusalem. Table of Links Abstract and Introduction Related Work Method Experiments Limitations and Conclusions A. Additional Experiments B. Implementation Details C. Societal Impact References Abstract and Introduction Abstract and Introduction Related Work Related Work Method Method Experiments Experiments Limitations and Conclusions Limitations and Conclusions A. Additional Experiments A. Additional Experiments B. Implementation Details B. Implementation Details C. Societal Impact C. Societal Impact References References B. Implementation Details In this section, we provide the implementation details that were omitted from the main paper. In Appendix B.1 we provide the implementation details of our method and the baselines. Then, in Appendix B.2 we provide the implementation details of the automatic metrics that we used to evaluate our method against the baselines. In Appendix B.3 we provide the implementation details and the statistical analysis for the user study we conducted. Lastly, in Appendix B.4 we provide the implementation details for the applications we presented. B.1. Method Implementation Details We based our method, and all the baselines (except ELITE [90] and BLIP-diffusion [42]) on Stable Diffusion XL (SDXL) [57], which is the state-of-the-art open source text-to-image model, at the writing of this paper. We used the official ELITE implementation, that uses Stable Diffusion V1.4, and the official implementation of BLIPdiffusion, that uses Stable Diffusion V1.5. We could not replace these two baselines to SDXL backbone, as the encoders were trained on these specific models. As for the rest of the baselines, we used the same SDXL architecture and weights. List of the third-party packages that we used: • Official SDXL [57] implementation by HuggingFace Diffusers [86] at https : / / github . com / huggingface/diffusers • Official SDXL LoRA DB implementation by HuggingFace Diffusers [86] at https : / / github . com / huggingface/diffusers. • Official ELITE [90] implementation at https : / / github.com/csyxwei/ELITE • Official BLIP-diffusion [42] implementation at https: / / github . com / salesforce / LAVIS / tree / main/projects/blip-diffusion • Official IP-adapter [93] implementation at https:// github.com/tencent-ailab/IP-Adapter • DINOv2 [54] ViT-g/14, DINOv1 [14] ViT-B/16 and CLIP [61] ViT-L/14 implementation by HuggingFace Transformers [91] at https : / / github . com / huggingface/transformers B.2. Automatic Metrics Implementation Details In order to automatically evaluate our method and the baselines quantitatively, we instructed ChatGPT [53] to generate prompts for characters of different types (e.g., animals, creatures, objects, etc.) in different styles (e.g., stickers, animations, photorealistic images, etc.). These prompts were then used to generate a set of consistent characters by our method and by each of the baselines. Next, these prompts were used to generate these characters in a predefined collection of novel contexts from the following list: • “a photo of [v] at the beach” • “a photo of [v] in the jungle” • “a photo of [v] in the snow” • “a photo of [v] in the street” • “a photo of [v] with a city in the background” • “a photo of [v] with a mountain in the background” • “a photo of [v] with the Eiffel Tower in the background” • “a photo of [v] near the Statue of Liberty” • “a photo of [v] near the Sydney Opera House” • “a photo of [v] floating on top of water” • “a photo of [v] eating a burger” • “a photo of [v] drinking a beer” • “a photo of [v] wearing a blue hat” • “a photo of [v] wearing sunglasses” • “a photo of [v] playing with a ball” • “a photo of [v] as a police officer” where [v] is the newly-added token that represents the consistent character. B.3. User Study Details As explained in Section 4.2, we conducted a user study to evaluate our method, using the Amazon Mechanical Turk (AMT) platform [2]. We used the same generated prompts and samples that were used in Section 4.1, and asked the evaluators to rate the prompt similarity and identity consistency of each result on a Likert scale of 1–5. For ranking the prompt similarity, the evaluators were instructed the following: “For each of the following images, please rank on a scale of 1 to 5 its correspondence to this text description: {PROMPT}. The character in the image can be anything (e.g., a person, an animal, a toy etc.” where {PROMPT} is the target text prompt (in which we replaced the special token with the word “character”). All the baselines, as well as our method, were presented in the same page, and the evaluators were asked to rate each one of the results using a slider from 1 (“Do not match at all”) to 5 (“Match perfectly”). Next, to assess identity consistency, we took for each one of the characters two generated images that correspond to different target text prompts, put them next to each other, and instructed the evaluators the following: “For each of the following image pairs, please rank on a scale of 1 to 5 if they contain the same character (1 means that they contain totally different characters and 5 means that they contain exactly the same character). The images can have different backgrounds”. We put all the compared images on the same page, and the evaluators were asked to rate each one of the pairs using a slider from 1 (“Totally different characters”) to 5 (“Exactly the same character”). We collected three ratings per question, resulting in 1104 ratings per task (prompt similarity and identity consistency). The time allotted per task was one hour, to allow the raters to properly evaluate the results without time pressure. The means and variances of the user study responses are reported in Table 1. In addition, we conducted a statistical analysis of our user study by validating that the difference between all the conditions is statistically significant using Kruskal-Wallis [40] test (p < 1e−28 for the text similarity test and p < 1e−76 for the identity consistency text). Lastly, we used Tukey’s honestly significant difference procedure [83] to show that the comparison of our method against all the baselines is statistically significant, as detailed in Table 2. B.4. Applications Implementation Details In Section 4.4, we presented three downstream applications of our method. Story illustration. Given a long story, e.g., “This is a story about Jasper, a cute mink with a brown jacket and red pants. Jasper started his day by jogging on the beach, and afterwards, he enjoyed a coffee meetup with a friend in the heart of New York City. As the day drew to a close, he settled into his cozy apartment to review a paper” , one can create a consistent character from the main character description (“a cute mink with a brown jacket and red pants”) , then they can generate the various scenes by simply rephrasing the sentence: Story illustration. “This is a story about Jasper, a cute mink with a brown jacket and red pants. Jasper started his day by jogging on the beach, and afterwards, he enjoyed a coffee meetup with a friend in the heart of New York City. As the day drew to a close, he settled into his cozy apartment to review a paper” (“a cute mink with a brown jacket and red pants”) “[v] jogging on the beach” “[v] drinking coffee with his friend in the heart of New York City” “[v] reviewing a paper in his cozy apartment” “[v] jogging on the beach” “[v] jogging on the beach” “[v] jogging on the beach” “[v] drinking coffee with his friend in the heart of New York City” “[v] drinking coffee with his friend in the heart of New York City” “[v] drinking coffee with his friend in the heart of New York City” “[v] reviewing a paper in his cozy apartment” “[v] reviewing a paper in his cozy apartment” “[v] reviewing a paper in his cozy apartment” Local image editing. Our method can be simply integrated with Blended Latent Diffusion [5, 7] for editing images locally: given a text prompt, we start by running our method to extract a consistent identity, then, given an input image and mask, we can plant the character in the image within the mask boundaries. In addition, we can provide a local text description for the character. Local image editing. Additional pose control. Our method can be integrated with ControlNet [97]: given a text prompt, we first apply our method to extract a consistent identity Θ = (θ, τ ), where θ are the LoRA weights and τ is a set of custom text embeddings. Then, we can take an off-the-shelf pre-trained ControlNet model, plug-in our representation Θ, and use it to generate the consistent character in different poses given by the user. Our method can be integrated with ControlNet [97]: given a text prompt, This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. This paper is available on arxiv under CC BY-NC-ND 4.0 DEED license. available on arxiv

The Chosen One: Consistent Characters in Text-to-Image Diffusion Models: Implementation Details

Desktop customization is a fun activity that many of you have likely tried at least once. There are various customization options, such as changing the boot screen logo, the start menu icon, or transforming the entire Windows interface to resemble macOS. Today, we are about to discuss one such customization, which is setting a different background wallpaper for each desktop (or monitor) in Windows 11 and 10 for aesthetics, productivity, creativity, or any other reason. Setting Different Backgrounds for Each Desktop on Windows 11, 10 In Windows, there is no limit on the number of desktops you can create from Task View (though more desktops will consume more system resources), allowing you to create as many desktops as you want and set different wallpapers for each. To customize which wallpaper appears on each desktop, 1. Press Win+Tab to open Task View. All your existing desktops will be listed below, with a "+" option to create more. 2. Right-click on the respective desktop (in my case, "Desktop 2") whose background wallpaper you want to change, and then select "Choose background" from the context menu. 3. It will redirect you to the "Personalize your background" section for that desktop, where you can choose a wallpaper from the recent images or select a new one from the file explorer. 4. Once the new background wallpaper for the desktop is set, you can confirm the changes by visiting the desktop. 5. To verify that each desktop has its own separate background wallpaper, you can take another peek into the Task View using Win+Tab shortcut. That's it! You've successfully set a different background wallpaper for each desktop on Windows 11 and 10. Bonus Tip! You can directly change the desktop background wallpaper from your current desktop by navigating to "Settings" -> "Personalization" -> "Background" and then right-clicking on any image to open a context menu with the "Set for desktop" option, allowing you to change the background wallpaper of your preferred desktop. Wrap Up That’s how you can change the background wallpaper for each desktop on Windows 11 and 10. You can further change all of your existing desktops background wallpaper by following this method. Desktop customization is a fun activity that many of you have likely tried at least once. There are various customization options, such as changing the boot screen logo, the start menu icon, or transforming the entire Windows interface to resemble macOS. Today, we are about to discuss one such customization, which is setting a different background wallpaper for each desktop (or monitor) in Windows 11 and 10 for aesthetics, productivity, creativity, or any other reason. Setting Different Backgrounds for Each Desktop on Windows 11, 10 In Windows, there is no limit on the number of desktops you can create from Task View (though more desktops will consume more system resources), allowing you to create as many desktops as you want and set different wallpapers for each. To customize which wallpaper appears on each desktop, 1. Press Win+Tab to open Task View. Win+Tab All your existing desktops will be listed below, with a "+" option to create more. All your existing desktops will be listed below, with a "+" option to create more. 2. Right-click on the respective desktop (in my case, " Desktop 2 ") whose background wallpaper you want to change, and then select " Choose background " from the context menu. Desktop 2 Choose background 3. It will redirect you to the " Personalize your background " section for that desktop, where you can choose a wallpaper from the recent images or select a new one from the file explorer. Personalize your background 4. Once the new background wallpaper for the desktop is set, you can confirm the changes by visiting the desktop. 5. To verify that each desktop has its own separate background wallpaper, you can take another peek into the Task View using Win+Tab shortcut. Win+Tab That's it! You've successfully set a different background wallpaper for each desktop on Windows 11 and 10. Bonus Tip! You can directly change the desktop background wallpaper from your current desktop by navigating to " Settings " -> " Personalization " -> " Background " and then right-clicking on any image to open a context menu with the " Set for desktop " option, allowing you to change the background wallpaper of your preferred desktop. Settings Personalization Background Set for desktop Wrap Up That’s how you can change the background wallpaper for each desktop on Windows 11 and 10. You can further change all of your existing desktops background wallpaper by following this method.

Setting Different Backgrounds for Each Desktop on Windows 11 and 10

Leveraging SwarmUI & Stable Diffusion 3 on Cloud Platforms: A Guide to Kaggle (No-Cost), Massed Comp

🔗 Tutorial Video Link :  ➡️ https://youtu.be/HKX8_F1Er_w https://youtu.be/HKX8_F1Er_w?embedable=true Do not skip any part of this tutorial to master how to use Stable Diffusion 3 (SD3) with the most advanced generative AI open-source APP SwarmUI. Automatic1111 SD Web UI or Fooocus are not supporting the SD3 yet. Therefore, I am starting to make tutorials for SwarmUI as well. StableSwarmUI is officially developed by StabilityAI, and your mind will be blown away after you watch this tutorial and learn its amazing features. StableSwarmUI uses ComfyUI as the back end, thus, it has all the good features of ComfyUI, and it brings you easy-to-use features of Automatic1111 StableDiffusion Web UI with them. I really liked SwarmUI and planning to do more tutorials for it. 🔗 The Public Post (no login or account required) Shown In The Video With The Links ➡️ https://www.patreon.com/posts/stableswarmui-3-106135985 0:00 Introduction to the Stable Diffusion 3 (SD3) and SwarmUI and what is in the tutorial 4:12 Architecture and features of SD3 5:05 What each different model files of Stable Diffusion 3 means 6:26 How to download and install SwarmUI on Windows for SD3 and all other Stable Diffusion models 8:42 What kind of folder path you should use when installing SwarmUI 10:28 If you get installation error how to notice and fix it 11:49 Installation has been completed and now how to start using SwarmUI 12:29 Which settings I change before start using SwarmUI and how to change your theme like dark, white, gray 12:56 How to make SwarmUI save generated images as PNG 13:08 How to find description of each settings and configuration 13:28 How to download SD3 model and start using on Windows 13:38 How to use model downloader utility of SwarmUI 14:17 How to set models folder paths and link your existing models folders in SwarmUI 14:35 Explanation of Root folder path in SwarmUI 14:52 VAE of SD3 do we need to download? 15:25 Generate and model section of the SwarmUI to generate images and how to select your base model 16:02 Setting up parameters and what they do to generate images 17:06 Which sampling method is best for SD3 17:22 Information about SD3 text encoders and their comparison 18:14 First time generating an image with SD3 19:36 How to regenerate same image 20:17 How to see image generation speed and step speed and more information 20:29 Stable Diffusion 3 it per second speed on RTX 3090 TI 20:39 How to see VRAM usage on Windows 10 22:08 And testing and comparing different text encoders for SD3 22:36 How to use FP16 version of T5 XXL text encoder instead of default FP8 version 25:27 The image generation speed when using best config for SD3 26:37 Why VAE of the SD3 is many times better than previous Stable Diffusion models, 4 vs 8 vs 16 vs 32 channels VAE 27:40 How to and where to download best AI upscaler models 29:10 How to use refiner and upscaler models to improve and upscale generated images 29:21 How to restart and start SwarmUI 32:01 The folders where the generated images are saved 32:13 Image history feature of SwarmUI 33:10 Upscaled image comparison 34:01 How to download all upscaler models at once 34:34 Presets feature in depth 36:55 How to generate forever / infinite times 37:13 Non-tiled upscale caused issues 38:36 How to compare tiled vs non-tiled upscale and decide best 39:05 275 SwarmUI presets (cloned from Fooocus) I prepared and the scripts I coded to prepare them and how to import those presets 42:10 Model browser feature 43:25 How to generate TensorRT engine for huge speed up 43:47 How to update SwarmUI 44:27 Prompt syntax and advanced features 45:35 How to use Wildcards (random prompts) feature 46:47 How to see full details / metadata of generated images 47:13 Full guide for extremely powerful grid image generation (like X/Y/Z plot) 47:35 How to put all downloaded upscalers from zip file 51:37 How to see what is happening at the server logs 53:04 How to continue grid generation process after interruption 54:32 How to open grid generation after it has been completed and how to use it 56:13 Example of tiled upscaling seaming problem 1:00:30 Full guide for image history 1:02:22 How to directly delete images and star them 1:03:20 How to use SD 1.5 and SDXL models and LoRAs 1:06:24 Which sampler method is best 1:06:43 How to use image to image 1:08:43 How to use edit image / inpainting 1:10:38 How to use amazing segmentation feature to automatically inpaint any part of images 1:15:55 How to use segmentation on existing images for inpainting and get perfect results with different seeds 1:18:19 More detailed information regarding upscaling and tiling and SD3 1:20:08 Seams perfect explanation and example and how to fix it 1:21:09 How to use queue system 1:21:23 How to use multiple GPUs with adding more backends 1:24:38 Loading model in low VRAM mode 1:25:10 How to fix colors over saturation 1:27:00 Best image generation configuration for SD3 1:27:44 How to apply upscale to your older generated images quickly via preset 1:28:39 Other amazing features of SwarmUI 1:28:49 Clip tokenization and rare token OHWX 🔗 Tutorial Video Link :  ➡️ https://youtu.be/HKX8_F1Er_w https://youtu.be/HKX8_F1Er_w https://youtu.be/HKX8_F1Er_w?embedable=true https://youtu.be/HKX8_F1Er_w?embedable=true Do not skip any part of this tutorial to master how to use Stable Diffusion 3 (SD3) with the most advanced generative AI open-source APP SwarmUI. Automatic1111 SD Web UI or Fooocus are not supporting the SD3 yet. Therefore, I am starting to make tutorials for SwarmUI as well. StableSwarmUI is officially developed by StabilityAI, and your mind will be blown away after you watch this tutorial and learn its amazing features. StableSwarmUI uses ComfyUI as the back end, thus, it has all the good features of ComfyUI, and it brings you easy-to-use features of Automatic1111 StableDiffusion Web UI with them. I really liked SwarmUI and planning to do more tutorials for it. 🔗 The Public Post (no login or account required) Shown In The Video With The Links ➡️ https://www.patreon.com/posts/stableswarmui-3-106135985 https://www.patreon.com/posts/stableswarmui-3-106135985 0:00 Introduction to the Stable Diffusion 3 (SD3) and SwarmUI and what is in the tutorial 4:12 Architecture and features of SD3 5:05 What each different model files of Stable Diffusion 3 means 6:26 How to download and install SwarmUI on Windows for SD3 and all other Stable Diffusion models 8:42 What kind of folder path you should use when installing SwarmUI 10:28 If you get installation error how to notice and fix it 11:49 Installation has been completed and now how to start using SwarmUI 12:29 Which settings I change before start using SwarmUI and how to change your theme like dark, white, gray 12:56 How to make SwarmUI save generated images as PNG 13:08 How to find description of each settings and configuration 13:28 How to download SD3 model and start using on Windows 13:38 How to use model downloader utility of SwarmUI 14:17 How to set models folder paths and link your existing models folders in SwarmUI 14:35 Explanation of Root folder path in SwarmUI 14:52 VAE of SD3 do we need to download? 15:25 Generate and model section of the SwarmUI to generate images and how to select your base model 16:02 Setting up parameters and what they do to generate images 17:06 Which sampling method is best for SD3 17:22 Information about SD3 text encoders and their comparison 18:14 First time generating an image with SD3 19:36 How to regenerate same image 20:17 How to see image generation speed and step speed and more information 20:29 Stable Diffusion 3 it per second speed on RTX 3090 TI 20:39 How to see VRAM usage on Windows 10 22:08 And testing and comparing different text encoders for SD3 22:36 How to use FP16 version of T5 XXL text encoder instead of default FP8 version 25:27 The image generation speed when using best config for SD3 26:37 Why VAE of the SD3 is many times better than previous Stable Diffusion models, 4 vs 8 vs 16 vs 32 channels VAE 27:40 How to and where to download best AI upscaler models 29:10 How to use refiner and upscaler models to improve and upscale generated images 29:21 How to restart and start SwarmUI 32:01 The folders where the generated images are saved 32:13 Image history feature of SwarmUI 33:10 Upscaled image comparison 34:01 How to download all upscaler models at once 34:34 Presets feature in depth 36:55 How to generate forever / infinite times 37:13 Non-tiled upscale caused issues 38:36 How to compare tiled vs non-tiled upscale and decide best 39:05 275 SwarmUI presets (cloned from Fooocus) I prepared and the scripts I coded to prepare them and how to import those presets 42:10 Model browser feature 43:25 How to generate TensorRT engine for huge speed up 43:47 How to update SwarmUI 44:27 Prompt syntax and advanced features 45:35 How to use Wildcards (random prompts) feature 46:47 How to see full details / metadata of generated images 47:13 Full guide for extremely powerful grid image generation (like X/Y/Z plot) 47:35 How to put all downloaded upscalers from zip file 51:37 How to see what is happening at the server logs 53:04 How to continue grid generation process after interruption 54:32 How to open grid generation after it has been completed and how to use it 56:13 Example of tiled upscaling seaming problem 1:00:30 Full guide for image history 1:02:22 How to directly delete images and star them 1:03:20 How to use SD 1.5 and SDXL models and LoRAs 1:06:24 Which sampler method is best 1:06:43 How to use image to image 1:08:43 How to use edit image / inpainting 1:10:38 How to use amazing segmentation feature to automatically inpaint any part of images 1:15:55 How to use segmentation on existing images for inpainting and get perfect results with different seeds 1:18:19 More detailed information regarding upscaling and tiling and SD3 1:20:08 Seams perfect explanation and example and how to fix it 1:21:09 How to use queue system 1:21:23 How to use multiple GPUs with adding more backends 1:24:38 Loading model in low VRAM mode 1:25:10 How to fix colors over saturation 1:27:00 Best image generation configuration for SD3 1:27:44 How to apply upscale to your older generated images quickly via preset 1:28:39 Other amazing features of SwarmUI 1:28:49 Clip tokenization and rare token OHWX

A Stable Diffusion 3 Tutorial With Amazing SwarmUI SD Web UI That Utilizes ComfyUI: Zero to Hero

Flutter vs NativeScript — Key Differences and Why we Switched in the Middle of the Road

Bitcoin is here to stay! In this article, I’ll explain Bitcoin in five levels of difficulty i.e. five different audiences. Explaining Bitcoin to a Child Bitcoin is like coins you can use on the internet. Imagine having coins in a video game that you can exchange for actual goods. You can purchase candies, save them for later, or trade your Bitcoin like fantasy coins. Explaining Bitcoin to a Teen Bitcoin is a currency you can use to buy things online or trade with other people. It’s different from real bills or physical coins because no bank or government prints or issues it. The Bitcoin network uses blockchain technology to track transactions and ensure everything is visible and safe. It functions similarly to a secret code that only you and the friend you're sending it to understand. Explaining Bitcoin to a College Student Bitcoin is a decentralized digital currency that uses cryptography to secure transactions. It runs on a chain of blocks (the blockchain). The blockchain acts as a public ledger that keeps track of every transaction and is incredibly transparent. Bitcoin does not require a central authority, unlike real money issued by a bank or government. You can buy, sell, and trade Bitcoin on a lot of platforms online and spend it on actual physical shops or online, like a credit card. This lack of authority makes it resistant to interference, monitoring, or censorship. Explaining Bitcoin to a Graduate Student Bitcoin is a revolutionary technology that operates on a decentralized, peer-to-peer network. This public distributed ledger network secures transparent and immutable transactions without relying on intermediaries or a central authority. To guarantee the integrity and immutability of the transactions, the design incorporates encryption and a distributed ledger system. Its decentralized disposition presents financial sovereignty. This is very attractive as an alternative to traditional banking systems. The emergence property of the blockchain adds new interesting properties different from traditional asset management. Bitcoin’s value fluctuates based on market demand, and people often see it as both a digital asset and a potential store of value. Explaining Bitcoin to an Expert Bitcoin is a decentralized digital currency that uses proof-of-work, a novel consensus method, to protect its network without the need for intermediaries. The decentralized implementation of Bitcoin along with its restricted and predetermined supply have consequential implications for global government, banking, and economics. Bitcoin brings financial sovereignty, resistance to censorship, and a hedge against traditional financial risks. The ledger is based on a public, durable, immutable, and transparent blockchain that records all transactions. To add new blocks to the blockchain, miners must solve challenging cryptographic problems. The miners get a prize reward encouraging them to keep the lights up. The protocol protects the transaction security and avoids double-spending with the miner's challenge. Similar to real gold, Bitcoin is a unique digital asset due to its deflationary nature and limited supply of fixed 21 million units. The halving event reduces the reward for mining Bitcoin blocks by half approximately every four years. This event controls the supply of Bitcoin preventing inflation. With every halving, the supply decreases. The last Bitcoin will be issued around the year 2140, based on the current rate of block rewards and the scheduled halving events. Are you a Bitcoin enthusiast? Bitcoin is here to stay! Bitcoin In this article, I’ll explain Bitcoin in five levels of difficulty i.e. five different audiences. Explaining Bitcoin to a Child Bitcoin is like coins you can use on the internet. Imagine having coins in a video game that you can exchange for actual goods. You can purchase candies, save them for later, or trade your Bitcoin like fantasy coins. Explaining Bitcoin to a Teen Bitcoin is a currency you can use to buy things online or trade with other people. It’s different from real bills or physical coins because no bank or government prints or issues it. The Bitcoin network uses blockchain technology to track transactions and ensure everything is visible and safe. It functions similarly to a secret code that only you and the friend you're sending it to understand. Explaining Bitcoin to a College Student Bitcoin is a decentralized digital currency that uses cryptography to secure transactions. It runs on a chain of blocks (the blockchain). The blockchain acts as a public ledger that keeps track of every transaction and is incredibly transparent. Bitcoin does not require a central authority, unlike real money issued by a bank or government. You can buy, sell, and trade Bitcoin on a lot of platforms online and spend it on actual physical shops or online, like a credit card. This lack of authority makes it resistant to interference, monitoring, or censorship. Explaining Bitcoin to a Graduate Student Bitcoin is a revolutionary technology that operates on a decentralized, peer-to-peer network. This public distributed ledger network secures transparent and immutable transactions without relying on intermediaries or a central authority. To guarantee the integrity and immutability of the transactions, the design incorporates encryption and a distributed ledger system. Its decentralized disposition presents financial sovereignty. This is very attractive as an alternative to traditional banking systems. The emergence property of the blockchain adds new interesting properties different from traditional asset management. Bitcoin’s value fluctuates based on market demand, and people often see it as both a digital asset and a potential store of value. Explaining Bitcoin to an Expert Bitcoin is a decentralized digital currency that uses proof-of-work, a novel consensus method, to protect its network without the need for intermediaries. The decentralized implementation of Bitcoin along with its restricted and predetermined supply have consequential implications for global government, banking, and economics. Bitcoin brings financial sovereignty, resistance to censorship, and a hedge against traditional financial risks. The ledger is based on a public, durable, immutable, and transparent blockchain that records all transactions. To add new blocks to the blockchain, miners must solve challenging cryptographic problems. The miners get a prize reward encouraging them to keep the lights up. The protocol protects the transaction security and avoids double-spending with the miner's challenge. Similar to real gold, Bitcoin is a unique digital asset due to its deflationary nature and limited supply of fixed 21 million units. The halving event reduces the reward for mining Bitcoin blocks by half approximately every four years. This event controls the supply of Bitcoin preventing inflation. With every halving, the supply decreases. The last Bitcoin will be issued around the year 2140, based on the current rate of block rewards and the scheduled halving events. Are you a Bitcoin enthusiast?

Explaining Bitcoin in 5 Levels of Difficulty

Authors:
(1) J. Quetzalcóatl Toledo-Marín, University of British Columbia, BC Children’s Hospital Research Institute Vancouver BC, Canada (Email: j.toledo.mx@gmail.com);
(2) James A. Glazier, Biocomplexity Institute and Department of Intelligent Systems Engineering, Indiana University, Bloomington, IN 47408, USA (Email: jaglazier@gmail.com);
(3) Geoffrey Fox, University of Virginia, Computer Science and Biocomplexity Institute, 994 Research Park Blvd, Charlottesville, Virginia, 22911, USA (Email: gcfexchange@gmail.com). Table of Links Abstract and Introduction Methods Results Discussion Conclusions and References 5 Conclusions When selecting a NN for a specific task, it is important to consider the function and requirements of the task at hand. There is currently no consensus on which NN is optimal for a given task, primarily due to the large number of NN options available, the rapidly evolving nature of the field, and the lack of a comprehensive deep learning theory. This leads to a reliance on empirical results. Our paper is an important step at establishing best practices for this type of problems. we focused on randomly placed sources with random fluxes which yield large variations in the field. Our method can be generalized for different diffusion equations. As part of future steps, we will increase the complexity of the problem being solved by considering conditions closer to real-problems, i.e., by considering less symmetrical sources, different diffusivities and different boundary conditions. In addition, further design is required for these models to be used in a production environment in a reliable way, i.e., how to deal with error performance edge cases on-the-fly? Ultimately, to be able to deploy the model for production, one requires a method to keep the performance error below a predefined bound. For instance, one can train an additional NN that takes the predicted stationary solution and predicts the initial condition, which is then compared with the ground truth initial condition. This framework allows a comparison between ground truth input and predicted input without requiring the ground truth steady-state solution. However, this approach would only be reliable if the input error is always proportional to the steady-state error and hence requires further investigation. A perhaps simpler approach consists in sampling the NN’s output for the same input using a drop-out feature, such that if the fluctuations from the samples are small enough, then the NN’s prediction is robust enough. Both cases require benchmark design. We have developed a process in this paper that can easily be replicated for more complicated problems of this type, and provided a variety of benchmarks. 6 Acknowledgements JQTM acknowledges a Mitacs Postdoctoral Fellowship. GCF acknowledges partial support from DOE DE-SC0023452 and NSF OAC-2204115. References [1] William E Schiesser. The numerical method of lines: integration of partial differential equations. Elsevier, 2012. [2] Amir Barati Farimani, Joseph Gomes, and Vijay S Pande. Deep learning the physics of transport phenomena. arXiv preprint arXiv:1709.02432, 2017. [3] Rishi Sharma, Amir Barati Farimani, Joe Gomes, Peter Eastman, and Vijay Pande. Weakly-supervised deep learning of heat transport via physics informed loss. arXiv preprint arXiv:1807.11374, 2018. [4] Haiyang He and Jay Pathak. An unsupervised learning approach to solving heat equations on chip based on auto encoder and image gradient. arXiv preprint arXiv:2007.09684, 2020. [5] Mohammad Edalatifar, Mohammad Bagher Tavakoli, Mohammad Ghalambaz, and Farbod Setoudeh. Using deep learning to learn physics of conduction heat transfer. Journal of Thermal Analysis and Calorimetry, pages 1–18, 2020. [6] Angran Li, Ruijia Chen, Amir Barati Farimani, and Yongjie Jessica Zhang. Reaction diffusion system prediction based on convolutional neural network. Scientific reports, 10(1):1–9, 2020. [7] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895, 2020. [8] Shengze Cai, Zhicheng Wang, Sifan Wang, Paris Perdikaris, and George Karniadakis. Physics-informed neural networks (pinns) for heat transfer problems. Journal of Heat Transfer, 2021. [9] Geoffrey Fox and Shantenu Jha. Learning everywhere: A taxonomy for the integration of machine learning and simulations. In 2019 15th International Conference on eScience (eScience), pages 439–448. IEEE, 2019. [10] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. arXiv preprint arXiv:1806.07366, 2018. [11] Christopher Rackauckas, Mike Innes, Yingbo Ma, Jesse Bettencourt, Lyndon White, and Vaibhav Dixit. Diffeqflux.jl - A julia library for neural differential equations. CoRR, abs/1902.02376, 2019. [12] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Scorebased generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020. [13] Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686–707, 2019. [14] Frank Noé, Alexandre Tkatchenko, Klaus-Robert Müller, and Cecilia Clementi. Machine learning for molecular simulation. Annual review of physical chemistry, 71:361–390, 2020. [15] Maciej Majewski, Adrià Pérez, Philipp Thölke, Stefan Doerr, Nicholas E Charron, Toni Giorgino, Brooke E Husic, Cecilia Clementi, Frank Noé, and Gianni De Fabritiis. Machine learning coarse-grained potentials of protein thermodynamics. arXiv preprint arXiv:2212.07492, 2022. [16] Mike Entwistle, Zeno Schätzle, Paolo A Erdman, Jan Hermann, and Frank Noé. Electronic excited states in deep variational monte carlo. arXiv preprint arXiv:2203.09472, 2022. [17] J Quetzalcóatl Toledo-Marín, Geoffrey Fox, James P Sluka, and James A Glazier. Deep learning approaches to surrogates for solving the diffusion equation for mechanistic real-world simulations. Frontiers in Physiology, 12, 2021. [18] Christoph Baur, Stefan Denner, Benedikt Wiestler, Nassir Navab, and Shadi Albarqouni. Autoencoders for unsupervised anomaly segmentation in brain mr images: A comparative study. Medical Image Analysis, page 101952, 2020. [19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. [20] Kishan KC, Rui Li, and MohammadMahdi Gilany. Joint inference for neural network depth and dropout regularization. Advances in Neural Information Processing Systems, 34, 2021. [21] Michael Innes, Elliot Saba, Keno Fischer, Dhairya Gandhi, Marco Concetto Rudilosso, Neethu Mariya Joy, Tejan Karmali, Avik Pal, and Viral Shah. Fashionable modelling with flux. CoRR, abs/1811.01457, 2018. [22] Christopher Rackauckas and Qing Nie. Differentialequations.jl–a performant and feature-rich ecosystem for solving differential equations in julia. Journal of Open Research Software, 5(1), 2017. [23] Andre˘ı Nikolaevich Tikhonov and Aleksandr Andreevich Samarskii. Equations of mathematical physics. Courier Corporation, 2013. [24] J. Quetzalcoatl Toledo-Marin. Steady state diffusion surrogate. https://github.com/jquetzalcoatl/ DiffSolver, 2023. [25] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. [26] Peter J Huber. Robust estimation of a location parameter. In Breakthroughs in statistics, pages 492–518. Springer, 1992. [27] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234–241. Springer, 2015. [28] Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. arXiv preprint arXiv:2206.14486, 2022. [29] Jeyan Thiyagalingam, Mallikarjun Shankar, Geoffrey Fox, and Tony Hey. Scientific machine learning benchmarks. Nature Reviews Physics, 4(6):413–420, 2022. This paper is available on arxiv under CC 4.0 license. Authors: (1) J. Quetzalcóatl Toledo-Marín, University of British Columbia, BC Children’s Hospital Research Institute Vancouver BC, Canada (Email: j.toledo.mx@gmail.com); (2) James A. Glazier, Biocomplexity Institute and Department of Intelligent Systems Engineering, Indiana University, Bloomington, IN 47408, USA (Email: jaglazier@gmail.com); (3) Geoffrey Fox, University of Virginia, Computer Science and Biocomplexity Institute, 994 Research Park Blvd, Charlottesville, Virginia, 22911, USA (Email: gcfexchange@gmail.com). Authors: Authors: (1) J. Quetzalcóatl Toledo-Marín, University of British Columbia, BC Children’s Hospital Research Institute Vancouver BC, Canada (Email: j.toledo.mx@gmail.com); (2) James A. Glazier, Biocomplexity Institute and Department of Intelligent Systems Engineering, Indiana University, Bloomington, IN 47408, USA (Email: jaglazier@gmail.com); (3) Geoffrey Fox, University of Virginia, Computer Science and Biocomplexity Institute, 994 Research Park Blvd, Charlottesville, Virginia, 22911, USA (Email: gcfexchange@gmail.com). Table of Links Abstract and Introduction Abstract and Introduction Methods Methods Results Results Discussion Discussion Conclusions and References Conclusions and References 5 Conclusions When selecting a NN for a specific task, it is important to consider the function and requirements of the task at hand. There is currently no consensus on which NN is optimal for a given task, primarily due to the large number of NN options available, the rapidly evolving nature of the field, and the lack of a comprehensive deep learning theory. This leads to a reliance on empirical results. Our paper is an important step at establishing best practices for this type of problems. we focused on randomly placed sources with random fluxes which yield large variations in the field. Our method can be generalized for different diffusion equations. As part of future steps, we will increase the complexity of the problem being solved by considering conditions closer to real-problems, i.e., by considering less symmetrical sources, different diffusivities and different boundary conditions. In addition, further design is required for these models to be used in a production environment in a reliable way, i.e., how to deal with error performance edge cases on-the-fly? Ultimately, to be able to deploy the model for production, one requires a method to keep the performance error below a predefined bound. For instance, one can train an additional NN that takes the predicted stationary solution and predicts the initial condition, which is then compared with the ground truth initial condition. This framework allows a comparison between ground truth input and predicted input without requiring the ground truth steady-state solution. However, this approach would only be reliable if the input error is always proportional to the steady-state error and hence requires further investigation. A perhaps simpler approach consists in sampling the NN’s output for the same input using a drop-out feature, such that if the fluctuations from the samples are small enough, then the NN’s prediction is robust enough. Both cases require benchmark design. We have developed a process in this paper that can easily be replicated for more complicated problems of this type, and provided a variety of benchmarks. 6 Acknowledgements JQTM acknowledges a Mitacs Postdoctoral Fellowship. GCF acknowledges partial support from DOE DE-SC0023452 and NSF OAC-2204115. References [1] William E Schiesser. The numerical method of lines: integration of partial differential equations. Elsevier, 2012. [2] Amir Barati Farimani, Joseph Gomes, and Vijay S Pande. Deep learning the physics of transport phenomena. arXiv preprint arXiv:1709.02432, 2017. [3] Rishi Sharma, Amir Barati Farimani, Joe Gomes, Peter Eastman, and Vijay Pande. Weakly-supervised deep learning of heat transport via physics informed loss. arXiv preprint arXiv:1807.11374, 2018. [4] Haiyang He and Jay Pathak. An unsupervised learning approach to solving heat equations on chip based on auto encoder and image gradient. arXiv preprint arXiv:2007.09684, 2020. [5] Mohammad Edalatifar, Mohammad Bagher Tavakoli, Mohammad Ghalambaz, and Farbod Setoudeh. Using deep learning to learn physics of conduction heat transfer. Journal of Thermal Analysis and Calorimetry, pages 1–18, 2020. [6] Angran Li, Ruijia Chen, Amir Barati Farimani, and Yongjie Jessica Zhang. Reaction diffusion system prediction based on convolutional neural network. Scientific reports, 10(1):1–9, 2020. [7] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895, 2020. [8] Shengze Cai, Zhicheng Wang, Sifan Wang, Paris Perdikaris, and George Karniadakis. Physics-informed neural networks (pinns) for heat transfer problems. Journal of Heat Transfer, 2021. [9] Geoffrey Fox and Shantenu Jha. Learning everywhere: A taxonomy for the integration of machine learning and simulations. In 2019 15th International Conference on eScience (eScience), pages 439–448. IEEE, 2019. [10] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. arXiv preprint arXiv:1806.07366, 2018. [11] Christopher Rackauckas, Mike Innes, Yingbo Ma, Jesse Bettencourt, Lyndon White, and Vaibhav Dixit. Diffeqflux.jl - A julia library for neural differential equations. CoRR, abs/1902.02376, 2019. [12] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Scorebased generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020. [13] Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686–707, 2019. [14] Frank Noé, Alexandre Tkatchenko, Klaus-Robert Müller, and Cecilia Clementi. Machine learning for molecular simulation. Annual review of physical chemistry, 71:361–390, 2020. [15] Maciej Majewski, Adrià Pérez, Philipp Thölke, Stefan Doerr, Nicholas E Charron, Toni Giorgino, Brooke E Husic, Cecilia Clementi, Frank Noé, and Gianni De Fabritiis. Machine learning coarse-grained potentials of protein thermodynamics. arXiv preprint arXiv:2212.07492, 2022. [16] Mike Entwistle, Zeno Schätzle, Paolo A Erdman, Jan Hermann, and Frank Noé. Electronic excited states in deep variational monte carlo. arXiv preprint arXiv:2203.09472, 2022. [17] J Quetzalcóatl Toledo-Marín, Geoffrey Fox, James P Sluka, and James A Glazier. Deep learning approaches to surrogates for solving the diffusion equation for mechanistic real-world simulations. Frontiers in Physiology, 12, 2021. [18] Christoph Baur, Stefan Denner, Benedikt Wiestler, Nassir Navab, and Shadi Albarqouni. Autoencoders for unsupervised anomaly segmentation in brain mr images: A comparative study. Medical Image Analysis, page 101952, 2020. [19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. [20] Kishan KC, Rui Li, and MohammadMahdi Gilany. Joint inference for neural network depth and dropout regularization. Advances in Neural Information Processing Systems, 34, 2021. [21] Michael Innes, Elliot Saba, Keno Fischer, Dhairya Gandhi, Marco Concetto Rudilosso, Neethu Mariya Joy, Tejan Karmali, Avik Pal, and Viral Shah. Fashionable modelling with flux. CoRR, abs/1811.01457, 2018. [22] Christopher Rackauckas and Qing Nie. Differentialequations.jl–a performant and feature-rich ecosystem for solving differential equations in julia. Journal of Open Research Software, 5(1), 2017. [23] Andre˘ı Nikolaevich Tikhonov and Aleksandr Andreevich Samarskii. Equations of mathematical physics. Courier Corporation, 2013. [24] J. Quetzalcoatl Toledo-Marin. Steady state diffusion surrogate. https://github.com/jquetzalcoatl/ DiffSolver, 2023. [25] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. [26] Peter J Huber. Robust estimation of a location parameter. In Breakthroughs in statistics, pages 492–518. Springer, 1992. [27] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pages 234–241. Springer, 2015. [28] Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. arXiv preprint arXiv:2206.14486, 2022. [29] Jeyan Thiyagalingam, Mallikarjun Shankar, Geoffrey Fox, and Tony Hey. Scientific machine learning benchmarks. Nature Reviews Physics, 4(6):413–420, 2022. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Generalizing Deep Learning Models for Varied Diffusion Equations

Understanding Factors Affecting Neural Network Performance in Diffusion Prediction

Authors:
(1) J. Quetzalcóatl Toledo-Marín, University of British Columbia, BC Children’s Hospital Research Institute Vancouver BC, Canada (Email: j.toledo.mx@gmail.com);
(2) James A. Glazier, Biocomplexity Institute and Department of Intelligent Systems Engineering, Indiana University, Bloomington, IN 47408, USA (Email: jaglazier@gmail.com);
(3) Geoffrey Fox, University of Virginia, Computer Science and Biocomplexity Institute, 994 Research Park Blvd, Charlottesville, Virginia, 22911, USA (Email: gcfexchange@gmail.com). Table of Links Abstract and Introduction Methods Results Discussion Conclusions and References 2 Methods The steady state of the diffusion equation is precisely the convolution of the equation’s associated Green’s function and the initial state. Hence a naive approach would consider a deep convolutional NN (CNN) where the height and width in each layer is kept fixed and equal to the input. CNNs are good at detecting edges and as the number of convolution layers increases, the CNN can detect more sophisticated elements. So can a CNN predict the stationary solution? We argue that it would require a large number of convolution layers such that each convolution would be proportional to a timestep. Adding many convolutions would lead to a rather harder and slower training, as it would increase the chances of under or overflow in the gradient. Furthermore, the number of convolution layers would need to be variable is some form or fashion in order to mimic the timesteps required to reach the stationary solution given an initial condition. There are ways to address each of the previous setbacks but there’s also a trade-off for every decision (e.g., increasing the number of convolutional layers increases the model’s instability which can be addressed by adding more regularization layers which makes the model larger and requires more time to train). Experience tells us that deep convolutional NNs where the height and width in each layer is kept fixed and equal to the input are not good for our task [17]. We designed and built an encoder-decoder CNN (ED-CNN) as our architecture. Fig. 2 provides a sketch of our NN architecture. We denote by |x〉 and |yˆ〉 the input and output images, that is the initial condition layout of the source cells and the predicted stationary solution of the diffusion equation, respectively. The input |x〉 goes through convolutional blocks depicted in Fig. 2 and generates the prediction |yˆ〉 which we then compare with the ground truth |y〉. The core feature of an encoder-decoder is the cone-like structure whereby the information is being compressed when it reaches the bottle neck and then it’s decompressed. Consider the following: A CNN can be seen as a set of operators {Oi} acting on the input, i.e., where the i tags the convolutional layer. Notice that in the particular case where {Oi} are unitary linear operators and all the same O = 1− Ad t, Eq. (1) becomes Notice that the previous Eq. is akin to a ResNet architecture [19]. By defining d t = t/n, where t is time, and taking the limit n → ∞ leads to |yˆ〉 = exp(At)|x〉. Hence, when |yˆ〉 equals the ground truth |y〉, the CNN has learnt the operator exp(At). Notice that taking the derivative with respect to time leads to the diffusion equation: Typically, numerical methods for searching for the stationary solution of a diffusion equation operate by comparing the solution at time t with the solution at time t +∆t, and halting the process when the difference between |y(t)〉 and |y(t+∆t)〉is smaller in magnitude than some pre-determined error value. However, the time t at which the stationary solution is reached depends on the initial conditions and the inhomogeneity term in the diffusion equation. This analogy suggests that, similar to traditional numerical methods, CNNs also require a flexible number of layers to reach the stationary solution, rather than an infinite number of layers. There have been recent developments in addressing the issue of a variable number of layers. For example, in reference [20], the authors propose a method in which the dropout parameters are trained in conjunction with the model parameters. In the context of ED-CNNs, it is a widely held belief that the cone-like architecture leads to improved performance due to the compression in the bottleneck. However, we argue that this is not the sole, if any, contributing factor. It is important to note that the ED-CNN architecture involves summing over the lattice and introducing a new variable, the depth. Additionally, past the bottleneck, the ED-CNN projects back to the original space. In this sense, we say the ED-CNN learns a propagator in a reciprocal space similar to methods such as Laplace and Fourier transform. In Table 1 we specify the neural network by specifying the type of operation and output shape through the sequence of blocks in the NN. Notice that the bottleneck block is 2048∗16∗16 which is twice the size of the input. We built and trained the NN in PyTorch. We used ADAM as the optimizer [25]. Deciding on a loss function is a critical choice in the creation of the surrogate. The loss function determines the types of error the surrogate’s approximation will make compared to the direct calculation and the acceptability of these errors will depend on the specific application. The mean squared error (MSE) error is a standard choice. However, it is more sensitive to larger absolute errors and therefore tolerates large relative errors at pixels with small values. In most biological contexts we want to have a small absolute error for small values and a small relative error for large values. We explored the NN’s performance when trained using a set of different loss functions with different hyperparameters. The set of different loss functions we used were MAE, MSE, Huber loss [26] and inverse Huber loss defined as: where α is 1 or 2 for MAE or MSE, respectively and β tags the tuple in the data set (input and target). Here 〈|〉 denotes the inner product and |i〉 is a unitary vector with the same size as |yβ〉 with all components equal to zero except the element in position i which is equal to one. |1〉 is a vector with all components equal to 1 and with size equal to that of |yβ〉. Then 〈i|yβ〉 is a scalar corresponding to the pixel value at the ith position in |yβ〉, whereas 〈i|1〉 = 1 for all i. Notice that high and low pixel values will have an exponential weight ≈ 1 and ≈ exp(−1/w), respectively. This implies that the error associated to high pixels will have a larger value than that for low pixels. Another prefactor we considered was a step-wise function, viz. This paper is available on arxiv under CC 4.0 license. Authors: (1) J. Quetzalcóatl Toledo-Marín, University of British Columbia, BC Children’s Hospital Research Institute Vancouver BC, Canada (Email: j.toledo.mx@gmail.com); (2) James A. Glazier, Biocomplexity Institute and Department of Intelligent Systems Engineering, Indiana University, Bloomington, IN 47408, USA (Email: jaglazier@gmail.com); (3) Geoffrey Fox, University of Virginia, Computer Science and Biocomplexity Institute, 994 Research Park Blvd, Charlottesville, Virginia, 22911, USA (Email: gcfexchange@gmail.com). Authors: Authors: (1) J. Quetzalcóatl Toledo-Marín, University of British Columbia, BC Children’s Hospital Research Institute Vancouver BC, Canada (Email: j.toledo.mx@gmail.com); (2) James A. Glazier, Biocomplexity Institute and Department of Intelligent Systems Engineering, Indiana University, Bloomington, IN 47408, USA (Email: jaglazier@gmail.com); (3) Geoffrey Fox, University of Virginia, Computer Science and Biocomplexity Institute, 994 Research Park Blvd, Charlottesville, Virginia, 22911, USA (Email: gcfexchange@gmail.com). Table of Links Abstract and Introduction Abstract and Introduction Methods Methods Results Results Discussion Discussion Conclusions and References Conclusions and References 2 Methods The steady state of the diffusion equation is precisely the convolution of the equation’s associated Green’s function and the initial state. Hence a naive approach would consider a deep convolutional NN (CNN) where the height and width in each layer is kept fixed and equal to the input. CNNs are good at detecting edges and as the number of convolution layers increases, the CNN can detect more sophisticated elements. So can a CNN predict the stationary solution? We argue that it would require a large number of convolution layers such that each convolution would be proportional to a timestep. Adding many convolutions would lead to a rather harder and slower training, as it would increase the chances of under or overflow in the gradient. Furthermore, the number of convolution layers would need to be variable is some form or fashion in order to mimic the timesteps required to reach the stationary solution given an initial condition. There are ways to address each of the previous setbacks but there’s also a trade-off for every decision (e.g., increasing the number of convolutional layers increases the model’s instability which can be addressed by adding more regularization layers which makes the model larger and requires more time to train). Experience tells us that deep convolutional NNs where the height and width in each layer is kept fixed and equal to the input are not good for our task [17]. We designed and built an encoder-decoder CNN (ED-CNN) as our architecture. Fig. 2 provides a sketch of our NN architecture. We denote by |x〉 and |yˆ〉 the input and output images, that is the initial condition layout of the source cells and the predicted stationary solution of the diffusion equation, respectively. The input |x〉 goes through convolutional blocks depicted in Fig. 2 and generates the prediction |yˆ〉 which we then compare with the ground truth |y〉. The core feature of an encoder-decoder is the cone-like structure whereby the information is being compressed when it reaches the bottle neck and then it’s decompressed. Consider the following: A CNN can be seen as a set of operators {Oi} acting on the input, i.e., where the i tags the convolutional layer. Notice that in the particular case where {Oi} are unitary linear operators and all the same O = 1− Ad t, Eq. (1) becomes Notice that the previous Eq. is akin to a ResNet architecture [19]. By defining d t = t/n, where t is time, and taking the limit n → ∞ leads to |yˆ〉 = exp(At)|x〉. Hence, when |yˆ〉 equals the ground truth |y〉, the CNN has learnt the operator exp(At). Notice that taking the derivative with respect to time leads to the diffusion equation: Typically, numerical methods for searching for the stationary solution of a diffusion equation operate by comparing the solution at time t with the solution at time t +∆t, and halting the process when the difference between |y(t)〉 and |y(t+∆t)〉is smaller in magnitude than some pre-determined error value. However, the time t at which the stationary solution is reached depends on the initial conditions and the inhomogeneity term in the diffusion equation. This analogy suggests that, similar to traditional numerical methods, CNNs also require a flexible number of layers to reach the stationary solution, rather than an infinite number of layers. There have been recent developments in addressing the issue of a variable number of layers. For example, in reference [20], the authors propose a method in which the dropout parameters are trained in conjunction with the model parameters. In the context of ED-CNNs, it is a widely held belief that the cone-like architecture leads to improved performance due to the compression in the bottleneck. However, we argue that this is not the sole, if any, contributing factor. It is important to note that the ED-CNN architecture involves summing over the lattice and introducing a new variable, the depth. Additionally, past the bottleneck, the ED-CNN projects back to the original space. In this sense, we say the ED-CNN learns a propagator in a reciprocal space similar to methods such as Laplace and Fourier transform. In Table 1 we specify the neural network by specifying the type of operation and output shape through the sequence of blocks in the NN. Notice that the bottleneck block is 2048∗16∗16 which is twice the size of the input. We built and trained the NN in PyTorch. We used ADAM as the optimizer [25]. Deciding on a loss function is a critical choice in the creation of the surrogate. The loss function determines the types of error the surrogate’s approximation will make compared to the direct calculation and the acceptability of these errors will depend on the specific application. The mean squared error (MSE) error is a standard choice. However, it is more sensitive to larger absolute errors and therefore tolerates large relative errors at pixels with small values. In most biological contexts we want to have a small absolute error for small values and a small relative error for large values. We explored the NN’s performance when trained using a set of different loss functions with different hyperparameters. The set of different loss functions we used were MAE, MSE, Huber loss [26] and inverse Huber loss defined as: where α is 1 or 2 for MAE or MSE, respectively and β tags the tuple in the data set (input and target). Here 〈|〉 denotes the inner product and |i〉 is a unitary vector with the same size as |yβ〉 with all components equal to zero except the element in position i which is equal to one. |1〉 is a vector with all components equal to 1 and with size equal to that of |yβ〉. Then 〈i|yβ〉 is a scalar corresponding to the pixel value at the ith position in |yβ〉, whereas 〈i|1〉 = 1 for all i. Notice that high and low pixel values will have an exponential weight ≈ 1 and ≈ exp(−1/w), respectively. This implies that the error associated to high pixels will have a larger value than that for low pixels. Another prefactor we considered was a step-wise function, viz. This paper is available on arxiv under CC 4.0 license. This paper is available on arxiv under CC 4.0 license. available on arxiv

Stories, News & Mentions about DRIFF

Driff’s Stories on HackerNoon

Driff's News & Mentions