Let me tell you a bit more about it! The Wim Hof Wethod in Action: Increased Mental Clarity: By practicing the Wim Hof breathing exercises, you can oxygenate your body, improving blood flow and ...

Enhance Your Well-being and Boost Your Performance with the Wim Hof Method

ZeroEyes, creators of the only proactive A.I.-based gun detection video analytics platform that holds the US Department of Homeland Security SAFETY Act Designation, announces that the United

ZeroEyes announces that USPTO has issued a patent for AI-based gun detection methodology

Officials are calling attention to the city’s Trap Neuter Return (TNR) Program, in an effort to effectively and humanely control the community cat population.

Peekskill police promote plan to feral or stray cats

Xsolla, a leading global video game commerce company celebrates opening a new office in Kuala Lumpur, one of the fastest-growing cities in Asia and the largest city in Malaysia. This will be the largest Xsolla office globally outside of Los Angeles,

Xsolla Announces New Office Location in Kuala Lumpur as Part of Continued Expansion and Commitment to Asia

We anticipate interested parties will argue the FTC has exceeded the scope of its authority and such a rule is within the province of the United States Congress, not a federal agency, says C. Larry Carbo III,

FTC Proposes Rule Banning Non-Compete Clauses With Workers

MultiSafepay, one of the leading European Payment Service Providers working with e-commerce merchants, announced the integration of MyBank on its platform today. MyBank is the pan-European e-payment solution,

Account-to-account Payments: MultiSafepay Adds MyBank to its Payment Method Mix

In December 2022, the Nilson Report, which monitors the payments industry, released a forecast indicating that U.S. losses from card fraud will total $165.1 billion over the next 10 years, plaguing every age group in every state.

Credit card fraud statistics

Boulder has seen a "dramatic" spike over the past six years in the number of methamphetamine-affected properties — places where the drug is used or produced.

How Boulder is managing its major meth problem

The Internet is a dangerous place, with many potential attacks that could be costly to your digital life. Here's how to keep your Mac, and your online identity, secure from threats.

Cybercriminals shall not pass: how to protect your Macs and yourself from the most popular cyber threats

The SPAC mania throughout the past two years saw most mergers plummet in value. Here are some of the outcomes.

How Did All Those Travel SPACs End Up Doing?

WETHOD

wethod

New Method Could Unlock AI's Power to See and Describe Images with Unmatched Detail

Authors:
(1) Hanoona Rasheed, Mohamed bin Zayed University of AI and equally contributing first authors;
(2) Muhammad Maaz, Mohamed bin Zayed University of AI and equally contributing first authors;
(3) Sahal Shaji, Mohamed bin Zayed University of AI;
(4) Abdelrahman Shaker, Mohamed bin Zayed University of AI;
(5) Salman Khan, Mohamed bin Zayed University of AI and Australian National University;
(6) Hisham Cholakkal, Mohamed bin Zayed University of AI;
(7) Rao M. Anwer, Mohamed bin Zayed University of AI and Aalto University;
(8) Eric Xing, Mohamed bin Zayed University of AI and Carnegie Mellon University;
(9) Ming-Hsuan Yang, University of California - Merced and Google Research;
(10) Fahad S. Khan, Mohamed bin Zayed University of AI and Linköping University. Editor's Note: This is Part 4 of 10 of a study detailing the development of an AI model that is designed to describe images to users. Read the rest below. Table of Links Abstract and 1 Introduction
2. Related Work
3. Method
4. Data Annotation Pipeline
5. Experiments
6. Conclusion and References Supplementary Material (Part 1) A. Additional Implementation Details
B. Additional Downstream Tasks
C. Additional Qualitative Results Supplementary Material (Part 2) D. Dataset Visualization
E. Limitations and Future Work
F. Ethics and Societal Impact 4. Data Annotation Pipeline We introduce our automated annotation pipeline used to create the Grounding-anything Dataset (GranD). GranD is a comprehensive, multi-purpose image-text dataset offering a range of contextual information, from fine-grained to high-level details. It aims to overcome challenges in image understanding and dense pixel-level grounding, thereby expanding the capabilities of visual instruction tuning in LMMs. The pipeline contains four distinct levels (see Fig. 4). i) Level-1 focuses on object localization and provides semantic labels, segmentation masks, attributes, and depth information. ii) Level-2 defines relationships between detected objects. iii) Level-3 organizes information from the first two levels into a hierarchical scene graph, used to generate dense captions using LLM with in-context examples. iv) Level-4 offers enriched contextual information for a deeper understanding of the scene, going beyond what’s observed (e.g., historical information of a landmark). Please refer to Appendix A.4 for pipeline implementation details. 4.1. Object Localization and Attributes (Level-1) In level-1, the focus is on detailed object identification within images. First, object-bounding boxes are identified using multiple SoTA object detection models. Classagnostic NMS is applied to each model to filter out false positives. After this step, bounding boxes from different models are compared using IoU, with a bounding box retained as an object only if detected by at least two other detection models. We also generate attributes for each filtered object using region-based vision-language models and incorporate depth information to contextualize each object’s relative position within the scene. 4.2. Relationships and Landmarks (Level-2) In level-2, multiple short textual descriptions of the overall scene are generated. Phrases extracted from these descriptions are grounded to specific objects in level-1 to form relationships. These relationships articulate connections between multiple objects or define an object’s role within the scene. Further, each scene is assigned a landmark category that includes a primary and a more specific sub-category (see Tab. 7 in Appendix 7). 4.3. Scene Graph and Dense Captioning (Level-3) In level-3, object attributes and labels from level-1 are combined with the relationships and phrases obtained from level-2 to form a hierarchical scene graph. This structured data serves as a query for LLM to generate dense image captions. To provide additional context, depth values and bounding box coordinates are used to assign each object to specific spatial layers within the scene, such as immediate foreground, foreground, midground, or background. Additionally, short scene-level captions are incorporated into the scene graph to enhance LLMs’ contextual understanding. Dense Captioning Verification: To enhance the fidelity of the LLM-generated dense captions, we implement an automatic verification pipeline using chain-of-thoughts prompting. This pipeline produces a checklist of objects derived from the generated dense caption expected to be present in the image. The associated caption is flagged as inaccurate if any object specified in the checklist is absent from the scene graph. Such captions are then regenerated, incorporating feedback from the initial assessment. 4.4. Extra Contextual Insights (Level-4) Level-4 builds on the scene graph from level-3 to obtain a more detailed visual understanding. we query LLM to extract extended contextual insights beyond basic object identification and relationships, including details about the landmarks, historical context, guidelines for interacting with the scene, and even predictive elements about future events. To facilitate this, we prompt LLM with in-context examples. Utilizing our automated annotation pipeline, we annotate a corpus of 11M SAM images [18], which are inherently diverse, high-resolution, and privacy-compliant. The resulting dataset comprises 810M regions, each associated with a segmentation mask, and includes 7.5M unique concepts. Further, the dataset features 84M referring expressions, 22M grounded short captions, and 11M densely grounded captions. To our knowledge, this is the first dataset of this scale generated entirely through an automated annotation pipeline (see Tab. 2 for details and Fig. 15 for dataset sample visualizations). We extend open-source datasets—namely Flickr30K [37], RefCOCOg [16], and PSG [49] by generating compatible GCG annotations. For RefCOCOg, we use the dataset’s referring expressions and their connected masks. These expressions offer concise descriptions of distinct objects in the image. With the aid of GPT-4, we seamlessly blend these referring expressions with contextual information from COCO captions, crafting detailed yet accurate grounded captions while preserving the original referring expressions. This ensures zero error in matching phrases with their corresponding segmentation masks. This technique yields approximately 24K GCG samples. For PSG, we leverage the dataset’s triplet structures, which describe relations between two objects in a scene. These triplets are integrated with COCO captions using GPT-4, resulting in densely annotated captions that can be mapped to segmentation masks. This gives us around 31K additional GCG samples. For Flickr-30K, we use the 158K Flickr captions and their referring expressions alongside associated bounding boxes. These boxes are then accurately segmented using HQ-SAM [17]. In addition, we contribute a minor, high-quality manual annotation set to benchmark the GCG task. Using GranD’s automatic annotations as a base, annotators refine referring expressions to match SAM GT masks, yielding around 1000 focused samples for training and 1000 for evaluation (refer to Appendix D and Fig. 14 for designed prompts and dataset visualizations). This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Hanoona Rasheed, Mohamed bin Zayed University of AI and equally contributing first authors; (2) Muhammad Maaz, Mohamed bin Zayed University of AI and equally contributing first authors; (3) Sahal Shaji, Mohamed bin Zayed University of AI; (4) Abdelrahman Shaker, Mohamed bin Zayed University of AI; (5) Salman Khan, Mohamed bin Zayed University of AI and Australian National University; (6) Hisham Cholakkal, Mohamed bin Zayed University of AI; (7) Rao M. Anwer, Mohamed bin Zayed University of AI and Aalto University; (8) Eric Xing, Mohamed bin Zayed University of AI and Carnegie Mellon University; (9) Ming-Hsuan Yang, University of California - Merced and Google Research; (10) Fahad S. Khan, Mohamed bin Zayed University of AI and Linköping University. Authors: Authors: (1) Hanoona Rasheed, Mohamed bin Zayed University of AI and equally contributing first authors; (2) Muhammad Maaz, Mohamed bin Zayed University of AI and equally contributing first authors; (3) Sahal Shaji, Mohamed bin Zayed University of AI; (4) Abdelrahman Shaker, Mohamed bin Zayed University of AI; (5) Salman Khan, Mohamed bin Zayed University of AI and Australian National University; (6) Hisham Cholakkal, Mohamed bin Zayed University of AI; (7) Rao M. Anwer, Mohamed bin Zayed University of AI and Aalto University; (8) Eric Xing, Mohamed bin Zayed University of AI and Carnegie Mellon University; (9) Ming-Hsuan Yang, University of California - Merced and Google Research; (10) Fahad S. Khan, Mohamed bin Zayed University of AI and Linköping University. Editor's Note: This is Part 4 of 10 of a study detailing the development of an AI model that is designed to describe images to users. Read the rest below. Editor's Note: This is Part 4 of 10 of a study detailing the development of an AI model that is designed to describe images to users. Read the rest below. Editor's Note: This is Part 4 of 10 of a study detailing the development of an AI model that is designed to describe images to users. Read the rest below. Editor's Note: This is Part 4 of 10 of a study detailing the development of an AI model that is designed to describe images to users. Read the rest below. Table of Links Abstract and 1 Introduction 2. Related Work 3. Method 4. Data Annotation Pipeline 5. Experiments 6. Conclusion and References Abstract and 1 Introduction Abstract and 1 Introduction 2. Related Work 2. Related Work 3. Method 3. Method 4. Data Annotation Pipeline 4. Data Annotation Pipeline 5. Experiments 5. Experiments 6. Conclusion and References 6. Conclusion and References Supplementary Material (Part 1) Supplementary Material (Part 1) A. Additional Implementation Details B. Additional Downstream Tasks C. Additional Qualitative Results A. Additional Implementation Details A. Additional Implementation Details B. Additional Downstream Tasks B. Additional Downstream Tasks C. Additional Qualitative Results C. Additional Qualitative Results Supplementary Material (Part 2) Supplementary Material (Part 2) D. Dataset Visualization E. Limitations and Future Work F. Ethics and Societal Impact D. Dataset Visualization D. Dataset Visualization E. Limitations and Future Work E. Limitations and Future Work F. Ethics and Societal Impact F. Ethics and Societal Impact 4. Data Annotation Pipeline We introduce our automated annotation pipeline used to create the Grounding-anything Dataset (GranD). GranD is a comprehensive, multi-purpose image-text dataset offering a range of contextual information, from fine-grained to high-level details. It aims to overcome challenges in image understanding and dense pixel-level grounding, thereby expanding the capabilities of visual instruction tuning in LMMs. The pipeline contains four distinct levels (see Fig. 4). i) Level-1 focuses on object localization and provides semantic labels, segmentation masks, attributes, and depth information. ii) Level-2 defines relationships between detected objects. iii) Level-3 organizes information from the first two levels into a hierarchical scene graph, used to generate dense captions using LLM with in-context examples. iv) Level-4 offers enriched contextual information for a deeper understanding of the scene, going beyond what’s observed (e.g., historical information of a landmark). Please refer to Appendix A.4 for pipeline implementation details. 4.1. Object Localization and Attributes (Level-1) In level-1, the focus is on detailed object identification within images. First, object-bounding boxes are identified using multiple SoTA object detection models. Classagnostic NMS is applied to each model to filter out false positives. After this step, bounding boxes from different models are compared using IoU, with a bounding box retained as an object only if detected by at least two other detection models. We also generate attributes for each filtered object using region-based vision-language models and incorporate depth information to contextualize each object’s relative position within the scene. 4.2. Relationships and Landmarks (Level-2) In level-2, multiple short textual descriptions of the overall scene are generated. Phrases extracted from these descriptions are grounded to specific objects in level-1 to form relationships. These relationships articulate connections between multiple objects or define an object’s role within the scene. Further, each scene is assigned a landmark category that includes a primary and a more specific sub-category (see Tab. 7 in Appendix 7). 4.3. Scene Graph and Dense Captioning (Level-3) In level-3, object attributes and labels from level-1 are combined with the relationships and phrases obtained from level-2 to form a hierarchical scene graph. This structured data serves as a query for LLM to generate dense image captions. To provide additional context, depth values and bounding box coordinates are used to assign each object to specific spatial layers within the scene, such as immediate foreground, foreground, midground, or background. Additionally, short scene-level captions are incorporated into the scene graph to enhance LLMs’ contextual understanding. Dense Captioning Verification: To enhance the fidelity of the LLM-generated dense captions, we implement an automatic verification pipeline using chain-of-thoughts prompting. This pipeline produces a checklist of objects derived from the generated dense caption expected to be present in the image. The associated caption is flagged as inaccurate if any object specified in the checklist is absent from the scene graph. Such captions are then regenerated, incorporating feedback from the initial assessment. Dense Captioning Verification: 4.4. Extra Contextual Insights (Level-4) Level-4 builds on the scene graph from level-3 to obtain a more detailed visual understanding. we query LLM to extract extended contextual insights beyond basic object identification and relationships, including details about the landmarks, historical context, guidelines for interacting with the scene, and even predictive elements about future events. To facilitate this, we prompt LLM with in-context examples. Utilizing our automated annotation pipeline, we annotate a corpus of 11M SAM images [18], which are inherently diverse, high-resolution, and privacy-compliant. The resulting dataset comprises 810M regions, each associated with a segmentation mask, and includes 7.5M unique concepts. Further, the dataset features 84M referring expressions, 22M grounded short captions, and 11M densely grounded captions. To our knowledge, this is the first dataset of this scale generated entirely through an automated annotation pipeline (see Tab. 2 for details and Fig. 15 for dataset sample visualizations). We extend open-source datasets—namely Flickr30K [37], RefCOCOg [16], and PSG [49] by generating compatible GCG annotations. For RefCOCOg, we use the dataset’s referring expressions and their connected masks. These expressions offer concise descriptions of distinct objects in the image. With the aid of GPT-4, we seamlessly blend these referring expressions with contextual information from COCO captions, crafting detailed yet accurate grounded captions while preserving the original referring expressions. This ensures zero error in matching phrases with their corresponding segmentation masks. This technique yields approximately 24K GCG samples. For PSG, we leverage the dataset’s triplet structures, which describe relations between two objects in a scene. These triplets are integrated with COCO captions using GPT-4, resulting in densely annotated captions that can be mapped to segmentation masks. This gives us around 31K additional GCG samples. For Flickr-30K, we use the 158K Flickr captions and their referring expressions alongside associated bounding boxes. These boxes are then accurately segmented using HQ-SAM [17]. In addition, we contribute a minor, high-quality manual annotation set to benchmark the GCG task. Using GranD’s automatic annotations as a base, annotators refine referring expressions to match SAM GT masks, yielding around 1000 focused samples for training and 1000 for evaluation (refer to Appendix D and Fig. 14 for designed prompts and dataset visualizations). This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Researchers Develop Groundbreaking Method to Teach AI to 'Understand' Images Like Never Before

Authors:
(1) Anton Razzhigaev, AIRI and Skoltech;
(2) Arseniy Shakhmatov, Sber AI;
(3) Anastasia Maltseva, Sber AI;
(4) Vladimir Arkhipkin, Sber AI;
(5) Igor Pavlov, Sber AI;
(6) Ilya Ryabov, Sber AI;
(7) Angelina Kuts, Sber AI;
(8) Alexander Panchenko, AIRI and Skoltech;
(9) Andrey Kuznetsov, AIRI and Sber AI;
(10) Denis Dimitrov, AIRI and Sber AI. Editor's Note: This is Part 7 of 8 of a study detailing the development of Kandinsky, the first text-to-image architecture designed using a combination of image prior and latent diffusion. Read the rest below. Table of Links Abstract and Introduction
Related Work
Demo System
Kandinsky Architecture
Experiments
Results
Conclusion & Limitations
Ethical Considerations, Acknowledgements and References 7 Conclusion We presented Kandinsky, a system for various image generation and processing tasks based on a novel latent diffusion model. Our model yielded the SotA results among open-sourced systems. Additionally, we provided an extensive ablation study of an image prior to design choices. Our system is equipped with free-to-use interfaces in the form of Web application and Telegram messenger bot. The pre-trained models are available on Hugging Face, and the source code is released under a permissive license enabling various, including commercial, applications of the developed technology. In future research, our goal is to investigate the potential of the latest image encoders. We plan to explore the development of more efficient UNet architectures for text-to-image tasks and focus on improving the understanding of textual prompts. Additionally, we aim to experiment with generating images at higher resolutions and to investigate new features extending the model: local image editing by a text prompt, attention reweighting, physics-based generation control, etc. The robustness against generating abusive content remains a crucial concern, warranting the exploration of real-time moderation layers or robust classifiers to mitigate undesirable, e.g. toxic or abusive, outputs. 8 Limitations The current system produces images that appear natural, however, additional research can be conducted to (1) enhance the semantic coherence between the input text and the generated image, and (2) to improve the absolute values of FID and image quality based on human evaluations. This paper is available on arxiv under CC BY 4.0 DEED license. Authors: (1) Anton Razzhigaev, AIRI and Skoltech; (2) Arseniy Shakhmatov, Sber AI; (3) Anastasia Maltseva, Sber AI; (4) Vladimir Arkhipkin, Sber AI; (5) Igor Pavlov, Sber AI; (6) Ilya Ryabov, Sber AI; (7) Angelina Kuts, Sber AI; (8) Alexander Panchenko, AIRI and Skoltech; (9) Andrey Kuznetsov, AIRI and Sber AI; (10) Denis Dimitrov, AIRI and Sber AI. Authors: Authors: (1) Anton Razzhigaev, AIRI and Skoltech; (2) Arseniy Shakhmatov, Sber AI; (3) Anastasia Maltseva, Sber AI; (4) Vladimir Arkhipkin, Sber AI; (5) Igor Pavlov, Sber AI; (6) Ilya Ryabov, Sber AI; (7) Angelina Kuts, Sber AI; (8) Alexander Panchenko, AIRI and Skoltech; (9) Andrey Kuznetsov, AIRI and Sber AI; (10) Denis Dimitrov, AIRI and Sber AI. Editor's Note: This is Part 7 of 8 of a study detailing the development of Kandinsky, the first text-to-image architecture designed using a combination of image prior and latent diffusion. Read the rest below. Editor's Note: This is Part 7 of 8 of a study detailing the development of Kandinsky, the first text-to-image architecture designed using a combination of image prior and latent diffusion. Read the rest below. Editor's Note: This is Part 7 of 8 of a study detailing the development of Kandinsky, the first text-to-image architecture designed using a combination of image prior and latent diffusion. Read the rest below. Table of Links Abstract and Introduction Related Work Demo System Kandinsky Architecture Experiments Results Conclusion & Limitations Ethical Considerations, Acknowledgements and References Abstract and Introduction Abstract and Introduction Related Work Related Work Demo System Demo System Kandinsky Architecture Kandinsky Architecture Experiments Experiments Results Results Conclusion & Limitations Conclusion & Limitations Ethical Considerations, Acknowledgements and References Ethical Considerations, Acknowledgements and References 7 Conclusion We presented Kandinsky, a system for various image generation and processing tasks based on a novel latent diffusion model. Our model yielded the SotA results among open-sourced systems. Additionally, we provided an extensive ablation study of an image prior to design choices. Our system is equipped with free-to-use interfaces in the form of Web application and Telegram messenger bot. The pre-trained models are available on Hugging Face, and the source code is released under a permissive license enabling various, including commercial, applications of the developed technology. In future research, our goal is to investigate the potential of the latest image encoders. We plan to explore the development of more efficient UNet architectures for text-to-image tasks and focus on improving the understanding of textual prompts. Additionally, we aim to experiment with generating images at higher resolutions and to investigate new features extending the model: local image editing by a text prompt, attention reweighting, physics-based generation control, etc. The robustness against generating abusive content remains a crucial concern, warranting the exploration of real-time moderation layers or robust classifiers to mitigate undesirable, e.g. toxic or abusive, outputs. 8 Limitations The current system produces images that appear natural, however, additional research can be conducted to (1) enhance the semantic coherence between the input text and the generated image, and (2) to improve the absolute values of FID and image quality based on human evaluations. This paper is available on arxiv under CC BY 4.0 DEED license. This paper is available on arxiv under CC BY 4.0 DEED license. available on arxiv

Russian Scientists Unveil Open-Source Image Generator With a Groundbreaking Diffusion Method

Table of Links Abstract and 1. Introduction 2 Related Work 3 Preliminaries 4 Method 4.1 Key Sample and Joint Editing 4.2 Edit Propagation Via TokenFlow 5 Results 5.1 Qualitative Evaluation and 5.2 Quantitative Evaluation 5.3 Ablation Study 6 Discussion 7 Acknowledgement and References A Implementation Details 5.1 QUALITATIVE EVALUATION Fig. 6 provides a qualitative comparison of our method to prominent baselines; please refer to SM for the full videos. Our method (bottom row) outputs videos that better adhere to the edit prompt while maintaining temporal consistency of the resulting edited video, while other methods struggle to meet both these goals. Tune-A-Video (second row) inflates the 2D image model into a video model, and fine-tunes it to overfit the motion of the video; thus, it is suitable for short clips. For long videos it struggles to capture the motion resulting with meaningless edits, e.g., the shiny metal sculpture. Applying PnP for each frame independently (third row) results in exquisite edits adhering to the edit prompt but, as expected, lack any temporal consistency. The results of Gen-1 (fourth row) also suffer from some temporal inconsistencies (the beak of the origami stork changes color). Moreover, their frame quality is significantly worse than that of a text-to-image diffusion model. The edits of Text2Video-Zero and Fate-Zero (fifth and sixth row) suffer from severe jittering as these methods rely heavily on the extended attention mechanism to implicitly encourage consistency. The results of Rerender-a-Video exhibit notable long-range inconsistencies and artifacts arising primarily from their reliance on optical flow estimation for distant frames (e.g. keyframes), which is known to be sub-optimal (See our video results in the SM; when the wolf turns its head, the nose color changes). We provide qualitative comparison to Text2LIVE and to a RGB propagation baseline in the SM. 5.2 QUANTITATIVE EVALUATION We evaluate our method in terms of: (i) edit fidelity measured by computing the average similarity between the CLIP embedding (Radford et al., 2021) of each edited frame and the target text prompt; (ii) temporal consistency. Following Ceylan et al. (2023); Lai et al. (2018a), temporal consistency is measured by (a) computing the optical flow of the original video using Teed & Deng (2020), warping the edited frames according to it, and measuring the warping error, and (b) a user study; We adopt a Two-alternative Forced Choice (2AFC) protocol suggested in Kolkin et al. (2019); Park et al. (2020), where participants are shown the input video, ours and a baseline result, and are asked to determine which video is more temporally consistent and better preserves the motion of the original video. The survey consists of 2000-3000 judgments per baseline obtained using Amazon mechanical turk. We note that warpingerror could not be measured for Gen1 since their product platform does not output the same number of input frames. Table 1 compares our method to baselines. Our method achieves the highest CLI score, showing a good fit between the edited video and the input guidance prompt. Furthermore, our method has a low warping error, indicating temporally consistent results. We note that Re-rendera-Video optimizes for the warping error and uses optical flow to propagate the edit, and hence has the lowest warping error; However, this reliance on optical flow often creates artifacts and longrange inconsistencies which are not reflected in the warping error. Nonetheless, they are apparent in the user study, that shows users significantly favoured our method over all baselines in terms of temporal consistency. Additionally, we consider the reference baseline of passing the original video through the LDM auto-encoder without performing editing (LDM recon.). This baseline provides an upper bound on the temporal consistency achievable by LDM auto-encoder. As expected, the CLIP similarity of this baseline is poor as it does not involve any editing. However, this baseline does not achieve zero warp error either due to the imperfect reconstruction of the LDM auto-encoder, which hallucinates high-frequency information We further evaluate our correspondences and video representation by measuring the accuracy of video reconstruction using TokenFlow. Specifically, we reconstruct the video using the same pipeline of our editing method, only removing the keyframes editing part. Table 2 reports the PSNR and LPIPS distance of this reconstruction, compared to vanilla DDIM reconstruction. As seen, TokenFlow reconstruction slightly improves DDIM inversion, demonstrating robust frame representation. This improvement can be attributed to the keyframe randomization; It increases robustness to challenging frames since each frame is reconstructed from multiple other frames during the generation. Notably, our evaluation focuses on accurate correspondences within the feature space during generation, rather than RGB frame correspondences evaluation, which is not essential to our method. This paper is available on arxiv under CC BY 4.0 DEED DEED license. Authors:
(1) Michal Geyer, Weizmann Institute of Science and Indicates equal contribution;
(2) Omer Bar-Tal, Weizmann Institute of Science and Indicates equal contribution;
(3) Shai Bagon, Weizmann Institute of Science;
(4) Tali Dekel, Weizmann Institute of Science. Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2 Related Work 2 Related Work 3 Preliminaries 3 Preliminaries 4 Method 4 Method 4.1 Key Sample and Joint Editing 4.1 Key Sample and Joint Editing 4.2 Edit Propagation Via TokenFlow 4.2 Edit Propagation Via TokenFlow 5 Results 5 Results 5.1 Qualitative Evaluation and 5.2 Quantitative Evaluation 5.1 Qualitative Evaluation and 5.2 Quantitative Evaluation 5.3 Ablation Study 5.3 Ablation Study 6 Discussion 6 Discussion 7 Acknowledgement and References 7 Acknowledgement and References A Implementation Details A Implementation Details 5.1 QUALITATIVE EVALUATION Fig. 6 provides a qualitative comparison of our method to prominent baselines; please refer to SM for the full videos. Our method (bottom row) outputs videos that better adhere to the edit prompt while maintaining temporal consistency of the resulting edited video, while other methods struggle to meet both these goals. Tune-A-Video (second row) inflates the 2D image model into a video model, and fine-tunes it to overfit the motion of the video; thus, it is suitable for short clips. For long videos it struggles to capture the motion resulting with meaningless edits, e.g., the shiny metal sculpture. Applying PnP for each frame independently (third row) results in exquisite edits adhering to the edit prompt but, as expected, lack any temporal consistency. The results of Gen-1 (fourth row) also suffer from some temporal inconsistencies (the beak of the origami stork changes color). Moreover, their frame quality is significantly worse than that of a text-to-image diffusion model. The edits of Text2Video-Zero and Fate-Zero (fifth and sixth row) suffer from severe jittering as these methods rely heavily on the extended attention mechanism to implicitly encourage consistency. The results of Rerender-a-Video exhibit notable long-range inconsistencies and artifacts arising primarily from their reliance on optical flow estimation for distant frames (e.g. keyframes), which is known to be sub-optimal (See our video results in the SM; when the wolf turns its head, the nose color changes). We provide qualitative comparison to Text2LIVE and to a RGB propagation baseline in the SM. 5.2 QUANTITATIVE EVALUATION We evaluate our method in terms of: (i) edit fidelity measured by computing the average similarity between the CLIP embedding (Radford et al., 2021) of each edited frame and the target text prompt; (ii) temporal consistency. Following Ceylan et al. (2023); Lai et al. (2018a), temporal consistency is measured by (a) computing the optical flow of the original video using Teed & Deng (2020), warping the edited frames according to it, and measuring the warping error, and (b) a user study; We adopt a Two-alternative Forced Choice (2AFC) protocol suggested in Kolkin et al. (2019); Park et al. (2020), where participants are shown the input video, ours and a baseline result, and are asked to determine which video is more temporally consistent and better preserves the motion of the original video. The survey consists of 2000-3000 judgments per baseline obtained using Amazon mechanical turk. We note that warpingerror could not be measured for Gen1 since their product platform does not output the same number of input frames. Table 1 compares our method to baselines. Our method achieves the highest CLI score, showing a good fit between the edited video and the input guidance prompt. Furthermore, our method has a low warping error, indicating temporally consistent results. We note that Re-rendera-Video optimizes for the warping error and uses optical flow to propagate the edit, and hence has the lowest warping error; However, this reliance on optical flow often creates artifacts and longrange inconsistencies which are not reflected in the warping error. Nonetheless, they are apparent in the user study, that shows users significantly favoured our method over all baselines in terms of temporal consistency. Additionally, we consider the reference baseline of passing the original video through the LDM auto-encoder without performing editing (LDM recon.). This baseline provides an upper bound on the temporal consistency achievable by LDM auto-encoder. As expected, the CLIP similarity of this baseline is poor as it does not involve any editing. However, this baseline does not achieve zero warp error either due to the imperfect reconstruction of the LDM auto-encoder, which hallucinates high-frequency information We further evaluate our correspondences and video representation by measuring the accuracy of video reconstruction using TokenFlow. Specifically, we reconstruct the video using the same pipeline of our editing method, only removing the keyframes editing part. Table 2 reports the PSNR and LPIPS distance of this reconstruction, compared to vanilla DDIM reconstruction. As seen, TokenFlow reconstruction slightly improves DDIM inversion, demonstrating robust frame representation. This improvement can be attributed to the keyframe randomization; It increases robustness to challenging frames since each frame is reconstructed from multiple other frames during the generation. Notably, our evaluation focuses on accurate correspondences within the feature space during generation, rather than RGB frame correspondences evaluation, which is not essential to our method. This paper is available on arxiv under CC BY 4.0 DEED DEED license. This paper is available on arxiv under CC BY 4.0 DEED DEED license. available on arxiv Authors: (1) Michal Geyer, Weizmann Institute of Science and Indicates equal contribution; (2) Omer Bar-Tal, Weizmann Institute of Science and Indicates equal contribution; (3) Shai Bagon, Weizmann Institute of Science; (4) Tali Dekel, Weizmann Institute of Science. Authors: Authors: (1) Michal Geyer, Weizmann Institute of Science and Indicates equal contribution; (2) Omer Bar-Tal, Weizmann Institute of Science and Indicates equal contribution; (3) Shai Bagon, Weizmann Institute of Science; (4) Tali Dekel, Weizmann Institute of Science.

Qualitative Evaluation and Quantitative Evaluation: Comparing Our Method to Others

Table of Links Abstract and 1. Introduction 2 Related Work 3 Preliminaries 4 Method 4.1 Key Sample and Joint Editing 4.2 Edit Propagation Via TokenFlow 5 Results 5.1 Qualitative Evaluation and 5.2 Quantitative Evaluation 5.3 Ablation Study 6 Discussion 7 Acknowledgement and References A Implementation Details 5 RESULTS We evaluate our method on DAVIS videos (Pont-Tuset et al., 2017) and on Internet videos depicting animals, food, humans, and various objects in motion. The spatial resolution of the videos is 384× 672 or 512×512 pixels, and they consist of 40 to 200 frames. We use various text prompts on each video to obtain diverse editing results. Our evaluation dataset comprises of 61 text-video pairs. We utilize PnP-Diffusion (Tumanyan et al., 2023) as the frame editing method, and we use the same hyper-parameters for all our results. PnP-Diffusion may fail to accurately preserve the structure of each frame due to inaccurate DDIM inversion (see Fig. 3, middle column, right frame: the dog’s head is distorted). Our method improves robustness to this, as multiple frames contribute to the generation of each frame in the video. Our framework can be combined with any diffusion-based image editing technique that accurately preserves the structure of the images; results with different image editing techniques (e.g. Meng et al. (2022); Zhang & Agrawala (2023)) are available in the SM. Fig. 5 and 1 show sample frames from the edited videos. Our edits are temporally consistent and adhere to the edit prompt. The man’s head is changed to Van-Gogh or marble (top left); importantly, the man’s identity and the scene’s background are consistent throughout the video. The patterns of the polygonal wolf (bottom left) are the same across time: the body is consistently orange while the chest is blue. We refer the reader to the SM for implementation details and video results. Baselines. We compare our method to state-of-the-art, and concurrent works: (i) Fate-Zero (Qi et al., 2023) and (ii) Text2Video-Zero (Khachatryan et al., 2023b), that utilize a text-to-image model for video editing using self-attention inflation. (iii) Re-render a Video (Yang et al., 2023) that edits keyframes by adding optical flow optimization to self-attention inflation of an image model, and then propagates the edit from the keyframes to the rest of the video using an off-the-shelf propagation method. (iv) Tune-a-Video (Wu et al., 2022) that fine-tunes the text-to-image model on the given test video. (v) Gen-1 (Esser et al., 2023), a video diffusion model that was trained on a large-scale image and video dataset. (vi) Per-frame diffusion-based image editing baseline, PnP-Diffusion (Tumanyan et al., 2023). We additionally consider the two following baselines: (i) Text2LIVE (Bar-Tal et al., 2022) which utilize a layered video representation (NLA) (Kasten et al., 2021) and perform test-time training using CLIP losses. Note that NLA requires foreground/background separation masks and takes ∼10 hours to train. (ii) Applying PnP-Diffusion on a single keyframe and propagating the edit to the entire video using Jamriska et al. (2019). This paper is available on arxiv under CC BY 4.0 DEED DEED license. Authors:
(1) Michal Geyer, Weizmann Institute of Science and Indicates equal contribution;
(2) Omer Bar-Tal, Weizmann Institute of Science and Indicates equal contribution;
(3) Shai Bagon, Weizmann Institute of Science;
(4) Tali Dekel, Weizmann Institute of Science. Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2 Related Work 2 Related Work 3 Preliminaries 3 Preliminaries 4 Method 4 Method 4.1 Key Sample and Joint Editing 4.1 Key Sample and Joint Editing 4.2 Edit Propagation Via TokenFlow 4.2 Edit Propagation Via TokenFlow 5 Results 5 Results 5.1 Qualitative Evaluation and 5.2 Quantitative Evaluation 5.1 Qualitative Evaluation and 5.2 Quantitative Evaluation 5.3 Ablation Study 5.3 Ablation Study 6 Discussion 6 Discussion 7 Acknowledgement and References 7 Acknowledgement and References A Implementation Details A Implementation Details 5 RESULTS We evaluate our method on DAVIS videos (Pont-Tuset et al., 2017) and on Internet videos depicting animals, food, humans, and various objects in motion. The spatial resolution of the videos is 384× 672 or 512×512 pixels, and they consist of 40 to 200 frames. We use various text prompts on each video to obtain diverse editing results. Our evaluation dataset comprises of 61 text-video pairs. We utilize PnP-Diffusion (Tumanyan et al., 2023) as the frame editing method, and we use the same hyper-parameters for all our results. PnP-Diffusion may fail to accurately preserve the structure of each frame due to inaccurate DDIM inversion (see Fig. 3, middle column, right frame: the dog’s head is distorted). Our method improves robustness to this, as multiple frames contribute to the generation of each frame in the video. Our framework can be combined with any diffusion-based image editing technique that accurately preserves the structure of the images; results with different image editing techniques (e.g. Meng et al. (2022); Zhang & Agrawala (2023)) are available in the SM. Fig. 5 and 1 show sample frames from the edited videos. Our edits are temporally consistent and adhere to the edit prompt. The man’s head is changed to Van-Gogh or marble (top left); importantly, the man’s identity and the scene’s background are consistent throughout the video. The patterns of the polygonal wolf (bottom left) are the same across time: the body is consistently orange while the chest is blue. We refer the reader to the SM for implementation details and video results. Baselines. We compare our method to state-of-the-art, and concurrent works: (i) Fate-Zero (Qi et al., 2023) and (ii) Text2Video-Zero (Khachatryan et al., 2023b), that utilize a text-to-image model for video editing using self-attention inflation. (iii) Re-render a Video (Yang et al., 2023) that edits keyframes by adding optical flow optimization to self-attention inflation of an image model, and then propagates the edit from the keyframes to the rest of the video using an off-the-shelf propagation method. (iv) Tune-a-Video (Wu et al., 2022) that fine-tunes the text-to-image model on the given test video. (v) Gen-1 (Esser et al., 2023), a video diffusion model that was trained on a large-scale image and video dataset. (vi) Per-frame diffusion-based image editing baseline, PnP-Diffusion (Tumanyan et al., 2023). We additionally consider the two following baselines: (i) Text2LIVE (Bar-Tal et al., 2022) which utilize a layered video representation (NLA) (Kasten et al., 2021) and perform test-time training using CLIP losses. Note that NLA requires foreground/background separation masks and takes ∼10 hours to train. (ii) Applying PnP-Diffusion on a single keyframe and propagating the edit to the entire video using Jamriska et al. (2019). Baselines. This paper is available on arxiv under CC BY 4.0 DEED DEED license. This paper is available on arxiv under CC BY 4.0 DEED DEED license. available on arxiv Authors: (1) Michal Geyer, Weizmann Institute of Science and Indicates equal contribution; (2) Omer Bar-Tal, Weizmann Institute of Science and Indicates equal contribution; (3) Shai Bagon, Weizmann Institute of Science; (4) Tali Dekel, Weizmann Institute of Science. Authors: Authors: (1) Michal Geyer, Weizmann Institute of Science and Indicates equal contribution; (2) Omer Bar-Tal, Weizmann Institute of Science and Indicates equal contribution; (3) Shai Bagon, Weizmann Institute of Science; (4) Tali Dekel, Weizmann Institute of Science.

TokenFlow: Taking a Look at Our Results and Evaluating Our Methods

Table of Links Abstract and 1. Introduction 2 Related Work 3 Preliminaries 4 Method 4.1 Key Sample and Joint Editing 5 Results 5.1 Qualitative Evaluation and 5.2 Quantitative Evaluation 5.3 Ablation Study 6 Discussion 7 Acknowledgement and References A Implementation Details 4 METHOD This paper is available on arxiv under CC BY 4.0 DEED DEED license. Authors:
(1) Michal Geyer, Weizmann Institute of Science and Indicates equal contribution;
(2) Omer Bar-Tal, Weizmann Institute of Science and Indicates equal contribution;
(3) Shai Bagon, Weizmann Institute of Science;
(4) Tali Dekel, Weizmann Institute of Science. Table of Links Abstract and 1. Introduction Abstract and 1. Introduction 2 Related Work 2 Related Work 3 Preliminaries 3 Preliminaries 4 Method 4 Method 4.1 Key Sample and Joint Editing 4.1 Key Sample and Joint Editing 5 Results 5 Results 5.1 Qualitative Evaluation and 5.2 Quantitative Evaluation 5.1 Qualitative Evaluation and 5.2 Quantitative Evaluation 5.3 Ablation Study 5.3 Ablation Study 6 Discussion 6 Discussion 7 Acknowledgement and References 7 Acknowledgement and References A Implementation Details A Implementation Details 4 METHOD This paper is available on arxiv under CC BY 4.0 DEED DEED license. This paper is available on arxiv under CC BY 4.0 DEED DEED license. available on arxiv Authors: (1) Michal Geyer, Weizmann Institute of Science and Indicates equal contribution; (2) Omer Bar-Tal, Weizmann Institute of Science and Indicates equal contribution; (3) Shai Bagon, Weizmann Institute of Science; (4) Tali Dekel, Weizmann Institute of Science. Authors: Authors: (1) Michal Geyer, Weizmann Institute of Science and Indicates equal contribution; (2) Omer Bar-Tal, Weizmann Institute of Science and Indicates equal contribution; (3) Shai Bagon, Weizmann Institute of Science; (4) Tali Dekel, Weizmann Institute of Science.

How TokenFlow Works: A Look At Our Method

Short on Budget? Use These Design Methodologies to Build Your First MVP Without a Full-time Designer

Knappes Budget? Verwenden Sie diese Designmethoden, um Ihr erstes MVP ohne Vollzeit-Designer zu erstellen

¿Tiene poco presupuesto? Utilice estas metodologías de diseño para crear su primer MVP sin un diseñador a tiempo completo

Anggaran Terbatas? Gunakan Metodologi Desain Ini untuk Membangun MVP Pertama Anda Tanpa Desainer Penuh Waktu

Hai poco budget? Usa queste metodologie di progettazione per creare il tuo primo MVP senza un designer a tempo pieno

予算が足りない？フルタイムのデザイナーなしで最初のMVPを構築するには、これらのデザイン手法を活用

Mokuse na Budget? Salelá ba Méthodologies oyo ya Design mpo na kotonga MVP na yo ya liboso Sans Designer ya tango mobimba

Weinig budget? Gebruik deze ontwerpmethodologieën om uw eerste MVP te bouwen zonder een fulltime ontwerper

E shkurtër për buxhetin? Përdorni këto Metodologji projektimi për të ndërtuar MVP-në tuaj të parë pa një projektues me kohë të plotë

Дар бораи буҷет кӯтоҳ? Ин усулҳои тарроҳиро барои сохтани аввалин MVP-и худ бидуни тарроҳии пурравақт истифода баред

Xana u komile eka Mpimanyeto? Tirhisa Maendlelo Lawa Ya Dizayini Ku Aka MVP Ya Wena Yo Sungula Handle Ka Muendli Wa Nkarhi Hinkwawo

Mfutshane kuhlahlo-lwabiwo mali? Sebenzisa ezi Ndlela zoYilo ukwakha iMVP yakho yokuqala ngaphandle koMyili osisigxina

预算不足？使用这些设计方法，无需全职设计师即可构建您的第一个 MVP

Table of Links Abstract and 1. Introduction Background and Related Work


Threat Model


Robust Style Mimicry


Experimental Setup


Results
6.1 Main Findings: All Protections are Easily Circumvented
6.2 Analysis


Discussion and Broader Impact, Acknowledgements, and References A. Detailed Art Examples B. Robust Mimicry Generations C. Detailed Results D. Differences with Glaze Finetuning E. Findings on Glaze 2.0 F. Findings on Mist v2 G. Methods for Style Mimicry H. Existing Style Mimicry Protections I. Robust Mimicry Methods J. Experimental Setup K. User Study L. Compute Resources F Findings on Mist v2 After responsibly disclosing our work to defense developers, authors from Mist brought to our attention the recent release of their latest Mist v2 with improved resilience (Zheng et al., 2023). As we did with Glaze v2.0 (see Section E), we reproduced some of our experiments with the latest protections to verify the success of robust mimicry. Their original implementation still uses the outdated version 1.5 of Stable Diffusion. We change to SD 2.1 to match our previous experiments[6]. Our findings, as we saw with Glaze v2.0, highlight that improved protections are still not effective against low-effort robust mimicry. More specifically, the latest version of Mist: introduces visible perturbations over the images. See Figure 23


does not improve protections against robust mimicry. See Figure 24


creates protection that are easily removable with Noisy Upscaling. See Figure 25. Authors:
(1) Robert Honig, ETH Zurich (robert.hoenig@inf.ethz.ch);
(2) Javier Rando, ETH Zurich (javier.rando@inf.ethz.ch);
(3) Nicholas Carlini, Google DeepMind;
(4) Florian Tramer, ETH Zurich (florian.tramer@inf.ethz.ch). This paper is available on arxiv under CC BY 4.0 license. [6] Both models share the same encoder for which protections are optimized. Table of Links Abstract and 1. Introduction Abstract and 1. Introduction Background and Related Work Threat Model Robust Style Mimicry Experimental Setup Results
6.1 Main Findings: All Protections are Easily Circumvented
6.2 Analysis Discussion and Broader Impact, Acknowledgements, and References Background and Related Work Background and Related Work Background and Related Work Threat Model Threat Model Threat Model Robust Style Mimicry Robust Style Mimicry Robust Style Mimicry Experimental Setup Experimental Setup Experimental Setup Results 6.1 Main Findings: All Protections are Easily Circumvented 6.2 Analysis Results Results 6.1 Main Findings: All Protections are Easily Circumvented 6.1 Main Findings: All Protections are Easily Circumvented 6.2 Analysis 6.2 Analysis Discussion and Broader Impact, Acknowledgements, and References Discussion and Broader Impact, Acknowledgements, and References Discussion and Broader Impact, Acknowledgements, and References A. Detailed Art Examples A. Detailed Art Examples B. Robust Mimicry Generations B. Robust Mimicry Generations C. Detailed Results C. Detailed Results D. Differences with Glaze Finetuning D. Differences with Glaze Finetuning E. Findings on Glaze 2.0 E. Findings on Glaze 2.0 F. Findings on Mist v2 F. Findings on Mist v2 G. Methods for Style Mimicry G. Methods for Style Mimicry H. Existing Style Mimicry Protections H. Existing Style Mimicry Protections I. Robust Mimicry Methods I. Robust Mimicry Methods J. Experimental Setup J. Experimental Setup K. User Study K. User Study L. Compute Resources L. Compute Resources F Findings on Mist v2 After responsibly disclosing our work to defense developers, authors from Mist brought to our attention the recent release of their latest Mist v2 with improved resilience (Zheng et al., 2023). As we did with Glaze v2.0 (see Section E), we reproduced some of our experiments with the latest protections to verify the success of robust mimicry. Their original implementation still uses the outdated version 1.5 of Stable Diffusion. We change to SD 2.1 to match our previous experiments[6]. Our findings, as we saw with Glaze v2.0, highlight that improved protections are still not effective against low-effort robust mimicry. More specifically, the latest version of Mist: introduces visible perturbations over the images. See Figure 23 does not improve protections against robust mimicry. See Figure 24 creates protection that are easily removable with Noisy Upscaling. See Figure 25. introduces visible perturbations over the images. See Figure 23 introduces visible perturbations over the images. See Figure 23 does not improve protections against robust mimicry. See Figure 24 does not improve protections against robust mimicry. See Figure 24 creates protection that are easily removable with Noisy Upscaling. See Figure 25. creates protection that are easily removable with Noisy Upscaling. See Figure 25. Authors: (1) Robert Honig, ETH Zurich (robert.hoenig@inf.ethz.ch); (2) Javier Rando, ETH Zurich (javier.rando@inf.ethz.ch); (3) Nicholas Carlini, Google DeepMind; (4) Florian Tramer, ETH Zurich (florian.tramer@inf.ethz.ch). Authors: Authors: (1) Robert Honig, ETH Zurich (robert.hoenig@inf.ethz.ch); (2) Javier Rando, ETH Zurich (javier.rando@inf.ethz.ch); (3) Nicholas Carlini, Google DeepMind; (4) Florian Tramer, ETH Zurich (florian.tramer@inf.ethz.ch). This paper is available on arxiv under CC BY 4.0 license. This paper is available on arxiv under CC BY 4.0 license. available on arxiv available on arxiv [6] Both models share the same encoder for which protections are optimized.

Mist v2 Fails to Defend Against Robust Mimicry Methods Like Noisy Upscaling

Table of Links Abstract and 1. Introduction Background and Related Work


Threat Model


Robust Style Mimicry


Experimental Setup


Results
6.1 Main Findings: All Protections are Easily Circumvented
6.2 Analysis


Discussion and Broader Impact, Acknowledgements, and References A. Detailed Art Examples B. Robust Mimicry Generations C. Detailed Results D. Differences with Glaze Finetuning E. Findings on Glaze 2.0 F. Findings on Mist v2 G. Methods for Style Mimicry H. Existing Style Mimicry Protections I. Robust Mimicry Methods J. Experimental Setup K. User Study L. Compute Resources 6.1 Main Findings: All Protections are Easily Circumvented We find that all existing protective tools create a false sense of security and leave artists vulnerable to style mimicry. Indeed, our best robust mimicry methods produce images that are, on average, indistinguishable from baseline mimicry attempts using unprotected art. Since many of our simple mimicry methods only use tools that were available before the protections were released, style forgers may have already circumvented these protections since their inception. Noisy upscaling is the most effective method for robust mimicry, with a median success rate above 40% for each protection tool (recall that 50% success indicates that the robust method is indistinguishable from a mimicry using unprotected images). This method only requires preprocessing images and black-box access to the model via a finetuning API. Other simple preprocessing methods like Gaussian noising or DiffPure also significantly reduce the effectiveness of protections. The more complex white-box method IMPRESS++ does not provide significant advantages. Sample generations for each method are in Appendix B. A style forger does not have to use a single robust mimicry method, but can test all of them and select the most successful. This “best-of-4” approach always beats the baseline mimicry method over unprotected images (which attempts a single method and not four) for all protections. Appendix A shows images at each step of the robust mimicry process (i.e., protections, preprocessing, and sampling). Appendix B shows example generations for each protection and mimicry method. Appendix C has detailed success rates broken down per artist, for both image style and quality. Authors:
(1) Robert Honig, ETH Zurich (robert.hoenig@inf.ethz.ch);
(2) Javier Rando, ETH Zurich (javier.rando@inf.ethz.ch);
(3) Nicholas Carlini, Google DeepMind;
(4) Florian Tramer, ETH Zurich (florian.tramer@inf.ethz.ch). This paper is available on arxiv under CC BY 4.0 license. Table of Links Abstract and 1. Introduction Abstract and 1. Introduction Background and Related Work Threat Model Robust Style Mimicry Experimental Setup Results
6.1 Main Findings: All Protections are Easily Circumvented
6.2 Analysis Discussion and Broader Impact, Acknowledgements, and References Background and Related Work Background and Related Work Background and Related Work Threat Model Threat Model Threat Model Robust Style Mimicry Robust Style Mimicry Robust Style Mimicry Experimental Setup Experimental Setup Experimental Setup Results 6.1 Main Findings: All Protections are Easily Circumvented 6.2 Analysis Results Results 6.1 Main Findings: All Protections are Easily Circumvented 6.1 Main Findings: All Protections are Easily Circumvented 6.2 Analysis 6.2 Analysis Discussion and Broader Impact, Acknowledgements, and References Discussion and Broader Impact, Acknowledgements, and References Discussion and Broader Impact, Acknowledgements, and References A. Detailed Art Examples A. Detailed Art Examples B. Robust Mimicry Generations B. Robust Mimicry Generations C. Detailed Results C. Detailed Results D. Differences with Glaze Finetuning D. Differences with Glaze Finetuning E. Findings on Glaze 2.0 E. Findings on Glaze 2.0 F. Findings on Mist v2 F. Findings on Mist v2 G. Methods for Style Mimicry G. Methods for Style Mimicry H. Existing Style Mimicry Protections H. Existing Style Mimicry Protections I. Robust Mimicry Methods I. Robust Mimicry Methods J. Experimental Setup J. Experimental Setup K. User Study K. User Study L. Compute Resources L. Compute Resources 6.1 Main Findings: All Protections are Easily Circumvented We find that all existing protective tools create a false sense of security and leave artists vulnerable to style mimicry. Indeed, our best robust mimicry methods produce images that are, on average, indistinguishable from baseline mimicry attempts using unprotected art. Since many of our simple mimicry methods only use tools that were available before the protections were released, style forgers may have already circumvented these protections since their inception. Noisy upscaling is the most effective method for robust mimicry, with a median success rate above 40% for each protection tool (recall that 50% success indicates that the robust method is indistinguishable from a mimicry using unprotected images). This method only requires preprocessing images and black-box access to the model via a finetuning API. Other simple preprocessing methods like Gaussian noising or DiffPure also significantly reduce the effectiveness of protections. The more complex white-box method IMPRESS++ does not provide significant advantages. Sample generations for each method are in Appendix B. A style forger does not have to use a single robust mimicry method, but can test all of them and select the most successful. This “best-of-4” approach always beats the baseline mimicry method over unprotected images (which attempts a single method and not four) for all protections. Appendix A shows images at each step of the robust mimicry process (i.e., protections, preprocessing, and sampling). Appendix B shows example generations for each protection and mimicry method. Appendix C has detailed success rates broken down per artist, for both image style and quality. Authors: (1) Robert Honig, ETH Zurich (robert.hoenig@inf.ethz.ch); (2) Javier Rando, ETH Zurich (javier.rando@inf.ethz.ch); (3) Nicholas Carlini, Google DeepMind; (4) Florian Tramer, ETH Zurich (florian.tramer@inf.ethz.ch). Authors: Authors: (1) Robert Honig, ETH Zurich (robert.hoenig@inf.ethz.ch); (2) Javier Rando, ETH Zurich (javier.rando@inf.ethz.ch); (3) Nicholas Carlini, Google DeepMind; (4) Florian Tramer, ETH Zurich (florian.tramer@inf.ethz.ch). This paper is available on arxiv under CC BY 4.0 license. This paper is available on arxiv under CC BY 4.0 license. available on arxiv available on arxiv

Art Protection Tools Fail Against Advanced AI Mimicry Methods

In this article, I will show you how you can test a restful “Get” method using the Robot Framework. If you are new to Robot Framework, please read the doc from the website to find out the setup guide. You can install Robot Framework using “pip install robotframework” in your command line. Please note that you must install pip on your system before running this command. To write test cases in Robot Framework, you should create a file with .robot extension, then you can write your scenario in the file to be run. Each .robot file usually contains three main parts: Settings, Variables, and Test cases like the below image. In settings, you can import the libraries, suit setup, and documentation. In the variables part, you can put your variables, and the scenarios are written in the test cases section. To test a simple Get request, we should import “RequestsLibrary” in settings because we want to make a request. In test cases, first, we should define a keyword for our test scenario, and then we should write the test steps in the keyword’s scope. See the picture below: Let’s automate a simple Get request together. Suppose we want to write a test case for the following curl. This test case will call the endpoint and assert whether: 1- The endpoint responds with 200.

2- The response body contains “Google“ word.

3- The value of Content-Type is “application/json” in response headers. curl --location 'https://api.restful-api.dev/objects' --header 'Content-Type: application/json' --data '' This is a simple request that returns some mobile names like the below picture: In this step, I am going to write a Robot Framework code to call the API and assert the response. First, I define a variable to hold the base address of the API. Second, I put the URI of the API in another variable. You may ask why I put the URI in another variable. You will get your answer in the creating get request step. As I mentioned earlier, it it important to choose a name for our keyword. I choose get_request for the keyword part and then write the codes in the keyword’s scope. To create a restful request, we should establish a session first. Robot Framework has a built-in function to establish a session, it is “Create Session”: Create Session    session_name    ${base_address} Then we can make a get request by using “Get Request“ built-in function: Get Request    session_name      ${uri} Get Request is a built-in function in Robot Framework that gets at least two arguments. This is why I put the URI in a separate variable. The code above calls the API and gets the response, so we must hold the response in a variable like: ${resp}=    Get Request    session_name     ${uri} Going through the steps that I have mentioned so far, will make get request, call the API, and save the response in the “resp” variable. Now we have the response and are ready to do assertions. One of the assertions in Robot Framework is “Should Be Equal“ built-in function. It gets two arguments and compares them. If they are the same, the assertion will return true. Should Be Equal    ${resp.status_code}    200 Note that integers in Robot Framework are considered as strings and ${resp.status_code} is returned as an integer. For example, in the code line above, 200 is an integer but Robot Framework considers it as a string so you should convert ${resp.status_code} to a string to compare it with 200. The built-in function for converting an integer to a string is “Convert To String“. Through the response of the API, we can get the response headers and response body by ${resp.headers} and ${resp.content} respectively. Once you have the body of the response and the headers of the response, you can assert them as well. For instance, suppose you want to check whether the response body contains a specific word “x“. It is possible to make this assertion by using a built-in function “Should Contain“. Should Contain    ${resp.content}    x Please note that ${resp.content} is the body of the response in bytes, so you should convert the word “x“ to byte to do the assertion. The built-in function for converting to byte is “Convert To Bytes“. To assert the headers, we should capture all headers from the response. ${resp.headers} will capture all response headers as a dictionary. As a result, to assert a specific value, we should get the value from the headers dictionary by using the key(we know that the dictionary holds keys and the related values; we can access any value by the related key). This can be done by “Get From Dictionary“ built-in function. ${content_type_value}=    Get From Dictionary     ${resp.headers}     Content-Type Now we have the value of Content-Type header and we can assert it. Should Be Equal    ${content_type_value}    application/json In the picture below, you can see all the assertions at a glance. Resources: https://restful-api.dev/ https://robotframework.org/ Image header: by brgfx on freepik In this article, I will show you how you can test a restful “Get” method using the Robot Framework. If you are new to Robot Framework, please read the doc from the website to find out the setup guide. You can install Robot Framework using “pip install robotframework” in your command line. Please note that you must install pip on your system before running this command. doc To write test cases in Robot Framework, you should create a file with .robot extension, then you can write your scenario in the file to be run. Each .robot file usually contains three main parts: Settings, Variables, and Test cases like the below image. In settings, you can import the libraries, suit setup, and documentation. In the variables part, you can put your variables, and the scenarios are written in the test cases section. To test a simple Get request, we should import “RequestsLibrary” in settings because we want to make a request. In test cases, first, we should define a keyword for our test scenario, and then we should write the test steps in the keyword’s scope. See the picture below: Let’s automate a simple Get request together. Suppose we want to write a test case for the following curl. This test case will call the endpoint and assert whether: 1- The endpoint responds with 200.

2- The response body contains “Google“ word.

3- The value of Content-Type is “application/json” in response headers. 1- The endpoint responds with 200.

2- The response body contains “Google“ word.

3- The value of Content-Type is “application/json” in response headers. curl --location 'https://api.restful-api.dev/objects' --header 'Content-Type: application/json' --data '' This is a simple request that returns some mobile names like the below picture: In this step, I am going to write a Robot Framework code to call the API and assert the response. First, I define a variable to hold the base address of the API. Second, I put the URI of the API in another variable. You may ask why I put the URI in another variable. You will get your answer in the creating get request step. As I mentioned earlier, it it important to choose a name for our keyword. I choose get_request for the keyword part and then write the codes in the keyword’s scope. To create a restful request, we should establish a session first. Robot Framework has a built-in function to establish a session, it is “Create Session”: Create Session    session_name    ${base_address} Create Session    session_name    ${base_address} Then we can make a get request by using “Get Request“ built-in function: Get Request    session_name      ${uri} Get Request    session_name      ${uri} Get Request is a built-in function in Robot Framework that gets at least two arguments. This is why I put the URI in a separate variable. The code above calls the API and gets the response, so we must hold the response in a variable like: ${resp}=    Get Request    session_name     ${uri} ${resp}=    Get Request    session_name     ${uri} Going through the steps that I have mentioned so far, will make get request, call the API, and save the response in the “resp” variable. Now we have the response and are ready to do assertions. One of the assertions in Robot Framework is “Should Be Equal“ built-in function. It gets two arguments and compares them. If they are the same, the assertion will return true. Should Be Equal    ${resp.status_code}    200 Should Be Equal    ${resp.status_code}    200 Note that integers in Robot Framework are considered as strings and ${resp.status_code} is returned as an integer. For example, in the code line above, 200 is an integer but Robot Framework considers it as a string so you should convert ${resp.status_code} to a string to compare it with 200. The built-in function for converting an integer to a string is “Convert To String“. Through the response of the API, we can get the response headers and response body by ${resp.headers} and ${resp.content} respectively. Once you have the body of the response and the headers of the response, you can assert them as well. For instance, suppose you want to check whether the response body contains a specific word “x“. It is possible to make this assertion by using a built-in function “Should Contain“. Should Contain    ${resp.content}    x Should Contain    ${resp.content}    x Please note that ${resp.content} is the body of the response in bytes, so you should convert the word “x“ to byte to do the assertion. The built-in function for converting to byte is “Convert To Bytes“. To assert the headers, we should capture all headers from the response. ${resp.headers} will capture all response headers as a dictionary. As a result, to assert a specific value, we should get the value from the headers dictionary by using the key(we know that the dictionary holds keys and the related values; we can access any value by the related key). This can be done by “Get From Dictionary“ built-in function. ${content_type_value}=    Get From Dictionary     ${resp.headers}     Content-Type ${content_type_value}=    Get From Dictionary     ${resp.headers}     Content-Type Now we have the value of Content-Type header and we can assert it. Should Be Equal    ${content_type_value}    application/json Should Be Equal    ${content_type_value}    application/json In the picture below, you can see all the assertions at a glance. Resources: https://restful-api.dev/ https://restful-api.dev/ https://robotframework.org/ https://robotframework.org/ Image header: by brgfx on freepik Image header: by brgfx on freepik brgfx

Stories, News & Mentions about WETHOD

Wethod’s Stories on HackerNoon

Wethod's News & Mentions