Table of Links Abstract and 1. Introduction Background and Related Work


Threat Model


Robust Style Mimicry


Experimental Setup


Results
6.1 Main Findings: All Protections are Easily Circumvented
6.2 Analysis


Discussion and Broader Impact, Acknowledgements, and References A. Detailed Art Examples B. Robust Mimicry Generations C. Detailed Results D. Differences with Glaze Finetuning E. Findings on Glaze 2.0 F. Findings on Mist v2 G. Methods for Style Mimicry H. Existing Style Mimicry Protections I. Robust Mimicry Methods J. Experimental Setup K. User Study L. Compute Resources 6.2 Analysis We now discuss key insights and lessons learned from these results. Glaze protections break down without any circumvention attempt. Results for Glaze without robust mimicry (see “Naive mimicry” row in Figure 4) show that the tool’s protections are often ineffective. Without any robustness intervention, 30% of the images generated with our off-the-shelf finetuning are rated as better than the baseline results using only unprotected images. This contrasts with Glaze’s original evaluation, which claimed a success rate of at most 10% for robust mimicry.[4] This difference is likely due to the protection’s brittleness to slight changes in the finetuning setup (as we illustrated in Section 4.1). With our best robust mimicry method (noisy upscaling) the median success rate across artists rises further to 40%, and our best-of-4 strategy yields results indistinguishable from the baseline for a majority of artists. Robust mimicry works for contemporary and historical artists alike. Shan et al. (2023b) note that one of IMPRESS’ main limitations is that “purification has a limited effect when tested on artists that are not well-known historical artists already embedded in original training data”. Yet, we find that our best-performing robust mimicry method—Noisy Upscaling—has a similar success rate for historical artists (42.2%) and contemporary artists with little representation in the model’s training set (43.5%). Protections are highly non-uniform across artists. As we observe from Figure 4, the effectiveness of protections varies significantly across artists: the least vulnerable artist (left-most whisker) enjoys much stronger mimicry protections than the median artist or the most vulnerable artist (right-most whisker). We find that robust mimicry is the least successful for artists where the baseline mimicry from unprotected images gives poor results to begin with (cf. results for artist A1 in Appendix C and Appendix K.1). Yet, since existing tools do not provide artists with a way to check how vulnerable they are, these tools still provide a false sense of security for all artists. This highlights an inherent asymmetry between protection tools and mimicry methods: protections should hold for all artists alike, while a mimicry method might successfully target only specific artists. Robust mimicry failures still remove protection artifacts. We manually checked the cases where all annotators ranked mimicry from unprotected art as better than robust mimicry with Noisy Upscaling. Figure 5 shows two examples. We find that in many instances, the model fails to mimic the style accurately even from unprotected art. In these cases, robust mimicry is still able to generate clear images that are similar to unprotected mimicry, but neither matches the original style well. Authors:
(1) Robert Honig, ETH Zurich (robert.hoenig@inf.ethz.ch);
(2) Javier Rando, ETH Zurich (javier.rando@inf.ethz.ch);
(3) Nicholas Carlini, Google DeepMind;
(4) Florian Tramer, ETH Zurich (florian.tramer@inf.ethz.ch). This paper is available on arxiv under CC BY 4.0 license. [4] The original evaluation in Glaze directly asks annotators whether a mimicry is successful or not, rather than a binary comparison between a robust mimicry and a baseline mimicry as in our setup. Shan et al. (2023a) report that mimicry fails in 4% of cases for unprotected images, and succeeds in 6% of cases for protected images. This bounds the success rate for robust mimicry—according to our definition in Equation (1)—by at most 10%. Table of Links Abstract and 1. Introduction Abstract and 1. Introduction Background and Related Work Threat Model Robust Style Mimicry Experimental Setup Results
6.1 Main Findings: All Protections are Easily Circumvented
6.2 Analysis Discussion and Broader Impact, Acknowledgements, and References Background and Related Work Background and Related Work Background and Related Work Threat Model Threat Model Threat Model Robust Style Mimicry Robust Style Mimicry Robust Style Mimicry Experimental Setup Experimental Setup Experimental Setup Results 6.1 Main Findings: All Protections are Easily Circumvented 6.2 Analysis Results Results 6.1 Main Findings: All Protections are Easily Circumvented 6.1 Main Findings: All Protections are Easily Circumvented 6.2 Analysis 6.2 Analysis Discussion and Broader Impact, Acknowledgements, and References Discussion and Broader Impact, Acknowledgements, and References Discussion and Broader Impact, Acknowledgements, and References A. Detailed Art Examples A. Detailed Art Examples B. Robust Mimicry Generations B. Robust Mimicry Generations C. Detailed Results C. Detailed Results D. Differences with Glaze Finetuning D. Differences with Glaze Finetuning E. Findings on Glaze 2.0 E. Findings on Glaze 2.0 F. Findings on Mist v2 F. Findings on Mist v2 G. Methods for Style Mimicry G. Methods for Style Mimicry H. Existing Style Mimicry Protections H. Existing Style Mimicry Protections I. Robust Mimicry Methods I. Robust Mimicry Methods J. Experimental Setup J. Experimental Setup K. User Study K. User Study L. Compute Resources L. Compute Resources 6.2 Analysis We now discuss key insights and lessons learned from these results. Glaze protections break down without any circumvention attempt. Results for Glaze without robust mimicry (see “Naive mimicry” row in Figure 4) show that the tool’s protections are often ineffective. Without any robustness intervention, 30% of the images generated with our off-the-shelf finetuning are rated as better than the baseline results using only unprotected images. This contrasts with Glaze’s original evaluation, which claimed a success rate of at most 10% for robust mimicry.[4] This difference is likely due to the protection’s brittleness to slight changes in the finetuning setup (as we illustrated in Section 4.1). With our best robust mimicry method (noisy upscaling) the median success rate across artists rises further to 40%, and our best-of-4 strategy yields results indistinguishable from the baseline for a majority of artists. Glaze protections break down without any circumvention attempt. Robust mimicry works for contemporary and historical artists alike. Shan et al. (2023b) note that one of IMPRESS’ main limitations is that “purification has a limited effect when tested on artists that are not well-known historical artists already embedded in original training data”. Yet, we find that our best-performing robust mimicry method—Noisy Upscaling—has a similar success rate for historical artists (42.2%) and contemporary artists with little representation in the model’s training set (43.5%). Robust mimicry works for contemporary and historical artists alike. Protections are highly non-uniform across artists. As we observe from Figure 4, the effectiveness of protections varies significantly across artists: the least vulnerable artist (left-most whisker) enjoys much stronger mimicry protections than the median artist or the most vulnerable artist (right-most whisker). We find that robust mimicry is the least successful for artists where the baseline mimicry from unprotected images gives poor results to begin with (cf. results for artist A1 in Appendix C and Appendix K.1). Yet, since existing tools do not provide artists with a way to check how vulnerable they are, these tools still provide a false sense of security for all artists. This highlights an inherent asymmetry between protection tools and mimicry methods: protections should hold for all artists alike, while a mimicry method might successfully target only specific artists. Protections are highly non-uniform across artists. Robust mimicry failures still remove protection artifacts . We manually checked the cases where all annotators ranked mimicry from unprotected art as better than robust mimicry with Noisy Upscaling. Figure 5 shows two examples. We find that in many instances, the model fails to mimic the style accurately even from unprotected art. In these cases, robust mimicry is still able to generate clear images that are similar to unprotected mimicry, but neither matches the original style well. Robust mimicry failures still remove protection artifacts Authors: (1) Robert Honig, ETH Zurich (robert.hoenig@inf.ethz.ch); (2) Javier Rando, ETH Zurich (javier.rando@inf.ethz.ch); (3) Nicholas Carlini, Google DeepMind; (4) Florian Tramer, ETH Zurich (florian.tramer@inf.ethz.ch). Authors: Authors: (1) Robert Honig, ETH Zurich (robert.hoenig@inf.ethz.ch); (2) Javier Rando, ETH Zurich (javier.rando@inf.ethz.ch); (3) Nicholas Carlini, Google DeepMind; (4) Florian Tramer, ETH Zurich (florian.tramer@inf.ethz.ch). This paper is available on arxiv under CC BY 4.0 license. This paper is available on arxiv under CC BY 4.0 license. available on arxiv available on arxiv [4] The original evaluation in Glaze directly asks annotators whether a mimicry is successful or not, rather than a binary comparison between a robust mimicry and a baseline mimicry as in our setup. Shan et al. (2023a) report that mimicry fails in 4% of cases for unprotected images, and succeeds in 6% of cases for protected images. This bounds the success rate for robust mimicry—according to our definition in Equation (1)—by at most 10%.

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

New Findings Show All Major Art Protection Tools Are Vulnerable to AI Forgery

Art Protection Tools Fail Against Advanced AI Mimicry Methods

Read My Stories

Too Long; Didn't Read

Make resilience your competitive advantage

The Struggle to Stop AI from Imitating Human Artists' Styles

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

User Study for Evaluating Robust Style Mimicry

New Research Reveals Vulnerabilities in Popular Art Protection Tools Against AI Theft

Why AI Style Protections Fall Short Against Advanced Mimicry Techniques

How AI Forgers Bypass Style Protections to Mimic Artists' Work

New Study Shows AI Can Now Mimic Art Styles More Accurately Than Ever

User Study for Evaluating Robust Style Mimicry

New Research Reveals Vulnerabilities in Popular Art Protection Tools Against AI Theft

Why AI Style Protections Fall Short Against Advanced Mimicry Techniques

How AI Forgers Bypass Style Protections to Mimic Artists' Work

New Study Shows AI Can Now Mimic Art Styles More Accurately Than Ever

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps