Kudzidza Kutsva Kunoratidza AI Zvino Inogona Kutevedzera Madhirowa Masitaira Zvakanyanya Kupfuura

Table of Links

Abstract uye 1. Nhanganyaya

2. Background uye Related Basa

3. Threat Model

4. Robust Style Mimicry

5. Experimental Setup

6. Results

   6.1 Yakanyanya Kuwanikwa: Yese Dziviriro Inotenderedzwa Zviri Nyore

   6.2 Ongororo

7. Hurukuro uye Kuwedzera Kwakawedzera, Kutenda, uye References

A. Detailed Art Mienzaniso

B. Robust Mimicry Generations

C. Detailed Results

D. Misiyano neGlaze Finetuning

E. Zvakawanikwa paGlaze 2.0

F. Zvakawanikwa paMist v2

G. Nzira dzeChimiro Mimicry

H. Existing Style Mimicry Dziviriro

I. Robust Mimicry Methods

J. Experimental Setup

K. Kudzidza Kwemushandisi

L. Compute Resources

4 Robust Style Mimicry

Isu tinoti dhizaini yekutevedzera yakasimba kana ichigona kutevedzera dhizaini ichishandisa mifananidzo yakachengetedzwa chete. Nepo nzira dzekutevedzera kwakasimba dzakatotaurwa, tinocherekedza huwandu hwezvipimo munzira idzi uye ongororo yavo muChikamu 4.1. Isu tinobva tapa isu pachedu nzira (Chikamu 4.3) uye ongororo (Chikamu 5) inogadzirisa izvi zvisingakwanisi.

4.1 Kuganhurirwa Kwekare Kwakasimba Kutevedzera Nzira uye Yekuongorora Kwadzo

(1) Mamwe edziviriro ekutevedzera haawanze pane ese finetuning seti . Matsotsi mazhinji ane vavariro dzisina kunaka sezvo vasingateereri zvikumbiro zvemaartist zvechokwadi zvekusashandisa hunyanzvi hwavo kugadzira AI (Heikkila¨, 2022). Dziviriro yakabudirira inofanirwa kuramba kuedza kunzvenga kubva kune mugadziri ane zviwanikwa anogona kuedza maturusi akasiyana. Asi, mukuedza kwekutanga, takaona kuti Glaze (Shan et al., 2023a) akaita zvakanyanya zvakanyanya kupfuura zvaitaurwa muongororo yekutanga, kunyangwe asati aedza kunzvenga. Mushure mekukurukurirana nevanyori veGlaze, takawana misiyano midiki pakati peiyo off-the-sherufu finetuning script, uye iyo yakashandiswa mukuongorora kwekutanga kweGlaze (iyo vanyori vakagovana nesu).[1] Iyi misiyano midiki mukunatsiridza inokwana kudzikisira zvakanyanya kudzivirira kweGlaze (ona Mufananidzo 2 wemienzaniso yemhando yepamusoro). Sezvo yedu ye-off-the-sherufu finetuning script yakanga isina kugadzirwa kuti idarike zvidziviriro zvekutevedzera, izvi zvawanikwa zvinotoratidza nezvekudzivirirwa kwepamusoro uye kwakaoma kunopihwa nemidziyo iripo: maartist haana simba pamusoro peiyo finetuning script kana hyperparameter yaizoshandiswa nefori, saka dziviriro. inofanira kuva yakasimba pane idzi sarudzo.

(2) Kuedza kutevedzera kwakasimba kuriko kuri kudiki. Kuongororwa kwedziviriro kusati kwatadza kuratidza kugona kwevagadziri vane hunyanzvi, vanoshandisa nzira dzemazuva ano (kunyangwe dzisiri pasherufu). Semuenzaniso, Mist (Liang et al., 2023) anoongorora achipokana neDiffPure kucheneswa achishandisa yekare uye yakaderera-resolution yekuchenesa modhi. Tichishandisa DiffPure neazvino modhi, tinoona kuvandudzwa kwakakosha. Glaze (Shan et al., 2023a) haiongororwe maererano neshanduro ipi neipi yeDiffPure, asi inodzivirira kudzivirirwa kubva kuCompressed Upscaling, iyo inotanga kudzvanya chifananidzo neJPEG yozoisimudza nemuenzaniso wakazvipira. Asi, isu ticharatidza kuti nekungochinjanisa iyo JPEG compression neGaussian ruzha, tinogadzira Noisy Upscaling seyakasiyana inobudirira zvakanyanya pakubvisa kutevedzera kudzivirira (ona Mufananidzo 26 wekuenzanisa pakati penzira mbiri idzi).

(3) Ongororo iripo haina kukwana. Kuenzanisa kusimba kwedziviriro yepakutanga kunonetsa nekuti ongororo dzekutanga dzinoshandisa seti dzakasiyana dzevaimbi, kukurumidza, uye kugadzirisa. Uyezve, kumwe kuongorora kunoenderana neotomatiki metrics (semuenzaniso, CLIP kufanana) iyo isingavimbike pakuyera maitiro ekutevedzera (Shan et al., 2023a,b). Nekuda kwehutete hwenzira dzekudzivirira uye kuzviisa pasi kweyekutevedzera ongororo, tinotenda kuti kuongororwa kwakabatana kunodiwa.

4.2 Ongororo Yakabatana uye Yakasimba yeRobust Mimicry Methods

Kuti tigadzirise zvipimo zvakaunzwa muChikamu 4.1, isu tinosuma yakabatana yekuongorora protocol kuti tiongorore nekuvimbika kuti dziviriro iripo inofambiswa sei kubva kune dzakasiyana siyana dzakapusa uye dzakasimba dzakasimba dzekutevedzera. Mhinduro dzedu kune imwe neimwe yezvipimo zvakaverengerwa pamusoro ndeiyi: (1) Anorwisa anoshandisa yakakurumbira "off-the-sherufu" finetuning script yeiyo yakasimba yakavhurika-sosi modhi iyo yese dziviriro inoti inoshanda kune: Yakagadzikana Diffusion 2.1. Iyi finetuning script inosarudzwa yakazvimirira pane chero yedziviriro iyi, uye tinoitora sebhokisi dema. (2) Isu tinogadzira nzira ina dzakasimba dzekutevedzera, dzinotsanangurwa muChikamu 4.3. Isu tinoisa pamberi kuve nyore uye nyore kushandisa kune yakaderera-hunyanzvi vanorwisa nekubatanidza zvakasiyana-siyana zve-off-the-sherufu maturusi. (3) Isu tinogadzira uye tinoitisa chidzidzo chemushandisi kuti tiongorore kuchengetedzwa kwega kwega kwekutevedzera kune imwe neimwe yakasimba nzira yekutevedzera pane yakajairwa seti yevanyori uye zvinokurudzira.

4.3 Yedu Yakasimba Mimicry Nzira

Isu zvino tinotsanangura nzira ina dzakasimba dzekutevedzera dzatakagadzira kuongorora kusimba kwedziviriro. Isu tinonyanya kukoshesa nzira dzakareruka dzinongoda preprocessing yakachengetedzwa mifananidzo. Nzira idzi dzinopa njodzi huru nekuti dzinowanikwa nyore, hadzidi hunyanzvi hwehunyanzvi, uye dzinogona kushandiswa mumabhokisi eblack-box (semuenzaniso kana finetuning ichipihwa sevhisi yeAPI). Kuti tive nekukwana, isu tinopamhidzira imwe chena-bhokisi nzira, yakafuridzirwa neIMPRESS (Cao et al., 2024).

Isu tinocherekedza kuti nzira dzatino kurudzira dzakatariswa (zvishoma muchikamu) mubasa rekutanga iro rakaona kuti haashande kurwisa maitiro ekutevedzera kudzivirira (Shan et al., 2023a; Liang et al., 2023; Shan et al., 2023b ). Asi, sezvatakaona muChikamu 4.1, ongororo idzi dzakatambura kubva kune dzinoverengeka dzisingakwanisi. Isu nokudaro tinoongorora zvakare nzira idzi (kana zvishoma zvishoma zvakasiyana) uye ticharatidza kuti dzakabudirira zvakanyanya pane zvakambotaurwa.

Black-bhokisi preprocessing nzira.

✦ Gaussian ruzha . Sedanho rakareruka rekugadzirisa, isu tinowedzera hushoma hweGaussian ruzha kumifananidzo yakachengetedzwa. Iyi nzira inogona kushandiswa pamberi pechero dema-bhokisi diffusion modhi.

✦ DiffPure . Isu tinoshandisa mifananidzo-kune-mufananidzo modhi kuti tibvise kuvhiringika kwakaunzwa nedziviriro, inonziwo DiffPure (Nie et al., 2022) (ona Appendix I.1). Iyi nzira ndeye dema-bhokisi, asi inoda mhando mbiri dzakasiyana: inochenesa, uye iyo inoshandiswa kutevedzera maitiro. Isu tinoshandisa Yakagadzikana Diffusion XL semuchenesi wedu.

✦ Noisy Upscaling . Isu tinosuma yakareruka uye inoshanda mutsauko weiyo-matanho maviri ekumusoro ekucheneswa anoonekwa muGlaze (Shan et al., 2023a). Nzira yavo inotanga kuita JPEG compression (kuderedza kukanganisa) uyezve inoshandisa Stable Diffusion Upscaler (Rombach et al., 2022) (kuderedza kuderedzwa muhutano). Asi, tinoona kuti upscaling inokudza JPEG compression artifacts pachinzvimbo chekubvisa. Kugadzira nzira yekuchenesa iri nani, tinoona kuti iyo Upscaler inodzidziswa pamifananidzo yakawedzerwa neruzha rweGaussian. Naizvozvo, isu tinochenesa mufananidzo wakachengetedzwa nekutanga kushandisa Gaussian ruzha uye tozoisa iyo Upscaler. Iyi Noisy Upscaling nzira inosuma zvinhu zvisinganzwisisike uye inoderedza zvakanyanya dziviriro (ona Mufananidzo 26 wemuenzaniso uye Appendix I.2 yeruzivo).

White-bhokisi nzira.

✦ IMPRESS ++. Kuti tive nekukwana, tinogadzira nzira yewhite-box yekuongorora kana dzimwe nzira dzakaoma dzinogona kuwedzera kusimba kwemaitiro ekutevedzera. Nzira yedu inovaka paIMPRESS (Cao et al., 2024) asi inotora basa rakasiyana-siyana rekurasikirwa uye rinowedzera kushandiswa kusina kunaka kukurudzira (Miyake et al., 2023) uye denoising kuvandudza kusimba kwemaitiro ekuenzanisa (ona Appendix I.3 uye Figure 27 kuti uwane ruzivo).

[1] Iwo maviri manyoro ekugadzirisa anonyanya kusiyana mukusarudza raibhurari, modhi, uye hyperparameter. Isu tinoshandisa yakajairwa HuggingFace script uye Yakagadzikana Diffusion 2.1 (iyo modhi yakaongororwa mubepa reGlaze).