Ababhali: UMGao, NVIDIA, iYunivesithi yaseToronto, iZiko leVector (jung@nvidia.com) UShen, NVIDIA, iYunivesithi yaseToronto, iZiko leVector (frshen@nvidia.com) UWanga, NVIDIA, iYunivesithi yaseToronto, iZiko leVector (zianw@nvidia.com) UChen, NVIDIA, iYunivesithi yaseToronto, iZiko leVector (wenzchen@nvidia.com) UYin, NVIDIA (kangxuey@nvidia.com) ULi, NVIDIA (daiqingl@nvidia.com) ULitany, NVIDIA (olitany@nvidia.com) UGojcic, NVIDIA (zgojcic@nvidia.com) UFidler, NVIDIA, iYunivesithi yaseToronto, iZiko leVector (sfidler@nvidia.com) Isishwankathelo Njengoko imishishini emininzi ihamba ngokubhekisele ekumodeleni amakhulu ehlabathi abonakalayo, imfuno ye tools zokudala umxholo ezinokukala ngobungakanani, umgangatho, kunye nobubanzi bomxholo we-3D iyabonakala. Kumnandisi wethu, sijonge ukuqeqesha iimodeli zokudala ze-3D ezisebenza kakuhle ezenza ii-mesh ezinemibala ezinokuthi zisetyenziswe ngqo kwiinjini zokuhambisa i-3D, ngoko ke zisetyenziswe kwangoko kwi-down-stream applications. Imisebenzi yangaphambili kwi-3D generative modeling ayinazo iinkcukacha zejometri, inemida kwi-topology ye-mesh enokuyenza, ngesiqhelo ayixhasi imibala, okanye isebenzise ii-renderers ze-neural kwinkqubo yokuyila, nto leyo eyenza ukusetyenziswa kwayo kwi-common 3D software kube nzima. Kulo msebenzi, sethula i-GET3D, imodeli yokudala eyakha ngokuthe ngqo ii-mesh ze- xplicit extured ezine-topology eyiyo, iinkcukacha zejometri ezizityebi, kunye nemibala ephezulu. Sijongana nempumelelo yamva nje kwimodeli ye-surface ebonakalayo, ukuhambisa okubonakalayo kunye nee-2D Generative Adversarial Networks ukuba siqeqeshe imodeli yethu kwiiqoqo zemifanekiso ye-2D. I-GET3D iyakwazi ukuvelisa ii-mesh ezinemibala ezikumgangatho ophezulu, ukusuka kwiimoto, izitulo, izilwanyana, iimoto ezimbini kunye nabalinganiswa babantu ukuya ezakhiweni, iphumeza ukuphucuka okukhulu kwiindlela zangaphambili. Iphepha lejelo lethu: E T 3D https://nv-tlabs.github.io/GET3D 1 Intshayelelo Umgangatho oceleceleyo, oceleceleyo we-3D ubaya uba ngowona ubaluleke kakhulu kwimishishini eyahlukeneyo, kuquka ukudlala imidlalo, iirobhothi, ubugcisa bokwakha, kunye namaqonga. Nangona kunjalo, ukudala okwenziwa ngesandla kwe-3D asset kubonisa ixesha elininzi kwaye kufuna ulwazi olukhethekileyo lobuchwepheshe kunye nezakhono zokumodela kobugcisa. Enye yeengxaki eziphambili kukunyuka - ngelixa unokufumana iimodeli ze-3D kwii-marketplaces ze-3D ezifana neTurbosquid [ ] okanye iSketchfab [ ], ukwenza iimodeli ezininzi ze-3D ukuthi, ukugcwalisa umdlalo okanye imuvi ngesihlwele sabalinganiswa ababonakala bebanga-mnye kusekho ixesha elininzi lokwenziwa kwabaculi. 4 3 Ukufaka isandla kwinkqubo yokudala umxholo kunye nokuyenza ifikeleleke kubasebenzisi abaninzi (abangeneleli), iinethiwekhi ze-3D ezivelisayo ezinokwenza ii-asset ze-3D ezikumgangatho ophezulu kunye nezahlukeneyo ziye zabangindawo esebenzayo yophando [ , , , , , , , , , , ]. Nangona kunjalo, ukuba zisebenziseka ngokwenene kwizicelo zangoku zehlabathi, iimodeli ze-3D ezingezizo kufanele ziqwalasele iimfuno ezilandelayo: Kufuneka zikwazi ukuvelisa izimo ezineenkcukacha zejometri kunye ne-topology eyiyo, Isiphumo kufuneka sibe yi-mesh enombala, eyona nto iphambili esetyenziswa ngamaqhaza omkhandi wegrafu njengeBlender [ ] kunye neMaya [ ], kwaye Kufuneka sikwazi ukusebenzisa imifanekiso ye-2D ukuyijonga, njengoko zifumaneka ngokubanzi kunezimo ze-3D ezicacileyo. 5 14 43 46 53 68 75 60 59 69 23 (a) (b) 15 1 (c) Umsebenzi wangaphambili kwi-3D generative modeling ugxile kumacandelo eemfuno ezingaphezu kwentetho, kodwa akukho ndlela ukuza kuthi ga ngoku iyazalisekisa zonke (Tab. ). Ngokomzekelo, iindlela ezenza ii-point clouds ze-3D [ , 68, 75] ngesiqhelo azivelisi imibala kwaye kufuneka ziguqulelwe kwi-mesh kwi-post-processing. 1 5 Iindlela ezivelisa ii-voxels zihlala zingena kwiinkcukacha zejometri kwaye azivelisi umbala [ , , , ]. Iimodeli zokudala ezisekelwe kwiinethiwekhi ze-neural fields [ , ] zigxile ekukhuphweni kwejometri kodwa azinakhathalelo kumbala. Uninzi lwazo lufuna kunye nokujonga okucacileyo kwe-3D. Ekugqibeleni, iindlela eziqondanga ukuvelisa ii-mesh ze-3D ezinemibala [ , ] ngesiqhelo zifuna iitemplate ze-shape ezichaziweyo kwangaphambili kwaye azikwazi ukuvelisa izimo ezinayo i-topology eyiyo kunye ne-genus eguquguqukayo. 66 20 27 40 43 14 54 53 Mva nje, ukuphumelela okukhawulezileyo kwi-neural volume rendering [ ] kunye nee-2D Generative Adversarial Networks (GANs) [ , , , , ] kuye kwakhokelela ekunyukeni kokuyila imifanekiso ye-3D-aware [ , , , , , ]. Nangona kunjalo, lo msebenzi uhambisa imifanekiso egqibeleleyo ngokusebenzisa ukuhambisa kwe-neural kwinkqubo yokuyila kwaye awuqinisekisi ukuba izimo ze-3D ezingatshatanga zinokwenziwa. Ngelixa i-mesh inokuthi ifunyanwe ukusuka kwimfanekiso ye-neural field ngokusebenzisa i-algorithm ye-marching cube [ ], ukukhupha umbala ohambelanayo akulula. 45 34 35 33 29 52 7 57 8 49 51 25 39 Kulo msebenzi, sethula indlela entsha ejolise ekuboneleleni zonke iimfuno zemodeli yokudala ye-3D esebenzayo. Ngokukodwa, siphakamisa i-GET3D, imodeli yokudala yezimo ze-3D eye ngokuthe ngqo ibonelele ii-mesh ze- xplicit extured ezineenkcukacha ezibalaseleyo zejometri kunye nombala kunye ne-topology ye-mesh eyiyo. Embindini wendlela yethu kukho inkqubo yokudala esebenzisa indlela yokukhupha umphezulu we- ebonakalayo [ ] kunye neteknoloji yokuhambisa ebonakalayo [ , ]. Eyokuqala isivumela ukuba sihlawule ngokuthe ngqo kwaye sibonelele ii-mesh ze-3D ezinemibala ezine-topology eyiyo, ngelixa eyesibini isivumela ukuba siqeqeshe imodeli yethu ngemifanekiso ye-2D, ngoko ke sisebenzise ii-discriminator ezinamandla nezivuthiweyo eziyilelwe ukuyila imifanekiso ye-2D. Ekubeni imodeli yethu ikhupha ii-mesh ngokuthe ngqo kwaye isebenzisa i-renderer yefotografi esebenza ngokugqithisileyo (ebonakalayo), sinokuyinyusa ngokulula imodeli yethu ukuze siqeqeshe ngomfanekiso E T 3D explicit 60 47 37 isombululo esingaphezulu kwi-1024 × 1024, esivumela ukuba sifunde iinkcukacha zejometri kunye nemibala ezikumgangatho ophezulu. Sibonisa ukusebenza kobugcisa obuphezulu bokudala umzobo we-3D omkhulu kwii-khathegori ezininzi ezinejometri eyiyo ukusuka kwi-ShapeNet [ ], iTurbosquid [ ] kunye neRenderpeople [ ], ezifana nezitulo, iimoto ezimbini, iimoto, abalinganiswa babantu, nezakhiwo. Nge-mesh ecacileyo njengesiphumo sokumelwa, i-GET3D ikwabonisa ukuguquguquka kwaye ingalungiswa ngokulula kwezinye izicelo, kuquka: ukufunda ukudala iziphumo zezinto kunye neziphumo zokukhanya ezixhomekeke kwimbono usebenzisa i-rendering ebonakalayo [ ], ngaphandle kokujonga, ukuyila komzobo we-3D ochazwe bubhalo usebenzisa i-CLIP [ ] embedding. 9 4 2 (a) 12 (b) 56 2 Umsebenzi Ohambelanayo Siphonononga ukuphucuka kwamva nje kwiimodeli zokudala ze-3D zejometri kunye nembonakalo, kunye nokuyila imifanekiso ye-3D-aware. Kule minyaka mihlanu idlulileyo, iimodeli zokudala ze-2D ziye zafikelela kumgangatho we-photorealistic ekuyileni imifanekiso ephakamileyo [ , , , , , , ]. Olu phuhliso lukwanikele ngenkalipho uphando kwizinto zokudala ze-3D. Iindlela zangaphambili zijonge ukwandisa ngokuthe ngqo ii-generators ze-CNN ze-2D kwii-voxel grids ze-3D [ , , , , ], kodwa isikhala esikhulu sememori kunye nobunzima bokubala be-3D convolutions bukhathaza inkqubo yokuyila kwi-resolution ephezulu. Njengenye indlela, eminye imisebenzi iye yajonga i-point cloud [ , , , ], i-implicit [ , ], okanye i-octree [ ] representations. Nangona kunjalo, le misebenzi ijonga ngaphezu kokudala ijometri kwaye ayinakhathalelo kwimbonakalo. Iziphumo zazo zokumelwa kufuneka zilungiswe ukuze zibe nenkqubo kunye nee-engines ze-graphics eziqhelekileyo. Iimodeli Zokudala Ze-3D 34 35 33 52 29 19 16 66 20 27 40 62 5 68 75 46 43 14 30 Ezinye izinto ezifanayo nomsebenzi wethu, iTextured3DGAN [ , ] kunye neDIBR [ ] zenze ii-mesh ze-3D ezinemibala, kodwa zenza ukuyila njengokuguqulelwa kwii-template mesh, nto leyo ebathintelayo ekwenzeni i-topology eyiyo okanye izimo ezine-genus eyahlukileyo, into eyenziwa yindlela yethu. I-PolyGen [ ] kunye neSurfGen [ ] zikwazi ukuvelisa ii-mesh nge-topology eyiyo, kodwa azivelisi imibala. 54 53 11 48 41 Kunyanzeliswe yimpumelelo ye-neural volume rendering [ ] kunye nokumelwa kwe-implicit [ , ], umsebenzi wamva nje uqalise ukujongana nengxaki yokuyila imifanekiso ye-3D-aware [ , , , , , , , , , ]. Nangona kunjalo, ii-network ze-neural volume rendering zihlala zicotha ukufuna, zikhokelela kwixesha elide lokuqeqesha [ , ], kwaye zenze imifanekiso yobungakanani obulinganiselweyo. I-GIRAFFE [ ] kunye ne-StyleNerf [ ] ziphucula ukusebenza kokuqeqesha nokuhambisa ngokwenza i-neural rendering kwinqanaba eliphantsi kwaye emva koko iphakame iziphumo nge-2D CNN. Nangona kunjalo, inkqubo yempumelelo iza ngenxalenye yokuncipha kokuhambelana kweembono ezininzi. Ngokusebenzisa i-discriminator ephindwe kabini, i-EG3D [ ] inganciphisa le ngxaki ngokuyinxalenye. Nokuba kunjalo, ukukhupha umphezulu onombala kwiindlela ezisekwe kwi-neural rendering yingxaki ekungekho lula ukuyilwa. Ngokungafaniyo, i-GET3D ngokuthe ngqo ibonelele ii-mesh ze-3D ezinemibala ezisetyenziswa ngokulula kwi-engines zemifanekiso eqhelekileyo. Ukuyila Imifanekiso Ye-3D-Aware 45 43 14 7 57 49 26 25 76 8 51 58 67 7 57 49 25 8 3 Indlela Ngoku sethula isakhiwo sethu se-GET3D sokuyila izimo ze-3D ezinemibala. Inkqubo yethu yokuyila yahlulwe yaba ngamacandelo amabini: igumbi lejometri, elikhupha ngokubonakalayo i-mesh yomphezulu ye-topology eyiyo, kunye negumbi lokubala elenza i-field yombala enokuthi ifunwe kwiindawo zomphezulu ukuvelisa imibala. Okokugqibela kungandiswa kwezinye iipropati zomphezulu ezifana, umzekelo, izinto (Sec. ). Ngexesha lokuqeqesha, i-rasterizer esebenza ngokugqithisileyo isetyenziswa ukuhambisa i-mesh enombala kwiifoto ze-2D ezikumgangatho ophezulu. Inkqubo yonke ibonakalayo, ivumela ukuqeqeshwa kwe-adversarial ukusuka kwimifanekiso (neemaski ezibonisa into enomdla) ngokudlulisa ii-gradients ukusuka kwi-2D discriminator ukuya kumagumbi omvelisi omabini. Imodeli yethu iboniswa kumzekelo. . Kwixesha elizayo, siphakamisa kuqala isidima sethu somvelisi we-3D kwi-Sec. , ngaphambi kokugqithisa ekuhambiseni okubonakalayo kunye nemisebenzi elahlekileyo kwi-Sec. . 4.3.1 2 3.1 3.2 3.1 Imveliso Yokudala Ye-3D Ezinemibala Ze-Mesh Sijonge ukufunda i-generator ye-3D = ( ) ukugqitha isampulu ukusuka kububanzi bokwahlula okujongwayo M, E G z ∈ N (0*,* ) kwi-mesh enemibala . z I M E Ekubeni i-jometri efanayo ingaba nemibala eyahlukileyo, kwaye umbala ofanayo ungasetyenziswa kwiijometri ezahlukileyo, sigqitha ii-vector ezingezizo 1 ∈ R512 kunye 2 ∈ R512. Ukulandela i-StyleGAN [ , , ], sisebenzisa iinethiwekhi zokumodela ezingekho ngqo geo kunye tex ukugqitha 1 kunye 2 kwi-vector ze-latent ezimaphakathi 1 = geo( 1) kunye 2 = tex( 2) ezisetyenziswa ngakumbi ukuvelisa ii- ezilawula ukuyila kwemilo ye-3D kunye nombala, ngokulandelanayo. Siphakamisa ngokusemthethweni isidima sokuyila kwejometri kwi-Sec. kunye nesidima sokuyila kombala kwi-Sec. . z z 34 35 33 f f z z w f z w f z styles 3.1.1 3.1.2 3.1.1 Isidima Sejometri Siyila isidima sejometri yethu ukubandakanya i-DMTet [ ], ummeli wemphezulu obonakalayo osandula ukunikezelwa. I-DMTet imela i-jometri njengomhlaba wobude obuchazwe (SDF) kwi-tetrahedral grid egudileyo [ , ], apho umphakamo ungahlolwa ngokubonakalayo ngokuphathwa kwemithetho [ ]. Ukugudisa i-grid ngokuhambisa ii-vertices zayo kuphucula ukusetyenziswa kwayo kwesombululo. Ngokwamkela i-DMTet ukukhuphwa komphezulu, singavelisa ii-mesh ezicacileyo ezine-topology eyiyo kunye ne-genus. Sijonge ukubonelela ngesishwankathelo esifutshane se-DMTet kwaye siye kumnini-nxaxheba iphepha eliyinxalenye kwiinkcukacha ezongezelelweyo. 60 22 24 17 Makhe ( ) ibonise umhlaba wonke we-3D apho into ikhoyo, apho zii-vertices kwi-tetrahedral grid . Yonke i-tetrahedron ∈ ichazwa kusetyenziswa ii-vertices ezine { }, kunye ∈ {1*, . . . , K*}, apho yingqokelela epheleleyo yee-tetrahedra, kwaye ∈ ∈ R3. Ngaphezulu kwezi coordinates zayo zangoku ze-3D, yonke i-vertex iqulethe ixabiso le-SDF ∈ R kunye nokugudiswa ∆ ∈ R3 ye-vertex ukusuka kwikhomponent yayo yoqobo. Lo mmeli uvumela ukufunyanwa kwe-mesh ecacileyo nge-marching tetrahedra ebonakalayo [ ], apho amaxabiso e-SDF kumhlaba ongaguqukiyo abalwa ngokudibanisa kwe-barycentric kwixabiso labo kwi-vertices egudisiweyo ′ = + ∆ . VT , T VT T Tk T v ak , v bk , v ck , v dk k K v ik VT , v ik v i si v i 60 si v v i v i Sigqitha 1 ∈ R512 kwiixabiso ze-SDF kunye nokugudiswa kwi-vertex nganye ngothotho lwee-convolutional layers ze-3D ezibandakanyayo kunye neengqokelela ezizeleyo. Ngokukodwa, siqala sisebenzise ii-convolutional layers ze-3D ukudala ivolume yefayile egqithisileyo kwi- 1. Emva koko sifuna ifayile kwi-vertex nganye ∈ usebenzisa i-trilinear interpolation kwaye siyifakele kwi-MLPs eyona ivumela i-SDF value kunye nokugudiswa ∆ . Kwiimeko apho ukumodela kwinqanaba eliphezulu kufuneka (umzekelo, iimoto ezimbalwa ezineenqwelwana ezibhityileyo kumavili), siphinda sisebenzise i-volume subdivision ngokulandela [ ]. Ubume beNethiwekhi w v i w v i VT si v i 60 Emva kokufumana kunye no∆ kuzo zonke ii-vertices, sisebenzisa i-marching tetrahedra algorithm ebonakalayo ukukhupha i-mesh ecacileyo. Ii-marching tetrahedra zibonisa i-topology yomphezulu kwi-tetrahedron ngayo ngokusekelwe kwiimpawu ze- . Ngokukodwa, ubuso be-mesh buyakhuphwa xa isign( ) /= isign( ), apho ibonisa ii-indices ze-vertices kwi-edge ye-tetrahedron, kwaye ii-vertices yobu buso zichazwa ngokuphakathi ngokulinganayo njengoko mi,j = v 0 i sj−v 0 j si sj−si . Qaphela ukuba ifomyula engentla ichazwa kuphela xa si 6= sj , ngoko ke ibonakalayo, kwaye igradient ukusuka ku mi,j ingabuyiselwa kwiixabiso ze-SDF si kunye nokugudiswa ∆vi . Ngolu mmeli, izimo ezine-topology eyiyo zinokwenziwa ngokulula ngokubona iimpawu ezahlukeneyo ze-si . Ukukhupha iMesh Ebonakalayo si v i si si sj i, j m i,j 3.1.2 Isidima Sombala Ukuyila ngokuthe ngqo i-map yombala ehambelana ne-mesh ephumayo ayilula, njengoko imilo eyenziweyo ingaba ne-genus kunye ne-topology eyiyo. Ngoko ke siyakha umbala njengombala wokombala [ ]. 50 Ngokukodwa, simodela i-field yombala kunye nomsebenzi ogqitha indawo ye-3D yendawo yomphezulu ∈ R3, egqithisile kwi- 2, ukuya kumbala we-RGB ∈ R3 kwindawo leyo. Ekubeni i-field yombala ixhomekeke kwijometri, songeza ukubandakanya olu qeqelo kwi-jometri latent code 1, ukuze = ( *,* 1 ⊕ 2), apho ⊕ ichaza ukudibanisa. ft p w c w c ft p w w Simela i-field yombala yethu kusetyenziswa ummeli we-tri-plane, ongasebenzi kakuhle kwaye uchaza kakuhle ukubuyisela imikhomo ye-3D [ ] kunye nokuyila imifanekiso ye-3D-aware [ ] . Ngokukodwa, silandela [ , ] kwaye sisebenzise inethiwekhi ye-convolutional neural ye-2D ukugqitha i-latent code 1 ⊕ 2 ukuya kwiinqaba zefayile ezintathu ezihambelana ne-axis ezine-size × × ( × 3), apho = 256 ibonisa ububanzi besantya kwaye = 32 inani leenqaba. Ubume beNethiwekhi 55 8 8 35 w w N N C N C Unikwe iinqaba zefayile, i-vector yefayile f t ∈ R 32 yendawo yomphezulu p ingafunyanwa njengoko f t = P e ρ(πe(p)), apho πe(p) iyimpahla yendawo p kwi-plane yefayile e kwaye ρ(·) ichaza i-bilinear interpolation yeefayile. Inqanaba elongezelelweyo eligqibeleleyo lisetyenziswa ukugqitha kwii-vector zefayile ezi-aggregated f t ukuya kumbala we-RGB c. Qaphela ukuba, ngokungafaniyo nezinye izifundo kwi-3D-aware image synthesis ezisebenzisa nommeli we-neural field, sifuna kuphela ukugqitha i-field yombala kwiindawo zeendawo zomphezulu (ngokuchaseneyo neesampulu ezininzi ngokwemva komkhondo). Oku kunciphisa kakhulu ubunzima bokubala bokuhambisa imifanekiso ephakamileyo kwaye kuqinisekisa ukuyila imifanekiso ehambelanayo ngokuhambelana ngokwakhiwa. 3.2 Ukuhanjiswa Okubonakalayo Nokuqeqeshwa Ukujonga imodeli yethu ngexesha lokuqeqesha, sifumana inkuthazo kwi-Nvdiffrec [ ] eyenza ukubuyisela kwakhona iimbono ezininzi ze-3D ngokusebenzisa i-renderer ebonakalayo. Ngokukodwa, sihlambalaza i-mesh ekhutshiweyo kunye ne-field yombala kwimifanekiso ye-2D sisebenzisa i-renderer ebonakalayo [ ], kwaye sijonge inethiwekhi yethu nge-2D discriminator, ejonga ukwahlula umfanekiso ukusuka kwinto eyiyo okanye ihanjiswe kwi-object eyenziweyo. 47 37 Sicinga ukuba ububanzi bekhamera C obusetyenziselwa ukufumana imifanekiso kwidatha yaziwa. Ukuhambisa ii-shapes ezenziweyo, sigqitha ikhamera ukusuka kwi-C, kwaye sisebenzise i-rasterizer ye-Nvdiffrast esebenza ngokugqithisileyo [ ] ukuhambisa i-mesh ye-3D kwinqaku le-2D kunye nomfanekiso apho yonke ipikseli iqulethe iikhomponenti zendawo ye-3D engu Ukuhanjiswa Okubonakalayo c 37 indawo kwi-mesh yomphezulu. Ezi khomponenti zisetyenziswa ngakumbi ukufuna i-field yombala ukufumana amaxabiso e-RGB. Ekubeni sisebenza ngqo kwi-mesh ekhutshiweyo, sinokuhambisa imifanekiso ephakamileyo ngokusebenza okukhulu, sivumela imodeli yethu ukuba iqeqeshe ngomgangatho womfanekiso ongaphaya kwe-1024×1024. Siyala imodeli yethu kusetyenziswa injongo ye-adversarial. Samkela ubume be-discriminator kwi-StyleGAN [ ], kwaye sisebenzise injongo ye-GAN engabonakaliyo efanayo kunye ne-R1 regularization [ ]. Sibona ngamava ukuba ukusebenzisa ii-discriminator ezimbini ezahlukeneyo, enye yemifanekiso ye-RGB kwaye enye yeziqwenga, kubonelela ngezona ziphumo zilungileyo kune-discriminator enye esebenza kwizombini. Makhe ibonise i-discriminator, apho ingaba yimifanekiso ye-RGB okanye inqaku. Injongo ye-adversarial ke ichazwa ngolu hlobo lulandelayo: I-Discriminator & Objective 34 42 Dx x apho ( ) ichazwa njengo ( ) = − log(1 +exp(− )), yindlela yokuhambisa imifanekiso eyiyo, ibonisa ukuhambisa, kwaye yiparamenta engekhoyo. Ekubeni ibonakalayo, ii-gradients zingabuyiselwa kwiifoto ze-2D ukuya kwiizidima zethu ze-3D. g u g u u px R λ R Ukususa iifremu zangaphakathi ezikadadayo ezingabonakaliyo kuyo nayiphi na iimbono, sibuyisela isidima sejometri ngolwazi oluyintloko oluchazwe phakathi kwezixabiso ze-SDF zezantya ezikufutshane [ ]: Ukuqiniswa 47 apho ibonisa ubhubha obuyintloko obubanzi kwaye ibonisa umsebenzi we-sigmoid. Isamba kwi-Eq. ichazwa phezu kweseti ye-unique edges S kwi-tetrahedral grid, apho isign( ) /= isign( ). H σ 2 e si sj Umgangatho wonke olahlekileyo ke uchazwa ngolu hlobo: apho iyiparamenta engekhoyo elawula inqanaba lokuqiniswa. µ 4 Uvavanyo Senza uvavanyo olunzulu ukuhlola imodeli yethu. Siyala kuqala umgangatho we-3D enombala ii-mesh eyenziwe yi-GET3D kwiindlela ezikhoyo kusetyenziswa ii-datasets zeShapeNet [ ] kunye neTurbosquid [ ]. Emva koko, sihlaba izinto esizilungelelaniseyo kwi-Sec. . Ekugqibeleni, sibonisa ukuguquguquka kwe-GET3D ngokuyiguqulela kwezinye izicelo kwi-Sec. . Iziphumo zovavanyo ezingezizo kunye neenkcukacha zokusebenza zinikezelwa kwi-Appendix. 9 4 4.2 4.3 4.1 Uvavanyo Kwi-Datasets Ezenziweyo Ukuhlola kwi-ShapeNet [ ], sisebenzisa iikhathegori ezintathu ezinejometri eyiyo – , , kunye , eziquka izakhiwo ezingama-7497, ezingama-6778, kunye nezili-337, ngokulandelanayo. Siyahlula ngokungajongwanga ishishini ngalinye kwisiqeqesho (70%), ukuqinisekiswa (10%), kunye novavanyo (20%), kwaye sigqitha kwi-test set ii-shapes ezinemibhalo ephindaphindayo kwisiqeqesho. Ukuhambisa idatha yokuqeqesha, sigqitha izimbono ezizenzekelayo ukusuka kwi-hemisphere ephezulu yesimo ngasinye. Kwiikhathegori ze- kunye ne- , sisebenzisa izimbono ezingama-24 ezingajongwanga, ngelixa kwi- sisebenzisa izimbono ezili-100 ngenxa yenani elincinci lezakhiwo. Njengoko iimodeli kwi-ShapeNet zineemibala elula kuphela, siphinda sivavanye i-GET3D kwi-dataset ye- (izakhiwo ezingama-442) eziqokelelwe kwi-TurboSquid [ ], apho imibala ineenkcukacha ngakumbi kwaye siyihlukanisa kwisiqeqesho, ukuqinisekiswa kunye novavanyo njengoko kuchaziwe ngentla. Ekugqibeleni, ukubonisa ukuguquguquka kwe-GET3D, sibonelela ngemiphumo eyenziweyo kwi-dataset ye- eqokelelwe kwi-Turbosquid (izakhiwo ezingama-563), kunye ne-dataset ye- evela kwiRenderpeople [ ] (izakhiwo ezingama-500). Siqeqesha imodeli eyahlukileyo kwicategory nganye. Iidatasets 9 Imoto Isitulo neMoto Car Chair Motorbike Animal 4 House Human Body 2 Siyala i-GET3D kwezinye iinkqubo ezimbini: ii-generative models ze-3D ezixhomekeke kwi-3D supervision: iPointFlow [ ] kunye neOccNet [ ]. Qaphela ukuba ezi ndlela zivelisa ijometri kuphela ngaphandle kombala. Iimodeli zokuyila imifanekiso ye-3D-aware: iGRAF [ ], iPiGAN [ ], kunye neEG3D [ ]. Iimbambano 1) 68 43 2) 57 7 8 Ukuhlola umgangatho wokuyila kwethu, sihlola ijometri kunye nombala wezimo ezivelisiweyo. Ngokubhekisele kwijometri, samkela imilinganiso ukusuka [ ] kwaye sisebenzisa zombini i-Chamfer Distance (CD) kunye ne-Light Field Distance [ ] (LFD) ukubala i-Coverage score kunye ne-Minimum Matching Distance. Kwi-OccNet [ ], iGRAF [ ], iPiGAN [ ] kunye neEG3D [ ], sisebenzisa i-marching cubes ukukhupha ijometri engaphantsi. KwiPointFlow [ ], sisebenzisa i-Poisson surface reconstruction ukuguqula i-point cloud ibe yi-mesh xa sihlola i-LFD. Ukuhlola umgangatho wombala, samkela i-FID [ ] metric esetyenziswa rhoqo ukuhlola ukuyila imifanekiso. Ngokukodwa, kwicategory nganye, sihlambalaza ii-shapes zovavanyo kwimifanekiso ye-2D, kwaye sinyanzelisa ii-3D shapes ezivelisiweyo ukusuka kwimodeli nganye kwimifanekiso eyi-50k sisebenzisa ububanzi bekhamera efanayo. Emva koko sibala i-FID kwiiqoqo zemifanekiso ezimbini. Njengoko ii-baselines ukusuka kwi-3D-aware image synthesis [ , , ] azikhuphi ii-mesh ezinemibala ngokuthe ngqo, sibala i-FID score ngeendlela ezimbini: ( ) sisebenzisa i-neural volume rendering yazo ukufumana imifanekiso ye-2D, esiyibiza ngokuba yi-FID-Ori, kunye ( ) sikhupha i-mesh kwi-neural field representation yazo sisebenzisa i-marching cubes, siyihambise, kwaye emva koko sisebenzise indawo ye-3D ye-pixel nganye ukufuna inethiwekhi ukufumana amaxabiso e-RGB. Sixela le nqaku, eyona ncamaka kakhulu kwisimo sangempela se-3D, njenge-FID-3D. Iinkcukacha ezongezelelweyo kwiimilinganiso zovavanyo ziyafumaneka kwi-Appendix . Iimilinganiso 5 10 43 57 7 8 68 28 57 7 8 i ii B.3 Sinikezela ngezinto ezibalulekileyo zetables. kunye nemizekelo eyenziweyo kumzekelo. kunye noMzekelo. . Iziphumo ezongezelelweyo ziyafumaneka kwi-vidiyo eneenkcukacha. Xa kuthelekiswa ne-OccNet [ ] esebenzisa i-3D supervision ngexesha lokuqeqesha, i-GET3D ifumana ukusebenza okungcono ngokubhekisele kububanzi (COV) kunye nomgangatho (MMD), kwaye ii-shapes zethu ezenziweyo zineenkcukacha ezininzi zejometri. Iziphumo zovavanyo 2 3 4 43 I-PointFlow [ ] igqithisa i-GET3D ngokubhekisele kwi-MMD kwi-CD, ngelixa i-GET3D ingcono kwi-MMD kwi-LFG. Sicinga ukuba oku kungenxa yokuba i-PointFlow ijongana ngokuthe ngqo kwiindawo zeepoyinti, ezikhetha i-CD. I-GET3D ikwanikeza iziphumo ezilungileyo xa kuthelekiswa neenkonzo zokuyila imifanekiso ze-3D-aware, sifumana ukuphucuka okubalulekileyo kwi-PiGAN [ ] kunye neGRAF [ ] ngokubhekisele kuzo zonke iimilinganiso kuzo zonke ii-datasets. Izakhiwo zethu ezenziweyo zineenkcukacha ezithe sawelayo zejometri kunye nombala. Xa kuthelekiswa nomsebenzi wamva nje i-EG3D [ ]. Sifumana ukusebenza okufanayo ekwenzeni imifanekiso ye-2D (FID-ori), ngelixa siphucula kakhulu kwisintesi ye-3D shape ngokubhekisele kwi-FID-3D, ebonisa ukusebenza kwemodeli yethu ekufundeni ijometri kunye nombala wangempela we-3D. 68 7 57 8 Ekubeni senze ii-mesh ezinemibala, sinokuwukhupha kwii-Blender . Sibonisa iziphumo zokuhambisa kumzekelo. kunye noMzekelo. . I-GET3D iyakwazi ukuvelisa ii-shapes ezinejometri kunye ne-topology eyahlukeneyo kunye nomgangatho ophezulu, iinkcukacha ezibhityileyo kakhulu (iimoto ezimbini), kunye nemibala eyiyo kwii-moto, izilwanyana, kunye nezindlu. 1 1 5 I-GET3D ikwayivumela ukuhambisa ii-shapes, ezingaba luncedo kwimisebenzi yokuhlela. Sijonga indawo ye-latent ye-GET3D kumzekelo. , apho sihlambalaza ii-codes ze-latent ukwenza yonke imilo ukusuka ekhohlo ukuya ekunene. I-GET3D iyakwazi ukwenza kakuhle ukuhanjiswa okubusheleli kunye nokuthetha ukusuka kwisimo esinye ukuya kwesinye. Siphinda sijonge indawo encinci ye-latent ngokucofa iikhowudi ze-latent kwicala elithile elingajongwanga. I-GET3D ivelisa ii-shapes ezintsha kunye nezahlukeneyo xa isebenzisa ukuhlela okuncinci kwindawo ye-latent (Umzekelo. ). Ukuhambisa ii-Shapes 6 7 4.2 Ukuqhwalelwa Siyakwahlula imodeli yethu ngeendlela ezimbini: kunye nangaphandle kokuhlukaniswa komthamo, ukuyila kusetyenziswa imiqobo yemifanekiso eyahlukileyo. Ukuqhwalelwa okongezekileyo kunikezelwa kwi-Appendix . 1) 2) C.3 Njengoko kuboniswe kwi-Tbl. , ukuhlukaniswa komthamo kuphucula kakhulu ukusebenza kwiikhathegori ezineenqwelwana ezibhityileyo (umzekelo, iimoto ezimbini), ngelixa kungabikho zinzuzo kwezinye iikhathegori. Sicinga ukuba isombululo sokuqala se-tetrahedral sifanelekile ukubamba i-jometri eneenkcukacha kwii-Chairs kunye nee-Cars, kwaye ke ukuhlukaniswa akukwazi ukubonelela ngokuphuculwa okongezelelweyo. Ukugqithiselwa koHlukaniso loMthamo 2 Ukugqithiselwa kweMiQobiso yemifanekiso eyahlukeneyo Siyakwahlula impembelelo yomqobo womfanekiso wokuqeqesha kwi-Tbl. . Njengoko kulindelekile, ukunyuka komqobo womfanekiso kuphucula ukusebenza ngokubhekisele kwi-FID kunye nomgangatho we-shape, njengoko inethiwekhi ingabona iinkcukacha ezininzi, ezingafumanekiyo kwimifanekiso yobungakanani obuncinci. Oku kuqinisekisa ubungqinelaniso bokubaluleka kokuqeqesha ngomqobo womfanekiso ophezulu, osoloko unzima ukuwusebenzisa kwiindlela ezisekwe kwi-implicit. 3 4.3 Izicelo 4.3.1 Ukuyila Izinto KwiZiphumo Zokukhanya Ezixhomekeke Kwimbono I-GET3D ingandiswa ngokulula ukuba yenze kwakhona izinto zomphezulu ezisetyenziswa ngqo kwi-engines zemifanekiso zangoku. Ngokukodwa, silandela i-Disney BRDF esetyenziswa ngokubanzi [ , ] kwaye sichaza izinto ngokubhekisele kumbala osisiseko (R3), iipropati zentsimbi (R), kunye nokugoba (R). Ngenxa yoko, sigqithisa isidima sethu sokubala ukwenza i-field yokuhlanganisa eyi-5-channel (endaweni nje ye-RGB). Ukwenzela ukuhambisa i-rendering ebonakalayo yezinto, samkela indlela yokuhambisa ye-spherical Gaussian (SG) esebenza ngokugqithisileyo [ ]. Ngokukodwa, sihlambulula i-field yokuhlanganisa kwi-G-buffer, kwaye sigqitha ngokungajongwanga i-HDR image ukusuka kwiseti ye-HDR panoramas zangoku zemithambeka Slight = { } , apho ∈ R32×7 ifunyenwe ngokufaka ii-lobes ezingama-32 ze-SG kwipanoramas nganye. I-SG renderer [ ] emva koko isebenzisa ikhamera ukuhambisa umfanekiso we-RGB kunye neziphumo zokukhanya ezixhomekeke kwimbono, esiyifaka kwi-discriminator ngexesha lokuqeqesha. Qaphela ukuba i-GET3D ayifuni ukujongwa kwezinto ngexesha lokuqeqesha kwaye ifunde ukwenza izinto ezahlukeneyo ngendlela engajongwanga. 6 32 12 LSG K LSG 12 c Sinikezela ngezimvo ezenziweyo zezinto zomgangatho womphezulu kumzekelo. . Nangona ingajongwanga, i-GET3D ifumanisa ukwahlulwa kwezinto ezingathandekiyo, umzekelo, ii-windows zichazwa ngokuchanekileyo ngenani elincinci lokugoba ukuze zibe nombala omkhulu kune-body yemoto, kwaye i-body yemoto ifunyanwa njengenye i-dielectric ngelixa i-window ingeyona i-metallic. 8 4.3.2 Ukuyila Kwe-3D Okuchazwe Ngumbhalo Ngokufanayo nee-image GANs, i-GET3D ikwayixhasa ukuyila komxholo we-3D ochazwe ngombhalo ngokulungisa imodeli eyenziwe ngaphambili phantsi kolwalathiso lwe-CLIP [ ]. Qaphela ukuba isiphumo sethu sokugqibela sokuyila yi-mesh ye-3D enombala. Ukwenza oku, silandela uyilo lwesidima esiphindwe kabini ukusuka kwi-stylegan-NADA [ ], apho ikopi enokuyilwa kunye nekhopi ekhethiweyo yesidima esenziwe ngaphambili ziyasetyenziswa. Ngexesha lokuphuculwa kokusebenza kunye zombini zihambisa iifoto ukusuka kwii-views ezili-16 ezingajongwanga. Unikwe umbuzo obhalisiweyo, sigqitha iiparesi ezingama-500 zenkqubo zengxolo 1 kunye no 2. Kwisampulu nganye, silungisa ii-parameters ze- ukunciphisa ulwandiso lwe-CLIP directional [ ] (ii-label zomthombo zithi "imolo", "isilwanyana" kunye "nenqaba" kwiikhathegori ezihambelanayo), kwaye sikhethe iisampulu ezine-loss ephantsi. Ukukhawulezisa le nkqubo, siqala sisebenzisa inani elincinci lokusebenza okokuqala kwiisampulu ezingama-500, emva koko sikhethe iisampulu eziphezulu ezingama-50 ezine-losses ezisezantsi, kwaye sisebenzise ukusebenza kangangeentambosi ezingama-300. Iziphumo kunye nokuthelekiswa kunye nendlela ye-mesh yenkqubo yokuyila echazwe yombhalo, iText2Mesh [ ], zinikezelwa kumzekelo. . Qaphela ukuba, [ ] ifuna i-mesh yesimo njengento efakiweyo kwindlela. Sinikezela ii-mesh zethu ezenziweyo ukusuka kwisidima esikhethiweyo njengeemesh ezifakiweyo kuyo. Njengoko ifuna ii-vertices zemesh ukuba zibe zininzi ukuze zenze iinkcukacha zomphezulu ngeentshukuthelo ze-vertex, songeza ii-mesh ezihlukanisiweyo nge-mid-point subdivision ukuze siqinisekise ukuba yonke i-mesh ine-50k-150k vertices ngokuphakathi. 56 21 Gt Gf Gt Gf z z Gt 21 44 9 44 5 Isigqibo Siphakamise i-GET3D, imodeli entsha yokudala ye-3D ekwazi ukwenza ii-mesh ze-3D ezinemibala ezikumgangatho ophezulu kunye ne-topology eyiyo. I-GET3D iqeqeshwe kusetyenziswa kuphela ii-2D images njengokujonga. Siye savavanya ukuphucuka okubalulekileyo ekwenzeni ii-3D shapes kwiindlela zangoku zobugcisa ezizizigaba ezininzi. Siyathemba ukuba lo msebenzi usisondeza inyathelo elilodwa elibheke ekufikeleleni ukwenziwa komxholo we-3D kusetyenziswa i-A.I.. Ngelixa i-GET3D yenza inyathelo elibalulekileyo kwimicimbi ye-3D generative model esebenzayo ye-3D enombala, isenazo ezinye iingxaki. Ngokukodwa, sixhomekeke kwii-silhouettes ze-2D kunye nolwazi lobubanzi bekhamera ngexesha lokuqeqesha. Ngenxa yoko, i-GET3D iye yavavanywa kuphela kwiidatha ezenziweyo. Ukuphuculwa okuthembisayo kungasebenzisa ukuphucuka kwi-instance segmentation kunye nokulinganisa kwesantya sekhamera ukunciphisa le ngxaki kwaye wandise i-GET3D kwiidatha zangempela. I-GET3D ikwazi ukuqeqeshwa nge-category; ukuyandisa kwiikhathegori ezininzi kwixesha elizayo, ingasinceda ukuba sibonise ngcono ububanzi obukwi-category. Iingxaki Siphakamise imodeli entsha yokudala ye-3D eyenza ii-mesh ze-3D ezinemibala, ezicwangciswe ukuba zingene kwii-engines zemifanekiso yangoku. Imodeli yethu iyakwazi ukuvelisa izimo ezine-topology eyiyo, imibala ekumgangatho ophezulu kunye neenkcukacha zejometri ezizityebi, ivula indlela yokwenza. Impembelelo ebanzi isixhobo se-AI sokwenziwa komxholo we-3D. Njengee-models zokufunda zoomatshini, i-GET3D ikwazichaphazeleka yiintlukuhlo ezibangelwa ziidatha zokuqeqesha. Ngoko ke, kufuneka kusetyenziswe inkathalo eninzi xa kujongwa iz
