```html Ababhali: UMN Gao, NVIDIA, iYunivesithi yaseToronto, iYunivesithi yeYunivesithi (jung@nvidia.com) UMX Shen, NVIDIA, iYunivesithi yaseToronto, iYunivesithi yeYunivesithi (frshen@nvidia.com) UMX Wang, NVIDIA, iYunivesithi yaseToronto, iYunivesithi yeYunivesithi (zianw@nvidia.com) UMX Chen, NVIDIA, iYunivesithi yaseToronto, iYunivesithi yeYunivesithi (wenzchen@nvidia.com) UMX Yin, NVIDIA (kangxuey@nvidia.com) UMX Li, NVIDIA (daiqingl@nvidia.com) UMX Litany, NVIDIA (olitany@nvidia.com) UMX Gojcic, NVIDIA (zgojcic@nvidia.com) UMX Fidler, NVIDIA, iYunivesithi yaseToronto, iYunivesithi yeYunivesithi (sfidler@nvidia.com) Isishwankathelo Njengoko iintlobo ezininzi zamashishini zihamba ngokumodela iindalo ezinkulu ze-3D, imfuneko yeendawo zokudala ezinokukala ngokomlinganiselo, umgangatho, kunye nokwahluka kwezinto ze-3D iyacaca. Kumsebenzi wethu, sijonge ukuqeqesha iimodeli ezenziwa nge-3D ezinokwenza iimethi ezine-textured ezinokuthi zisetyenziswe ngokuthe ngqo kwiinjini zokudala ze-3D, ngoko ke zingasetyenziswa kwizicelo ezisezantsi. Imisebenzi yangaphambili kwimodeli ye-3D egqamileyo ayinayo iinkcukacha zejometri, inemida kwi-topology yomethi e-3D enokuvelisa, ngokuqhelekileyo ayixhasi iitexures, okanye isebenzisa iinjjini zokudala ze-neural kwinkqubo yokuyila, eyenza ukusetyenziswa kwayo kwi-software ye-3D eqhelekileyo kungabi lula. Kulo msebenzi, sethula i-GET3D, imodeli ye- enerative eyenza ngokuthe ngqo iimethi ze- xplicit extured ezine-topology eyahlukahlukeneyo, iinkcukacha zejometri ezipheleleyo, kunye neetexures eziphezulu. Sibophelela impumelelo yakutshanje kwimodeli yoqikelelo oluchazayo, ukudala okuchazayo, kunye ne-2D Generative Adversarial Networks ukuze siqeqeshe imodeli yethu kusuka kwiingqokelelo zemifanekiso ye-2D. I-GET3D iyakwazi ukwenza iimethi ezine-textured ze-3D ezikumgangatho ophezulu, ukusuka kwiimoto, izitulo, izilwanyana, iimotorbikes kunye nabalinganiswa babantu ukuya ezakhiweni, sifumana ukuphucuka okubalulekileyo kwizithintelo zangaphambili. Iphepha lethu leprojekthi: G E T 3D https://nv-tlabs.github.io/GET3D 1 Intshayelelo Ubungqingqwa, umgangatho ophezulu womxholo we-3D ubangela ukuba ube nefuthe elikhulayo kwiintlobo ezininzi zamashishini, kuquka imidlalo, iirobotics, ubugcisa, kunye namaqonga onxibelelwano. Nangona kunjalo, ukwenza izinto ezingathandekiyo ze-3D ngesandla kuthatha ixesha elininzi kwaye kufuna ulwazi oluthile lobuchwepheshe kunye nezakhono zemodeli zobugcisa. Enye yezona ngxaki ziphambili yile - ngelixa unokufumana iimodeli ze-3D kwiimarikethi ze-3D ezifana neTurbosquid [ ] okanye iSketchfab [ ], ukwenza iimodeli ezininzi ze-3D ukuze, sayo, ugcwalise umdlalo okanye ifilimu ngesihlwele sabalinganiswa ababonakala ngokwahlukileyo kusefuna ixesha elibalulekileyo labasebenzi bokudala. 4 3 Ukukhululeka kwenkqubo yokudala imixholo kwaye uyenze ifumaneke kuluhlu lwabasebenzisi (abaqalayo), iinethiwekhi ze-3D ezenziwayo ezinokwenza izinto ze-3D ezikumgangatho ophezulu kunye nezahlukeneyo ziye zabangindawo esebenzayo yophando [ , , , , , , , , , , ]. Nangona kunjalo, ukuze zibe luncedo kwiimfuno zangoku zangoku, iimodeli ze-3D ezenziwayo kufuneka zifezekise iimfuno ezilandelayo: Kufuneka zibe namandla okwenza imilo eneenkcukacha zejometri kunye ne-topology engakhethekiyo, Umveliso kufuneka ube ngumethi we-3D, ongumboniso ophambili osetyenziswa yiipakethi zesofthiwe yoyilo eqhelekileyo ezifana neBlender [ ] kunye neMaya [ ], kwaye Kufuneka sikwazi ukusebenzisa imifanekiso ye-2D ukuze sijongwe, njengoko ifumaneka kakhulu kuneemilo ze-3D ezicacileyo. 5 14 43 46 53 68 75 60 59 69 23 (a) (b) 15 1 (c) Umsebenzi wangaphambili kwimodeli ye-3D egqamileyo ugxile kwiindidi ezincinci zeemfuno ezingentla, kodwa akukho ndlela nanamhlanje iyifezekisayo. (Ithebula. ). Ngokomzekelo, iindlela ezenza ii-point clouds ze-3D [ , 68, 75] ziqhele ukungalungisi iitexures kwaye kufuneka ziguqulelwe kumethi emva kokucubungula. 1 5 Iindlela ezenza ii-voxels zihlala zingenayo iinkcukacha zejometri kwaye azivelisi itexture [ , , , ]. Iimodeli ze-Generative ezisekelwe kwiifields ze-neural [ , ] zigxile ekukhupheni ijometri kodwa azinaki inkangeleko. Uninzi lwazo luphinde ludinge ujongwa olucacileyo lwe-3D. Ekugqibeleni, iindlela eziphuma ngokuthe ngqo kwiimethi ezine-textured ze-3D [ , ] ziqhele ukufuna imodeli eshicilelweyo kwaye ayinakwenza imilo eneenkcukacha zejometri kunye ne-topology eguquguqukayo. 66 20 27 40 43 14 54 53 Mva nje, ukuqhubela phambili ngokukhawuleza kwii-neural volume rendering [ ] kunye ne-2D Generative Adversarial Networks (GANs) [ , , , , ] kuye kwakhokelela ekunyukeni kwemifanekiso ye-3D-aware [ , , , , , ]. Nangona kunjalo, lo msebenzi ujolise ekwenzweni kwemifanekiso ethobelana nemibono emininzi kusetyenziswa iinjjini zokudala ze-neural kwinkqubo yokuyila kwaye ayiqinisekisi ukuba iimilo ezingqongileyo ze-3D ezingaqondakaliyo zinokwenziwa. Ngelixa umethi unokufunyanwa kwi-neural field representation esisiseko kusetyenziswa i-marching cube algorithm [ ], ukukhuphela itexture ehambelanayo akulula. 45 34 35 33 29 52 7 57 8 49 51 25 39 Kulo msebenzi, sethula indlela entsha ejolise ekuboneleleni ngazo zonke iimfuno zemodeli ye-3D egqamileyo enokuba luncedo. Ngokukodwa, sethula i-GET3D, imodeli ye- enerative yeemilo ze-3D eyenza ngokuthe ngqo iimethi ze- xplicit extured ezineenkcukacha eziphezulu zejometri kunye netexture kunye ne-topology yomethi enga-vumelekanga. Entliziyweni yendlela yethu kukho inkqubo yokuyila esebenzisa indlela yokukhuphela umphezulu ochazayo [ ] kunye ne-differentiable rendering technique [ , ]. Eyokuqala isivumela ukuba sisebenzise ngokuthe ngqo kwaye senze iimethi ezine-textured ze-3D ezinemivuzo enga-vumelekanga, ngelixa eyesibini isivumela ukuba siqeqeshe imodeli yethu ngemifanekiso ye-2D, ngoko ke sisebenzisa abagxeki abanamandla kunye nabo bavuthiweyo abaveliselwe ukuyila imifanekiso ye-2D. Ekubeni imodeli yethu yenza ngokuthe ngqo iimethi kwaye isebenzisa injini yoyilo efanelekileyo (echazayo), sinokuyala imodeli yethu ukuba iqeqeshe ngemifanekiso G E T 3D 60 47 37 isibonelelo esingangoko 1024 × 1024, esivumelayo ukuba sifunde iinkcukacha zejometri kunye netexture ezikumgangatho ophezulu. Sibonisa ukusebenza kwe-state-of-the-art kwisizukulwana se-3D esingaqinisekanga kwiindidi ezininzi ezinemilo eyahlukahlukeneyo ukusuka kwi-ShapeNet [ ], iTurbosquid [ ] kunye ne-Renderpeople [ ], ezifana neezitulo, iimotorbikes, iimoto, abalinganiswa babantu, kunye nezakhiwo. Ngomboniso womethi ocacileyo njengomboniso wokukhupha, i-GET3D ikwabonakala ilungele kwaye ingasetyenziselwa ezinye izicelo, kuquka: ukufunda ukuvelisa iziphumo zematerial kunye neziphumo zokukhanyisa ezixhomekeke kumbono kusetyenziswa i-rendering echazayo [ ], ngaphandle kokujongwa, ukuyila kwe-3D okukhokhelwa yimibhalo kusetyenziswa i-CLIP [ ] embedding. 9 4 2 (a) 12 (b) 56 2 Umsebenzi Ohambelanayo Sijonga ukuqhubela phambili kwamva nje kwiimilo ze-3D zejometri kunye nenkangeleko, kunye nemifanekiso ye-3D-aware egqamileyo. Kumashumi eminyaka yakutshanje, iimodeli ze-2D ezizizizukulwane ziye zafikelela umgangatho we-photorealistic kwisizukulwana semifanekiso ephezulu [ , , , , , , ]. Le mpumelelo ikwakhuthaze uphando kwisizukulwana semixholo ye-3D. Iindlela zangaphambili zijolise ekwandiseni ngokuthe ngqo abavelisi be-2D CNN kwiigridi ze-3D voxel [ , , , , ], kodwa inkulu yokusetyenziswa kwememori kunye nobunzima bokubala be-3D convolutions buyayibambezela inkqubo yokuyila kwisibonelelo esiphezulu. Njengenye indlela, eminye imisebenzi iye yajonga i-point cloud [ , , , ], i-implicit [ , ], okanye i-octree [ ] representations. Nangona kunjalo, le misebenzi ijolise ikakhulu ekwenzeni ijometri kwaye ayinaki inkangeleko. Iimboniso zabo ziphinde zifune ukucubungulwa kwakhona ukwenzela ukuba zihambelane neenjini zoyilo eziqhelekileyo. Iimilo ze-3D 34 35 33 52 29 19 16 66 20 27 40 62 5 68 75 46 43 14 30 Ngaphezu komsebenzi wethu, i-Textured3DGAN [ , ] kunye ne-DIBR [ ] yenza iimethi ezine-textured ze-3D, kodwa bayilungelelanisa inkqubo njengokujika kwemethi yeshumi, nto leyo eyenza ukuba bangakwazi ukwenza i-topology eyahlukahlukeneyo okanye iimilo ezinezigaba eziguquguqukayo, ezenziwa yindlela yethu. I-PolyGen [ ] kunye ne-SurfGen [ ] ingavelisa iimethi ezine-topology engakhethekiyo, kodwa ayenzi iitexures. 54 53 11 48 41 Ephuhliswe yimpumelelo ye-neural volume rendering [ ] kunye ne-implicit representations [ , ], umsebenzi wamva nje uqale ukujongana nengxaki yokusintyezwa kwe-3D-aware [ , , , , , , , , , ]. Nangona kunjalo, iinethiwekhi ze-neural volume rendering ziqhele ukuba maziye ngokukhawuleza, zikhokelela kumaxesha okufundisa ende [ , ], kwaye yenza imifanekiso yesibonelelo esilinganiselweyo. I-GIRAFFE [ ] kunye ne-StyleNerf [ ] ziphucula ukusebenza kokufundisa kunye nokudala ngokwenza i-neural rendering kwinqanaba eliphantsi kwaye emva koko zikhuphule iziphumo nge-2D CNN. Nangona kunjalo, ukuphuculwa kokusebenza kuza ngexabiso lokunciphisa ukuhambelana kweembono ezininzi. Ngokusebenzisa umgxeki ophindwe kabini, i-EG3D [ ] inganciphisa ngoku-thile le ngxaki. Nangona kunjalo, ukukhuphela umphezulu one-textured kwiindlela ezisekelwe kwi-neural rendering yinto enzima ukuyenza. Ngokungafaniyo, i-GET3D yenza ngokuthe ngqo iimethi ezine-textured ze-3D ezinokuthi zisetyenziswe ngokulula kwiinjini zoyilo eziqhelekileyo. Ukusintyezwa kwemifanekiso ye-3D 45 43 14 7 57 49 26 25 76 8 51 58 67 7 57 49 25 8 3 Indlela Ngoku sethula isakhelo sethu se-GET3D sokwenza iimilo ze-3D ezinemibala. Inkqubo yethu yokuyila yahlulwe yaba ngamacandelo amabini: igalelo lejometri, elenza ngokuchazayo umethi womphezulu we-topology engakhethekiyo, kunye negalelo letshathi elenza i-texture field engakhiyelwa kwiindawo zomhlaba ukuvelisa imibala. Eyokugqibela ingandiswa kwezinye iipropathi zomhlaba ezifana, umzekelo, izinto (iSec. ). Ngexesha lokuqeqeshwa, i-rasterizer echazayo efanelekileyo iyasetyenziswa ukwenza imifanekiso ye-2D ephezulu kunye nemibala. Inkqubo yonke iyachazwa, ivumela uqeqesho olungakholelekiyo kusuka kwimifanekiso ye-2D (eneemaski ezibonisa into enomdla) ngokudlulisa ii-gradients ukusuka kumgxeki we-2D ukuya kwii-galelo zombini zenziwayo. Imodeli yethu iboniswe kwi-Fig. . Koku kulandelayo, siqala sethula isizukulwana sethu se-3D kwi-Sec , phambi kokuba siqhubeke nokudala okuchazayo kunye nemisebenzi yokulahlekelwa kwi-Sec . 4.3.1 2 3.1 3.2 3.1 Isizukulwana Seshishini Se-3D Ezinemibala Sijolise ekufundeni isizukulwana esizayo = ( ) ukudlulisa isampuli ukusuka kumlinganiselo weGaussian M, E G z ∈ N (0*,* ) ukuya kumethi enetexure . z I M E Ngokuba ikona enye ijometri ingaba neetexures ezahlukileyo, kwaye itexture enye ingasetyenziswa kwiijometri ezahlukileyo, siyakhetha ii-vectors zokungena ezingabonakaliyo 1 ∈ R512 kunye 2 ∈ R512. Ukulandela iStyleGAN [ , , ], sisebenzisa iinethiwekhi zokudala ezingezizo-linear geo kunye tex ukudlulisa 1 kunye 2 kwiivectors zangaphakathi ezibalulekileyo 1 = geo( 1) kunye 2 = tex( 2) ezisetyenziswa ngokuphinda zivelise izitayile ezilawula ukwenziwa kweemilo ze-3D kunye netexture, ngokulandelanayo. Siyilungelelanisa ngokusemthethweni imodeli yomvelisi wejometri kwi-Sec. kunye nemvelisi yetshathi kwi-Sec. . z z 34 35 33 f f z z w f z w f z 3.1.1 3.1.2 3.1.1 Umvelisi Wejometri Siyila isizukulwana sethu sejometri ukubandakanya i-DMTet [ ], imboniso yomphezulu echazayo ethi mva nje yasuswa. I-DMTet imele ijometri njengenkqubo yomgama echazayo (SDF) echazwe kwigridi ye-tetrahedral echazayo [ , ], apho umphezulu ungabuyiselwa ngokuchazayo nge-marching tetrahedra [ ]. Ukudibanisa igridi ngokuhambisa iivetes zakayo kubonelela ngokusetyenziswa okungcono kwesibonelelo sayo. Ngokwamkela i-DMTet yokukhuphela umphezulu, sinokwenza iimethi ezicacileyo ezinemivuzo engakhethekiyo kunye ne-genus. Sibe siphinda sibonelele ngesishwankathelo esifutshane se-DMTet kwaye sibhekisele kwiphepha lasekuqaleni ukuze ufumane iinkcukacha ezongezelelweyo. 60 22 24 17 Masithi ( ) ibonisa indawo epheleleyo ye-3D apho into ikhoyo, apho ziivetes kwigridi ye-tetrahedral . Yonke itetrahedron ∈ ichazwa kusetyenziswa iivetes ezine { }, kunye ∈ {1*, . . . , K*}, apho yeyona nani yeenkcukacha zetetrahedra, kwaye ∈ ∈ R3. Ngaphandle kweeyure zayo 3D, yonke ivete iqulethe ixabiso le-SDF ∈ R kunye ne-deformation ∆ ∈ R3 yevete ukusuka kwikhompyutha yayo yoqobo. Le mboniso ivumela ukubuyiswa kwemethi ecacileyo nge-marching tetrahedra [ ] echazayo, apho amaxabiso e-SDF kwindawo eqhubekayo abalwa ngokudibanisa okubalaseleyo kwe-barycentric yamaxabiso abo kwiivetes ezidibeneyo ′ = + ∆ . VT , T VT T Tk T v ak , v bk , v ck , v dk k K v ik VT , v ik v i si v i 60 si v v i v i Sidlulisa i-w1 ∈ R512 kumaxabiso e-SDF kunye ne-deformations kwiivetes nganye ngokuphindaphinda imijikelo ye-3D convolutional kunye neelayini ezidibeneyo ngokupheleleyo. Ngokukodwa, siphinda sisebenzise ii-convolutional layers ze-3D ukuvelisa ivolume yeempawu ezixhomekeke kwi-w1. Sibe siphinda siphendule iimpawu kwiivetes nganye ∈ kusetyenziswa i-trilinear interpolation kwaye siyifake kwi-MLPs enika ixabiso le-SDF kunye ne-deformation ∆ . Kwiimeko apho kufuneka imodeli kwisibonelelo esiphezulu (umzekelo, i-motorbike enezakhiwo ezincinci kumavili), siphinda sisebenzise ukuhlukana kwevolum ngokwe [ ]. I-Network Architecture v i v i VT si v i 60 Emva kokufumana kunye no-∆ kuzo zonke iivetes, sisebenzisa i-marching tetrahedra algorithm echazayo ukukhuphela imethi ecacileyo. I-marching tetrahedra ichaza i-topology yomphezulu ngaphakathi kutetrahedron nganye ngokusekelwe kwii-signes ze-si. Ngokukodwa, ubuso bomethi buyakhutshelwa xa sign( ) /= sign( ), apho ibonisa ii-indices zezikhondo eziphethe ubuso betetrahedron, kunye neevetes obo buso zichazwa ngokudibanisa okungafaniyo njengoko mi,j = v 0 i sj−v 0 j si sj−si . Qaphela ukuba ifomula engentla ihlolwa kuphela xa si 6= sj , ngoko ke ichazwa, kwaye ii-gradients ukusuka mi,j zingadlalwa kwakhona kwiixabiso le-SDF si kunye ne-deformations ∆vi . Ngale mboniso, iimilo ezinemivuzo engakhethekiyo zinokwenziwa ngokulula ngokulungisa ii-signes ezahlukileyo ze-si. Ukhupulo Lwemethi Echazayo si v i si sj i, j m i,j 3.1.2 Umvelisi Wetshathi Ukwenza ngokuthe ngqo itshathi yetexure ehambelana nemethi evelisiweyo akulula, njengoko imilo eyenziweyo ingaba ne-genus kunye ne-topology engakhethekiyo. Ngoko ke silungiselela itexure njengenkundla yetexure [ ]. 50 Ngokukodwa, sibeka imodeli yenkundla yetexure ngomsebenzi ophumelela indawo ye-3D yenxalenye yomphezulu ∈ R3, exhomekeke kwi-w2, kumbalo obomvu kunye noluhlaza okwesibhakabhaka ∈ R3 kwindawo leyo. Ngokuba itexure field ixhomekeke kwi-geometry, sibandakanya kwakhona le mapping kwikhowudi yegometri 1, ukuze = ( *,* 1 ⊕ 2), apho ⊕ ithetha ukudityaniswa. ft p c w c ft p w w Sibonisa imeko yetshathi yetexure kusetyenziswa imboniso ye-tri-plane, egqamileyo kwaye iyavakalisa ekwakheni kwakhona iimilo ze-3D [ ] kunye nokwenza imifanekiso ye-3D-aware [ ] . Ngokukodwa, silandela [ , ] kwaye sisebenzisa i-2D convolutional neural network enokuqhubeka nokwenza ikhowudi yedibaniso 1 ⊕ 2 ukuya kwiinqanaba ezintathu zempawu ezithintekayo ze-axis ze-N × N × ( × 3), apho = 256 ibonisa isibonelelo sesithuba kunye = 32 yeenkqubo. Inani lezikhondo. I-Network Architecture 55 8 8 35 w w C N C Ngemiphumo yempawu, iveti yempawu f t ∈ R 32 yenxalenye yomphezulu p ingafumaneka njenge f t = P e ρ(πe(p)), apho πe(p) iyimpawu zenx
