```html Abagwebi: Jun Gao, NVIDIA, University of Toronto, Vector Institute (jung@nvidia.com) Tianchang Shen, NVIDIA, University of Toronto, Vector Institute (frshen@nvidia.com) Zian Wang, NVIDIA, University of Toronto, Vector Institute (zianw@nvidia.com) Wenzheng Chen, NVIDIA, University of Toronto, Vector Institute (wenzchen@nvidia.com) Kangxue Yin, NVIDIA (kangxuey@nvidia.com) Daiqing Li, NVIDIA (daiqingl@nvidia.com) Or Litany, NVIDIA (olitany@nvidia.com) Zan Gojcic, NVIDIA (zgojcic@nvidia.com) Sanja Fidler, NVIDIA, University of Toronto, Vector Institute (sfidler@nvidia.com) Isishwancwelo Njengokuba iintlobo ezininzi zezoshishino zisiya ekubumbeni iindalo ezingaphakathi kwe-3D ezinkulu, imfuno yezixhobo zokudala imixholo ezinokukwazi ukwanda ngokobungakanani, umgangatho, kunye nokwahluka kweemixholo ze-3D iyabonakala. Kwimisebenzi yethu, sijolise ekqeqesheni iimodeli ezenza i-3D ezinokukwazi ukwenza i-meshes ezinezibhakabhaka ezinokuthi zingasetyenziswa ngqo yiinjini zokudweba i-3D, ngaloo ndlela zisebenziseke kwizicelo ezilandelayo. Imisebenzi yangaphambili kwimveliso ye-3D iphethe iinkcukacha ze-geometric, inomda kwi-topology ye-mesh enokuyenza, ngokwesiqhelo ayikuxhasi izibhakabhaka, okanye isebenzisa ii-renderers zenetwork kwinkqubo yokuphuma, eyenza ukusetyenziswa kwayo kwisoftware eqhelekileyo ye-3D kube nzima. Kulo msebenzi, siza kuzisa i-GET3D, imodeli yokwenza enerative enza ngokuthe ngqo i- xplicit extured meshes kunye ne-topology eyinkimbinkimbi, iinkcukacha ezicacileyo ze-geometric, kunye nezibhakabhaka ezikumgangatho ophezulu. Siyidibanisa impumelelo yakutshanje kwimodeli eyahlukileyo ye-surface, ukudweba okwahlukileyo kunye nee-2D Generative Adversarial Networks ukuqeqesha imodeli yethu kusuka kwiingqokelelo zemifanekiso ye-2D. I-GET3D iyakwazi ukwenza i-meshes ezenza i-3D ezikumgangatho ophezulu, ezisukela kwiimoto, izitulo, izilwanyana, iimoto ezimbini kunye nabalinganiswa babantu ukuya ezakhiweni, iphumeze ukuphuculwa okubalulekileyo phezu kweendlela zangaphambili. Iphepha leprojekthi yethu: G E T 3D https://nv-tlabs.github.io/GET3D 1 Intshayelelo Imixholo ye-3D eyahlukahlukeneyo, enomgangatho omkhulu iyanda ukubaluleka kwiintlobo ezininzi zezoshishino, kubandakanya imidlalo, iirobotics, ulwakhiwo, kunye namaqonga. Nangona kunjalo, ukwenziwa kwezinto ze-3D ngesandla kuthatha ixesha elininzi kwaye kufuna ulwazi oluthile lobuchwepheshe kunye neziphumo zobugcisa. Enye yeengxaki eziphambili ngoko ke yindawo - ngelixa unokufumana iimodeli ze-3D kwiimarike ze-3D ezifana neTurbosquid [ ] okanye iSketchfab [ ], ukwenza iimodeli ezininzi ze-3D ukuthi, ukugcwalisa umdlalo okanye ifilimu ngesihlwele sabalinganiswa ababonakala ngendlela eyahlukileyo kusafuneka kuthathwe ixesha elininzi lomzobi. 4 3 Ukulungiselela inkqubo yokudala imixholo nokuyenza ifumaneke kwiintlobo ezahlukeneyo zabasebenzisi (abasaqalayo), iinetswekhi ze-3D ezenza i-3D ezinokwenza izinto ze-3D ezikumgangatho ophezulu kunye nezahlukeneyo ziye zabangummandla osele ubalasele wophando [ , , , , , , , , , , ]. Nangona kunjalo, ukuze zisetyenziswe ngokufanelekileyo kuzo zonke izicelo zangoku zehlabathi, iimodeli ze-3D ezenza i-3D kufanele zifezekise iimfuno ezilandelayo: Kufuneka zibe namandla okwenza imilo eneenkcukacha ze-geometric kunye ne-topology engalawulekiyo, Isiphumo kufuneka sibe yi-mesh enezibhakabhaka, eyona ndlela yokumela esetyenziswa ziipakeji eziqhelekileyo zesoftware yemizobo efana neBlender [ ] kunye neMaya [ ], kwaye Kufuneka sikwazi ukusebenzisa imifanekiso ye-2D ukuze sifundiswe, njengoko ifumaneka kakhulu kunamathala ezimo ezingaphandle ze-3D. 5 14 43 46 53 68 75 60 59 69 23 (a) (b) 15 1 (c) Umsebenzi wangaphambili kwimveliso ye-3D ugxile kwiingongoma ezithile zezimfuno ezingentla, kodwa akukho ndlela ukuza kuthi ga ngoku iyifezekisileyo zonke (Tab. ). Ngokomzekelo, iindlela ezivelisa ii-point clouds ze-3D [ , 68, 75] ngokwesiqhelo azivelisi izibhakabhaka kwaye kufuneka ziguqulwe zibe yi-mesh kwinkqubo yokugqibela. 1 5 Iindlela ezivelisa ii-voxels kaninzi azikho iinkcukacha ze-geometric kwaye azivelisi izibhakabhaka [ , , , ]. Iimodeli zokwenza imiba ye-neural fields [ , ] zigxile ekukhupheni i-geometry kodwa azinaki izibhakabhaka. Kakhulu kwezi zifuna inkxaso ye-3D ecacileyo. Ekugqibeleni, iindlela ezivelisa ngokuthe ngqo ii-meshes ezenza i-3D ezinezibhakabhaka [ , ] ngokwesiqhelo zifuna iitemplate zemilo ezichazwe kwangaphambili kwaye azikwazi ukwenza imilo enetopology eyinkimbinkimbi kunye ne-genus eyahlukahlukeneyo. 66 20 27 40 43 14 54 53 Kutshanje, ukuqhubela phambili ngokukhawuleza kwi-neural volume rendering [ ] kunye nee-2D Generative Adversarial Networks (GANs) [ , , , , ] kuye kwakhokelela ekubeni kukhutshwe imifanekiso ye-3D-aware synthesis [ , , , , , ]. Nangona kunjalo, lo msebenzi ugxile ekwenzeni imifanekiso emininzi ye-view-consistent ngokusebenzisa i-neural rendering kwinkqubo yokwenza kwaye awuqinisekisi ukuba iimilo ze-3D eziqondakalayo zinokwenziwa. Ngelixa i-mesh ingafunyanwa kwi-representation yenetwork yenetwork kusetyenziswa indlela ye-marching cube [ ], ukukhupha isibhakabhaka esihambelanayo akulula. 45 34 35 33 29 52 7 57 8 49 51 25 39 Kulo msebenzi, siza kuzisa indlela entsha ejolise ukujongana zonke izimfuno zemodeli ye-3D eyenza imisebenzi. Ngokukodwa, siza kuzisa i-GET3D, imodeli yokwenza enerative yeemilo ze-3D ezivelisa ngokuthe ngqo ii- xplicit extured meshes eneenkcukacha eziphezulu ze-geometric kunye ne-texture kunye ne-topology ye-mesh engalawulekiyo. Kwingqiqo yendlela yethu yinkqubo yokwenza esebenzisa indlela yokukhupha umphezulu [ ] kunye netekhnoloji yokudweba eyahlukileyo [ , ]. Eyokuqala isivumela ukuba siqwalasele ngokuthe ngqo kwaye senze ii-meshes ze-3D ezinezibhakabhaka ezinayo nayiphi na i-topology, ngelixa eyokugqibela isivumela ukuba siqeqeshe imodeli yethu ngemifanekiso ye-2D, ngaloo ndlela sisebenzise ii-discriminators ezinokubaluleka kwaye ziphuculwe ezenzelwe ukwenziwa komfanekiso we-2D. Njengoko imodeli yethu ivelisa ngokuthe ngqo ii-meshes kwaye isebenzisa umdwebi onobuchule (owahlukileyo) wezobukrekrele, singakwazi ukwandisa imodeli yethu ukuze iqeqeshwe ngemifanekiso G E T 3D eyahlukileyo 60 47 37 isinyaniso esingangama-1024 × 1024, esisivumela ukuba sifunde iinkcukacha eziphezulu ze-geometric kunye ne-texture. Sibonisa umgangatho wobugcisa kwisayensi phambili kwisayensi yokwenza imilo ye-3D engaphandle kwisigaba esithile kwiikhetegori ezininzi ezinemilo eyinkimbinkimbi ezivela kwiShapeNet [ ], iTurbosquid [ ] kunye neRenderpeople [ ], ezifana nezigulo, iimoto ezimbini, iimoto, abalinganiswa babantu, nezakhiwo. Ngokwenza i-mesh ecacileyo njengemveliso yokubonisa, i-GET3D ikwanomoya kakhulu kwaye ingalungiswa ngokulula kwezinye izicelo, kubandakanya: ukufunda ukwenza imathiriyeli ehlulweyo kunye neziphumo zokukhanya ezixhomekeke kumbono kusetyenziswa ukudweba okuhlulekayo okungaboniswanga [ ], ngaphandle kwenkxaso, ukwenziwa kwe-3D okukhokhelwa yimibhalo kusetyenziswa i-CLIP [ ] embedding. 9 4 2 (a) 12 (b) 56 2 Umsebenzi Ohambelanayo Siyiphonononga ingqalelo yophuhliso olutshanje kwiimodeli ze-3D ezenza izinto zombini ze-geometry kunye nokubonisa, kunye nokwenziwa kwe-3D-aware generative image synthesis. Kule minyaka ingoku, iimodeli ze-2D ezenza izinto ziye zavuza umgangatho we-photorealistic ekwenzeni imifanekiso ephezulu-ye-resolution [ , , , , , , ]. Le nkqubela phambili iphinde yakhuthaza uphando kwisizukulwana semixholo ye-3D. Iindlela zokuqala zijolise ekwandiseni ngokuthe ngqo i-2D CNN generators ukuya kwii-3D voxel grids [ , , , , ], kodwa umthwalo omkhulu wenkumbulo kunye nobunzima bokubala be-3D convolutions butshabalalisa inkqubo yokwenza kwinqanaba eliphezulu. Njengenye indlela, eminye imisebenzi iye yaphonononga i-point cloud [ , , , ], engabonakaliyo [ , ], okanye i-octree [ ] representations. Nangona kunjalo, le misebenzi igxile kakhulu ekwenzeni i-geometry kwaye ayinaki ukubonakala. Imiboniso yayo yesiphumo ikwafuna ukwenziwa kwinkqubo yokugqibela ukuze ihambelane neenjini zeqashiso eziqhelekileyo. Iimodeli ze-3D ezenza izinto 34 35 33 52 29 19 16 66 20 27 40 62 5 68 75 46 43 14 30 Ngaphezu kwemisebenzi yethu, i-Textured3DGAN [ , ] kunye ne-DIBR [ ] yenza ii-meshes ezenza i-3D ezinezibhakabhaka, kodwa zibeka imveliso njengokukhutshwa kwe-mesh yetemplate, okuthintela ukwenziwa kwemilo enetopology eyinkimbinkimbi okanye imilo eno-genus eyahlukileyo, nto leyo imodeli yethu enokuyenza. I-PolyGen [ ] kunye ne-SurfGen [ ] zingavelisa ii-meshes ezinama-topology angalawulekiyo, kodwa azivelisi izibhakabhaka. 54 53 11 48 41 Kukhuthazwe yimpumelelo ye-neural volume rendering [ ] kunye ne-implicit representations [ , ], imisebenzi yakutshanje iye yaqala ukujongana nengxaki yokuboniswa kwe-3D-aware image synthesis [ , , , , , , , , , ]. Nangona kunjalo, iinetswekhi ze-neural volume rendering zidla ngokuba nezilumko ukuzibuza, zikhokelela kumaxesha aqeqeshayo abandayo [ , ], kwaye zenze imifanekiso yesinyaniso esingaphandle. I-GIRAFFE [ ] kunye ne-StyleNerf [ ] ziphucula uqoqosho lokuqeqesha kunye nokudweba ngokudweba i-neural kwinqanaba eliphantsi kwaye emva koko uphucule iziphumo nge-CNN ye-2D. Nangona kunjalo, inzuzo yokusebenza iza ngexabiso lokunciphiswa kokuhambelana kwe-multi-view. Ngokusebenzisa i-discriminator emibini, i-EG3D [ ] ingayinciphisa le ngxaki. Nangona kunjalo, ukukhupha umphezulu onezibhakabhaka kwiindlela ezisekwe kwi-neural rendering yingxaki enzima ukuyijongana nayo. Ngokungafaniyo, i-GET3D ivelisa ngokuthe ngqo ii-meshes ze-3D ezinezibhakabhaka ezisetyenziswayo kwiinjini zeqashiso eziqhelekileyo. Ukuboniswa kwe-3D okwenza imiba eyahlukeneyo 45 43 14 7 57 49 26 25 76 8 51 58 67 7 57 49 25 8 3 Indlela Ngoku siza kwazisa isakhelo sethu se-GET3D sokwenza iimilo ze-3D ezinezibhakabhaka. Inkqubo yethu yokwenza ihlukaniswe yaba ziingxenye ezimbini: i-geometry branch, eyivelisa ngokuhluleka i-mesh yemphezulu enama-topology angalawulekiyo, kunye ne-texture branch eyenza intsimi yesibhakabhaka enokuthi ibuzwe kwiindawo zomphezulu ukuze ifumane imibala. Ezi zokugqibela zingandiswa kwezinye iipropathi zomphezulu ezifana, umzekelo, izinto (u-Sec. ). Ngexesha lokuqeqesha, i-rasterizer eyi-differentiable esebenza kakuhle iyakusetyenziswa ukudweba i-mesh enezibhakabhaka eyenziweyo kwimifanekiso ye-2D ephezulu-ye-resolution. Inkqubo yonke iyahluleka, ivumela uqeqesho lwe-adversarial kusuka kwimifanekiso (ngemaski ebonisa into enomdla) ngokusasaza ii-gradients ukusuka kwi-2D discriminator ukuya kwi-generator branches zombini ze-2D. Imodeli yethu iboniswe kuMzekelo . Ngokulandelayo, siqala sibonise i-generator yethu ye-3D kwi-Sec , ngaphambi kokuba siqhubeke nokudweba okuhlulekayo kunye nemisebenzi elahlekileyo kwi-Sec . 4.3.1 2 3.1 3.2 3.1 Imveliso ye-3D ye-3D enezibhakabhaka Sijolise ekufundeni i-generator ye-3D = ( ) ukubeka isampuli ukusuka kwindawo enye ye-Gaussian M, E G z ∈ N (0*,* ) ukuya kwi-mesh enesibhakabhaka . z I M E Njengoko i-geometry efanayo inokuba nezibhakabhaka ezahlukeneyo, kwaye isibhakabhaka esifanayo singasetyenziswa kwii-geometries ezahlukileyo, siyasampula ii-vectors zokufaka ezingahleliweyo 1 ∈ R512 kunye 2 ∈ R512. Ngokulandela i-StyleGAN [ , , ], ke sisebenzisa iinetswekhi zokudibanisa ezingagciniyo geo kunye tex ukudibanisa 1 kunye 2 kwi-vectors elingaphakathi kwe-latent 1 = geo( 1) kunye 2 = tex( 2) eziya kusetyenziswa ngokubhekayo ukwenza izitayile * ezinokulawula imveliso yeemilo ze-3D kunye nesibhakabhaka, ngokulandelanayo. Siyibonisa ngokusemthethweni i-generator ye-geometry kwi-Sec. kunye ne-texture generator kwi-Sec. . z z 34 35 33 f f z z w f z w f z 3.1.1 3.1.2 3.1.1 Imveliso ye-Geometry Siyilwe imveliso yethu ye-geometry ukubandakanya i-DMTet [ ], imelemphezulu eyahlukileyo eyaphunyezwa kutshanje. I-DMTet imela i-geometry njenge-signed distance field (SDF) echazwe kwi-grid yethitettrahedral ehambelayo [ , ], apho umphezulu ungakwazi ukufunyanwa ngokuhluleka nge-marching tetrahedra [ ]. Ukudibanisa i-grid ngokuhambisa ii-vertices zayo kubangela ukusetyenziswa okungcono kwesinyaniso sayo. Ngokwamkela i-DMTet yokukhupha umphezulu, singavelisa ii-meshes ezicacileyo enama-topology kunye ne-genus angalawulekiyo. Emva koko sibonelela ngesishwankathelo esifutshane se-DMTet kwaye sibhekise umfundi kwiphepha lokuqala ngeenkcukacha ezongezelekileyo. 60 22 24 17 Masithi ( ) zichaze indawo epheleleyo ye-3D apho into ikhoyo, apho zii-vertices kwi-grid yethitettrahedral . Ilitertrahedron ngalinye ∈ lichazwa kusetyenziswa ii-vertices ezine { }, kunye ∈ {1*, . . . , K*}, apho lilonke inani litertrahedra, kunye ∈ ∈ R3. Ngaphezu kwezi-3D zayo coordinates, i-vertex ngalinye liqulethe ixabiso le-SDF ∈ R kunye nokudibanisa ∆ ∈ R3 kwe-vertex ukusuka kwikhomputha yayo yokuqala yesiqhelo. Le mbeleko ivumela ukufunyanwa kwe-mesh ecacileyo nge-marching tetrahedra [ ] eyahlukileyo, apho amaxabiso e-SDF kwindawo eqhubekayo abalwa ngokudibanisa kwe-barycentric ixabiso labo kwi-vertices ezidibene ′ = + ∆ . VT , T VT T Tk T v ak , v bk , v ck , v dk k K v ik VT , v ik v i si v i 60 si v v i v i Sibeka 1 ∈ R512 kwiixabiso ze-SDF kunye nezi-deformations kwi-vertex nganye ngeenethiwekhi ezithile ze-3D convolutional kunye ne-fully connected layers. Ngokukodwa, siqala sisebenzise iinetswekhi ze-3D convolutional ukwenza umthamo we-feature oxhomekeke kwi- 1. Emva koko sibuza i-feature kwi-vertex nganye ∈ kusetyenziswa i-trilinear interpolation kwaye siyibeke kwi-MLPs eyenza ixabiso le-SDF kunye ne-deformation ∆ . Kwiimeko apho ukubumba kwinqanaba eliphezulu lifuneka (umzekelo, imoto eneendawo ezimxinwa kumavili), siphinda sisebenzise ukwahlula komthamo ngokulandela [ ]. Ukuyilwa kweNetwọkhi w v i w v i VT si v i 60 Emva kokufumana kunye ∆ kuzo zonke ii-vertices, sisebenzisa indlela yokukhupha i-mesh ecacileyo eyenziwe nge-marching tetrahedra. I-Marching tetrahedra igqiba i-topology yomphezulu ngaphakathi kwitertrahedron ngalinye ngokusekelwe kwii-signs ze- . Ngokukodwa, ubuso be-mesh buyakhutshelwa xa sign( ) /= sign( ), apho zibonisa ii-indices zii-vertices emacaleni etertrahedron, kwaye ii-vertices zayo buso zigqitywa ngokudibanisa komgca njengoko mi,j = v 0 i sj−v 0 j si sj−si . Qaphela ukuba ifomula engentla ingasetyenziswa kuphela xa si 6= sj , ngoko ke iyahluleka, kwaye igradient evela kwi-mi,j ingabuyiselwa kwiixabiso ze-SDF si kunye nezi-deformations ∆vi . Ngale mbeleko, iimilo ezinama-topology angalawulekiyo zinokwenziwa ngokulula ngokubeka ii-signs ezahlukeneyo ze-si. Ukukhupha iMesh okuhlulekayo si v i si si sj i, j m i,j 3.1.2 Imveliso ye-Texture Ukukwenza ngokuthe ngqo i-texture map ehambelana ne-mesh eyenziweyo akulula, njengoko imilo eyenziweyo ingaba ne-genus kunye ne-topology engalawulekiyo. Ngoko ke silinganisa i-texture field [ ]. 50 Ngokukodwa, sibumba i-texture field kunye nomsebenzi oya kwenza indawo ye-3D yendawo yomphezulu ∈ R3, exhomekeke kwi- 2, kwimibala ye-RGB ∈ R3 kwindawo leyo. Njengoko i-texture field ixhomekeke kwi-geometry, songeza ukubeka iliso kwesi dibaniso kwikhowudi ye-geometry latent 1, ukuze = ( *,* 1 ⊕ 2), apho ⊕ kuthetha ukudibanisa. ft p w c w c ft p w w Siyimela i-texture field yethu kusetyenziswa imelemphezulu ye-tri-plane, ekhuthazayo kwaye ichaza kakhulu ekubuyiseni iimilo ze-3D [ ] kunye nokwenza imifanekiso ye-3D-aware [ ]. Ngokukodwa, silandela [ , ] kwaye sisebenzisa inetswekhi ye-2D convolutional neural network ukubeka ikhowudi ye-latent 1 ⊕ 2 kwiintsika ezintathu ze-feature ezi-orthogonal ezilingana nobungakanani obuyi- × × ( × 3), apho = 256 ibonisa isiganeko sobukhulu kwaye = 3 Ukuyilwa kweNetwọkhi 55 8 8 35 w w N N C N C
