Ngaphakathi kweMonolith yeByteDance: Injini Enika Amandla iiFeeds eziSmart, eziKhawulezayo zeMidiya

```html ``` Abaholi: Zhuoran Liu, Bytedance Inc. Leqi Zou, Bytedance Inc. Xuan Zou, Bytedance Inc. Caihua Wang, Bytedance Inc. Biao Zhang, Bytedance Inc. Da Tang, Bytedance Inc. Bolin Zhu∗, Fudan University Yijie Zhu, Bytedance Inc. Peng Wu, Bytedance Inc. Ke Wang, Bytedance Inc. Youlong Cheng†, Bytedance Inc. (youlong.cheng@bytedance.com) ISISHUNGO Ukwakha inkqubo yokucebisa enamandla kwaye ekhoyo ngexesha kubalulekile kumashishini amaninzi axhaswa yimpendulo yabathengi exhomekeke kwixesha, njengokuhlela iividiyo ezimfutshane okanye iintengiso ezikwi-intanethi. Ngaphandle kokwamkelwa ngokubanzi kwemisebenzi yenkqubo yokufunda enzulu efana ne-Ten-sorFlow okanye i-PyTorch, le misebenzi yeenjongo ezibanzi ayanelisi iimfuno zoshishino kwiimeko zokucebisa ngezizathu ezahlukeneyo: ngokubhekisele kwenye indawo, ukulungisa iinkqubo ngokusekelwe kwizinto ezingaguqukiyo kunye nokubala okubalulekileyo kokucebisa ngeempawu eziguquguqukayo nezicaba kunobutyelelo lomgangatho wobume; kwelinye icala, imisebenzi enjalo yenzelwe iqela lokuqeqesha kunye nenqanaba lenkonzo lihlukeneyo ngokupheleleyo, kuthintelwa ubume ukuba bungasebenzisani nempendulo yomthengi ngexesha lokwenyani. Le ngxaki yasenza sahluza iindlela zemveli kwaye sahluza ukhetho olwahlukileyo kuyilo. Kwiphepha elinye, sibonisa i- 1, inkqubo eyenzelwe ukuqeqesha kwi-intanethi. Uyilo lwethu luqhutywa ngokuphawula kwemisebenzi yesicelo sethu kunye nendawo yokuvelisa ebonisa ukuhla kakhulu kwiinkqubo zokucebisa ezinye. Igalelo lethu likhulu: Okokuqala, sakha itheyibhile yokufaka engenako ukungqubana kunye nokusetyenziswa kwezinto ezifana neendawo zokubeka ezichanekileyo kunye nokucoca rhoqo ukunciphisa isiseko sayo sememori; okwesibini, sibonelela ngolwakhiwo lokuqeqesha olukwi-intanethi olulungele ukuveliswa kunye nokubonelela okukhulu ngobutyala; ekugqibeleni, sabonisa ukuba ukuthembeka kwenkqubo kungatshintshwa ukuze kufundwe ngexesha lokwenyani. IMonolith iphumelele ukubekwa kwimveliso yeBytePlus Recommend2. Monolith 1 INTSHAYELELO Iminyaka elishumi idlulileyo ibone ubuninzi bamashishini axhaswa ngeendlela zokucebisa. Ekufuneni inkonzo engcono yomthengi, ukubonelela ngomxholo owenzelwe umntu wonke njengempendulo yexesha lokwenyani yinjongo eqhelekileyo kwezi zicelo zoshishino. Ukwenza oku, ulwazi oluvela kwingeniso yokugqibela yomsebenzisi lusetyenziswa rhoqo njengenkxaso ephambili yokuqeqesha ubume, njengoko luya kubonisa kakuhle umfanekiso womsebenzisi kwaye benze izibikelelo zenzalo yomsebenzisi kunye nezenzo zexesha elizayo. Ukuqeqeshwa okunzulu kuye kwakhalipha iindidi zokucebisa [ , , , , , ] njengoko isixa esikhulu sedatha yomsebenzisi ilungele iindidi zemodeli ezinoxolelo olukhulu. Nangona kunjalo, iinzame zokusebenzisa amandla okufunda okunzulu kwiinkqubo zokucebisa zeshishini zidibana rhoqo neengxaki ezivela kwiimpawu ezizodwa zedatha evela kwizenzo zomsebenzisi zehlabathi. Le datha yahlukile kakhulu kwezo zisetyenziswa kwiingxaki zemveli zokufunda enzulu ezifana nokumodela ulwimi okanye umbono womfundi ngekhompyutha ngeendlela ezimbini: 5 6 10 12 20 21 (1) Iimpawu zihlala zicaba, ziyabonakala kwaye zitshintsha ngokungaguqukiyo; (2) Unikezelo olungaphantsi lodumo lokufunda aluzinzanga, okanye i-Concept Drift [8]. Oko kwahlukileyo kuye kwabeka imingeni eyodwa kubaphandi kunye neenjineli ezisebenza kwiinkqubo zokucebisa. 1.1 Ukucaba kunye nokungaguquguquki Idatha yokucebisa iqulethe iimpawu ezipeyileyo zodumo, ezinye zazo zibonakala rhoqo. Ukuziqhelisa rhoqo ukuzibeka kwindawo enkulu yendawo yokuqokelela kwiindawo ezininzi kuya kubangela ingxaki ezininzi: • Ngokungafaniyo neenodeli zolwimi apho inani lezicwengu zegama linemingcelelo, inani labasebenzisi kunye nezinto zokuhlela likhulu kakhulu. Itheyibhile enje enkulu yokuqokelela kunzima ukuba ihlangane kwimemori yomgcini omnye; • Ngaphezu koko, ubukhulu betheyibhile yokufaka kulindeleke ukuba bukhule ngokuhamba kwexesha njengoko abasebenzisi kunye nezinto ezininzi zongezwa, ngelixa imisebenzi efana ne [ , ] isebenzisa izinto ezibonelelweyo ezinesayizi enye ukumela itheyibhile yokufaka. 1 17 Ngaphandle koko, iinkqubo ezininzi zisebenzisa i-hashing enoko-ngqubana [ , ] njengendlela yokunciphisa isiseko sememori kunye nokuvumela ukukhula kwezisazisi. Oku kuxhomekeke kwisimo esingekho nqeneha sokuba izazisi etheyibhile yokufaka zisasazeka ngokulinganayo kwintlonipho, kwaye ukungqubana akulimazi ubungakanani bobume. Ngelishwa oku kunqabile ukuba kube yinyani kwinkqubo yokucebisa ye-real-world, apho iqela elincinci labasebenzisi okanye izinto zinomdla ngaphezu kwezinye. Ngokukhula komzimba kobukhulu betheyibhile yokufaka, amathuba okungqubana kwezikhuni ze-hash ayakwanda kwaye ibangele ukudodobala kobungakanani bobume [ 3 6 [3]. Ngoko ke, yimfuneko yendalo kwiinkqubo zokucebisa zoshishino ukuba zibe namandla okubamba zonke iimpawu kwiiparameter zazo, kwaye zikwanamandla okukhulisa ngokulinganayo inani labasebenzisi kunye nezinto abazama ukuzigcina. 1.2 Ukusasazeka okungazinzanga Imifanekiso yobona kunye neyolwimi ayinakuthi iphumelele kwixesha lembali, ngelixa umsebenzisi ofanayo onomdla kwisihloko esithile angatshintsha umdla wakhe rhoqo ngomzuzu. Ngenxa yoko, ulwabiwo olungaphantsi lwedatha yomsebenzisi aluzinzanga, into edla ngokubizwa ngokuba yi-Concept Drift [8]. Ngokucacileyo, ulwazi oluvela kwimbali yakutshanje lunganegalelo ngokusebenzayo ekubikeleleni utshintsho kwizenzo zomsebenzisi. Ukunciphisa impembelelo ye-Concept Drift, iimodeli zenkonzo zifuna ukuhlaziywa kwiimpendulo zabathengi abatsha njengoko kukhawuleza kangakanani na ngokusekelwe kwizinto ezithandwa ngumsebenzisi. Ngokukhanyisa olu kwahlukana kunye nokuphawula kwiingxaki ezivela kwimveliso yethu, sakha i- , inkqubo enkulu yokucebisa ukujongana nale ngxaki. Senze uvavanyo olubanzi ukuqinisekisa kunye nokuhlaziya uyilo lwethu kwindawo yokuvelisa. IMonolith iyakwazi Monolith (1) Ukubonelela ngamandla apheleleyo weempawu ezicaba ngokuyila itheyibhile ye-hash engenako ukungqubana kunye nenkqubo yokugqibezela iimpawu eziguquguqukayo; (2) Ukujikeleza imbuyekezo yenkonzo kwinkqubo ukuqeqesha ngexesha lokwenyani kunye nokuqeqesha kwi-intanethi. Ixhaswe ngala mandla olwakhiwo, iMonolith iqhubeka nokugqitha iinkqubo ezisebenzisa iihash-tricks kunye nokungqubana ngobukhulu obufanayo bememori, kwaye ifumana i-AUC yokukhonza kwi-intanethi ngokugqwesileyo ngaphandle kokubeka umthwalo omkhulu kumandla okubala kwee-servaka zethu. Elinye iphepha lihlelwa ngolu hlobo. Siyichaza kuqala inkcukacha zoyilo malunga nendlela iMonolith ejongana ngayo nemingeni ekhoyo kunye netheyibhile ye-hash engenako ukungqubana kunye nokuqeqesha ngexesha lokwenyani kwiCandelo le-2. Uvavanyo kunye neziphumo ziya kuboniswa kwiCandelo 3, kunye nezicombululo ezivavanyiweyo zemveliso kunye nokuxoxa ngeengozi phakathi kobuchule bexesha, ukuthembeka, kunye nomgangatho wobume. Icandelo 4 lishwankathela umsebenzi ofanayo kwaye lisiqhathanisa kunye neMonolith. Icandelo 5 lishwankathela lo msebenzi. 2 UYILO Ulwakhiwo lwonke lweMonolith lulandela ngokubanzi inkqubo yeTensorFlow eyahlulahluliweyo yeWorker- arameter erver (uMgcawo Kwindlela yeWorker-PS, oomatshini banikwa imisebenzi eyahlukileyo; oomatshini beWorker banoxanduva lokwenza izibalo njengoko kuchaziwe kwigrafu, kwaye oomatshini bePS bagcina iziparameter kwaye bazihlaziye ngokusekelwe kwii-gradients ezibaliweyo yiWorkers. P S 2). Kwiindidi zokucebisa, iziparameter zihlelwa kwiinqanaba ezimbini: ezicaba nezicaba. Iiparameter ezicaba ziziqhakalisa/izinto eziguquguqukayo kwinethiwekhi enzulu ye-neural, kwaye iziparameter ezipeyileyo zibhekisa kwiitheyibhile zokufaka ezihambelana neempawu ezipeyileyo. Kwindalo yethu, iziparameter ezicaba nezicaba zombini ziyinxalenye yeGrafu yeTensorFlow, kwaye zigcinwa kwiiparameter server. Ngokufanayo neVariable yeTensorFlow yeeparameter ezicaba, sakha iseti yeHashTable ezisebenza kakhulu, ezingenawo ukungqubana, kunye nezinamathuba eninzi zeempawu ezipeyileyo. Njengongezo kuthintelweyo kweTensor-Flow okuvela ekuhlukaneni kokuqeqesha nokuhlalutya, ukuqeqesha okukhulu kweMonolith okuguquguqukayo kwi-intanethi kwenzelwe ukuhambelana ngokusebenzayo iziparameter ukusuka ku-training-PS ukuya kwi-serving-PS kwiimeko ezimfutshane, kunye nesiqinisekiso sobume obuqinileyo obubonelelwa yinkqubo yokuhlangana nokungaphumeleli. 2.1 Itheyibhile yeHash Umgaqo wokuqala kwindalo yethu yeengcaciso zesiparameter esicaba kukuphepha ukucofa ulwazi oluvela kuzo zonke izisazisi kwindawo yokugcina efanayo. Ukulinganisa itheyibhile yokufaka enobukhulu obuguquguqukayo kunye neVariable yeTensorFlow engaphandle kwebhokisi kuya kubangela ukungqubana kwezazisi, okwenza kubi njengoko izazisi ezintsha zifikayo kwaye itheyibhile ikhula. Ngoko ke, endaweni yokwakha phezu kweVariable, saphuhlisa iHashTable entsha yesitshixo-ixabiso yeeparameter zethu ezicaba. IHashTable yethu isebenzisa iCuckoo Hashmap [ ] ngaphakathi, evumela ukufakwa kwezikhuni ezintsha ngaphandle kokungqubana nezikhoyo. I-Cuckoo Hashing ifumana ixesha elibi kakhulu 𝑂(1) lexabiso kunye nokukhutshwa, kunye nexesha elilindelekileyo elithotyelweyo 𝑂(1) lofakelo. Njengoko ibonisiwe kuMgcawo igcina iitheyibhile ezimbini 𝑇0, 𝑇1 kunye nemisebenzi eyahlukileyo ye-hash ℎ0(𝑥), ℎ1(𝑥), kwaye into iya kugcinwa kwenye yazo. Xa uzama ukufaka into 16 3 𝐴 kwii-𝑇0, kuqala izama ukubeka 𝐴 kwii-ℎ0(𝐴); Ukuba ℎ0(𝐴) ihlelwe nenye into 𝐵, iya kukhupha 𝐵 kwi-𝑇0 kwaye izame ukufaka 𝐵 kwii-𝑇1 ngendlela efanayo. Le nkqubo iya kuphinda ize zonke izinto zizinzise, okanye i-rehash yenzeke xa ukufakelo lungena kwisehlobo. Ukunciphisa isiseko sememori nayo yinto ebalulekileyo kwindalo yethu. Indlela elula yokufaka yonke isazisi entsha kwiHashTable iya kunciphisa imemori ngokukhawuleza. Ukuphawula kwiindidi zemveliso zangempela kukhokelele kwiingcamango ezimbini: (1) Izazisi ezibonakala kuphela ngamaxesha ambalwa zinegalelo elincinci ekuphuculeni ubungakanani bobume. Ukuphawula okubalulekileyo kukunika izisazisi ezithandwayo zenzeka izigidi zeehlandlo ngelixa ezingathandwa kakhulu zenzeka zingadlulanga izihlandlo ezilishumi. Ukufakela okuhambelana kwezi zazisi ezingavamileyo azilunganga ngenxa yokusweleka kwedatha yokuqeqesha kwaye ubume ngeke bakwazi ukwenza umcimbi olungileyo ngokusekelwe kuzo. Ekupheleni kosuku ezi zazisi azinakubangela impembelelo kwisiphumo, ngoko ubungakanani bobume ngekhe buhlungwe kukuqhawuka kwezi zazisi ezinenani elincinci lokuvela; (2) Izazisi ezindala ezivela kwimbali ekude kunqamle iingxowa ezincinci kwiimodeli zangoku njengoko uninzi lwazo lungazange lutyelelwe. Oku kungaba ngenxa yomsebenzisi ongasekhoyo, okanye ividiyo emfutshane iphelelwe lixesha. Ukugcina izifakelo kwezi zazisi akunakunceda ubume nangayiphi na indlela ngaphandle kokuchitha imemori ye-PS yethu ngeze. Ngokusekelwe kwezi zibakala, sakha iindidi ezininzi zokucoca iimpawu zezazisi ukuze zisebenze kakhulu kwiHashTable esebenzisa imemori kancinci: (1) Izazisi ziyacocwa ngaphambi kokuba zivunyelwe kwiitheyibhile zokufaka. Sineendlela ezimbini zokucoca: Kuqala sicoca ngokweenkozo zazo ngaphambi kokuba zifakwe njengezikhuni, apho umda wokubala khona ungumqondiso ongaguquguqukayo owahlukileyo kwiimodeli nganye; Ukongeza, sisebenzisa isicoci esithile esinceda ukunciphisa ngakumbi ukusetyenziswa kwememori; ![#Figure 4: Streaming Engine. Isilawuli sempendulo yolwazi ukusuka ku [Umsebenzisi → Model Server → Training Worker → Model Server → Umsebenzisi] siya kuchitha ixesha elide xa sithatha indlela yeBatch Training, ngelixa iOnline Training iya kuvala isilawuli ngokukhawuleza.](https://cdn.hackernoon.com/images/InxBRjRIs6M1kdhuWcyNHiiUrxm1-78b3ep4.jpeg) (2) Izazisi zibhalwa kwaye zibekwe ukuba ziphele emva kokuba zingasebenzi ixesha elibekiweyo. Ixesha lokuphelelwa kukwanamathuba okugcinwa kwitheyibhile nganye yokufaka ukuvumela ukwahlula iimpawu ngeempawu ezahlukeneyo zoxolelo kwimbali. Kwintlungu yethu, iHashTable yenziwa njengomsebenzi we-TensorFlow wezixhobo. Ngokufanayo ne-Variable, ukukhangela kunye nokuhlaziya nako kwenziwa njengeenkqubo zeTensorFlow zemveli zokudibanisa lula kunye nokuhambelana okungcono. 2.2 UQEQESHO KWI-INTANETHI EMonolith, ukuqeqesha kwahlulwe kwiinqanaba ezimbini (uMgcawo 1): (1) Iqela lokuqeqesha ibatch. Eli qonga lisebenza njengesiqhelo se-TensorFlow training loop: Kwinqanaba ngalinye lokuqeqesha, umqeqeshi wokuqeqesha ufunda ibhetshi encinci yezibonelo zokuqeqesha kwindawo yokugcina, ucela iziparameter kwi-PS, ubale phambili nangasemva, kwaye ekugqibeleni ucofe iziparameter ezihlaziyiweyo kwi-training PS. Ngokungafaniyo nezinye iingxaki ezikhulu zokufunda enzulu, siqeqesha idatha yethu yeqela kuphela. Ukuqeqesha ibatch kuluncedo ekuqeqesheni idatha yembali xa silungisa ulwakhiwo lwethu lobume kwaye siyiqeqeshe kwakhona; (2) Iqonga lokuqeqesha kwi-intanethi. Emva kokuba ubume bubekwe kwinkonzo ye-intanethi, ukuqeqesha akuyeki kodwa kungena kwinqanaba lokuqeqesha kwi-intanethi. Endaweni yokufunda iibhetshi ezincinci ezivela kwindawo yokugcina, umqeqeshi wokuqeqesha usebenzisa idatha yexesha lokwenyani ngokukhawuleza kwaye uhlaziye i-training PS. I-training PS ihambelana rhoqo iziparameter zayo kwi-serving PS, eziya kusebenza kumsebenzisi ngoko nangoko. Oku kuvumela ubume bethu ukuba busebenzisane ngokuzimeleyo ngokwempendulo yomsebenzisi ngexesha lokwenyani. IMonolith yakhelwe ngamandla okutshintsha ngokungenamthungo phakathi kokuqeqesha ibatch kunye nokuqeqesha kwi-intanethi. Oku kuvunywa kuyilo lwethu lwenjini yokusasaza njengoko iboniswe kuMgcawo 2.2.1 Injini yeStreaming. 4. Kwindalo yethu, sisebenzisa umjikelo omnye weKafka [ ] ukubhalisa izenzo zabasebenzisi (Umzek., Cofa kwinto okanye uthande into njalo) kwaye omnye umjikelo weKafka weempawu. Kwingingqi yenjini yindawo yokusasaza ye-Flink [ ] yomxhunyi weempawu zexesha lokwenyani. Umxhunyi we-intanethi udibanisa iimpawu kunye nezilumkiso ezivela kwiizenzo zabasebenzisi kwaye ivelise imizekelo yokuqeqesha, eze ezibhalwa kumjikelezo weKafka. Umjikelo wezibonelo zokuqeqesha ufunyenwa zombini ukuqeqesha kwi-intanethi kunye nokuqeqesha ibatch: 13 4 Kukuqeqesha kwi-intanethi, umqeqeshi wokuqeqesha ufunda idatha ngokuthe ngqo kumjikelo weKafka; Kukuqeqesha ibatch, umsebenzi wokudumisa idatha uza kuqala kudumisa idatha kwi-HDFS [ ]; Emva kokuba idatha kwi-HDFS iqokeleleke kwinqanaba elithile, umqeqeshi wokuqeqesha uza kufumana idatha kwi-HDFS kwaye enze uqeqesho lwebatch. 18 Iiparameter ezihlaziyiweyo kwi-training PS ziya kuthunyelwa kwi-serving PS ngokuhambelana neshedyuli yokuhambelana kweeparameter. Kwiinkqubo zoshishino, isenzo somsebenzisi kunye neempawu zisasazeka kumxhunyi we-intanethi (uMgcawo ngaphandle kwesiqinisekiso ngolandelelwano lwamaxesha. Ngoko ke sisebenzisa isitshixo esikhethekileyo senqanaba ngalinye ukuze isenzo somsebenzisi kunye neempawu zikwazi ukudibana ngokuchanekileyo. 2.2.2 Uxhunyi lwe-intanethi. 5) Ukulibaziseka kwesenzo somsebenzisi kungaba yingxaki. Umzekelo, umsebenzisi angathatha iintsuku ezimbalwa ngaphambi kokuba anqume ukuthenga into awayiboniswe kwii-days ezidlulileyo. Oku kuyingxaki kumxhunyi kuba ukuba zonke iimpawu zigcinwa kwikheshi, ayiyi kuhlangana kwimemori. Kwindlela yethu, indawo yokugcina yesitshixo-ixabiso kwidisk isetyenziselwa ukugcina iimpawu ezilinde ixesha elithile. Xa ingeno lodumo lomsebenzisi lifika, kuqala likhangela ikheshi yangaphakathi, kwaye emva koko likhangela indawo yokugcina yesitshixo-ixabiso xa kukho ikheshi elahlekileyo. Enye ingxaki eyenzeka kwizicelo zangempela kukuba ukusasazeka kweziboniso ezibi nezintle akulingani, apho inani labangaphambili lingaba ngaphezu kwama-apositive amashumi. Ukuthintela iziboniso ezintle ukuba zingadlwelwa ziziboniso ezibi, inkqubo eqhelekileyo kukwenza i-negative sampling. Oku kuya kutshintsha ubume obungaphantsi bobudumo obuqeqeshweyo, bukwalise ubume ukuya kwi-probability ephezulu yokwenza izibikelelo ezintle. Njengonxibelelwano, sisebenzisa ukulungisa i-log odds [ ] ngexesha lenkonzo, siqinisekisa ukuba ubume be-intanethi buyisibikeleli esingeyotyala sobume obuqhelekileyo. 19 Ngexesha lokuqeqesha kwi-intanethi, iqela lokuqeqesha leMono-lith liqhubeka nokufumana idatha kwicandelo lenkonzo ye-intanethi kwaye luhlaziye iziparameter kwi-training PS. Inyathelo elibalulekileyo lokuvumela i-serving PS ye-intanethi ukuba izuze kwiizi pararameter ezintsha eziqeqeshweyo luhambelano lweeparameter ezihlaziyiweyo. Kwindawo yokuvelisa, sifumana iingxaki ezininzi: 2.2.3 Ukuhambelana kweziko. Iimodeli kwi-serving PS ye-intanethi akufuneki ziyeke ukukhonza xa zihlaziya. Iimodeli zethu kwimveliso zihlala zingangeetereabytes, kwaye ngenxa yoko ukutshintsha zonke iziparameter kuthatha ixesha. Kuya kuba akunakunyamezeleka ukuyeka i-PS ye-intanethi ukukhonza ubume ngexesha lenkqubo yokutshintsha, kwaye uhlaziyo kufuneka lwenziwe ngokukhawuleza; Ukuhambisa ubume be-multi-terabyte ngokupheleleyo ukusuka kwi-training PS ukuya kwi-serving PS ye-intanethi kungabeka uxinzelelo olukhulu kwinqanaba lenethiwekhi kunye nememori kwi-PS, njengoko kufuna imemori yobukhulu obubini ukwamkela ubume obufikayo. Ukuze ukuqeqesha kwi-intanethi kukwazi ukufikelela kubungakanani besimo soshishino lwethu, sakha inkqubo yokuhambelana kweeparameter ezincinci ngokuchanekileyo ngokuchanekileyo ngokusekelwe kwiimpawu ezithile zeemodeli zethu: (1) Iiparameter ezicaba zithatha ubukhulu beemodeli zokucebisa; (2) Ngenxa yomgangatho omncinci wewindows, iqela elincinci lezisazisi kuphela eliqeqeshwe kwaye izifakelo zalo zihlaziyiwe; (3) Izinto eziguquguqukayo ezicaba zihamba kancinci kuneentlobo zokufaka. Oku kungenxa yokuba kwi-optimizers ezisekelwe kwi-momentum, ukuqokelelana kwe-momentum yezi zinto zigxininiswa bubukhulu obukhulu bedatha yokuqeqesha yokucebisa, ngelixa izifakelo ezicaba ezimbalwa kuphela zifumana uhlaziyo kwi-batch enye yedatha. (1) kunye (2) zisebenzisa uhlaziyo olucaba kuzo zonke izisazisi. KwiMonolith, sigcina iseti ye-hash yezikhuni ezichaphazelekayo, ezimele izisazisi izifakelo zazo zafumana uqeqesho ukusukela kuhambelano lokugqibela lweeparameter. Sikhupha iqela elincinci leeparameter ezicaba ezizikhuni zazo zikwiqela lezikhuni ezichaphazelekayo kunye nexesha elincinci lothintelo ukusuka kwi-training PS ukuya kwi-serving PS. Eli qela loluhlelo oluncinci loluhlelo olungenamthwalo kwiintshukumo zenethiwekhi kwaye aliyi kubangela ukunyuka okukhulu kwememori ngexesha lokuhambelana. Siphinde sisebenzise (3) ukunciphisa ngakumbi ukusetyenziswa kwe-network I/O kunye nememori ngokubeka ishedyuli yokuhambelana eyahlukileyo kwiiparameter ezicaba, ngelixa uhlaziyo lwezi zinto lwenziwa rhoqo. Oku kungasibangela imeko apho izinto ezicaba esizisebenzisayo ziyinguqulo endala ngokuthelekiswa nenxalenye ecaba. Nangona kunjalo, ukungangqinelani okunjalo kunokunyamezeleka ngenxa yesizathu esichaziweyo kwi (3) njengoko kungabikho kulahleka okubonakalayo kubonwa. 2.3 UKUNGELELA NGOKUNGAPHUMELELI Njengenkqubo yokuvelisa, iMonolith yenzelwe ngamandla okuphinda ibuyise i-PS xa ingaphumeleli. Ukhetho oluqhelekileyo lokuphuma kukungqubana kokugcina imeko yobume rhoqo, kwaye ubuyisele kwisigcino sokugqibela xa ingaphumeleli kwe-PS kufunyanwa. Ukhetho lwefrequency yesigcino lunempembelelo ezimbini eziphambili: (1) Umgangatho wobume. Ngokucacileyo, umgangatho wobume ubonisa ukulahleka okuncinci kwembali yakutshanje ngokunyuka kwefrequency yesigcino. (2) Uxinzelelo lokubala. Ukugcina ubume obubanzi bee-terabytes akulunganga. Kufuneka iikopi ezinkulu zememori kunye ne-disk I/O. Njengokutshintshana phakathi komgangatho wobume kunye noxinzelelo lokubala, iMonolith igcina zonke ii-PS zokuqeqesha imihla ngemihla. Nangona i-PS ilahlekelwa yimihla edlulileyo yohlaziyo xa ingaphumeleli, sifumana ukuba ukudodobala kokusebenza kunokunyamezeleka ngovavanyo lwethu. Siya kuhlaziya impembelelo yokuthembeka kwe-PS kwiCandelo elilandelayo. 3 UKUVAVANYA Ukufumana ukuqonda okungcono kwezibonelelo kunye neengozi eziziswa kuyilo lwethu olucetywayo, senze uvavanyo oluninzi ngesikali sokuvelisa kunye novavanyo lwe-A/B kunye ne-traffic yokukhonza ebukhweni ukuvavanya kunye nokuqinisekisa iMonolith kwiinkalo ezahlukeneyo. Sijonge ukuphendula imibuzo elandelayo ngeevavanyo zethu: (1) Singazuza kangakanani kwiHashTable engenako ukungqubana? (2) Kubaluleke kangakanani ukuqeqesha kwi-intanethi ngexesha lokwenyani? (3) Ngaba uyilo lweMonolith lokuhambelana kweeparameter lunamandla ngokwaneleyo kwisimo sokuvelisa esikhulu? Kweli candelo, kuqala sibonisa izicwangciso zethu zovavanyo kwaye emva koko sixoxe ngezona ziphumo kunye nezifundo zethu ngokweenkcukacha. 3.1 Ukucwangciswa kokuVavanya Njengoko kuchaziwe kwiCandelo iitheyibhile zokufaka kwiMonolith zenziwa njengeHashTable ezingenawo ukungqubana. Ukubonisa imfuneko yokuphepha ukungqubana kwiitheyibhile zokufaka kunye nokubala inzuzo ukusuka kwinto yethu engenayo ukungqubana, senze amaqela amabini ovavanyo kwi-Movielens dataset kunye nenkqubo yethu yangaphakathi yemveliso ngokulandelelana: 3.1.1 Itheyibhile yokufaka. 2.1, (1) ml-25m dataset [ ]. Le yidatha kawonke-wonke yomgangatho wokulinganisa iifilimu, equlethe izilinganisi ezizizigidi ezingama-25 ezibandakanya abasebenzisi abamalunga nama-162,000 kunye neefilimu ezingama-62,000. I-MovieLens 11 • . Izilumkiso zangoku zivela Ukulungisa izilumkiso 0.5 ukuya ku-5.0, ngelixa kwimveliso imisebenzi yethu inguninqulo yokufumana imiyalezo emibini evela kubasebenzisi. Ukulinganisa imodeli yethu yemveliso ngcono, siguqula izilumkiso zengqokelelo kwizilumkiso zemibini ngokujonga amanqaku ≥ 3.5 njengeziboniso ezintle kwaye ezinye ziboniso ezimbi. • Imeko kunye neengozi. Senze imodeli eqhelekileyo ye-DeepFM [9], imodeli eqhelekileyo esetyenziswa kwiingxaki zokucebisa. Inokwakheka kwenkalo ye-FM kunye nenxalenye ecacileyo (uMgcawo 6). Izibikelelo zivavanywa yi-AUC [2] njengoko oku kuyona measurements ephambili yeemodeli zangempela. • Ukungqubana kokufaka. Le dataset iqulethe malunga ne-160K user IDs kunye ne-60K movie IDs. Ukuthelekisa inguqulo engenako ukungqubana yokwenza itheyibhile yokufaka, senze enye iqela lovavanyo apho izisazisi zilungiswe nge-MD5 hashing kwaye zafakwa kwindawo encinci yesazisi. Ngenxa yoko, ezinye izisazisi ziya kwabelana nge-embedding yazo nabanye. Itheyibhile 1 ibonisa iinkcukacha ezipheleleyo zabathengi kunye nee-movie IDs phambi naphambili kokuhahla. (2) . Ingaphakathi Recommendation dataset Siphinde senze uvavanyo kwi-model yokucebisa kwindawo yokuvelisa. Le model ngokubanzi ilandela ulwakhiwo lweenqaba ezininzi, kunye nenqaba nganye inoxanduva lokufunda ukubikezela uhlobo oluthile lwesenzo somsebenzisi. Imodeli nganye inee-embedding tables ezimalunga ne-1000, kwaye ukusasazeka kobukhulu beetheyibhile zokufaka akufani kakhulu; Indawo yokuqala yesazisi yetheyibhile yokufaka yayiyi-248. Kwi-baseline yethu, sisebenzise i-hashing trick ngokudala ukunciphisa ubukhulu betheyibhile yokufaka. Ukuba nengcaciso eninzi, sisebenzisa iitheyibhile ezimbini zokufaka ezincinci endaweni yeyodwa enkulu ukuvelisa i-embedding eyodwa kumsebenzisi ngamnye ngokudibanisa kwe-vector: apha 𝐸𝑟, 𝐸𝑞 zizifakelo ezihambelana ne-𝐼𝐷𝑟, 𝐼𝐷𝑞. Oku kunciphisa ngokufanelekileyo ubukhulu betheyibhile yokufaka ukusuka kwi-248 ukuya kwi-225; • Le model iyakhonza kwimveliso yangempela, kwaye ukusebenza kolu vavanyo lulinganiswa yi-AUC yangaphakathi kunye ne-traffic yokukhonza yangempela. Ngexesha lokuqeqesha kwi-intanethi, sihlaziya ii-PS zethu zokukhonza nge-set entsha yeeparameter ngeeminyutha. Sakha amaqela amabini ovavanyo ukuqinisekisa umgangatho wobume kunye nokuthembeka kwenkqubo. 3.1.2 UQEQESHO KWI-INTANETHI. (1) . Ukujonga imfuneko yokuphucula iminutha, senze uvavanyo oluhambelanisa iziparameter ukusuka kubume bokuqeqesha ukuya kubume bokubikezela ngeemfama ezahlukeneyo. Ukuhlaziya iFrequency Idatha esiyisebenzisayo yidatha yeCriteo Display Ads Challenge dataset3, idatha enkulu eyenzelwe ukubeka iliso kwiimodeli zeCTR. Ineedatha ezilandelelanayo zeentsuku ezisixhenxe ebhalisa iimpawu kunye nezenzo zokucofa. Kulo vavanyo, sisebenzisa imodeli yeqhelo ye-DeepFM njengoko ichaziwe ku [9] 6. Ukulinganisa ukuqeqesha kwi-intanethi, senze le nkqubo yokulungisa idatha. Sithatha iintsuku ezisixhenxe zedatha kwi-dataset, kwaye siyihlukanise kwiindawo ezimbini: iintsuku ezihlanu zedatha yokuqeqesha ibatch, kunye neentsuku ezimbini zokuqeqesha kwi-intanethi. Siyihlukanisa ngakumbi iintsuku ezimbini zedatha kwi-𝑁 shards ngokulandelelana. Ukuqeqesha kwi-intanethi kulinyazwa yi-algorithm 1. Njengale nto, silinganisa ukuhambelana kweeparameter ezifundisiweyo kwi-online serving PS kunye ne-interval ethintwe yinombolo yee-shards zedatha. Sisebenze nge-𝑁 = 10, 50, 100, ezihambelana ngqo nemixesha yokuhlaziya ye-5h, 1h, kunye ne-30min. (2) . Ukongeza, senze uvavanyo olubukhoma kunye ne-traffic yokukhonza yangempela ukubonisa ngakumbi ukubaluleka kokuqeqesha kwi-intanethi kwisicelo sangempela. Eli vavo le-A/B lithelekisa ukuqeqesha kwi-intanethi kunye nokuqeqesha ibatch kwi-model yethu ye-Ads kwimveliso. Uvavanyo olubukhoma 3.2 Iziphumo kunye noHlalutyo Iziphumo ezivela kwi-MovieLens dataset kunye ne-Internal recommendation dataset zombini zibonisa ukuba ukungqubana kweziko kuya kuhlambalaza umgangatho wobume. 3.2.1 Impembelelo yokungqubana kweziko. (1) Iimodeli ezineHashTable engenako ukungqubana zihlala zigqitha ezo zinako ukungqubana. Esi sigqibo siyasigcinwa nokuba Ukonyuka kwenani leqela lokuqeqesha. Njengoko iboniswe kuMgcawo imodeli ene-embedding table engenako ukungqubana inee-AUCs eziphezulu ukusuka kwi-epoch yokuqala kwaye ifikelela kwinqanaba eliphezulu; 7, Utshintsho lolwabiwo kunye nokuhamba kwexesha ngenxa ye-Concept Drift. Njengoko iboniswe kuMgcawo iimodeli ezine-embedding table engenako ukungqubana nazo zinamandla njengoko ixesha lihamba kwaye izixeko/izinto zitshintsha. 8, (2) Ukucaba okubangelwa yi-embedding table engenako ukungqubana akubangeli i-model overfitting. Njengoko iboniswe kuMgcawo imodeli ene-embedding table engenako ukungqubana ayiphethe kakubi emva kokuba ifikelele kwi-convergence. 7, ![#Figure 8: Effect of Embedding Collision On A Recommendation Model In Production Silinganisa ukusebenza kwale model yokucebisa nge-AUC yokukhonza kwi-intanethi, echaphazeleka ngokwahlukileyo phakathi kweentsuku ezahlukeneyo ngenxa ye-concept-drift.](https://cdn.hackernoon.com/images/InxBRjRIs6M1kdhuWcyNHiiUrxm1-d9i3eb9.jpeg) ![#Figure 9: Online training v.s. Batch training on Criteo dataset. Imigca eluhlaza: AUC yeemodeli eziqeqeshwe kwi-intanethi; Imigca emthubi: AUC yeemodeli zokuqeqesha ibatch ezivavanywe ngokuchasene nedatha yokusasaza.](https://cdn.hackernoon.com/images/InxBRjRIs6M1kdhuWcyNHiiUrxm1-o0j3ej8.jpeg) Safumana ukuba i-frequency ephezulu yokuhambelana kweeparameter ihlala iluncedo ekuphuculeni i-AUC yokukhonza kwi-intanethi, kwaye iimodeli zenkonzo ye-intanethi zinokunyamezela ukulahleka kwezibini ii-shards ze-PS ngaphezu kokuba silindele. 3.2.2 UQEQESHO KWI-INTANETHI: Ukutshintshana ngoButyala ukuze kufunyanwe ixesha. (1) . Kuvu-vanyo lwethu lokuqeqesha i-streaming kwi-intanethi kunye nedatha yeCriteo Display Ads Challenge, umgangatho wobume buhlala buphucuka ngokunyuka kwefrequency yokuhambelana kweeparameter, njengoko kubonakala ngokuthelekiswa kweembono ezimbini: Impembelelo yeParameter Synchronization Frequency (1) • Iimodeli ezine-training kwi-intanethi zisebenza ngcono kunezo zingenawo. UMgcawo uthelekisa i-AUC yeemodeli zokuqeqesha kwi-intanethi ezivavanywe yi-shard elandelayo yedatha ngokuchasene neemodeli zokuqeqesha ibatch ezivavanywe yi-shard nganye yedatha; 9a, 9b, 9c • Iimodeli ezine-interval yokuhambelana kweeparameter ezincinci zisebenza ngcono kunalezo zine-interval enkulu. UMgcawo kunye neTable zithelekisa i-AUC yokukhonza kwi-intanethi kwiimodeli ezine-sync interval ye-5hr, 1hr, kunye ne-30min ngokulandelelana. 10 2 Uvavanyo olubukhoma lwe-A/B phakathi kokuqeqesha kwi-intanethi kunye nokuqeqesha ibatch kwi-model ye-Ads kwimveliso ikwabonisa ukuba kukho ukunyuka okubalulekileyo kwi-AUC yokukhonza kwi-intanethi (I-Table 3). Iphefumlelwe yeli qwalaselo, sihlaziya iziparameter ezicaba kwi-serving PS yeemodeli zethu zemveliso rhoqo kangangoko kunokwenzeka (ngoku nge-minute-level), ukuya kuthi ga apho uxinzelelo lokubala kunye nokuthembeka kwenkqubo kunganyamezela. Khumbula ukuba izinto ezicaba zifuna uhlaziyo olungafaniyo njengoko kuchaziwe ku sizihlaziya kwi-day-level. Ngokwenza oku, singanciphisa uxinzelelo lokubala lwethu kwindawo ephantsi kakhulu. Ngokucinga ukuba i-100,000 IDs ihlaziywa ngomzuzu, kwaye ubukhulu bokufaka bubukhulu be-1024, ubuninzi bedatha ekufuneka idluliselwe buyi-4KB × 100,000 ≈ 400MB ngomzuzu. Kwiiparameter ezicaba, njengoko zihambelana mihla le, sikhetha ukucwangcisa ukuhambelana xa i-traffic ingekho phezulu (umzek., phakathi kobusuku). 2.2.3, (2) Impembelelo yokuthembeka kwe-PS. Ngokuhambelana kweeparameter kwi-minute-level, saqala sakhetha isigcino esifuna rhoqo kwi-training PS ukuhambelana nohlaziyo lwexesha lokwenyani. Ngokumangalisayo, sandisa ixesha lesigcino ukuya kosuku kwaye sasingafumani kulahleka okuncinci kumgangatho wobume. Ukufumana ulungiso olufanelekileyo phakathi komgangatho wobume kunye noxinzelelo lokubala kunzima kwiinkqubo zokuhlela eziyenziwe umntu kuba abasebenzisi bayaziva kakhulu ngomgangatho wokucebisa. Ngokwemveli, iinkqubo ezinkulu zinqenayo ukubeka ishedyuli yesigcino rhoqo yeemodeli zazo, ezigweba izixhobo zokubala ngenxa yokunciphisa ilahleko kumgangatho wobume. Siphinde senze uphando olulunge ngakumbi malunga noku kwaye ngokumangalisayo, umgangatho wobume unobutyebi kunezingafaniyo. Ngomlinganiselo wokungaphumeleli we-0.01% yomatshini we-PS ngosuku, sifumana ubume ukusuka kosuku oludlulileyo busebenza kakuhle kakhulu. Oku kuyakwazi ukuchazwa ngenxa yale calculatio elandelayo: Cinga ukuba iziparameter zobume ziqhekezwe kwi-1000 PS, kwaye zifota mihla le. Ngenxa yomlinganiselo wokungaphumeleli we-0.01%, enye yazo iya kwehla rhoqo emva kweentsuku ezili-10 kwaye silahlekelwa zonke izihlaziye kule PS enye iintsuku ezili-1. Ukucela i-DAU ye-15 Million kunye nokuhanjiswa okulunge ngakumbi kwezisazisi zomsebenzisi kwi-PS nganye, silahlekelwa yimpendulo yosuku olunye ukusuka kubasebenzisi abali-15,000 rhoqo kwiintsuku ezili-10. Oku kuyakwamkeleka kuba (a) Kwiimpawu ezicaba eziyeyomsebenzisi, oku kufana nokulahlekelwa yinxalenye encinci ye-0.01% ye-DAU; (b) Kwiiparameter ezicaba, njengoko zihlaziywa kancinci njengoko sichazile ku-2.2.3, ukulahlekelwa yosuku olunye uhlaziyo kwi-1000 PS kubalulekile. Ngokusekelwe kweli qwalaselo kunye ne-calculatio, sinciphise kakhulu i-frequency yethu yokufota kwaye ngenxa yoko sakhulula kakhulu kwiindleko zokubala. 4 UMSEBENZI OFANAYO Ukusukela izicelo zokuqala eziphumeleleyo zokufunda enzulu kwiinkqubo zokucebisa zeshishini [ , ], abaphandi kunye neenjineli baye basebenzisa iindlela ezahlukeneyo ukuphucula iingxaki ezikhankanyiweyo kwiCandelo 6 10 1. Ukujongana nengxaki yeengcaciso ezipeyileyo, [ , ] zisebenzisa itheyibhile yokufaka enobukhulu obugqityiweyo kunye ne-hash-trick. Kukho namaphulo ekwenzeni uhlobo lwe-hashing ukuze kuncitshiswe ukungqubana [ , ]. Ezinye iincwadi zisebenzisa itheyibhile ye-hash yesitshixo-ixabiso yendalo ukuvumela ukukhula okuguquguqukayo kobukhulu betheyibhile [ , , , ]. Ezi zinto zenziweyo zakhiwe kwi-TensorFlow kodwa zixhomekeke kwinkqubo yesoftware eyenzelwe ngokukhethekileyo [ , , ] okanye ihardware [ ] ukufikelela kunye nokulawula iitheyibhile zabo ze-hash. Xa kuthelekiswa nezi zicombululo, i-hash-table yeMonolith iseyinye inkqubo ye-TensorFlow yemveli. Inobuhlobo kunye nomsebenzisi kwaye inokuhambelana okungcono kwis platform, okwenza kube kufanelekile kwizicelo ze-ToB. Ukudibanisa okungokwemvelo kunye nokuqinileyo kunye ne-TensorFlow kuvumela ukulungiswa okulula kokusebenza kokubala. 3 6 3 7 12 15 20 21 14 15 20 21 Ukudibanisa igalelo phakathi kokuqeqesha kunye nenkonzo kunye nokunciphisa i-Concept Drift [ ] yenye imiba enomdla. Ukuxhasa uhlaziyo lwe-intanethi kunye nokuphepha iingxaki zememori, zombini [ ] kunye [ ] zakha iinkqubo zokugqibezela iimpawu ukuze ziguquguquke ubukhulu betheyibhile yokufaka. Zombini [ ] kunye [ ] zixhasa uhlobo oluthile lokuqeqesha kwi-intanethi, apho iziparameter ezifundisiweyo zihambelaniswa nenkonzo ngexesha elifutshane xa kuthelekiswa nokuqeqesha ibatch yemveli, kunye neenkqubo zokungaphumeleli. IMonolith yathatha indlela efanayo yokwamkela nokugqibezela ngokulinganayo iimpawu, ngelixa inenkqubo yokuhambelana kweeparameter engenamthwalo ukuqinisekisa umgangatho wobume. 8 12 20 12 14 5 ISISHWANKATHELO Kweli phetshana, sibuyekeze ezinye zeengxaki ezibaluleke kakhulu kwiinkqubo zokucebisa ze-level yeshishini kwaye sibonise inkqubo yethu kwimveliso, iMonolith, ukujongana nazo kwaye yafumana ukusebenza okungcono xa kuthelekiswa nezicombululo ezikhoyo. Sabonisa ukuba itheyibhile yokufaka engenako ukungqubana ibalulekile kumgangatho wobume, kwaye sabonisa ukuba ukwenziwa kwethu kweteyibhile yokufaka ye-Cuckoo HashMap esekelwe kuyo yenziwe yasebenza kakuhle kwi-memory kwaye iluncedo ekuphuculeni izikali zokukhonza kwi-intanethi. Siphinde sabonisa ukuba inkonzo ye-real-time ibalulekile kwiinkqubo zokucebisa, kwaye ukuba i-parameter synchronization interval kufuneka ibe nekhawuleza kangangoko kunokwenzeka ukuze kufumaneke ukusebenza okugqibeleleyo kobume. Isicombululo sethu senkonzo ye-realtime kwi-intanethi kwiMonolith sinenkqubo yokuhambelana kweeparameter eyenzelwe ngobuchule kunye nenkqubo yokungaphumeleli: Kwisicombululo sethu sokuhambelana kweeparameter, sabonisa ukuba ukungaguquguquki kwenguquko kuzo zonke iindawo zeeparameter kunokutshintshwa ukuze kuncitshiswe ukusetyenziswa kwe-bandwidth yenethiwekhi; Kwindalo yokungaphumeleli, sabonisa ukuba inkqubo yethu yokutshintshana ngobutyala bokungaphumeleli kwe-PS ukuze kufunyanwe ixesha yisisombululo esiqinileyo. Ukushwankathela, iMonolith iphumelele ekuboneleleni ngesisombululo esibanzi kwiinkqubo zokucebisa ngesikali sokuvelisa. UKUQAPHELISISA U-Hanzhi Zhou ubonelele ngeengcebiso eziyimfuneko malunga nokuhlaziywa kweli phepha. IMIBHALO [1] Martín Abadi, Paul Barham, Jianmin Chen, Z. Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek Gordon Murray, Benoit Steiner, Paul A. Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zhang. 2016. TensorFlow: A system for large-scale machine learning. abs/1605.08695 (2016). ArXiv [2] Andrew P. Bradley. 1997. The use of the area under the ROC curve in the evaluation of machine learning algorithms. 30 (1997), 1145–1159. Pattern Recognit. [3] Thomas Bredillet. 2019. Core modeling at Instagram. https://instagram-engineering.com/core-modeling-at-instagram-a51e0158aa48 [4] Paris Carbone, Asterios Katsifodimos, Stephan Ewen, Volker Markl, Seif Haridi, and Kostas Tzoumas. 2015. Apache Flink™: Stream and Batch Processing in a Single Engine. 38 (2015), 28–38. IEEE Data Eng. Bull. [5] Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishikesh B. Aradhye, Glen Anderson, Gregory S. Corrado, Wei Chai, Mustafa Ispir, Rohan Anil, Zakaria Haque, Lichan Hong, Vihan Jain, Xiaobing Liu, and Hemal Shah. 2016. Wide & Deep Learning for Recommender Systems. (2016). Proceedings of the 1st Workshop on Deep Learning for Recommender Systems [6] Paul Covington, Jay K. Adams, and Emre Sargin. 2016. Deep Neural Networks for YouTube Recommendations. (2016). Proceedings of the 10th ACM Conference on Recommender Systems [7] Alexandra Egg. 2021. Online Learning for Recommendations at Grubhub. (2021). Fifteenth ACM Conference on Recommender Systems [8] João Gama, Indre˙ Žliobaite˙, Albert Bifet, Mykola Pechenizkiy, and A. Bouchachia. 2014. A survey on concept drift adaptation. 46 (2014), 1 – 37. ACM Computing Surveys (CSUR) [9] Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017. DeepFM: A Factorization-Machine based Neural Network for CTR Prediction. In . IJCAI [10] Udit Gupta, Xiaodong Wang, Maxim Naumov, Carole-Jean Wu, Brandon Reagen, David M. Brooks, Bradford Cottel, Kim M. Hazelwood, Bill Jia, Hsien-Hsin S. Lee, Andrey Malevich, Dheevatsa Mudigere, Mikhail Smelyanskiy, Liang Xiong, and Xuan Zhang. 2020. The Architectural Implications of Facebook’s DNN-Based Personalized Recommendation. (2020), 488–501. 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA) [11] F. Maxwell Harper and Joseph A. Konstan. 2015. The MovieLens Datasets: History and Context. 5 (2015), 19:1–19:19. ACM Trans. Interact. Intell. Syst. [12] Biye Jiang, Chao Deng, Huimin Yi, Zelin Hu, Guorui Zhou, Yang Zheng, Sui Huang, Xinyang Guo, Dongyue Wang, Yue Song, Liqin Zhao, Zhi Wang, Peng Sun, Yu Zhang, Di Zhang, Jinhui Li, Jian Xu, Xiaoqiang Zhu, and Kun Gai. 2019. XDL: an industrial deep learning framework for high-dimensional sparse data. (2019). Proceedings of the 1st International Workshop on Deep Learning Practice for High-Dimensional Sparse Data [13] Jay Kreps. 2011. Kafka : a Distributed Messaging System for Log Processing. [14] Xiangru Lian, Binhang Yuan, Xuefeng Zhu, Yulong Wang, Yongjun He, Honghuan Wu, Lei Sun, Haodong Lyu, Chengjun Liu, Xing Dong, Yiqiao Liao, Mingnan Luo, Congfei Zhang, Jingru Xie, Haonan Li, Lei Chen, Renjie Huang, Jianying Lin, Chengchun Shu, Xue-Bo Qiu, Zhishan Liu, Dongying Kong, Lei Yuan, Hai bo Yu, Sen Yang, Ce Zhang, and Ji Liu. 2021. Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameters. abs/2111.05897 (2021). ArXiv [15] Meituan. 2021. Distributed Training Optimization for TensorFlow in Recom-mender Systems (in Chinese). https://tech.meituan.com/2021/12/09/meituan-tensorflow-in-recommender-systems.html [16] R. Pagh and Flemming Friche Rodler. 2001. Cuckoo Hashing. In . ESA [17] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In . NeurIPS [18] Konstantin V. Shvachko, Hairong Kuang, Sanjay R. Radia, and Robert J. Chansler. 2010. The Hadoop Distributed File System. (2010), 1–10. 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST) [19] HaiYing Wang, Aonan Zhang, and Chong Wang. 2021. Nonuniform Negative Sampling and Log Odds Correction with Rare Events Data. In . Advances in Neural Information Processing Systems [20] Minhui Xie, Kai Ren, Youyou Lu, Guangxu Yang, Qingxing Xu, Bihai Wu, Jiazhen Lin, Hongbo Ao, Wanhong Xu, and Jiwu Shu. 2020. Kraken: Memory-Efficient Continual Learning for Large-Scale Real-Time Recommendations. (2020), 1–17. SC20: International Conference for High Performance Computing, Networking, Storage and Analysis [21] Weijie Zhao, Jingyuan Zhang, Deping Xie, Yulei Qian, Ronglai Jia, and Ping Li. 2019. AIBox: CTR Prediction Model Training on a Single Node. (2019). Proceedings of the 28th ACM International Conference on Information and Knowledge Management This paper is under CC by 4.0 Deed (Attribution 4.0 International) license. available on arxiv