Zviri Mukati Nhanganyaya Ruzivo Rwakasviba Distillation Procedure Edzai paMNIST Kuwedzera kuedza paMNIST 1. Nhanganyaya Muchinyorwa chino, ini ndichaongorora maitiro ekuziva distillation muAI - mabatiro aanoita mukuzara, kukosha kwayo, uye zvikonzero zvekuishandisa. Tinogona sei kumanikidza uye kuendesa ruzivo kubva kune hombe modhi kana ensemble yemamodheru (ayo akadzidziswa pamaseti makuru kwazvo kuti abvise chimiro kubva kune data) kuenda kune imwechete modhi isina kunyura mukuita? Asi nei tichida kuita izvi? Nei tichida diki modhi kana hombe modhi kana ensemble modhi iri kutopa mibairo mikuru pabvunzo data? Panguva yekudzidziswa isu tinowanzo kudzidzisa hombe / mubatanidzwa wemamodheru nekuti chinangwa chikuru ndechekubvisa chimiro kubva mumaseti makuru kwazvo. Tinogona zvakare kushandisa zvinhu zvakawanda senge kudonha, kuwedzera data panguva yechitima kudyisa aya mahombe mhando dzese data. Asi panguva yekufanotaura chinangwa chedu chakasiyana zvachose. Tinoda kuwana mhinduro nekukurumidza sezvinobvira. Saka kushandisa hombe/ensemble yemamodheru kunodhura zvakanyanya uye kunotadzisa kutumirwa kune nhamba huru yevashandisi. Saka, zvino mubvunzo ndewekuti tingadzvanyirira sei ruzivo kubva kune iyi modhi hombe kuita diki modhi inogona kuiswa nyore. Geoffrey Hinton, Oriol Vinyals, naJeff Dean kubva kugoogle kuburikidza ravo vakauya neimwe mhando yekudzidziswa inonzi kuendesa ruzivo urwu kune diki modhi. Iyi ndiyo nzira imwe chete iyo yekumbundira kumeso yakashandiswa mukuita kwavo . nebepa distillation kweDistill BERT Kana tikakwanisa kudzidzisa iyi modhi diki kuita nenzira yakafanana neyemhando hombe, saka iyi modhi diki yakadzidziswa nenzira iyi ichaita zvirinani pane diki modhi yakadzidziswa pane imwecheteyo data asi nenzira yakajairika. Iyi ndiyo imwe yemisimboti kuseri kweDistillation zvakazara mikuru 2. Rima Zivo Kazhinji, muMuchina kudzidza, modhi inodzidza kusarura pakati pehuwandu hukuru hwemakirasi, chinangwa chikuru chekudzidzisa ndechekuwedzera avhareji yelogi mukana wemhinduro chaiyo. Semuyenzaniso, tora muenzaniso wedataset yeMNIST apo chinangwa chiri chekuisa mufananidzo kuti 1 kana 2 kana ... 9. Saka kana mufananidzo chaiwo uri 2 saka chinangwa chemhando ipi zvayo ndechekuwedzera (inogona kuverengwa semukana wekuti mumwe mufananidzo unopiwa mufananidzo wacho 2). Asi iyo modhi inopawo mukana kune ese mhinduro dzisiridzo kunyangwe iwo mukana ari madiki, mamwe acho akakura kupfuura mamwe. Pfungwa ndeyekuti kunyangwe izvi zvingangove zvidiki, mukana wemhinduro dzisina kunaka unotiudza zvakawanda nezve maitirwo angaite modhi. Kuti tinzwisise, ngatitarisei muenzaniso uri pasi apa. P (yayo 2/mufananidzo) Mumufananidzo uri pamusoro, iyi vhezheni ye2 yakapihwa mukana we10-6 wekuve 3 uye 10-9 yekuve 7 nepo kune imwe vhezheni inogona kunge iri imwe nzira. Urwu ruzivo rwakakosha runotsanangura hupfumi hwakafanana chimiro pamusoro peiyo data (kureva iyo inoti iyo 2 inoita senge 3's uye inoita senge 7's) asi ine simba shoma pamuchinjiko-entropy mutengo webasa panguva yekufambisa nhanho nekuti mikana yacho iri pedyo ne zero. Asi tisati taenderera mberi nekuita distillation, ngatipedze nguva tichiona kuti modhi yacho yakanyatso gadzira sei zvinobuda. Apa ndipo panouya softmax activation. Yekupedzisira nhanho yekugadzirisa modhi ndeye softmax uye chikamu ichi ndicho chinopa mukana wekubuda. Input kune softmax inonzi logits uye tinogadzira iyo yekupedzisira layer yeNN nenzira yekuti nhamba yezvikamu zvakavanzika = nhamba yemakirasi atinoda kuisa. Iyo formula yekuverenga softmax inopiwa se Equation iri pamusoro apa inopa mikana kune imwe neimwe uye chiyero chezvose zvingangoitika zvakaenzana yakaenzana 1. Munguva yekudzidzira nguva, kurasikirwa kwemuenzaniso mumwe chete wekudzidzisa kunoverengwa nekuenzanisa izvi zvingangoitika zve softmax nezvinangwa zvakaoma (mabhii) uye kushandisa backpropagation coefficients inovandudzwa kusvikira kurasikirwa kuduku. i i Sezvinoonekwa pamusoro apa softmax inopa mukana wepamusoro kune yechokwadi label uye yakaderera mikana kune zvisirizvo zvinyorwa. Isu tinoona zvakare kuti mukana wemhinduro dzisiridzo kunyangwe diki, dzine ruzivo rwakawanda rwakavigwa mazviri izvo zvinobatsira modhi kuita zvakazara. Izvi tinozvidaidza kuti Zivo Yakasviba 3. Distillation Procedure Sekureva kwebepa, nzira yakanakisa yekuendesa hunyanzvi hweiyo modhi hombe kune diki modhi ndeye kushandisa mukana wekirasi wakagadzirwa neinorema modhi sezvinangwa zvekudzidzisa modhi diki. zvakapfava Saka maitiro ndeaya anotevera: Tora iyo yepakutanga seti yekudzidziswa iyo yakashandiswa kudzidzisa hombe modhi wobva wapfuudza iyo data yekudzidziswa kuburikidza neiyo hombe modhi uye uwane zvingangoita softmax pamakirasi akasiyana. Sezvinoonekwa pamusoro, label yechokwadi inowana mukana wepamusoro uye mavara asiri iwo anowana yakaderera mikana. Asi takaona mikana yakaderera iyi ine ruzivo rwakawanda rwakavanda mavari. Saka kukudza kukosha kweizvi zvingangoitika vanyori vemapepa vakashandisa shanduko inonzi Temperature(T) kugovanisa zvese matanda vasati vapfuura nepasoftmax. Izvi zvinogadzira yakapfava mukana wekugovera pamusoro pemakirasi. Tinogona kuona pazasi Kubuda kwekuisa softmax ine tembiricha(T) ndiyo yatinodaidza kuti Soft targets. Izvi ndizvo zvinonzi nevanyori . distillation Kufananidza nekubvisa tsvina mumvura nekuwedzera tembiricha Ruzivo rwakawanda pamusoro pebasa rakadzidzwa kubva kumhando huru inogara mureshiyo yezvingabvira zvidiki-diki muzvinangwa zvakapfava. Mamwe mazwi: - kuburitsa kubva kune hombe modhi mushure mekupisa T yaiswa panguva yesoftmax Zvinangwa zvakapfava - kuburitsa kubva kudiki modhi mushure mekupisa T yaiswa panguva yesoftmax Kufembera kwakapfava - kubuda kubva kudiki modhi kana tembiricha T = 1 (yakajairika softmax) Kufanotaura kwakaoma - zvinangwa chaizvo kubva pakudzidziswa seti Zvinangwa zvechokwadi Pazasi pane kuyerera kwemaitiro ese ekudzidziswa Saka nzira yekudzidzira yemuenzaniso mudiki ine 2 kurasikirwa mabasa. Yekutanga kurasikirwa basa inotora zvese zvakapfava kufanotaura uye zvinyoro zvinangwa uye ndiyo yemuchinjikwa-entropy kurasikirwa basa. Iyi ndiyo nzira iyo kugona kwegeneralization kunotamiswa kubva kune hombe modhi kuenda kudiki modhi nekuyedza kuenzanisa zvipfuro zvakapfava. Kune iyi yekurasikirwa basa, ese softmax anoshandisa tembiricha ye 'T'. Vanyori vakaonawo kuti kushandisa modhi diki kuenzanisa zvinangwa zvechokwadi kunobatsira. Izvi zvinosanganiswa mune yechipiri mutengo basa. Mutengo wekupedzisira chiyero chakayerwa cheaya maviri mutengo mabasa ane hyper-paramita alpha uye beta. 4. Edza paMNIST Vanyori vakashandisa dataset yeMNIST kuyedza nzira iyi. Vakashandisa zvivakwa zviviri zveizvi izvo zvinosiyana chete munhamba yezvikamu zvakavanzika muzvikamu zvepakati. Vanyori vakashandisa 2 yakavanzika layer neural network mune ese ari maviri kesi Modhi diki inogona kutariswa se784 -> 800 -> 800 -> 10 (apo 784 yakavhurwa zviyero zvemufananidzo, 800 inhamba yezvikamu zvakavanzwa zvine RELU activation uye gumi ndiyo nhamba yemakirasi atiri kufanotaura). Iyi modhi yakapa 146 bvunzo zvikanganiso pasina kurongeka. Hombe modhi inogona kutariswa se784 -> 1200 -> 1200 -> 10 (apo 784 yakavhurwa zviyero zvemufananidzo, 1200 inhamba yezvikamu zvakavanzika zvine RELU activation uye gumi ndiyo nhamba yemakirasi atiri kufanotaura). Iyi modhi inodzidziswa paMNIST ichishandisa dropout, weight-constraints, uye jittering mapikicha ekuisa uye mambure aya akawana 67 zvikanganiso zvebvunzo. Tinogona here kuendesa kuvandudzwa uku mumhando huru kune modhi diki? Vanyori zvino vashandisa zvinangwa zvakapfava zvakawanikwa kubva kune mambure makuru uye zvinangwa zvechokwadi pasina kudonhedza uye pasina jittering yemifananidzo kureva kuti, mambure madiki akagadziriswa chete nekuwedzera rimwe basa rekufananidza zvinangwa zvakapfava zvinogadzirwa nemambure makuru patembiricha ye20 uye mhedzisiro yacho. uchishandisa 784 -> 800 -> 800 -> gumi 74 bvunzo zvikanganiso Izvi zvinoratidza kuti zvinangwa zvakapfava zvinogona kuendesa ruzivo rwakakura kune diki modhi, kusanganisira ruzivo rwekuita generalize iyo inodzidzwa kubva kune yakashandurwa data data. Nemamwe manzwi, bhenefiti yatakawana kubva mukusandura mambure ekuisa kune mambure madiki kunyangwe tisiri kushandura mambure madiki. Zvinonyatsozivikanwa kuti kushandura mapindiro neshanduko dzakasiyana-siyana kunoita kuti modhi iwande zvirinani uye mune yedu ruzivo rwekuita rwuri kuratidzwa muruzivo rweRima uye izvi zvakavanda mune zvakapfava zvinangwa. Hapana yeruzivo urwu rwuri mu True targets. Saka nekushandisa ruzivo kubva kune zvakapfava zvinangwa mambure edu madiki ari kuita zvirinani. ruzhinji Mambure makuru achishandisa zvinyoro zvinyoro akadzidza metric yakafanana iyo yakadzidza 'zvakaita sei' uye nekufambisa ruzivo uku, tiri kuudza mambure madiki 'zvakaita sei' Zvese zviri pamusoro kuyedza paMNIST zvakapfupikiswa pazasi 5. Kuwedzera kuedza paMNIST Pamusoro pezvo, vanyori vakaedzawo kusiya mienzaniso yedhijiti 3 pakudzidzisa modhi diki vachishandisa distillation. Saka kubva pamaonero eiyo modhi diki, 3 idijiti yengano yaasati amboona. Sezvo iyo diki modhi isati yamboona 3 panguva yekudzidziswa, isu tinotarisira kuti iite zvikanganiso zvakawanda paunosangana ne3 muyedzo seti. Zvisinei neizvi, iyo distilled modhi yakangoita 206 zvikanganiso zvebvunzo izvo 133 zviri pa1010 threes muyedzo set. Izvi zvinoratidza pachena kuti hunyanzvi hwekuita hwemhando huru hwakaendeswa kune iyo diki modhi panguva yedistillation uye izvi zvinoita kuti iyo diki modhi ifanotaura nemazvo 3 kazhinji. Saka tsika dzenyaya yacho. Kushandura mapikicha ekuisa kunonatsiridza generalization. Kusandura zvibodzwa zvinewo mhedzisiro yakakura uye kana tikakwanisa kuwana zvipfuro zvakapfava kubva kune imwe nzvimbo zvakachipa zvakanyanya sezvo isu tichigona kuwana kuita kwakafanana neiyo diki modhi. References: Hinton, Geoffrey, Oriol Vinyals, naJeff Dean. "Kuburitsa ruzivo mune neural network." Zivo distillation by intellabs.
