Ababhali: Junaid Syed, Georgia Institute of Technology Sai Shanbhag, Georgia Institute of Technology Vamsi Krishna Chakravarthy, Georgia Institute of Technology I-Automated Essay Scoring (AES) ngumsebenzi we-NLP oye wafundwa iminyaka emininzi. I-AES inokubaluleka okuninzi okusebenzayo kunye namandla amakhulu oqoqosho - i-AES lilitye lembombo kwiimviwo ezinkulu zokhuphiswano (umz. SAT, GRE) kunye nemarike yokufunda kwi-intanethi ekhulayo. Imibutho emininzi ye-philanthropic kunye nengenzi nzuzo efana neBill & Melinda Gates Foundation kunye neZuckerberg-Chan Initiative baye baxhasa ngemali ukhuphiswano oluninzi lweKaggle kwi-AES [6, 7, 8]. Nangona le nzame, nangona kunjalo, ingxaki isekude ukuba isonjululwe ngenxa yobunzima obusisiseko bokufumana amanqaku esincoko. Ukuphonononga isincoko kuxhomekeke kakhulu kwaye kubandakanya imiba engaqondakaliyo efana nokunamathelana, igrama, ukufaneleka, njl.njl. ekunzima ukuyibala. Ngenxa yoko, ukufumana iilebhile zedatha yoqeqesho enomlinganiselo wegranular wesincoko kuzo zonke iimpawu ezifana negrama, ukuhambelana, njl, kubiza kakhulu. Ngenxa yoko, iseti yedatha yoqeqesho ilinganiselwe kakhulu xa kuthelekiswa neminye imisebenzi ye-NLP efana (nemaski) iimodeli zoLwimi, i-NER, ukuthegiswa kwePOS, ukuguqulelwa komatshini, njl. abancedi abafundi kwinkqubela yabo. Ke ngoko, iinzame zangoku zijolise ekuvavanyeni isincoko kwimiba yegranular kunenqaku elinye. Oku kukwanceda ukuphepha ukufaneleka ngokugqithisileyo kuba imodeli yokuqikelela ngoku kufuneka iqhube kakuhle kuzo zonke iimethrikhi kwaye hayi i-metric enye, ngokusisiseko, umntu unokucinga ngale nto njengemodeli yemisebenzi emininzi. Kuphononongo lwangoku, sigxininisa kwiimetriki ezintandathu: ukudibanisa, i-syntax, isigama, i-phraseology, igrama, kunye nemigaqo. 1.1 uPhando loNcwadi Phambi koo-2010, uninzi lweemodeli ze-AES zazixhomekeke kwiifitsha ezenziwe ngezandla eziyilwe ziingcali zeelwimi ezibaliweyo [10, 4]. Nangona kunjalo, le mifuziselo yayidla ngokuthambekela kwiimpawu ezithile (umz. ubude besincoko) kwaye ayikwazanga ukwenza ngokubanzi izihloko kunye neemetrics ngokubanzi. Ukuthambekela kwiimpawu ezenziwe ngesandla kwasonjululwa ngokuthi endaweni yazo kufakwe izihlomelo zamagama ezifundwe ziimodeli zolwimi ezifana neWord2Vec kunye neGloVe. Ngokusekwe kolu fakelo lwamagama, amanqaku esincoko aye aqikelelwa njengemisebenzi yokubuyisela umva kunye nokuhlelwa ngokongeza inethiwekhi ye-neural ezantsi komjelo wegama lokuzinzisa. Ngokusebenzisa izifakelo eziqeqeshwe kwikhophusi enkulu, uphuculo olubonakalayo lubonwa kumanqaku esincoko kuzo zonke iimetriki kunye namanqaku ewonke [11]. Nangona kunjalo, kwawona magama afakelweyo ebebalulekile kuphuculo lwentsebenzo angqineke angowona mda mkhulu wemodeli. Njengoko uzinziso luvela kwindlela ye-Bag-of-Words, abakwazanga ukuthabatha naluphi na ulwazi lomxholo oluthinjwe ngokuyinxenye ziimpawu zolwimi ezenziwe ngesandla kwiimodeli zangaphambili. Esikhundleni sokongeza iimpawu ezenziwe ngesandla kunye nokukwazi ukwazisa kwakhona iintsilelo zeemodeli zangaphambili, ingxaki yokunqongophala kolwazi lomxholo yalungiswa ngendlela yokuqwalaselwa kusetyenziswa i-LSTM [13] kunye nezakhiwo ze-transformer. Umsebenzi kaVaswani kunye nePolosukhin [14] uphuhlise ngempumelelo imodeli ye-BERT usebenzisa i-transformers. Ikhuthazwe yimpumelelo yemodeli ye-BERT kunye noyilo lwe-transformer, uthotho lwemizekelo yolwimi esekwe kuqwalaselo lwaphuhliswa. Ngoku, endaweni yokufakela amagama, umntu unokufumana isivakalisi okanye ukufakwa kwenqanaba loxwebhu olubamba ulwazi lomxholo. Ukusebenzisa olu lwazi lunzulu, iimodeli zenethiwekhi ye-neural ziphuhliswa ukuqikelela amanqaku esincoko (zombini njengokuhlelwa kunye nemisebenzi yokubuyisela umva). 1.2 Unyino kwiiNdlela zangoku Ngaphandle kwale nkqubela phambili, kukho imida engqongqo ngokusebenzisa imodeli ye-BERT. Lottridge et al. (2021) [10] ubonise ukunqongophala kokuqina kwemodeli yezincoko zomdlalo, ukushupa okungahleliwe, kunye nezincoko zaseBhabheli. Ukusebenza kuyahluka kakhulu kwiiklasi ezahlukeneyo kunye neemetrics. Ukujongana nale ntsilelo, kolu phando, siya kumodela zonke iimetriki ngaxeshanye ngokufunda imisebenzi emininzi. Olunye umda obalulekileyo wohlalutyo olusekelwe kwi-BERT kukuba ubude bethokheni bukhawulelwe kwi-512 kwimodeli ye-BERT. Sifuna ukulungisa oku ngokusebenzisa izakhiwo eziphucukileyo ezifana ne-Longformer evumela ukuya kuma-tokens angama-4096 ngoxwebhu. Kwiseti yedatha eqwalaselwe kolu phononongo (iinkcukacha kwiCandelo 2.1), ngaphezu kwe-40% yamaxwebhu angaphezulu kwamathokheni angama-512 ubude. Ke ngoko, ukucutha uxwebhu lube ngamathokheni angama-512 kuphela ngemodeli ye-BERT eqhelekileyo kuya kubangela ilahleko enkulu kumxholo. Isithintelo sesithathu esiphambili sezifundo ezahlukeneyo yiseti yedatha elinganiselweyo - nangona izifundo ezininzi zijolise kwi-AES, nganye yezo datha inikwe amanqaku ngokwahlukileyo, kwaye ngenxa yoko, iimodeli azikwazi ukuqeqeshwa ngokulula kuzo zonke iiseti zedatha. Ke ngoko, kolu phononongo, siphanda usetyenziso lweenethiwekhi ze-autoencoder ukuqeqesha kuzo zonke iiseti zedatha kwaye sisebenzise i-encodings eyenziwe nge-autoencoder ukwenza imisebenzi ye-AES. Isishwankathelo, olu phononongo luphanda ifuthe leekhowudi ezahlukeneyo ezisekelwe kuxwebhu olusekwe kumaxwebhu kumanqaku esincoko azenzekelayo. Iseti yedatha, indlela yokusebenza, imifuniselo, kunye nokufakela okunzulu okuqwalaselwa kolu phononongo kwaziswa kwiCandelo 2. Ngaphandle kokwahluka kokuzinzisa okunzulu, sihlalutya iindlela zokudibanisa iiseti zedatha ze-AES ngokuqeqesha iikhowudi ezinzulu kwinethiwekhi ye-Autoencoder. Iziphumo eziphuma kuzo zonke ezi ndlela zinikwe kwiCandelo lesi-3 kwaye izigqibo kwakunye nemiyalelo yophando olubhekele phaya zinikiwe kwiCandelo lesi-4. 2. Ulwazi-nkqubo 2.1 Idatha ILebhu ye-Arhente yokuFunda, iYunivesithi yaseGeorgia State, kunye neYunivesithi yaseVanderbilt iqokelele inani elikhulu leencoko ezivela kwii-arhente zezemfundo zelizwe kunye nelizwe, kunye nemibutho engenzi nzuzo. Ukusuka kule ngqokelela, baye baphuhlisa i-Persuasive Essays for Rating, Selecting, and Understanding Argumentative and Discourse Elements (PERSUADE) corpus, equlathe izincoko eziphikisanayo ezibhalwe ngabafundi abakumabakala 6-12, kunye nolwimi lwesiNgesi loMfundi iInsight, ubuchule kunye noVavanyo lweZakhono. (ELLIPSE) ikhompusi, equka izincoko ezibhalwe nguMfundi woLwimi lwesiNgesi (ELLs) kumabakala 8-12. I-ELLIPSE corpus iqulethe ngaphezu kwe-7,000 yezincoko ezibhalwe ngu-ELLs kumabanga 8-12. Ezi zincoko zabhalwa njengenxalenye yeemvavanyo zokubhala ezisemgangathweni zikarhulumente ukusuka kunyaka wesikolo ka-2018-19 no-2019-20. Izincoko ezikwikhophu yeELLIPSE ziye zachazwa ngabalinganisi babantu kumanqanaba obuchule bolwimi besebenzisa irubrikhi yamanqaku amahlanu equka zombini izikali ezipheleleyo kunye nezihlalutyayo. Isikali esipheleleyo sigxininise kwinqanaba lobuchule bolwimi xa lilonke elibonakaliswe kwizincoko, ngelixa isikali socazululo siquka iireyithingi zokunamathelana, isintaksi, isigama, isigama, igrama kunye nemigaqo. Amanqaku omlinganiselo ngamnye wokuhlalutya avela kwi-1.0 ukuya kwi-5.0 ngokunyuka kwe-0.5 kunye namanqaku amakhulu ahambelana nobugcisa obukhulu kuloo mlinganiso. I-ELLIPSE corpus: I-PERSUADE corpus iqulethe ngaphezu kwe-25,000 yezincoko eziphikisanayo ezibhalwe ngabafundi base-US kumabanga 6-12. Ezi zincoko zabhalwa njengenxalenye yovavanyo lokubhala olusemgangathweni lukazwelonke nolukarhulumente ukusuka ngo-2010-2020. Isincoko ngasinye kwikhophusi ye-PERSUADE sichazelwe ngamaxabiso abantu kwiingxoxo kunye neengxoxo kunye nobudlelwane obuphezulu phakathi kwezinto eziphikisanayo. Irubrikhi yesichasiselo yaphuhliswa ukuchonga nokuvavanya imiba yentetho edla ngokufunyanwa kubhalo lwengxoxo. I-PERSUADE corpus: Kule projekthi, sisebenzisa i-ELLIPSE corpus kwaye ngaxeshanye siqikelela inqaku kumanyathelo amathandathu ohlalutyo: ukudibanisa, i-syntax, isigama, i-phraseology, igrama, kunye nemigaqo. Ukongeza, sizama ukuphucula ukuchaneka koqikelelo lwethu ngokusebenzisa i-autoencoder. Umbono kukuqeqesha i-autoencoder usebenzisa i-ELLIPSE kunye ne-PERSUADE corpus. Ngale nkqubo, into ecinezelweyo yevektha evela kwi-autoencoder ingakwazi ukuthabatha iimpawu zezincoko eziyimfuneko ekufumaneni amanqaku emodeli yolwimi oluqeqeshelwe kwangaphambili iimpawu ezinokuphoswa. 2.2 Indlela Njengoko bekutshiwo ngaphambili, injongo yale projekthi kukuqikelela inqaku lemilinganiselo emithandathu yohlalutyo: ukuhambelana, ulandelelwano, isintaksi, isigama, ibinzana lamagama, igrama, kunye nemigaqo yezincoko eziphikisanayo ezibhalwe ngabafundi besiNgesi bebakala lesi-8 ukuya kwele-12. Kulo msebenzi, siqala siphuhlise isiseko kwaye emva koko sisebenzise iimodeli ezininzi eziqeqeshwe kwangaphambili ukuphucula kwisiseko. : Isiseko siphuhliswe kusetyenziswa uthungelwano lweglove kunye nothungelwano lwe-LSTM oluphindwe kabini. Kwimodeli esisiseko, kuqala sicoca idatha okt ukususwa kweziphumlisi, ukususwa kwesithuba esimhlophe, njl.njl usebenzisa ithala leencwadi le-regex kwaye emva koko, sisebenzise igama elithi tokenizer elisuka kwi-NLTK ukwenza umqondiso wezincoko. Uthungelwano lwe-LSM luqeqeshelwa ukhowudo lwe-GloVe kwizincoko ukuvelisa i-vektha yobude besi-6 emele amanqaku kwimilinganiselo yohlalutyo emithandathu engentla. Sisebenzisa iMean Squared Error loss (MSELoss) ukuqeqesha inethiwekhi ye-neural. Isiseko : I-DistilBERT imodeli encinci, ekhawulezayo, kunye nekhaphukhaphu yeTransformer eqeqeshwe yi-distill base-BERT. Ine-40% yeeparamitha ezimbalwa kune-bert-base-uncased kwaye iqhuba i-60% ngokukhawuleza ngelixa igcina ngaphezu kwe-95% yentsebenzo ye-BERT njengoko ilinganisiwe kwi-benchmark yokuqonda ulwimi lwe-GLUE. I-BERT isebenzisa ukuzijonga ukuba ibambe ulwazi lomxholo kulo lonke ulandelelwano [2]. Oku kuphucula ukukwazi kwemodeli yokuvavanya iisampulu zesincoko kunye nokubonelela ngamanqaku achanekileyo. Kule modeli, sisebenzisa i-auto tokenizer ukwenza umqondiso wezincoko kwaye emva koko sigqithise la mathokheni kwimodeli ye-DistilBERT eqeqeshwe kwangaphambili ukufumana umboniso wevector wezincoko. Emva koko siqeqesha inethiwekhi ye-neural enemigangatho emibini usebenzisa i-MSLoss ukubuyisela i-6-dimensional output vector emele amanqaku nganye kwiimpawu ezintandathu zokubhala ezichazwe ngasentla. I-DistilBERT : I-T5 okanye i-Text-To-Text Transfer Transformer iyimodeli ye-encoder-decoder eqeqeshwe kwangaphambili kumxube wemisebenzi emininzi engalawulwayo kunye nemisebenzi ephantsi kolawulo kwaye apho umsebenzi ngamnye uguqulelwa kwifomathi yokubhaliweyo ukuya kumbhalo. Nge-BERT, eqeqeshwe kwangaphambili kwi-Masked LM kunye nenjongo yokuQikelela kwesivakalisi esilandelayo, kufuneka silungise ngokwahlukeneyo iimeko ezahlukeneyo zemodeli eqeqeshwe kwangaphambili kwimisebenzi eyahlukeneyo esezantsi efana nokuhlelwa ngokulandelelana. I-T5's text-to-text framework inika indlela elula yokuqeqesha imodeli enye kwiintlobo ezininzi zemisebenzi yesicatshulwa usebenzisa umsebenzi ofanayo wokulahlekelwa kunye nenkqubo yokuguqula. Esi sikhokelo soqeqesho lwangaphambili sinika imodeli ngenjongo eqhelekileyo "yolwazi" oluphucula ukusebenza kwayo kwimisebenzi esezantsi [12]. Sasebenzisa i-auto-tokenizer ukwenza i-tokenize izincoko kwaye emva koko sidlulisele ezi tokens kwimodeli ye-T5-Base eqeqeshwe kwangaphambili ukufumana ukumelwa kwe-vector yezincoko. Emva koko siqeqesha inethiwekhi ye-neural enemigangatho emibini usebenzisa i-MSELOss ukubuyisela i-6-dimensional output vector (efana ne-DistilBERT). T5 : I-RoBERTa yenye imodeli yolwimi oluyi-BERT efana ne-mask ephuhliswe ngu-Facebook. Kwimeko ye-RoBERTa, imaski eguqukayo isetyenziswa kulo lonke uqeqesho lwazo zonke ii-epochs, ngelixa kwi-BERT imaski imile. Ngale nto, imodeli ifunda amathokheni amaninzi ngakumbi kune-BERT. Uphuculo olongezelelweyo lokwenziwa komsebenzi luphunyezwa ngoqeqesho kwikophusi enkulu yedatha kune-BERT (10x) kunye neseti yesigama esikhulu. Ngolu tshintsho kuqeqesho, i-RoBERTa idlula i-BERT kwimisebenzi emininzi ye-GLUE kunye ne-SQuAD [9]. I-RoBERTa-base : I-Longformer imodeli ye-BERT efana ne-transformer eyavela kwindawo yokuhlola i-RoBERTa kwaye yaqeqeshwa njengeModeli yoLwimi lweMasked (MLM) kumaxwebhu amade. Ixhasa ukulandelelana kobude ukuya kuma-4,096 amathokheni. Ngokuqhelekileyo, iimodeli ezisekwe kwi-transformer ezisebenzisa i-self-attention mechanism ayikwazi ukwenza ulandelelwano olude ngenxa yokuba imemori kunye neemfuno zokubala zikhula ngobude be-quadratically. Oku kwenza ukuba kube nzima ukwenza ngokufanelekileyo ulandelelwano olude. Abadlali bexesha elide bajongana nalo mda ubalulekileyo ngokuzisa indlela yokuqwalaselwa elinganisa ngokulandelelana kunye nobude bokulandelelana [1]. Isebenzisa ifestile etyibilikayo kunye nefestile yokutyibiliza evulelekileyo ukuze ibambe imeko yasekhaya neyehlabathi. Kwimodeli yeLongformer, sisebenzisa indlela efanayo neDistilBERT. Sisebenzisa i-auto-tokenizer ukulinganisa izincoko kwaye emva koko sidlulisele ezi mpawu kwimodeli ye-Longformer eqeqeshwe kwangaphambili ukuze sifumane ukubonakaliswa kwe-vector yezincoko. Emva koko siqeqesha inethiwekhi ye-neural enemigangatho emibini usebenzisa i-MSELOss ukubuyisela i-6-dimensional output vector (efana ne-DistilBERT). I-Longformer Sikwasebenzise ukuqokelelwa kwe-gradient ukuqeqesha iimodeli zethu kubungakanani bebhetshi enkulu kunokuba i-GPU yethu ye-Colab yokuqhuba ikwazile ukungena kwinkumbulo yayo. Ngenxa yobukhulu obukhulu bemodeli yeLongformer, sasilinganiselwe kubungakanani bebhetshi ezimbini kuphela. Ubungakanani bebhetshi encinci ngolo hlobo buya kubangela ubalo lwemithamo engazinzanga. Siyayithintela le nto ngokuqokelelwa komgangatho - endaweni yokubuyisela umva ilahleko emva kokuphindwaphindwa, siqokelela ilahleko kwaye sibuyisela umva impazamo emva kwenani elithile leebhetshi ukuphucula uzinzo lohlaziyo lwegradient [3]. 2.3 Uvandlakanyo Ukuvavanya ukuchaneka komzekelo wethu wamanqaku aqikelelweyo, siza kusebenzisa ingcambu yentsingiselo ethetha impazamo ephindwe kabini (MCRMSE) njengemetric. Umethrikhi ubalwa ngolu hlobo: 2.4 Iimvavanyo Emva kokuphumeza iimodeli ezichazwe ngasentla, sizame imifuniselo embalwa yokuphucula impazamo yokuxela kwangaphambili kwezi modeli. Iinkcukacha zale mifuniselo zezi zilandelayo: : Kwi-ELLIPSE corpus, inqaku lomlinganiselo ngamnye wokuhlalutya ukusuka kwi-1.0 ukuya kwi-5.0 ngokunyuka kwe-0.5 kunye namanqaku amakhulu ahambelana nobuchule obukhulu kulo mlinganiso. Siyilungisile inethiwekhi yethu ye-neural kangangokuba imveliso iyanyanzelwa phakathi kwe-1 kunye ne-5. Senze oku ngokubandakanya umaleko we-sigmoid apho imveliso idlulayo kwaye siphinda-phinda le mveliso ngo-4 kwaye songeze i-1 kuyo. Ngaphezu koko, xa iziphumo ziveliswa kwi-neural net senza i-mathematical operation ukuqinisekisa ukuba imveliso inyuka kumanyathelo e-0.5 kuphela. Lo msebenzi ujolise ekuphindaphindeni ifomathi yamanqaku okuqala kwaye ukhangele ukuba ukuguqulwa okunjalo kuphucula ukuchaneka. Ubungakanani besiphumo score = int [(2 * score + 0.5) / 2] : Kwi-ELLIPSE corpus, inqaku lomlinganiselo wohlalutyo ngalunye luvela kwi-1.0 ukuya kwi-5.0 ngokunyuka kwe-0.5. Nangona kunjalo, ukuhanjiswa kwenqaku ngalinye kwi-dataset akufani. Amanqaku athile anje ngo-2.5, 3, kunye no-3.5 ayenzeka rhoqo kwidathasethi yethu yemilinganiselo yohlalutyo ngelixa amanqaku afana no-1, kunye no-5 esenzeka ngokungaqhelekanga kulo lonke uluhlu lwedatha. Ukunika ingxelo ngoku kungalingani sisebenzise ingcambu ene-weighted mean imposiso yesikwere (WRMSE) umsebenzi apho uguqulo lophindaphindo lwamanqaku athile lusetyenziswa njengobunzima kwaye siyabuqhawula obu bunzima ukuba bubephezulu kakhulu xa buthelekiswa nobunye ubunzima. Ubunzima be-RMSE : Njengoko kukhankanyiwe kwicandelo langaphambili, ekubeni ukuhanjiswa kwenqaku ngalinye kwi-dataset akufani, siye sazama ukuba nenethiwekhi ye-neural echanekileyo yomgangatho wesibini ukuqikelela amanqaku. Ke endaweni yentloko yemveliso enye eqikelela ixabiso lamanqaku ahlukeneyo ama-6, siphumeze iintloko zeziphumo ezi-6 ezahlukeneyo ukuqikelela amanqaku kumlinganiselo wohlalutyo ngamnye. I-MultiHead Architecture : Isethi yedatha enikezelweyo yomsebenzi wangoku wamanqaku amaninzi eklasi yesincoko malunga neesampuli ze-4k kuphela. Nangona kunjalo, kwi-ELLIPSE kunye ne-PERSUADE corpus kunye, kukho ngaphezu kwe-180k izincoko zeminye imisebenzi ye-AES, njengamanqaku angamanye kuzo zonke izincoko, kunye neenxalenye zezincoko. Ke ngoko, ii-autoencoders zisetyenziselwa ukukhulisa le database enkulu kwaye zenze ufundo oluphantsi kweliso. Ngokufutshane, iikhowudi ezisuka kwiimodeli zolwimi ezifana ne-BERT, T5 zigqithiswa kuthungelwano lwe-autoencoder oluqeqeshwe kusetyenziswa zonke iisampuli ze-180k. Emva koko, mhlawumbi i-bottleneck layer encoding okanye i-denoised language model encodings ukusuka kwi-decoder part ye-autoencoder isetyenziselwa ukuqikelela amanqaku eklasi ezininzi kusetyenziswa i-2-layer neural network yentloko yokubuyisela, efana nescenario egadiweyo ngokupheleleyo. Ke ngoko, ngokusebenzisa iseti enkulu yedatha engabhalwanga ukuqeqesha i-autoencoder njenge-preprocessor, sifuna ukuphucula uqikelelo lokufunda olubekwe esweni. Kolu phononongo, sithathele ingqalelo zombini iikhowudi ezidityanisiweyo ezisekwe kwiikhowudi zeDistilBERT. I-Autoencoder 3. Iziphumo kunye neNgxoxo : Itheyibhile 1 ishwankathela i-metric yokusebenza efunyenwe ngokuhluka kweemodeli eziqeqeshwe kwangaphambili ezichazwe kwiCandelo 2.2. Kule miba, ii-encodings ezivela kwiimodeli eziqeqeshwe kwangaphambili zigqithiswa ngokuthe ngqo kwi-2-layer neural network eqeqeshwe ngokusebenzisa ilahleko ye-MSE, kwaye akukho nanye yezinto eziphuculweyo ezixutyushwa kwiCandelo 2.4. Njengoko oku kukuhlehliswa kweeklasi ezininzi, ukusebenza kweemodeli zemetric nganye yamanqaku kuboniswe kwiThebhile 3. Impembelelo yeeKhowudi eziQeqeshiweyo Phakathi kwezakhiwo zetransformer ezidweliswe kwiThebhile yoku-1, sibona ukuba iimodeli zolwimi ezifihliweyo iDistilBERT, RoBERTa, kunye neLongformer ziqhuba ngcono kunemodeli yemveliso T5 - mhlawumbi ngenxa yokuba iimodeli ezigqunyiweyo zilungelelaniswe ngakumbi kwimisebenzi yocalucalulo eneziphumo zamanani. Uphando olongezelelweyo luyimfuneko ukuze kuqukunjelwe ukuba oku kunokwenziwa ngokubanzi kwiimodeli ezininzi zolwimi oluvelisayo. Lilonke, i-RoBERTa inelona nqaku liqikelelwa kakuhle phakathi kweemodeli ezahlukeneyo, ngenxa yoqeqesho lwayo olukhulu kunye nokumaski okubalaseleyo. Itheyibhile 1: Amanqaku e-MCRMSE ewonke kwiimodeli ezahlukeneyo Umzekelo Imetriki ye-MCRMSE Isiseko 1.36 DistilBERT 0.4934 T5-isiseko 0.5320 ROBERTa 0.4746 Umntu owadala ixesha elide 0.4899 : Ngaphambili, siye sahlola umphumo wamagalelo ahlukeneyo kwintloko yokubuyisela (okt, ngokutshintsha iimodeli eziqeqeshwe kwangaphambili kunye ne-encodings kuyo), ngelixa ubambe uqeqesho lwentloko yokubuyisela rhoqo. Kweli candelo, siphonononga umphumo wokuhluka koqeqesho lwentloko yokubuyisela ngelixa ubambe i-encodings rhoqo. ICandelo 2.4 lidwelisa iinguqu ezahlukeneyo kuqeqesho lokuhlehla eziphononongwayo kolu phando. Qaphela ukuba kulo lonke eli candelo, imodeli ye-DistilBERT isetyenziswa kuba iyeyona modeli ikhawulezayo kwaye ineemfuno ezisezantsi ze-GPU. Iziphumo zezikim zoqeqesho/uphuculo olwahlukeneyo zibonisiwe kwiThebhile 2. Impembelelo yokuphuculwa kwentloko yokubuyisela Itheyibhile 2: Amanqaku e-MCRMSE kwiimodeli ezahlukeneyo Zama I-MCRMSE Ubungakanani bemveliso 0.5294 Ubunzima be-RMSE 0.5628 MultiHead Architecture 0.508 I-Autoencoder Denoising 0.575 Ngelishwa, akukho nanye kwezi yantlukwano ekuqeqesheni imodeli yokuhlehla ikhokelela ekuphuculeni okubonakalayo kokuchaneka kwengqikelelo xa kuthelekiswa neemodeli zethu zokuqala. Enyanisweni, i-metric yokusebenza ekuqinisekiseni okusetwe kwiThebhile 2 ibonisa ukuhla kwentsebenzo ngolu tshintsho. Akucaci ukuba kutheni oku kuncitshiswa kwenzeka kwaye ukuqhubela phambili uphononongo kunye ne-dataset enkulu kubalulekile ukuqinisekisa ukuba oku kuncitshiswa kokusebenza akusiyo i-artifact. Kuzo zonke iinguqu kwi-encoding yombhalo kunye nokuqeqeshwa kwentloko yokubuyisela, siqaphela ukusuka ekuqinisekiseni amanqaku e-MCRMSE kumanyathelo omntu ngamnye ukuba ukudibanisa kunye negrama ibonakala iyona nto inzima ukuqikelela kuyo yonke imifuziselo (jonga iTheyibhile 3). Oku kunokuba ngumda kwiimodeli zolwimi eziqeqeshwe kwangaphambili ezisetyenziswa kwi-AES kungekhona umfuziselo wethu. UKim et al. (2020) [5] ubonisa imida kwimizekelo yolwimi lwangoku ekufumaneni ulwazi lwegrama kunye nokubonelela ngezalathiso zenkqubela phambili kwimizekelo yolwimi. Itheyibhile 3: Inqaku le-MCRMSE lomlinganiselo wohlalutyo lomntu ngamnye Imodeli (okanye Exp.) Umanyano Isivakalisi Isigama I-Praseology Igrama Iindibano Isiseko 1.37 1.35 1.32 1.34 1.44 1.36 distilBERT 0.54 0.51 0.46 0.52 0.57 0.49 T5-Base 0.55 0.52 0.48 0.54 0.58 0.53 ROBERTa 0.51 0.47 0.42 0.47 0.51 0.46 Umntu owadala ixesha elide 0.54 0.48 0.46 0.49 0.53 0.47 distilBERT + imveliso quantization 0.55 0.53 0.48 0.53 0.57 0.51 distilBERT + WRMSE 0.56 0.56 0.55 0.56 0.61 0.53 distilBERT + Multi Head Arch. 0.53 0.50 0.45 0.51 0.56 0.49 Autoencoder + distilBERT 0.59 0.56 0.52 0.56 0.61 0.55 4. Isiphelo Kulo msebenzi, siphande isiphumo soyilo olwahlukeneyo oluqeqeshwe kwangaphambili kunye neendlela zokuqeqesha intloko yokubuyisela umva kumsebenzi we-Automated Essay Scoring, apho siphawula isincoko ngasinye kwisikali soku-1 ukuya kwisi-5 kwiimetriki zolwimi ezintandathu (umz., ukuhambelana, igrama, isigama. njl.). I-dataset ithathwa kwi-ELLIPSE corpus, ngokukodwa i-subset yedatha edweliswe kukhuphiswano lwe-Kaggle. Sithathele ingqalelo iindlela ezintlanu zokufunda ezinzulu kunye neendlela ezintlanu zokuqeqesha intloko yokubuyisela umva kwaye sajonga ukusebenzisa i-RoBERTa-base enomgangatho olula we-2-layer feed-forward layer ukuqikelela amanqaku njengemveliso yeeklasi ezininzi inike esona siphumo silungileyo. Njengoko bekulindelekile, i-architectures ye-transformer igqwesile ngokubonakalayo imodeli yesiseko yeGlove+LSTM. Ngaphaya koko, kulwakhiwo lwe-transformer, sibona ukuba iimodeli zolwimi ezigqunyiweyo (DistilBERT, RoBERTa, Longformer) zinika ukusebenza okuphezulu xa kuthelekiswa nemodeli yolwimi oluzalayo T5. Nangona olu qwalaselo lungenzeki ngokubanzi kuzo zonke iimodeli eziveliswayo, ngokuqondayo ukongamela kweMLM kubonakala kungaguquguquki njengoko beqeqeshelwa iziphumo zamanani. Olunye uqwalaselo olunomdla olu phando kukuba ukuhluka kokuqeqeshwa kwentloko yokubuyisela ngokuguqula imisebenzi yokulahlekelwa, ukunyanzeliswa kweziphumo, kunye nokunciphisa ubungakanani obusekelwe kwi-autoencoder-based dimensionality / denoising, kunye nokwandiswa kwedatha, ayizange iphucule imodeli yokusebenza. Oku kunokuba kungalindelekanga, kwaye asiziqondi ngokupheleleyo izizathu ezibangela le nto. Kuphononongo lwexesha elizayo, ezi ndlela zingaphinda ziphindwe nge-dataset enkulu - oku kunceda ukufumanisa ukuba ngaba oku qwalaselo malunga nokuqeqeshwa kwentloko yokubuyisela ingenziwa ngokubanzi. Isishwankathelo, siqaphela ukuba ukusebenzisa i-encodings ye-RoBERTa ene-2-layer feed-forward neural net ukuqikelela amanqaku amathandathu ngaxeshanye, afana nokufunda ngemisebenzi emininzi, kubonelela ngeyona ntsebenzo ibalaseleyo. Ngokukodwa, kunikwe ubungakanani obuncinci bedathasethi, umphumo wokusebenzisa imodeli eqinile yokuqeqeshwa kwangaphambili ibonakala iphucula kakhulu ukusebenza kwangaphambili kwemodeli. Kwakhona, intsebenzo ekuvavanyeni igrama yesincoko imbi kakhulu kunayo nayiphi na enye imetriki yovandlakanyo, kwaye oku kuhambelana nemodeli yolwimi. Kungoko, imisebenzi yexesha elizayo kufuneka igxininise ekuphuculeni imizekelo yolwimi ukuze ibambe ngcono imiba yegrama yolwimi. Iimbekiselo Iz Beltagy, Matthew E Peters, kunye no-Arman Cohan. 2020. Longformer: Isiguquli samaxwebhu amade. . arXiv preprint arXiv:2004.05150 UJacob Devlin, uMing-Wei Chang, uKenton Lee, kunye noKristina Toutanova. 2018. I-BERT: Ukuqeqeshwa kwangaphambili kweenguqu ezinzulu ze-bidirectional zokuqonda ulwimi. . arXiv preprint arXiv:1810.04805 UJoeri R Hermans, uGerasimos Spanakis, kunye noRico Möckel. 2017. Ukuqhelaniswa kwegradient eqokelelweyo. , amaphepha 439–454. PMLR. KwiNkomfa yaseAsia ngokuFunda koomatshini UZixuan Ke kunye noVincent Ng. 2019. Amanqaku esincoko esizenzekelayo: Uvavanyo lwemeko yobugcisa. Kwi , vol. 19, iphepha 6300-6308. -IJCAI UTaeuk Kim, uJihun Choi, uDaniel Edmiston, kunye noSang-goo Lee. 2020. Ngaba iimodeli zolwimi eziqeqeshwe kwangaphambili ziyawazi amabinzana? Iziseko ezilula kodwa ezinamandla zoqheliso lwegrama. ILebhu ye-Arhente yokuFunda. 2022a. Ibhaso lempendulo-ukufunda ulwimi lwesiNgesi. ILebhu ye-Arhente yokuFunda. 2022b. Ibhaso lengxelo-Ukuphonononga ukubhala kwabafundi. ILebhu ye-Arhente yokuFunda. 2022c. Ibhaso lengxelo-Ukuqikelela iingxoxo ezisebenzayo. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, kunye noVeselin Stoyanov. 2019. URoberta: Indlela yoqeqesho ephuculweyo yebert. arXiv preprint arXiv:1907.11692. Sue Lottridge, Ben Godek, Amir Jafari, kunye noMilan Patel. 2021. Ukuthelekisa ukomelela kwemfundo enzulu kunye neendlela zamanqaku ezizihambelayo zeklasi kubuchule bokudlala. Ingxelo yobuchwephesha-Cambium Assessment Inc. UHuyen Nguyen kunye noLucio Dery. 2016. Uthungelwano lwe-Neural lwe-automated essay grading. IiNgxelo ze-CS224d zaseStanford: 1-11. UAdam Roberts kunye noColin Raffel. 2020. Ukuphonononga ukudluliselwa kokufunda kunye ne-T5: i-text-to-text transfer transformer. Ifumaneka kumaphepha 23–07. UKaveh Taghipour kunye noHwee Tou Ng. 2016. Indlela ye-neural kumanqaku esincoko esizenzekelayo. Kwiinkqubo zenkomfa ye-2016 malunga neendlela ezisebenzayo kwi-natural language processing, iphepha 1882-1891. UNoam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N. Gomez Łukasz Kaiser Vaswani, Ashish kunye no-Illia Polosukhin. 2017. Ingqalelo yiyo yonke into oyifunayo. Ukuqhubela phambili kwiinkqubo zokucwangcisa ulwazi lwe-neural, 30.
