I-OPT-175B iyahambelana ne-GPT-3 ngenkathi kufuneka kuphela i-1 / 7th ye-Carbon Footprint

OPT: Open Pre-trained Transformer Language Models OPT: Open Pre-Trained Transformer Izilimi Amamodeli Umbhali: Susan Zhang, Meta AI (susanz@fb.com) Stephen Roller, Meta AI (roller@fb.com) Naman Goyal, Meta AI (naman@fb.com) U-Mikel Artetxe, u-Meta AI Moya Chen, Meta AI Shuohui Chen, Meta AI Christopher Dewan, Meta AI Mona Diab, Meta AI Xian Li, Meta AI Xi Victoria Lin, Meta AI U-Todor Mihaylov, u-Meta AI Myle Ott, Meta AI Sam Shleifer, u-Meta AI U-Kurt Shuster, uMeta AI U-Daniel Simig, uMeta AI Punit Singh Koura, Meta AI Anjali Sridhar, Meta AI Tianlu Wang, Meta AI ULuke Zettlemoyer, Meta AI Umbhali: Susan Zhang, Meta AI (susanz@fb.com) Stephen Roller, Meta AI (roller@fb.com) Naman Goyal, Meta AI (naman@fb.com) U-Mikel Artetxe, u-Meta AI Moya Chen, Meta AI Shuohui Chen, Meta AI UChristopher Dewan, Meta AI Mona Diab, Meta AI Xian Li, Meta AI Xi Victoria Lin, Meta AI U-Todor Mihaylov, u-Meta AI Myle Ott, Meta AI Sam Shleifer, u-Meta AI U-Kurt Shuster, uMeta AI U-Daniel Simig, uMeta AI Punit Singh Koura, Meta AI Anjali Sridhar, Meta AI Tianlu Wang, Meta AI ULuke Zettlemoyer, Meta AI abstract Amamodeli amabhizinisi amakhulu, amakhulu amahlathi amahlathi amahlathi amahlathi amabhizinisi amahlathi amabhizinisi amabhizinisi amabhizinisi amabhizinisi amabhizinisi amabhizinisi amabhizinisi amabhizinisi amabhizinisi amabhizinisi amabhizinisi amabhizinisi amabhizinisi amabhizinisi amabhizinisi amabhizinisi amabhizinisi amabhizinisi amabhizinisi amabhizinisi amabhizinisi amabhizinisi amabhizinisi amabhizinisi amabhizinisi amabhizinisi amabhizinisi amabhizinisi amabhizinisi amabhizinisi amabhizinisi amabhizinisi amabhizinisi amabhizinisi. 1 Ukulinganisa I-Language Models (LLMs) eyenziwe nge-massive text collections has shown surprising emerging capabilities to generate text and perform zero- and few-shot learning (Brown et al., 2020; Lieber et al., 2021; Smith et al., 2022; Rae et al., 2021; Chowdhery et al., 2022). Nakuba eminye izimo umphakathi angakwazi ukuxhumana nezimodeli ngokusebenzisa ama-API ezamahala, ukufinyelela kwe-model ephelele ikakhulukazi ihlanganisa kuphela amabhizinisi amakhulu ezisebenzayo.2 Ukufinyelela okuhlobene kuncike amandla abacwaningi ukubuyekeza kanjani futhi ukuthi amamodeli amakhulu amabhizinisi asebenza, okuvimbela ukuthuthukiswa kwezimpendulo ezaziwayo ezivela kumazwe afana ne-robustness, i-bias, kanye ne-toxicity. Kulesi hlobo lwezobuchwepheshe, sinikeza i-Open Pretrained Transformers (OPT), isikhokelo se-transformers e-decoderonly e-pre-trained ezisuka ku-125M kuya ku-175B ama-parameter, okuyinto sinikeza ngokugcwele futhi ngokufanele nabasebenzi abacwaningi abacwaningi. Sinikeza ama-OPT amamodeli ukuze kulinganise ukusebenza kanye nemingcele ye-GPT-3 yama-model, Izici ze-risk, i-harm, i-bias, ne-toxicity, njll, kufanele zithunyelwe yi-collective research community ngokugcwele, okuyinto iyatholakala kuphela lapho amamodeli zithunyelwe ukuhlola. Sithunyelwe zonke amamodeli zethu phakathi kwama-125M ne-66B ama-parameter, futhi sithunyelwe ukufinyelela kokufunda kwe-OPT-175B ngokuhambisana ne-request. Ukufinyelela ku-academic izidakamizwa ziye zithunyelwe; ama-organisations e-government, i-civil society, ne-academic; ne-in-industrial research laboratories. Sithunyelwa futhi i-logbook ye-model creation yethu kanye ne-codebase, i-metaseq,3 eyenza uku Ukusebenza okuhle, futhi ukusetshenziswa kwama-hardware ye-NVIDIA edlulile, sinikeza i-OPT-175B usebenzisa kuphela i-1 / 7th ye-carbon footprint ye-GPT-3. Nakuba lokhu kuyinto ukufinyelela okuhle, i-energy cost of creating such a model is still non-trivial, futhi izixazululo ezihlangeneyo yokuguquka imodeli yesi-size ngeke kuphela ukwandisa ukucubungula ukucubungula kwama-LLMs. Sinikholelwa ukuthi wonke umphakathi we-AI - abacademic abacademics, ibhizinisi be-civil society, abaphathi-policy, futhi umkhakha - kufanele zihlanganisa ukuvelisa izicelo ezizodwa zokusebenza kwe-AI ngokuvamile kanye ne-LLM 2 Uhlobo 2.1 Imodeli Ukubonisa imiphumela kumamodeli ye-Transformer ye-language angama-125 million kuya ku-175 billion ama-parameters. Imibuzo ye-architectural ifakwe ku-Table 1. Ngenxa yobuchwepheshe, futhi ukunciphisa ingozi kwezimpendulo zokusebenza, amamodeli zethu futhi ama-hyperparameters zihlanganisa kakhulu ku-Brown et al. (2020), nge izinguquko emikhulu yama-batch ikakhulukazi ukufinyelela ukusebenza okuphakeme. 2.2 Ukulungiselela ukuqeqeshwa Ukuze ukulungiselela ubuningi, sincoma izindlela ezivamile ezivumelanayo ezivumelanayo e-Megatron-LM codebase,4 usebenzisa ukuguqulwa okwenziwe nge-zero average ne-standard deviation of 0.006. Ukuguqulwa okwenziwe ngama-output layers kubhalwe nge- 1.0/ √ 2L isikhathi lapho i-L iyona inani elikhulu le-layers. Zonke izimo zokudlela ziye ziye ziye ziye ziye zihlanganiswa nge-0, futhi zonke amamodeli zihlanganiswa nge-ReLU activation ne-sequence length of 2048. Thina usebenzisa i-AdamW optimizer (Loshchilov and Hutter, 2017) nge- (β1, β2) ibekwe ku- (0.9, 0.95), futhi ukunciphisa isisindo se-0.1 Thola isitimela se-learning rate linear, okushisa kusuka ku-0 kuya ku-maximum learning rate ngaphansi kwezinyathelo zokuqala ze-2000 e-OPT-175B, noma ngaphezu kwe-375M tokens emakhileni ethu encane, futhi ukunciphisa ku-10% ye-maximum ye-LR phezu e-tokens e-300B. Izinhlelo eziningi ze-mid-flight ze-LR ziye zidingo (bheka i-Section 2.5). Usayizi zethu ze-batch zihlanganisa kusuka ku-0.5M kuya ku-4M ngokuhambisana ne-model size (bheka Table 1) futhi Thina usebenzisa i-dropout ye-0.1 emhlabeni wonke, kodwa sinikeze i-dropout ye-embeddings. Sinikeza izinga le-gradient ku-1.0, ngaphandle kwezinye izinguquko ze-midflight ezincinciphisa le threshold kusuka ku-0.0 kuya ku-0.3 (bheka i-Section 2.5). Sinikeza futhi i-gradient predivide factor ukunciphisa ingozi ye-over/underflows lapho ukulawula i-gradient kuzo zonke izilinganiso (ukubalwa ukuxuba kwe-world-size ye-N ku-two-division operations nge- √ N). 2.3 I-Pre-Training Corpus I-pre-training corpus iqukethe i-concatenation ye-datasets ezisetshenziselwa ku-RoBERTa (Liu et al., 2019b), i-Pile (Gao et al., 2021a), ne-PushShift.io Reddit (Baumgartner et al., 2020; Roller et al., 2021). Zonke i-corpora ziye zihlanganiswe noma zihlanganiswe ukuze zihlanganise ngokuvamile ingxubevange le-English, kodwa inani elincane le-data e-non-English iyatholakala ku-corpus ngokusebenzisa i-CommonCrawl. Sihlanganisa i-documents ezimbini kuzo zonke i-datasets ngokuvimbela i-documents ngokusebenzisa i-MinhashLSH (Rajaraman and Ullman, 2011) nge-J Thola i-BookCorpus (Zhu et al., 2015) ne-Stories (Trinh and Le, 2018) subset ye-ROBERTa corpus futhi isetshenziswe inguqulo esisha ye-CCNews, ebonakalayo izindaba ezivela ku-September 28, 2021. Le CCNews v2 corpus yasungulwa ngokufanayo ne-original i-ROBERTa CCNews (Liu et al., 2019b). RoBERTa Sihlanganisa subset ye-Pile (Gao et al., 2021a), kuhlanganise: CommonCrawl, DM Mathematics, Project Gutenberg, HackerNews, OpenSubtitles, OpenWebText2, USPTO futhi Wikipedia. Ezinye subset ye-Pile ziye zithunyelwe njengokufunda ukuthi zihlanganisa ingozi yokuphazamiseka, ngokuvumelana ne-tendence yokusabela izinga ze-gradient ku-scale 1.3B, noma zihlanganiswa okungagunyaziwe. Zonke subset ziye zithunyelwe ngokuvumelanisa okunye ad-hoc ye-whitespace. The Pile Ngezinye i-subset ye-Pushshift.io corpus eyenziwe ngu-Baumgartner et al. (2020) futhi eyenziwe ngempumelelo ngu-Roller et al. (2021). Ukuguqulwa kwezibambo zokuxhumana kumadokhumenti eyenziwe ngegama-model-friendly, sinikeza inethiwekhi elide yama-comments e-trade ngamunye kanye nokunciphisa zonke izindlela ezingaphezu kuka-tree. Lokhu ukunciphisa i-corpus nge-approx. 66%. PushShift.io Reddit 2.4 Ukusebenza kwe-Training Thola i-OPT-175B ku-992 80GB A100 i-GPU, ngokusebenzisa I-Full Sharded Data Parallel (Artetxe et al., 2021) nge-Megatron-LM Tensor Parallelism (Shoeybi et al., 2019). Thola ukusetshenziswa kwe-147 TFLOP / s ngalinye i-GPU. Sinikeza isimo se-Adam ku-FP32, njengoko sinikeza ku-hosts zonke, kanti izinga le-model lihlala ku-FP16. Ukuze ukunciphise underflows, sisebenzisa ukucindezeleka kwe-dynamic loss, njenge-Micikevicius et al. (2017). 2.5 Izinsizakalo zokusebenza Ngiyaziwa lapha izincwajana zokusebenza eziyinhloko ezivela ngesikhathi se-OPT-175B pre-training. Ngokuvamile, imibuzo ye-hardware yandisa okungenani ama-35 ama-reboot manual kanye ne-cycling angaphezu kwama-100 ama-hosts ngexesha le-2 izinyanga. Nge-reboot manual, isinyathelo se-training lashukumisa, futhi isizinda se-diagnostic zihlanganisa ama-nodes ezinemibuzo. Ama-nodes e-flagged ziye zihlanganiswa futhi ukuqeqeshwa lashukumisa ukusuka kwindawo yokushicilela sokugqibela. Ngenxa yokuhlanganisa inani le-host eyenziwe futhi inani le-reboot manual, sincoma i-70+ ama-reboot automatic ngenxa ye-hardware. Hardware Failures I-Divergent Loss iyiphi ingozi emfundo yethu. Uma i-Divergent Loss, siphinde ukuthi ukunciphisa izinga lokufunda kanye nokuguqulwa kwegama elandelayo okukhuthaza umsebenzi ukuguqulwa nokuhlela. Sinikeza ukuxhumana phakathi kwe-Divergent Loss, ukuguqulwa kwe-Dynamic Loss Scalar ku-0, ne-l2 -norm ye-activations ye-spiking ye-layer ephambili. Lezi zihlanganisa sinikeza izindawo zokufunda lapho i-Dynamic Loss Scalar etholakalayo emzimbeni ye-"healthy" (≥ 1.0), futhi emva kwalokho izinga lwethu lokusebenza lihlanganisa ngezansi ngaphandle kokukhula. Isikhathi lwethu se-LR empirical ibonisa ku-F Loss Divergences Ukuphendula ku-vanilla SGD (i-optimization plateaued ngokushesha, futhi siphindezile ku-AdamW); ukuguqulwa kwe-dynamic loss scalar (eyenza ukuguqulwa kwezinye kodwa akuyona zonke ama-divergences); kanye nokuguqulwa ku-Megatron ye-version ezintsha (eyenza ukucindezeleka kwezinga lokusebenza kanye nokuphuculwa kwama-performance). Other Mid-flight Changes 3 Ukuhlolwa 4.1 Ukukhuthaza & Ukukhuthaza I-HellaSwag (Zellers et al., 2019), i-StoryCloze (Mostafazadeh et al., 2016), i-PIQA (Bisk et al., 2020), i-ARC Easy and Challenge (Clark et al., 2018), i-OpenBookQA (Mihaylov et al., 2018), i-WinoGrad (Levesque et al., 2011), i-WinoGrande (Sakaguchi et al., 2020), ne-SuperGLUE (Wang et al., 2019). Sihlola i-GPT-3 (Brown et al., 2020) ngokusebenzisa ama-prompts kanye nokufaka kwe-experimental jikelele. Sihlanganisa ikakhulukazi ne-GPT-3, ngokufanele ukuguqulela izicelo zabo zokuhlola, kodwa kuhlanganise ukusebenza kwezinye i-LLM Ukubuyekezwa kwebhizinisi we-zero-shot ephakeme kuzo zonke izigaba ezingu-14. Ngokuphathelene, sikholelwa ukuthi imiphumela yethu ephakeme ivela imizuzu ye-GPT-3. Nokho, imiphumela ingahlukile ngokuvamile kuzo zonke izicelo: ukuze kuhlukaniswe ngokuphelele, bheka i-Appendix A. Qaphela ukuthi sinikeze MultiRC ne-WIC kusuka kumayizi ezingu-14, njengoko lezi zamasethe ze-OPT zihlanganisa ngokuvamile imiphumela ye-GPT-3 noma i-OPT ngokulinganiselwe. Ukusebenza kwethu lihlanganisa ngokuvamile i-GPT-3 ngezikhathi ezingu-10, futhi i-underperformed ku-3 izigaba (i-ARC Challenge ne-MultiRC). Kwi-WIC, sin Zero-shot I-Chinchilla (Hoffmann et al., 2022) ne-Gopher (Rae et al., 2021) isebenza ngokuvamile ngokuhambisana nabanye ngenxa yama-parameter sizes, lapho i-PaLM (Chowdhery et al., 2022) ngokuvamile isebenza kahle kuzo zonke izimo, ngisho lapho ukulawula inani kwezinhlayiyana. Sinikeza ukusebenza okusezingeni eliphezulu le-PaLM ikhona ikakhulukazi izinga okusezingeni eliphezulu kanye ne-diversity ye-pre-training data. Ukusebenza okuphezulu kwe-incontext ye-multi-shot ibonisa ku-Figure 4 (ngaphezu kwalokho, ukunciphisa i-MultiRC ne-WIC), kanye nokusebenza okwengeziwe iboniswe ku-Appendix A. Ngokuphathelene nezinqubo ze-OPT, sinikeza ukuthi imodeli ze-OPT zokusebenza ngokufana ne-GPT-3 amamodeli. Nokho, njenge-zerooshot, ukunciphisa imiphumela yayo ngamahhala ibonisa isihloko eyahlukile: ku-set efanayo ye-10 ye-datasets njengoba i-zero-shot, sinikeza ukusebenza efanayo kumamodeli amabili. Ezinye ama-datasets ezahlukile zibonisa ukusebenza engahlukile ngokufana ne-model size ye-OPT ne-G One-shot and Few-shot 3.2 Ukuxhumana Ngokusho ukuthi i-LLM ayaziwa ukuba ingxenye enhle yeemodeli ye-dialog modern (Adiwardana et al., 2020; Roller et al., 2021; Thoppilan et al., 2022; Rae et al., 2021; Chowdhery et al., 2022), sinikezela i-OPT-175B ku-open source data sets. Ngokuvamile, sinikezela i-Roller et al. (2021), futhi i-evaluate ku-ConvAI2 (Dinan et al., 2020b), i-Wizard of Wikipedia (Dinan et al., 2019b), i-Empatic Dialogues (Rashkin et al., 2019), neBlended Skill Talk (Smith et al., 2020). Sinikezela ku-Wizard ye-Internet dataset (Kiliome et al., 2021). Ngaphezu kwalokho, sinikeza ku-R2C2 BlenderBot, imodeli we-Parameter BlenderBot ye-2.7B eyenziwe ngu-Shuster et al. (2022). Sinikeza i-Perplexity ne-Unigram F1 (UF1), ngokulandelana ne-metrics ye-ConvAI2 Competition (Dinan et al., 2020b). Ukulawula i-tokenization eyahlukile kumamodeli ngamunye, sinikeza zonke ama-perplexities ukuba abe emkhakheni ye-GPT-2 tokenizer (Radford et al., 2019). Sinikeza futhi amamodeli abacindezelwe ngokulandelana nemisebenzi yesi-dialogue futhi abacindezelwe. Ukuze i-OPT-175B, zonke imizuzu asebenza ngokusebenzisa ukucubungula kwe- Ukubonisa imiphumela eTabela 2. Sihlola ukuthi i-OPT-175B ibonise kakhulu imodeli we-Reddit 2.7B ebonakalayo kuzo zonke izivakashi, futhi ibonise ngempumelelo nge-Modeli we-BlenderBot 1 ebonakalayo ngokuphelele, ikakhulukazi ku-ConvAI2 dataset. On i-Wizard-of-Internet dataset, ebonakalayo ngokuphelele kuzo zonke ama-models, sihlola ukuthi i-OPT-175B ibonise ubuningi ukuphazamiseka kodwa i-UF1 etholakalayo kunezinhlelo zokulawula kwe-Wizard-of-Wikipedia. Ngokuvamile, singathanda ukuthi imibuzo ye-OPT-175B engabonakaliwe ngokuvumelana ne-BlenderBot 1 ku-ConvAI2 dataset. Lokhu kungathanda ukuchithwa kwe-ConvAI2 dataset kuya ku-pre-training corpus jikelele noma ngisho ku-validation data njengoba ibhaliswe ku-Table 2. Ukuze uthole ukunakekelwa kwe-leakage, thola i-BlenderBot 1 ye-pre-training corpus yayo yokuqala e-ConvAI2 dataset, kodwa siphinde akukho ukuxuba. Ngaphezu kwalokho, siphindeze i-OPT-175B ku-ConvAI2 hidden test set, okuyinto ayidlulile ngokubanzi, futhi siphinde i-10.7 ppl ne-185 UF1, enikezela ukusebenza 4 I-Bias & I-Evaluation ye-Toxicity Ukuze uthole ukuphazamiseka okufakiwe kwe-OPT-175B, sincoma inqaku oluhambisana ne-hate speech detection, ukunakekelwa kwe-stereotype, kanye nokukhiqizwa kwe-content ye-toxic. Nakuba kunezinye iziphumo ezivamile ezivela ku-benchmarks (Blodgett et al., 2021; Jacobs and Wallach, 2021), lezi zinyathelo yokuqala ekutholeni izinzuzo ze-OPT-175B. Sihlanganisa ikakhulukazi ku-GPT-3 Davinci, njengoba lezi zinyathelo zangaphandle ziye zihlanganiswe ku-Brown et al. (2020). 4.1 Ukukhishwa kwe-Hate Speech Ngokusebenzisa i-ETHOS dataset etholakalayo e-Mollas et al. (2020) futhi i-instrumentated ngu-Chiu ne-Alexander (2021), sincoma umthamo we-OPT-175B ukucacisa ukuba noma kungekho izihloko ezithile zebhanki zihlanganisa ukuthi zihlanganisa ukuthi zihlanganisa ukuthi zihlanganisa ukuthi zihlanganisa ukuthi zihlanganisa ukuthi zihlanganisa ukuthi zihlanganisa ukuthi zihlanganisa ukuthi zihlanganisa ukuthi zihlanganisa ukuthi zihlanganisa ukuthi zihlanganisa ukuthi zihlanganisa ukuthi zihlanganisa ukuthi zihlanganisa ukuthi zihlanganisa ukuthi zihlanganisa ukuthi zihlanganisa ukuthi zihlanganisa ukuthi zihlanganisa ukuthi zihlanganisa ukuthi zihlanganisa ukuthi zihlanganisa ukuthi zihlanganisa ukuthi zihl Iziphumo zithunyelwe embhedeni 3. Nge zonke izakhiwo zethu one-shot nge-few-shot, i-OPT175B isebenza kahle kakhulu kune-Davinci. Sinikeza ukuthi oku kuthathwa kusuka emakhasini ezimbili: (1) Ukubuyekezwa nge-API ye-Davinci kungase kuholele izindlela zokulawula ukhuseleko ngaphezu kwemodeli yokuqala ye-175B GPT-3 e-Brown et al. (2020); futhi (2) ukunikezelwa okuphakeme kwezindaba ze-social media ku-pre-training dataset kunikeza izinzuzo ezengeziwe ezihambayo ezisiza ngezinqubo ze-classification. 4.2 I-CrowS-Pairs I-CrowSPairs (Nangia et al., 2020) iyisisombululo ye-crowdsourcing esebenzayo ekuthuthukiseni izinzuzo ze-intrasentence ngezigaba ezingu-9: isondo, isonto, isilwane / umbala, ukucindezeleka kwesondo, ubudala, isizwe, ukucindezeleka, ukucindezeleka, ukucindezeleka kwezilwane, kanye ne-status ye-socioeconomic. Yonke isibonelo iqukethe inqanawa esibonakalayo esibonakalayo, noma anti-stereotype, ngokuhambisana neqembu elilodwa, ngokusho ukucindezeleka ukucindezeleka kwama-model ngokuphathelene ne-stereotypical expressions. Izinqubo ezingaphezulu zihlanganisa ukucindezeleka okuphak Uma kuqhathaniswa ne-Davinci ku-Table 4, i-OPT175B ibonisa ukujabulela okungenani ama-stereotypical ngezigaba ezingaphezu kwe-religion. Ngaphezu kwalokho, lokhu kungenzeka ngenxa yokuhlukaniswa kwedatha zokusebenza; Nangia et al. (2020) zibonisa ukuthi i-Pushshift.io Reddit corpus ine-incidence rate ephezulu ye-stereotypes kanye ne-text discriminatory kunezinye i-corpora (isib. I-Wikipedia). Ngokusho ukuthi lokhu isithombe se-data yokuqala ye-OPT-175B, isampula ingathola izinhlangano ezingaphezu kwe-discriminatory, okuyinto ikhasimende esebenzayo ku-CrowS-Pairs. 4.3 Ukuhlobisa Ngemva noLieber et al. (2021) noArtetxe et al. (2021), sicela usebenzisa i-StereoSet (Nadeem et al., 2021) ukucubungula i-stereotypical bias ku- 4 izigaba: isivumelwano, isondo, isonto kanye nesilinganiso. Ngaphandle kokucubungula kwe-intrasentence (ngama-CrowSPairs), i-StereoSet ihlanganisa ukucubungula kwinqanaba le-intersentence ukuhlola ikhono le-model yokuxhumana ne-context eyengeziwe. Ukubuyekeza ukucubungula okungagunyaziwe phakathi kokucubungula kwe-bias ne-lingual modeling, i-StereoSet ihlanganisa izinga ezimbili: I-Language Modeling Score (LMS) ne-Stereotype Score (SS), eyenziwe ngempumelelo ukuze ifakwe i-Idealized Context Association Test Score (ICAT). Ngokungafani ne-Lieber et al. (2021), sinamathela ama-score nge-token count, engaphansi kwe-character count, okuyinto zihlanganisa ukuthuthukisa ama-metric for amamodeli amaningi. Iziphumo zibonakalisa ku-Table 5. Sinikeza ukuthi i-Davinci ne-OPT-175B zihlanganisa ama-score efana ne-aggregate (i-ICAT ngokuvamile iyatholakala kakhulu phakathi kwama-two). Ngokuvamile, i-Davinci iyatholakala kakhulu kumazwe kanye ne-race, lapho i-OPT-1 4.4 I-RealToxicityPrompts I-Future destirtirtiry i-tendence ye-OPT-175B yokusabela nge-language ye-toxic via i-RealToxicityPrompts (Gehman et al., 2020) dataset. Ngemuva kwe-PaLM (Chowdhery et al., 2022), sinikeza izinjongo ezingu-25 ze-20 tokens usebenzisa i-nucleus sampling (Holtzman et al., 2020) (p = 0.9) ngamunye amayunithi angama-10,000 eyenziwe ngempumelelo kusuka ku-RTP, futhi sinikeza izinga lokuqinisekisa lokuqinisekisa lokuqinisekisa, ezihlaziywa phakathi kwezinto ze-bucket toxicities ze-prompts yokuqala. Ngokuvamile, sinikeza izinga lokuqinisekisa lokuqinisekisa kw 4.5 Ukubuyekezwa kwe-Dialogue Safety Okokuqala, i-SafeDialogues (Ung et al., 2021) ibonise amandla yokuguqulwa kwezimpumelelo zokhuseleko, ngokuvamile ngaphansi kwe-excuse noma ukujabulela imiphumela yayo. Okokuqala, i-SafeBench Unit Tests (Dinan et al., 2021) ibonise ukuthi impendulo yama-modeli kuyinto emangalisayo, eyahlukile phakathi kwezingane ezingu-4 ze-theme sensitivity: I-Safe, i-Realistic, i-Unsafe, ne-Adversarial. Njengoba ne-other dialogue assessments (i-Section 3.2), sinezihlanganisa nama-models ezivamile ze-open-source dialogue. Iziphumo ezimbini zokusebenza ziboniswe embhedeni 6. Sinikezele ukuthi i-OP 5 Izinzuzo Kwi-Section 3.1 ne-4, sinikeza ukucubungula okwengeziwe kwama-models ezivela eziningana. Sinikeza ukucubungula kwama-performance ye-standard assessment datasets ezisetshenziselwa amamodeli we-GPT-3. Ngaphezu kwalokho, sinikeza ukucubungula zokhuseleko, ukucubungula, futhi ukucubungula ukucubungula, futhi sinikeza ukucubungula okwengeziwe ngokuvumelana nezinhlangano ezithile zokusebenza kwe-toxicity kanye ne-hate speech detection. Nokho, izibuyekezo ezivamile angakwazi ukucubungula ngokugcwele izinzuzo zezi amamodeli. Ngokuvamile, sinamathele ukuthi i-OPT-175B inesilinganiso efanayo Ukubuyekeza ngezinyathelo ezivela ekukhiqizeni i-simulation ye-dialogue eqala nge-instruksiyon efana nokuveliswa kwe-instruksiyon. Ukusebenza okusheshayo ku-instruction learning, ku-InstructGPT (Ouyang et al., 2022), kungase ukunciphisa lezi zimpendulo. I-OPT-175B ingathandeki ukuguqulwa futhi ingathandeki ngokushesha. Nakuba ukuguqulwa kwe-sampling ingaba ukunciphisa i-incidence rate ye-repetitive behavior (Holtzman et al., 2020), thina ngempumelelo siye ukunciphisa ngokuphelele lapho i-sampling ye-generation eyodwa kuphela. Ukusebenza okusheshayo ingathandakanya izindlela ezintsha zokunciphisa ukuhlinzeka nok Ngokusho namanye i-LLMs, i-OPT-175B ingakwazi ukukhiqiza izihloko ezingenalutho (i-Adiwardana et al., 2020; Brown et al., 2020; Roller et al., 2021; Rae et al., 2021; Chowdhery et al., 2022; Thoppilan et al., 2022). Lokhu kungenziwa ikakhulukazi emibi lapho ukucindezeleka kwebhizinisi kubaluleke, njenge-healthcare kanye nokukhwabanisa kwenzululululwazi (Weidinger et al., 2021b). Emva kweminyaka embalwa, izixazululo eziningana zihlanganisa ukuthi amamodeli ezihlangene nokukhwabanisa kungcono ukuphucula ukucindezeleka kwe-LLMs (Lewis et al., 2020; Komeili et al., 2021 I-Dathathri et al., 2019; Dinan et al., 2019a; Sheng et al., 2019a; Dinan et al., 2020a; Liu et al., 2019a; Krause et al., 2020; Xu et al., 2020; Liang et al., 2021; Dinan et al., 2021a; Dhamala et al., 2021a; Schick et al., 2021; Ouyang et al., 2022). Ngokusho izicelo ezilandelayo, izicelo ezilandelayo ze-OPT-175B zange kufuneka usebenzise izindlela ezintsha zokuxhumana, ikakhulukazi ngaphambi kokufaka kwelinye izwe zangempela. Njengoba izidingo yethu yokuqala njengama-replication ye-GPT-3, sincoma ukuba asebenzise lezi zixazululo yokuqala. Ngokuvamile, sincoma ukuthi ubuchwepheshe le-OPT-1 Nangona kuhlanganise data sheets and model cards, thina ucwaninga ukunambitheka i-training data nge-data characterization kanye ne-selection criteria ezengeziwe ukuze isetshenziswe idatha ngokufanelekayo. Isetshenzisweni esilandelayo kuyinto ukunambitheka i-model nge-data eningi kunoma futhi ukunambitheka okungenani phakathi kwezi datasets. Ngaphandle kokuqinisekisa okuhlobene, thina ngempumelelo ukuba izakhi zokusebenza zokusebenza eziningana nokuphathelela nokuvamile zezinqubo zokusebenza zokusebenza. Izinzuzo zokusebenza zokusebenza izakhiwo kanye nezinhlayiyana zokusebenza zokufundisa ku-in-context yokufundisa kungenziwa izinzuzo eziningana neziphumo. Thina sincoma ukuthi ukubuy 6 Izinzuzo zokuzimela Ngokusho ama-recommendations kumacwaningi ngamunye eyenziwe yi-Partnership for AI,7 kanye ne-governance guidance eyakhelwe yi-NIST,8 sinikeza zonke izifundo ezihlangene nokufundisa i-OPT-175B ngokusebenzisa i-logbook yethu,9 i-code yethu, futhi inikeza abacwaningi ukufinyelela kwama-model weights ye-OPT-175B, kanye ne-setup ye-baselines ezincinane ezivela ku-OPT175B. Ngokusho ukufinyelela ngokuphelele kwelanga lokusebenza kwe-OPT-175B, futhi kuphela ngokusebenzisa ukwanda kwe-transparency phakathi kwe-LLM yokusebenza, singakwazi ukuqala ukucacisa izinzuzo kanye nezimo ze-LLM ngaphambi kokusebenza okwengeziwe. Ngokuveza ukubuyekeza okuhlobene kwinkqubo yethu yokulungiselela ngosuku, sinikeza kuphela inani le-computer esetshenziselwa ukuqeqesha inguqulo se-OPT-175B yamanje, kodwa futhi i-overhead yabantu ebonakalayo lapho isakhiwo esifundeni noma inqubo yokulungiselela ngokuvumelana ngokuvumelana. Lezi zincazelo zihlukaniswe ngokuvamile emahlathi ezidlulileyo, ngokuvamile ngenxa ye-incapacity yokushintshwa okwenziwe ngempumelelo Ngaphandle kwezi zincwadi, i-metaseq encodebase ngokuvamile iyisisombululo sokugqibela emininzi yezinqubo yethu yokukhiqiza. Ngokuvumela encodebase yethu yokukhiqiza, sinikezela ukuncintisana kwezinqubo ezithile zokukhiqiza okuyinto bakwazi ukufakelwa ngokuvumelanisa kule iphepha, njengoko kubhekwa njenge-detail ye-standard practice emkhakheni, noma kuphela isithombe esizayo. Le encodebase esithakazelisayo kuyinto isisombululo esifundeni esifundeni esifundeni esifundeni yokukhiqiza i-transformer ye-decoderonly okuyinto ≥175B ama-parameters ngaphandle kokusebenzisa i-pipeline parallelism ku-NVIDIA GPUs. Ukuze kusebenze ukuhlolwa kwama-175B, sinikeza abacwaningi ukufinyelela okuqondile kumaparametrithi we-OPT-175B. Isisombululo lapha kuyinto ezimbili: ukufinyelela ukucwaninga kwe-I.I.I. ku-LLM kanye nokunciphisa imiphumela ye-environmental yenza ucwaningo kule-scale. Kukho ingozi esikhulu sokuphelelwa kwezimpendulo ze-ethical ne-social kusuka ku-deploying amamodeli ze-language ne-emergent capabilities ku-scale (Weidinger et al., 2021a; Bommasani et al., 2021; Dinan et al., 2021; Kenton et al., 2021). Ngokunciphisa ukufinyelela kwe-OPT-175B ku-community yokucwaninga nge-license Ngaphezu kwalokho, kunezinto eziningi zokucwaninga nokushintshwa kwe-carbon yokuguqulwa kwama-models yesi-size. Nakuba i-OPT-175B yakhelwe nge-escalated carbon footprint (CO2eq) ye-75 tons,i-10 GPT-3 yakhelwe ukusetshenziswa ama-ton 500 (Patterson et al., 2021), kanti i-Gopher yakhelwe ama-ton 380 (Rae et al., 2021). Lezi zibonelelo engathunyelwa ngokuvamile, futhi izindlela zokucwaninga zokucwaninga zihlanganisa lezi zihlanganisa ayikho. Ngaphezu kwalokho, ukwakhiwa kwe-model iyinhlanganisela kuphela ye-carbon footprint jikelele ye-AI izinhlelo; kufuneka kwenziwe futhi ukuhlolwa nokushintshwa kokuqinisekisa okungenani, Ngokusha i-logbook yethu, sincoma ukucindezela ingxubevange phakathi kwe-cost estimate ye-carbon eyenziwe ngaphandle kwe-hardware failures noma ukucindezeleka kwezimpendulo, kuya ku-cost that is aiming to include the entire LLM development lifecycle. Thina kufuneka ukwazi ukukhiqizwa kwe-carbon (noma eyenziwe) yezinhlelo zezi (Gupta et al., 2021) njengoba zithunyelwe kakhulu kakhulu, futhi sincoma ukuthi iphepha lethu kungasiza umsebenzi elandelayo ekuphumeni izimo ezengeziwe ezidingekayo ekubunjweni umphumela we-scale ku-environment. Ngaphezu kwalokho, ngokukhiqiza inani le-base-liners ku-scale emininzi, sincoma ukuba ibhizinisi elikhulu lwezenzulululwazi ukuhlola imiphumela nama-models ngokuhambisana ne-scale kuphela. Njengoba zithunyelwe ku-Hoffmann et al. (2022), eziningi zezi-LLM ziye ziye ziye zithunyelwe ngaphansi kwezibalo zokufundisa ezisetshenziselwa, okuyinto ibonisa ukuthi ukuhlanganisa idatha ezingaphezu futhi ukufundisa lezi zibonelelo zokusebenza kungenzeka ukwandisa ukusebenza. Kukhona futhi iziphumo ukuthi izinguquko ze-step-function zokusebenza zingatholakala ku-scale engaphansi kakhulu kuka-175B (Wei et al., 2021), okuvumela ukuba kuhlolwa ububanzi be-scale eziningana nezinsizak 7 Ukusebenza okuqukethwe Ngemva kokubhalwa kwe-Transformer Architecture (Vaswani et al., 2017) ne-BERT (Devlin et al., 2019), indawo ye-NLP iye yatholakala isixazululo esikhulu ekusetshenziswa kwe-LLMs nge-auto-supervised pre-training. Amamodeli amaningi e-masked langauge, kuhlanganise ne-T5 (Raffel et al., 2020) ne-MegatronLM (Shoeybi et al., 2019), babonisa ukuthuthukiswa okuqhubekayo ngokusebenzisa ububanzi. Lezi izinzuzo ze-scaling asekelwe kuphela ukwandisa inani lwezinhlayiyana ezivamile ezivamile ezivame ama-models, kodwa futhi inani kanye nenkhiqizo yedatha ze-pre-training (Liu et al., 2019b; Hoffmann et Ukusabela okuphezulu kwe-generative fluency kanye ne-quality eyakhelwe kuqala ku-GPT-2 (Radford et al., 2019) futhi kwenziwe ngempumelelo nge-GPT-3 (Brown et al., 2020) kanye namamodeli ezilandelayo. Nangona amamodeli amakhulu kakhulu (ngaphezulu kwama-100B amaparametre) amamodeli amamodeli amamodeli amamodeli amamodeli amamodeli amamodeli amamodeli amamodeli amamodeli (Lieber et al., 2021; Rae et al., 2021; Thoppilan et al., 2022; Smith et al., 2022; Chowdhery et al., 2022), bonke zihlanganisa isisindo esifundeni futhi zitholakala kuphela ngempumelelo noma nge-API amamodeli Lezi amamodeli zihlanganisa nezimodeli ze-OPT ku-pre-training idatha, izilimi zokusiza kanye nezimo zokusebenza, okwenza kube lwezinhlangano yokuhlola izindlela ezahlukahlukene zokusebenza ngaphambi kokufunda. Kusukela ku-Brown et al. (2020), i-principal assessment criterion ye-LLM has a prompt-based (Black et al., 2022; Rae et al., 2021; Chowdhery et al., 2022), njengoko kwenziwa kule iphepha. Lokhu kubaluleke kakhulu ngenxa yokushintshwa kwezidingo eziningi ngaphandle kokufundisa izicelo ezizodwa. Prompting inesikhathi eside: izivumelwano ze-cloze zihlanganisa eminyakeni eziningi (Chambers and Jurafsky, 2008; Mostafazadeh Nangona kunjalo, ukwakhiwa kwe-prompt efanelekayo ibhizinisi lokuphendula. Iziphumo zihlukile kakhulu futhi engatholakali ngokuchofoza i-prompt (Lu et al., 2021) , futhi amamodeli akukwazi ukuxhumana ne-prompt ngokugcwele njengokuthanda (Webson and Pavlick, 2021) . Ngaphezu kwalokho, kubaluleke ukuxhumana ne-prompt ngaphandle kwe-development set, okuholela imibuzo malunga nokuphumelela kokuphumelela kokuphumelela kokuphumelela kokuphumelela kokuphumelela (Perez et al., 2021) . Thina akufuna ukuxolisa lezi zengxaki ze-prompting, futhi kuncike kuphela ukunikezela ukubuyekeza kwe-OPT-175B emangalisayo. Nokho, 8 Ukuphakama Kule isifundo se-Technical, sinikeza i-OPT, isakhiwo se-auto-regressive language models ezingu-125M kuya ku-175B parameters. Umthombo yethu kuyinto ukuguqulwa ukusebenza kanye nemingcele ye-GPT-3 class of models, futhi isetshenziswe izindlela ezinhle ezintsha zokucwaninga idatha kanye nokucwaninga lokusebenza. Sinikeza izici zokucwaninga, ukucwaninga ukusebenza kwezinhlangano eziningi ze-NLP ne-dialogue, kanye nokuhlanganisa izimo ngokuhambisana ne-bias, i-toxicity kanye ne-hate speech. Sinikeza futhi izimo ezininzi ezinye ezifakiwe ezine ze-models, kanye nokuhlanganisa inqumo esikhulu sokubuyekezwa kwezobuchwepheshe ez Ukubuyiselwa Thina sincoma uScott Jeschonek, Giri Anantharaman, Diego Sarina, Joaquin Colombo, Chris Bray, Stephen Roylance, Kalyan Saladi, Shubho Sengupta, noBrian O’Horo ukunceda ukuguqulwa kwe-infrastructure blockers ngokulandelanayo; Percy Liang, Rishi Bommasani, futhi Emily Dinan ngenxa yobugcisa kwezimfuneko ezinobuchwepheshe; Carole-Jean Wu ngenxa yobugcisa kwezimo ze-sustainability kanye nezinkinga ze-carbon footprint; Srini Iyer, Ramakanth Pasunuru, noShruti Bhosale ngenxa yobugcisa ezidlulayo ezivamile; Benjamin Lefaudeux, Geeta Chauhan, Natalia Gimelshein, Horace, futhi Sam G Ukuhlobisa Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. 2020. Ukuhamba ku-human-like open-domain chatbot. arXiv preprint arXiv:2001.09977. Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, Giri Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Mona T. Diab, Zornitsa Kozareva, futhi Ves Stoyanov. 2021. Ukuhlolwa kwebhizinisi ngokushesha nge-expert mix. CoRR,/2112.684.104. U-Jason Baumgartner, uSavvas Zannettou, uBrian Keegan, uMegan Squire, no-Jeremy Blackburn. 2020. I-pushshift reddit dataset. CoRR, abs/2001.08435. Emily M Bender, Timnit Gebru, Angelina McMillanMajor, futhi Shmargaret Shmitchell. 2021. On the dangers of stochastic parrot: Can language models be too big? In Proceedings of 2021 ACM Conference on Fairness, Accountability, and Transparency, amakhasi 610-623 Yonatan Bisk, Rowan Zellers, Ronan Le bras, Jianfeng Gao, noYejin Choi. 2020. Piqa: Ukucaciswa kwezingane zezingane emzimbeni emzimbeni emzimbeni emzimbeni emzimbeni emzimbeni emzimbeni emzimbeni emzimbeni emzimbeni emzimbeni emzimbeni emzimbeni emzimbeni, 34(05):7432–7439. Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, noSamuel Weinbach. 2022. Gpt-neox-20b: I-open-source model ye-autoregressive language. Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, noHanna Wallach. 2021. I-Stereotyping of Norwegian Salmon: A inventory of traps in fairness benchmark datasets. In I-Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (I-Volume 1: Long Papers), amakhasi 1004-1015, Online. I-Association for Computational Linguistics. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie S. Chen, Kathleen Creel, Jared Quincy Davis, Noah D. Goodman, Shelby Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li FeiFei, Chelsea Finn, Trevor Gale, Lauren Gespillie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsun U-Sebastian Borgeaud, uArthur Mensch, uJordan Hoffmann, uTrevor Cai, uEliza Rutherford, uKatie Millican, uGeorge van den Driessche, uJean-Baptiste Lespiau, uBogdan Damoc, uAidan Clark, et al. 2021. Ukuphucula imodeli ye-language ngokufaka kusuka ku-trillions of tokens. arXiv preprint arXiv:2112.04426. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel HerbertVoss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, futhi Dario Amodei. 2020. Amamodeli zezilwanyana abaneminyaka embalwa. In Advances in Neural Information Processing Systems, Volume 33, 1877-1901. Curran Associates, Inc. U-Nathanael Chambers no-Dan Jurafsky. 2008. Ukufundwa okuzenzakalelayo kwe-narrative event chains. In Proceedings of ACL-08: HLT, amakhasi 789-797 Columbus, Ohio. I-Association for Computational Linguistics. Ke-Li Chiu noRohan Alexander. 2021. Ukukhishwa kwe-hate speech nge-gpt-3. arXiv preprint arXiv:2103.12407. Aakanksha Diawon Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Ghemawat, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sun Devipa, Henryk Michalewski, Xavier Garcia, Vedantra Mis, Kevin Robinson, Liam Fedus, Jason Zhou, Den Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, no-Oyvind Tafjord. 2018. Uyakwazi ukuyisombulula umbuzo ngokuphendula? uchungechunge arc, i-AI2 ukubuyekeza. CoRR, abs/1803.05457. Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, futhi Rosanne Liu. 2019. Plug and play language models: A simple approach to controlled text generation. arXiv preprint arXiv:1912.02164. Jacob Devlin, Ming-Wei Chang, Kenton Lee, noKristina Toutanova. 2019. BERT: Ukuqeqeshwa kokuqala kwe-deep bidirectional transformers for language understanding. Ku-North American Association for Computational Linguistics (NAACL). U-Jwala Dhamala, uTony Sun, uVarun Kumar, uSatyapriya Krishna, uYada Pruksachatkun, uKai-Wei Chang, uRahul Gupta. 2021. I-Bod: Dataset ne-metric for measuring biases in open-end language generation. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, amakhasi 862-872. Emily Dinan, Gavin Abercrombie, A Stevie Bergman, Shannon Spruit, Dirk Hovy, Y-Lan Boureau, futhi Verena Rieser. 2021. Ukuhlinzeka kwezimo zokhuseleko ku-e2e conversational ai: Framework futhi tooling. arXiv preprint arXiv:2107.03451. Emily Dinan, Angela Fan, Adina Williams, Jack Urbanek, Douwe Kiela, noJason Weston. 2020a. I-Queens iyakwazi futhi: Ukunciphisa i-gender bias ku-dialogue generation. In I-Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), amakhasi 8173-8188, Online. I-Association for Computational Linguistics. U-Emily Dinan, uSamuel Humeau, uBharath Chintagunta, noJason Weston. 2019a. Isakhiwo se-break it fix it for dialogue safety: I-robustness kusuka ku-attack ye-adversarial. arXiv preprint arXiv:1908.06083. Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, Shrimai Prabhumoye, Alan W. Black, Alexander Rudnicky, Jason Williams, Joelle Pineau, Mikhail Burtsev, noJason Weston. 2020b. Umthamo we-2 we-Conversational Intelligence (ConvAI2). In The NeurIPS '18 Competition, amakhasi 187-208, Cham. Springer International Publishing. U-Emily Dinan, uStephen Roller, uKurt Shuster, uAngela Fan, uMichael Auli, uJason Weston. 2019b. I-Wizard ye-Wikipedia: Izinto ze-conversational ezisebenzayo ze-Knowledge. Ngo-Proceedings of the International Conference on Learning Representations. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, futhi Connor Leahy. 2021a. I-stack: I-dataset ye-800gb ye-text eyenziwe nge-modeling ye-language. CoRR, abs/2101.00027. Tianyu Gao, Adam Fisch, noDanqi Chen. 2021b. Ukwenza amamodeli amahlathi amahlathi amahlathi amahlathi amakhulu amahlathi amahlathi amahlathi amahlathi amahlathi amahlathi amahlathi amahlathi amahlathi amahlathi amahlathi amahlathi amahlathi amahlathi amahlathi amahlathi amahlathi amahlathi amahlathi amahlathi amahlathi amahlathi amahlathi amahlathi amahlathi amahlathi amahlathi amahlathi amahlathi amahlathi amahlathi amahlathi amahlathi amahlathi amahlathi amahlathi amahlathi amahlathi amahlat Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, noKate Crawford. 2021. Idatasheets for datasets. Common. ACM, 64(12):86-92. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, no. Noah A. Smith. 2020. RealToxicityPrompts: Ukubuyekezwa kwe-neural toxic degeneration kumamodeli ye-language. In Iziphumo ze-Association for Computational Linguistics: EMNLP 2020, amakhasi 3356-3369, Online. I-Association for Computational Linguistics. Udit Gupta, Young Geun Kim, Sylvia Lee, Jordan Tse, Hsien-Hsin S Lee, Gu-Yeon Wei, David Brooks, no-Carole-Jean Wu. 2021. Chasing carbon: Ukuhlobisa okwelashwa kwe-computing. IEEE International Symposium on High-Performance Computer Architecture (HPCA 2021). Kaiming He, Xiangyu Zhang, Shaoqing Ren, noJian Sun. 2016. Ukufundwa okuqhubekayo yokufakelwa kwe-image recognition. In I-Proceedings of the IEEE conference on computer vision and pattern recognition, amakhasi 770-778. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, futhi Laurent Sifre. 2022. Ukuqeqesha amamodeli amabhizinisi amakhulu. U-Ari Holtzman, uJan Buys, uMaxwell Forbes, noYejin Choi. I-2020. I-Curious Case of Neural Text Degeneration. I-ArXiv, i-abs/1904.09751. Abigail Z. Jacobs noHanna Wallach. 2021. Ukukhipha nokuhlakazeka. Ngo-Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, page 375-385, New York, NY, USA. I-Association for Computing Machinery. U-Zachary Kenton, uTom Everitt, uLaura Weidinger, uIason Gabriel, uVladimir Mikulik, uGeoffrey Irving. 2021. Ukuhlobisa ama-agent ye-language. CoRR, abs/2103.14659. Mojtaba Komeili, Kurt Shuster, noJason Weston. 2021. Ukwakhiwa kwe-internet-augmented dialogue. CoRR, abs/2107.07566. Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, futhi Nazneen Fatema Rajani. 2020. GEDI: Generative discriminator guided sequence generation. arXiv preprint arXiv:2009.06367. Brian Lester, Rami Al-Rfou, noNoah Constant. 2021. Umthamo we-scale ye-parameter-efficient prompt tuning. CoRR, abs/2104.08691. Hector J Levesque, Ernest Davis, futhi Leora Morgenstern. 2011. The Winograd schema challenge. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, umugqa 46, iphepha 47. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Ukukhishwa kwe-Retrieval-Augmented for knowledge-intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459–9474. Xiang Lisa Li noPercy Liang. 2021. I-Prefix-Tuning: Ukuqhathanisa Izimpendulo Zezimali Zezimali. amakhasi 4582–4597. Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, noRuslan Salakhutdinov. 2021. Ukuxhumana nokunciphisa izinzuzo zomphakathi kumamodeli yebhizinisi. In International Conference on Machine Learning, amakhasi 6565-6576. PMLR. Opher Lieber, noma Sharir, Barak Lenz, futhi Yoav Shoham. 2021. I-Jurassic-1: Imibuzo yobuchwepheshe nokushintshwa. Imininingwane yobuchwepheshe, I-AI21 Labs. Haochen Liu, Jamell Dacon, Wenqi Fan, Hui Liu, Zitao Liu, no Jiliang Tang. 2019a. Ingabe umzimba umzimba umzimba umzimba umzimba umzimba umzimba umzimba umzimba umzimba umzimba umzimba. arXiv preprint arXiv:1910.10486. U-Haokun Liu, uWilliam Huang, uDhara Mungra, noSamuel R. Bowman. I-2020. I-Task Formalization ye-Winograd Schema Assessments. In I-Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), amakhasi 8275-8280, Online. I-Association for Computational Linguistics. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, futhi Weizhu Chen. 2021. Yintoni izibonelo ezinhle zokusebenza kwe-gpt-3? CoRR, abs/2101.06804. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, futhi Veselin Stoyanov. 2019b. Roberta: Isisombululo esebenzayo esebenzayo esebenzayo. arXiv preprint arXiv:1907.11692. I-Ilya Loshchilov noFrank Hutter. 2017. Ukuguqulwa kwe-regularization ye-weight decay ku-adam. CoRR, abs/1711.05101. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, futhi Pontus Stenetorp. 2021. Imibuzo emangalisayo emangalisayo futhi lapho ukufumana: Ukuphumelela emibuzo emangalisayo emangalisayo emangalisayo emangalisayo. Clara Meister, Tim Vieira, futhi Ryan Cotterell. 2020. Ukusebenza kwe-Association for Computational Linguistics, 8:795–809. U-Paulius Micikevicius, uSharan Narang, uJonah Alben, uGregory Diamos, uErich Elsen, uDavid Garcia, uBoris Ginsburg, uMichael Houston, uOleksii Kuchaiev, uGanesh Venkatesh, et al. 2017. Ukuqeqeshwa kwe-precision. arXiv preprint arXiv:1710.03740. Todor Mihaylov, Peter Clark, Tushar Khot, no-Ashish Sabharwal. 2018. Ingaba isikhwama se-armor ivela umugqa? I-dataset entsha yokufaka imibuzo se-open book. CoRR, abs/1809.02789. Tomas Mikolov, Jiri Kopecky, Lukas Burget, Ondrej Glembek, et al. 2009. Imodeli ye-network ye-neural-based language for highly inflective languages. Ngo-2009 i-IEEE International Conference on acoustics, speech and signal processing, amakhasi 4725-4728 IEEE. Sewon Min, Mike Lewis, Luke Zettlemoyer, futhi Hannaneh Hajishirzi. 2021. Metaicl: Ukufundisa ukufundisa ku-context. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, noLuke Zettlemoyer. 2022. Ukubuyekeza umphumela we-demonstrations: Yintoni okwenza ukufundisa kwegama? arXiv preprint arXiv:2202.12837. Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, futhi Timnit Gebru. 2018. I-Model Cards for Model Reporting. CoRR, abs/1810.03993. Ioannis Mollas, Zoe Chrysopoulou, Stamatis Karlos, futhi Grigorios Tsoumakas. 2020. ETHOS: idatha yedatha yokubonisa umculo we-hate speech online. CoRR, abs/2006.08328. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, noJames F. Allen. 2016. I-corpus ne-evaluation framework ye-common sense stories. CoRR, abs/1604.01696. Moin Nadeem, Anna Bethke, futhi Siva Reddy. 2021. StereoSet: Ukubalwa imibuzo stereotypical emadolobheni amamodeli pre-trained. In Association for Computational Linguistics (ACL). U-Reiichiro Nakano, uJacob Hilton, uSuchir Balaji, uJeff Wu, uLong Ouyang, uChristina Kim, uChristopher Hesse, uShantanu Jain, uVineet Kosaraju, uWilliam Saunders, et al. 2021. Webgpt: Ukuphendula kwe-browser nge-feedback yabantu. arXiv preprint arXiv:2112.09332. Nikita Nangia, Clara Vania, Rasika Bhalerao, noSamuel R Bowman. 2020. I-Crows-pairs: A challenge dataset for measuring social biases in masked language models. arXiv preprint arXiv:2010.00133. U-Erik Nijkamp, uBo Pang, uHiroaki Hayashi, uLifu Tu, uHuan Wang, uYingbo Zhou, uSilvio Savarese, no-Caiming Xiong. 2022. I-paradigm ye-conversational ye-program synthesis. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Ukwakhiwa kwezilimi amamodeli ukuhlola imiyalezo nge umnikelo womuntu. arXiv preprint arXiv:2203.02155. UDavid Patterson, uJoseph Gonzalez, uQuoc Le, uChen Liang, uLuis-Miquel Munguia, uDaniel Rothchild, uDavid So, uMaud Texier, noJeff Dean. 2021. Ukuhlukaniswa kwe-carbon kanye nokufunda kwebhizinisi elikhulu ye-neuronal. arXiv preprint arXiv:2104.10350. U-Ethan Perez, uDouwe Kiela, noKyunghyun Cho. 2021. Ukufundwa okwenziwe ngempumelelo nge-model ye-language. Advances in Neural Information Processing Systems, 34. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, no-Alexander Miller. Imodeli ye-lingu as a knowledge base? I-Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLPIJCNLP), amakhasi 2463-2473, Hong Kong, China. I-Association for Computational Linguistics. U-Alec Radford, u-Karthik Narasimhan, u-Time Salimans, u-Ilya Sutskever. 2018. Ukuphucula ukubuyekezwa kwe-language nge-learning engaphandle. Imininingwane ye-Technical, i-OpenAI. U-Alec Radford, uJeffrey Wu, uRewon Child, uDavid Luan, uDario Amodei, uIliya Sutskever. 2019 Imodeli ye-Language akuyona abalandeli abalandeli abalandeli abalandeli. Isitifiketi se-Technical, i-OpenAI. Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, H. Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Amy Wu, Erich Elsen, Siddhant M. Jayakumar, Po-Sen Huang, Amelia Glaese, David Budden, Johannes Welbl, Sumanth Dathathri, Chris Saffron Huang, Laurent Sifre Huang, Jonathan Uesayato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant M. Jayakumar, Elena Buchatskaya, David Budden, Igor Sutherland, Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, no. Peter J. Liu. 2020. Ukuhlola izinzuzo zokuxhumana nge-transformer eyodwa ye-text-to-text. I-Journal of Machine Learning Research (JMLR), 21:1-67. U-Anand Rajaraman no-Jeffrey David Ullman. 2011. Ukushintshwa kwama-datasets amakhulu. I-Cambridge University Press. U-Hannah Rashkin, u-Eric Michael Smith, u-Margaret Li, no-Y-Lan Boureau. 2019. Ukusuka kwama-empathic opendomain conversational models: A new benchmark and dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, amakhasi 5370-5381, Florence, Italy. I-Association for Computational Linguistics. Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, noJason Weston. 2021. Ukulungiswa kokwakha i-chatbot e-open-domain. I-Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, amakhasi 300-325, Online. I-Association for Computational Linguistics Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, noYejin Choi. 2020. I-Winogrande: A adversarial vineyard schema challenge on scale. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, I-10th AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, amakhasi 8732-8740. AAAI Press. Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry Jason, Alan Fries, Ryan Te U-Teven Le Scao no-Alexander M. Rush. 2021. U-Data Points is a prompt value? amakhasi 2627– 2636. Timo Schick noHinrich Schütze. 2020. It is not only size that matters: Imodeli amabhizinisi amancane ama-learners. CoRR, abs/2009.07118. Timo Schick, Sahana Udupa, futhi Hinrich Schütze. 2021. Self-diagnosis kanye self-debiasing: A proposal for reducing corpus-based bias in nlp. Transactions of the Association for Computational Linguistics, 9:1408-1424. U-Rico Sennrich, uBarry Haddow, no-Alexandra Birch. 2016. Ukuguqulwa kwe-Neural Machine ye-rare words nge-subword units. In I-Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (I-Volume 1: Long Papers), amakhasi 1715-1725, Berlin, Germany. I-Association for Computational Linguistics. Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, futhi Nanyun Peng. 2019. Umbhali wasebenza njenge-baby-sitter: On Izinzuzo ku-language generation. arXiv preprint arXiv:1909.01326. U-Taylor Shin, uYasaman Razeghi, uRobert L. Logan IV, uEric Wallace, noSameer Singh. 2020. I-AutoPrompt: Ukukhishwa kwe-Knowledge kusuka ku-Language Models nge-Automatically Generated Prompts. amakhasi ze-4222-4235. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, noBryan Catanzaro. 2019. Megatron-lm: Ukuqeqesha ama-billion-parameter language models usebenzisa model parallelism. arXiv preprint arXiv:1909.08053. UKurt Shuster, Mojtaba Komeili, uLeonard Adolphs, uStephen Roller, uArthur Szlam, noJason Weston. 2022. Imodeli ye-language eyenza ulwazi: Ukuhlola kwe-Modular & Generation for dialogue and prompt completion. arXiv preprint arXiv:2203.13224. U-Eric Smith, u-Mary Williamson, u-Kurt Shuster, u-Jason Weston, no-Y-Lan Boureau. U-Y-Lan Boureau. I-2020. Ungayifaka yonke into: Ukubuyekeza umthengi we-conversational’s ability to blend skills. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. ACL. Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zheng, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, noBryan Catanzaro. 2022. Ukusebenzisa isivinini futhi megatron ukuqeqesha megatron-turing NLG 530b, A enkulu-scale generative model isilimi. CoRR, abs/2201.11990. U-Romal Thoppilan, uDaniel De Freitas, uJamie Hall, uNoam Shazeer, uApoorv Kulshreshtha, uHeng-Tze Cheng, uAlicia Jin, uTaylor Bos, uLeslie Baker, uYu Du, et al. 2022. Lamda: Imodeli ye-language ye-dialog application. arXiv preprint arXiv:2201.08239. U-Trieu H. Trinh noQuoc V. Le. 2018. Isinyathelo esisodwa sokucindezeleka. CoRR, abs/1806.02847. Megan Ung, Jing Xu, no-Y-Lan Boureau. 2021. I-SafeDialogue: Ukuthatha imibuzo emangalisayo emva kokuphumelela kwezobuchwepheshe zokhuseleko. ArXiv, abs/2110.07518. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, futhi Illia Polosukhin. 2017. Ukuhlobisa kuyinto konke ufuna. In Advances in neural information processing systems. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, no-Samuel R. Bowman. 2019. SuperGLUE: Umhlahlandlela olufanelekayo kwezinhlelo zokusetshenziswa kwezilimi. arXiv preprint 1905.00537. U-Albert Webson no-Ellie Pavlick. 2021. Amamodeli amamodeli amamodeli amamodeli amamodeli amamodeli amamodeli amamodeli amamodeli amamodeli amamodeli? arXiv preprint arXiv:2109.01247. Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, futhi Quoc V. Le. 2021. Imodeli ye-language eyenziwe ngempumelelo. CoRR, abs/2109.01652. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zac Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks, William Isaac, Sean Legassick, Geoffrey Irving, and Iason Gabriel. 2021a. Ethical and social risks of harm from language models. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. 2021b. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359. Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, noJason Weston. 2020. Ukwakhiwa kwe-neural text nge ukuqeqeshwa kwe-unprobability. Ku-International Conference on Learning Representations. Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga Behram, James Huang, Charles Bai, Michael Gschwind, Anurag Gupta, Myle Ott, Anastasia Melnikov, Salvatore Candido, David Brooks, Geeta Chauhan, Benjamin Lee, Hsien-Hsin S. Lee, Bugra Akyildiz, Maximilian Balandat, Joe Spisak, Ravi Jain, Mike Rabbat, and Kim Hazelwood. 2022. Sustainable AI: environmental implications, challenges and opportunities. In Proceedings of the Conference on Machine Learning and Systems. Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, futhi Emily Dinan. 2020. Ukulungiselela ukhuseleko kwi-chatbots e-open-domain. arXiv preprint arXiv:2010.07079. Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, futhi Emily Dinan. 2021a. I-bot-adversarial dialogue for safe conversational agents. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, amakhasi 2950–2968, Online. I-Association for Computational Linguistics. UJing Xu, uArthur Szlam, noJason Weston. 2021b. Ngaphandle kweMemori ye-Goldfish: Ukuxhumana kwe-open-domain eminingi. arXiv preprint arXiv:2107.07567. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, noYejin Choi. 2019. Hellaswag: Ingaba umatshini ukugcina isihloko sakho? I-Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28-2 August 2019, Volume 1: Long Papers, amakhasi 4791-4800. I-Association for Computational Linguistics. Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, futhi Sanja Fidler. 2015. Ukuqhathanisa imibhalo nezifilimu: Ukuqhathanisa izifundo ezibonakalayo ngokufunda imibhalo kanye nokufunda imibhalo. CoRR, abs/1506.06724. Ukuhlolwa okwengeziwe B Izinzuzo Ukuqeqeshwa • Umklamo wokuqala: Susan Zhang • Ukwakhiwa kwe-Infrastructure ne-Ablations yokuqala: Naman Goyal, Myle Ott, Stephen Roller, Sam Shleifer, Susan Zhang • Training efficiency: Naman Goyal, Myle Ott, Sam Shleifer • Data curation kanye deduplication: Shuhoi Chen, Myle Ott, Stephen Roller • Ukwakhiwa nokulawula OPT-175B: Mikel Artetxe, Moya Chen, Naman Goyal, Punit Singh Koura, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Stephen Roller, Susan Zhang • Training 125M–66B baselines: Naman Goyal, Stephen Roller, Susan Zhang Ukuhlolwa • NLP: Xian Li, Xi Victoria Lin, Todor Mihaylov, Stephen Roller, Anjali Sridhar Umbhali: Stephen Roller • Ukubuyekezwa kwe-AI ezigcwele: Punit Singh Koura, Stephen Roller, Tianlu Wang U-Moya Chen, uStephen Roller, uLuke Zettlemoyer, uSusan Zhang Paper writing: U-Christopher Dewan, u-Susan Zhang Code release preparation: Mona Diab, U-Susan Zhang Responsible AI conduct: C Datasheet Thola izixazululo ze-Gebru et al. (2021) futhi sinikeza ikhadi le-dataset esebenzisana ekuthuthukiseni amamodeli ze-OPT. C.1 Ukuphendula I-pre-training data yokulungisa imodeli ye-OPT-175B yasungulwa yi-union ye-5 datasets, kuhlanganise ama-datasets ezintathu ezisetshenziselwa ku-ROBERTa (Liu et al., 2019b), a subset ye-Pile (Gao et al., 2021a), kanye ne-Pushshift.io Reddit dataset eyenziwe ku- (Baumgartner et al., 2020) futhi isetshenziselwa ku- (Roller et al., 2021). Lezi izicelo zokwakha lokhu dataset zihlanganisa ukulungiselela imodeli yebhizinisi ku-corpus esikhulu se-text, ngokucindezela ku-human-generated text. • For what purpose was the dataset created? Was there a specific task in mind? Was there a specific gap that needed to be filled? Please provide a description. Ukucaciswa kwe-AI • Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)? Ukucaciswa kwe-AI • Who funded the creation of the dataset? If there is an associated grant, please provide the name of the grantor and the grant name and number. • Ukusebenza Yini? Any other comments? C.2 Isakhiwo The instances are textual documents. The overall dataset is composed from a union of the following datasets: • What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Are there multiple types of instances (e.g., movies, users, and ratings; people and interactions between them; nodes and edges)? Please provide a description. – BookCorpus (Zhu et al., 2015) iqukethe ezingaphezu kuka-10K amabhuku ebuthunyelwe – CC-Stories (Trinh and Le, 2018) contains a subset of CommonCrawl data filtered to match the story-like style of Winograd schemas – I-Pile (Gao et al., 2021a) ebandayo: * Imininingwane-CC * I-OpenWebText2 * Ukusebenza * I-Project Gutenberg * I-OpenSubtitles * I-Wikipedia * DM Mathematics * HackerNews – Pushshift.io Reddit dataset that was developed in Baumgartner et al. (2020) and processed in Roller et al. (2021). – CCNewsV2 okuqukethe inguqulo eyakhiwe ye-English ingxenye ye-CommonCrawl News dataset eyenziwa ku-RoBERTa (Liu et al., 2019b) Idatha ye-training iqukethe i-token ye-180B efana ne-800 GB yedatha. • How many instances are there in total (of each type, if appropriate)? I-CC-stories dataset iqukethe i-subset ye-CommonCrawl idatha eyenziwe ngokubambisana ne-story-like style ye-Winograd schemes. I-rest of the dataset iyatholakala emithonjeni angaphambili, i-reformatted, ne-deduplicated. • Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? If the dataset is a sample, then what is the larger set? Is the sample representative of the larger set (e.g., geographic coverage)? If so, please describe how this representativeness was validated/verified. If it is not representative of the larger set, please describe why not (e.g., to cover a more diverse range of instances, because instances were withheld or unavailable). Konke isibonelo iqukethe data raw text. • What data does each instance consist of? “Raw” data (e.g., unprocessed text or images) or features? In either case, please provide a description. Yini? • Is there a label or target associated with each instance? If so, please provide a description. Yini? • Is any information missing from individual instances? If so, please provide a description, explaining why this information is missing (e.g., because it was unavailable). This does not include intentionally removed information, but might include, e.g., redacted text. Akukho ukuxhumana okuhlobene phakathi kwezinye izigaba. • Are relationships between individual instances made explicit (e.g., users’ movie ratings, social network links)? If so, please describe how these relationships are made explicit. Thina ubhalisele i-validation set ye-approx. 200MB kusuka ku-pre-training data, eyenziwe ngama-sampled ngokuhambisana ne-biggest ye-dataset e-pre-training corpus. • Are there recommended data splits (e.g., training, development/validation, testing)? If so, please provide a description of these splits, explaining the rationale behind them. Ngaphandle kwe-duplication ebonakalayo eyenziwe ngempumelelo phakathi kwama-datasets, akukho ezinye i-redundancies, ama-errors, noma izilimi ze-noise etholakalayo. • Are there any errors, sources of noise, or redundancies in the dataset? If so, please provide a description. • Ukusebenza Kuyinto self-contained Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? Izigaba ze-dataset zihlanganisa i-subset ye-Common Crawl data, kanye ne-subset ye-Reddit data, okuqukethe ama-phrases, okuyinto, uma ibhekwa ngqo, kungase zihlanganisa, zihlanganisa, zihlanganisa, noma kungase kungase kusiza ukuphazamiseka. • Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? If so, please describe why. Izidakamizwa ezithile zezi zebhizinisi zihlanganisa abantu, ezifana nezindaba, izifundo ze-Wikipedia, njll. • Does the dataset relate to people? If not, you may skip the remaining questions in this section. Noma, idatha asebenza ngokuvumelana ne-subpopulation identification. • Does the dataset identify any subpopulations (e.g., by age, gender)? If so, please describe how these subpopulations are identified and provide a description of their respective distributions within the dataset. Yini? • Any other comments? I-C.3 Inqubo ye-Collection N/A. I-dataset iyinhlanganisela yezinhlanganisela ze-5 ze-datasets ezivamile zitholakala. • How was the data associated with each instance acquired? Was the data directly observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey responses), or indirectly inferred/ derived from other data (e.g., part-of-speech tags, model-based guesses for age or language)? If data was reported by subjects or indirectly inferred/derived from other data, was the data validated/verified? If so, please describe how. • Ukusebenza Izixhumi zithunyelwe kusuka ku-internet. What mechanisms or procedures were used to collect the data (e.g., hardware apparatus or sensor, manual human curation, software program, software API)? How were these mechanisms or procedures validated? Sicela uchofoze imibuzo ezidlulileyo mayelana nokufaka indlela yokufaka idatha. • If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)? Lezi zebhizinisi zihlanganiswa, zihlanganiswa futhi zihlanganiswa ngamathuluzi. • Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)? I-CC-News dataset ibandakanya i-articles ye-newspapers e-English ezivela phakathi kwe-September 2016 ne-September 2021. • Over what timeframe was the data collected? Does this timeframe match the creation timeframe of the data associated with the instances (e.g., recent crawl of old news articles)? If not, please describe the timeframe in which the data associated with the instances was created. Yini? • Does the dataset relate to people? If not, you may skip the remainder of the questions in this section. • Ukusebenza N / A Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources (e.g., websites)? N / A • Were the individuals in question notified about the data collection? If so, please describe (or show with screenshots or other information) how notice was provided, and provide a link or other access point to, or otherwise reproduce, the exact language of the notification itself. N / A • Did the individuals in question consent to the collection and use of their data? If so, please describe (or show with screenshots or other information) how consent was requested and provided, and provide a link or other access point to, or otherwise reproduce, the exact language to which the individuals consented. N/A. • If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or for certain uses? If so, please provide a description, as well as a link or other access point to the mechanism (if appropriate). Izinguquko ezithile ze-toxicity ne-bias ziye zitholakala. Sicela uxhumane ifomu lokuqala kanye ne-model card ukuze uthole imininingwane. • Has an analysis of the potential impact of the dataset and its use on data subjects (e.g., a data protection impact analysis) been conducted? If so, please provide a description of this analysis, including the outcomes, as well as a link or other access point to any supporting documentation. Yini? • Any other comments? C.4 Preprocessing / Ukunciphisa / Ukuhlobisa Iziqu ze-component datasets zithunyelwe izindlela ezivamile ze-cleaning ne-re-formatting, kuhlanganise ukunciphisa umbhalo we-repetitive/non-informative njenge-”Chapter One,” noma “This eBook by Project Gutenberg.” • Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? If so, please provide a description. If not, you may skip the remainder of the questions in this section. I-dataset ye-component "i-raw" iyatholakala ngokuvumelana nezindawo zayo (ama-details ziyaziwa kumadokhumenti zayo zihlanganisa ngezifundo). • Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data (e.g., to support unanticipated future uses)? If so, please provide a link or other access point to the “raw” data. Yini? • Any other comments? C.5 Ukusetshenziswa Yes, le dataset isetshenziselwa pre-training i-OPT amamodeli. • Has the dataset been used for any tasks already? If so, please provide a description. • Ukusebenza https://github.com/facebookresearch/ i-metaseq Is there a repository that links to any or all papers or systems that use the dataset? If so, please provide a link or other access point. Lezi zebhizinisi zingasetshenziselwa ukulungiselela amamodeli zebhizinisi, okuyinto zihlanganisa izinhlelo eziningi zokusebenza zebhizinisi. • What (other) tasks could the dataset be used for? I-pipeline yokwenza le dataset ivula indlela yokwakha isakhiwo se-scalable ye-mining datasets. • Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses? For example, is there anything that a future user might need to know to avoid uses that could result in unfair treatment of individuals or groups (e.g., stereotyping, quality of service issues) or other undesirable harms (e.g., financial harms, legal risks) If so, please provide a description. Is there anything a future user could do to mitigate these undesirable harms? Ngaphandle kwalokho, akuyona okungaziwa ngoku. • Are there tasks for which the dataset should not be used? If so, please provide a description. Yini? • Any other comments? C.6 Ukusabalalisa Ngicabanga lokhu isikhathi. • Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? If so, please provide a description. • Ukusebenza N / A How will the dataset will be distributed (e.g., tarball on website, API, GitHub)? Does the dataset have a digital object identifier (DOI)? N / A • When will the dataset be distributed? N / A • Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? If so, please describe this license and/or ToU, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms or ToU, as well as any fees associated with these restrictions. N / A • Do any export controls or other regulatory restrictions apply to the dataset or to individual instances? If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any supporting documentation. Yini? • Any other comments? C.7 Ukusetshenziswa Ukucaciswa kwe-AI • Who is supporting/hosting/maintaining the dataset? • Ukusebenza Kusukela ku-Document Main How can the owner/curator/manager of the dataset be contacted (e.g., email address)? N / A • Is there an erratum? If so, please provide a link or other access point. Okungenani akuyona inqubo yokushicilela. • Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)? If so, please describe how often, by whom, and how updates will be communicated to users (e.g., mailing list, GitHub)? N / A • If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were individuals in question told that their data would be retained for a fixed period of time and then deleted)? If so, please describe these limits and explain how they will be enforced. N / A • Will older versions of the dataset continue to be supported/hosted/maintained? If so, please describe how. If not, please describe how its obsolescence will be communicated to users. Okwamanje, akukho mechanism itholakala. • If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so? If so, please provide a description. Will these contributions be validated/ verified? If so, please describe how. If not, why not? Is there a process for communicating/ distributing these contributions to other users? If so, please provide a description. Yini? • Any other comments? D Model ikhadi Ngemva Mitchell et al. (2018), sinikeza ikhadi model ye-OPT-175B. D.1 Model Izindaba • Ukusebenza I-OPT-175B yasungulwa yi-Meta AI. Person or organization developing model: • Ukusebenza I-OPT-175B yasungulwa ngo-May 3, 2022. Model date: • Ukusebenza I-OPT-175B ebonakalayo kulesi isihloko se-version 1.0.0. Model version: I-OPT-175B iyimodeli enkulu ye-decoder-only transformer language. • Model type: I-OPT-175B yasungulwa nge-AdamW ngenxa ye-parameter sizes kusuka ku-125M kuya ku-175B. Sicela ikhadi le-Data Card (I-Appendix C) ukuze uthole ulwazi malunga ne-training data kanye ne-Section 2.2 - 2.5 ukuze uthole ulwazi malunga ne-training process. • Information about training algorithms, parameters, fairness constraints or other applied approaches, and features: Thola inkinobho esilandelayo le-OPT-175B kanye ne-post efanelekayo ku-Meta AI Research Blog. Izinzuzo ezilandelayo zitholakala futhi ku-metaseq, i-open-source repository yethu.12 • Paper or other resource for more information: I-OPT-175B kanye nemodeli ezincinane ze-base-line zithunyelwe ngokusebenzisa isivumelwano se-license ye-non-commercial use esithunyelwe ku-model license yethu.13 • License: • Ukusebenza Sicela uqhagamshelane nombhali efanelekayo {susanz,roller,namangoyal}@fb.com ngezinye imibuzo noma imibuzo. Where to send questions or comments about the model: D.2 Ukusetshenziswa Ukukhishwa kwe-OPT-175B ukuze kusetshenziswe izifundo ze-Language Models, ikakhulukazi ngenxa ye-Responsible AI. Funda kabanzi ku-Section 6 ukuze uthole izibuyekezo ezininzi ze-Release. Ubuchwepheshe mayelana nokusetshenziswa kwe-model zingatholakala ku-metaseq, i-open-source repository yethu. • Primary intended uses: Thola ikakhulukazi abacwaningi nabasebenzi abacwaningi nabasebenzi abacwaningi. • Primary intended users: • Ukusebenza I-OPT-175B ayithunyelwa kokusebenzisa ekukhiqizeni noma ukulethwa kwezwe mhlaba. Njengoba sisaziwe ku-Section 5, i-OPT-175B, njenge-model ye-language enkulu efanayo, inezinhlanganisela eziningana ezingenalutho yokusebenzisa kwebhizinisi. Out-of-scope use cases: D.3 Izici, Izimfuneko, nezimfuneko Ukuqeqeshwa idatha ye-OPT-175B yasungulwa ngokusekelwe ukuhlanganiswa ububanzi kanye nokufinyelela. Vumela i-Data Card yethu (i-Appendix C) ukuze uthole ulwazi oluthile mayelana nezidatha ezisetshenziselwa ukuqeqeshwa imodeli yethu. • Data selection for training: • Ukusebenza Ukubuyekezwa kuleli khasi kusetshenziselwa ukubuyekeza ukusebenza ngokuvumelana nezimodeli ezinama-scale efana ne-literature. Ngokuvumelana nezimonyo e-community malunga ne-safety kanye ne-equity ye-big-language models ngokuvamile, sinikeza ngokuvumelana ngokuvumelana ne-Responsible AI (bheka i-Section 4). Data selection for evaluation: Ngokusho amanye amamodeli amabhizinisi amakhulu angama-diversity (noma inexistence) ye-training data enikeza imiphumela emanzini amamodeli yethu, i-OPT-175B inesibopho emangalisayo kanye ne-safety. I-OPT-175B inokuthi inesibopho emangalisayo emangalisayo emangalisayo emangalisayo kanye ne-hallucination. Ngokuvamile, i-OPT-175B ayikho emangalisayo emangalisayo emangalisayo emangalisayo amamodeli amabhizinisi amabhizinisi amasha. Ngokuvumela isicelo esisodwa, sincommercial, sincoma ukwandisa ukuxhumana, ukuhlaziywa, kanye nokuhlola imibuzo ye-model amabhizinisi am • Limitations: Sicela inqaku 6 ukuze uthole okwengeziwe mayelana nezimo zethu ze-Release, kuhlanganise ukubuyekeza izindlela ezingenalutho zophando ezikhuthazayo ngokuvula imodeli yethu kwamanye amazwe zophando. Thina sincoma ukuthi ukubuyekeza i-OPT-175B, kanye nezifundo mayelana nenqubo yethu yokufundisa amamodeli, iyakwazi ukwandisa isayensi ebonakalayo ezimbini amamodeli amakhulu e-spesific ne-natural language processing kanye ne-deep learning ngokuvamile. • Recommendations for future work: I-E Sample Model Ukusuka Ukuze zonke imiphumela isampula, ucwaningo lokuqala iboniswe ngokunambitheka futhi elinye kuyinto isixazululo. Lezi imiphumela isampula zihlanganiswe ngokuzimela ukuze ukubonisa iziphumo kanye neziphumo zokusebenza kwe-OPT-175B model. Okuzenzakalelayo iyatholakala ku-archiv ngaphansi kwe-license ye-CC by 4.0 Deed (i-Attribution 4.0 International). Okuzenzakalelayo iyatholakala ku-archiv ngaphansi kwe-license ye-CC by 4.0 Deed (i-Attribution 4.0 International).