Ukufinyelela ukusebenza-of-the-art ngokusekelwe ukuqeqeshwa kuphela ku-Data e-Public Available

LLaMA: Open and Efficient Foundation Language Models LLaMA: Izinhlelo zokusungula ezisebenzayo ze-Foundation Language Models Umbhali: I-Hugo Touvron, i-Meta AI Thibaut Lavril, Meta AI Gautier Izacard, Meta AI U-Xavier Martinet, Meta AI U-Marie-Anne Lachaux, u-Meta AI Timothee Lacroix, u-Meta AI Baptiste Rozière, Meta AI Naman Goyal, Meta AI U-Eric Hambro, uMeta AI Faisal Azhar, Meta AI U-Aurelien Rodriguez, u-Meta AI Armand Joulin Edouard Grave, Meta AI Guillaume Lample, Meta AI Authors: I-Hugo Touvron, i-Meta AI Thibaut Lavril, Meta AI Gautier Izacard, Meta AI U-Xavier Martinet, Meta AI U-Marie-Anne Lachaux, u-Meta AI Timothee Lacroix, u-Meta AI Baptiste Rozière, Meta AI Naman Goyal, Meta AI U-Eric Hambro, uMeta AI Faisal Azhar, Meta AI U-Aurelien Rodriguez, u-Meta AI Armand Joulin Edouard Grave, Meta AI Guillaume Lample, Meta AI abstract Thola i-LLaMA, isakhiwo se-fundamental language models ezingu-7B kuya ku-65B parameters. Thola amamodeli zethu ku-trillions of tokens, futhi ibonisa ukuthi kungenzeka ukuqeqesha amamodeli e- state-of-the-art usebenzisa ama-datasets ezivamile, ngaphandle kokufaka ama-datasets e-proprietary ne-inaccessible. Ngokuvamile, i-LLaMA-13B inikeza i-GPT-3 (175B) ku-benchmarks ezininzi, futhi i-LLaMA65B ibhizinisi nomamodeli ezinhle, i-Chinchilla-70B ne-PaLM-540B. Sithumela amamodeli bethu ku-research community1 . 1 Ukulinganisa I-Large Tokuages Models (LLMs) eyenziwe nge-corporate enkulu ye-texts zibonakalisa ikhono lwayo yokusebenza izicelo ezintsha kusuka ku-text or from a few examples (Brown et al., 2020). Lezi izakhiwo ezincinane ziye ziye ziye ziye ziye ziye ziye zihlanganisa amamodeli kuze kube ubukhulu efanele (Kaplan et al., 2020), okwenza isilinganiso yokusebenza esihlanganisa ukuthuthukiswa okuqhubekayo amamodeli (Chowdhery et al., 2022; Rae et al., 2021). Lezi zokusebenza zihlanganisa ukuthi ama-parametres ezininzi zihlanganisa ukusebenza okuhle. Nokho, umsebenzi olutsha we-Hoffmann et al. (2022) ibonisa ukuthi, ngenxa ye Umhlahlandlela lokusebenza le-LlaMA kuyinto ukuqeqeshwa isilinganiso se-lingu model enikezela ukusebenza okungenani kunezimali eziningana, ngokusekelwe ukuqeqeshwa ku-tokens ezingaphezu kuka-usetshenziswa ngokuvamile. Amamodeli eyenziwe, ebizwa ngokuthi i-LLaMA, ihlanganisa ama-parameters angu-7B kuya ku-65B nge-performance ephakeme kunezimali ezingcono. Ngokwesibonelo, i-LLaMA-13B inikeza i-GPT-3 ku-benchmarks amaningi, ngaphandle kokubili okungenani angu-10x. Sinikeza ukuthi le model uziza ukwandisa ukufinyelela kanye nokufunda i-LLMs, njengoba kungenziwa ku-GPU eyodwa. E-up-end-scale, imodeli yethu ye- Kwezinye izinzuzo, ikakhulukazi i-OPT (Zhang et al., 2022), i-GPT-NeoX (Black et al., 2022), i-BLOOM (Scao et al., 2022) ne-GLM (Zeng et al., 2022), kodwa ayikho izinzuzo ezivela ku-PaLM-62B noma i-Chinchilla. Kwi-papiers esilandelayo, sinikeza ukubuyekeza izinguquko etholakalayo ekukhiqizeni isakhiwo se-transformer (Vaswani et al., 2017), kanye nenkqubo yethu yokulungisa. Sishayele ukusebenza kwamamodeli bethu futhi zihlanganisa nezinye i-LLMs ku-set ye-benchmarks ezijwayelekile. Ekugcineni, sinikeza ezinye izinzuzo nezinto ezikhodile ezivela em 2 Ukulungiswa Ukuqeqeshwa kwethu kuyinto efana nezindlela ezijwayelekile ezijwayelekile (Brown et al., 2020; Chowdhery et al., 2022), futhi kusungulwa ngezinsizakalo ze-Chinchilla Scaling (Hoffmann et al., 2022). Thola ama-transformers ezinkulu ku-umthombo emikhulu yedatha ye-text usebenzisa i-optimizer ye-standard. 2.1 I-Pre-Training Data I-Training Dataset yethu iyinhlanganisela ye-sources eziningana, ikakhulukazi ku-Table 1, okuyinto zihlanganisa isixazululo esikhulu se-domains. Ngokuvamile, sinikeza izithombe ze-data ezisetshenziselwa ukuqeqeshwa kwamanye LLMs, ne-limit ye-using kuphela ama-data eyenziwa ngokubanzi, futhi i-open source-compatible. Lokhu kuholele ku-mix ye-data elilandelayo kanye ne-percentage eyenziwa ku-training set: . Thina preprocesses amabili CommonCrawl dumps, ukusuka 2017 kuya 2020, nge CCNet pipeline (Wenzek et al., 2020). Le nqubo deduplicates idatha ngesilinganiso, isebenze isifinyezo lwezilinganiso nge fastText linear classifier ukunciphisa amakhasi non-English futhi ifilimu ubuningi ngekhwalithi ngegama ngama-ngram model. Ngaphezu kwalokho, sinikezele imodeli linear ukuqhathanisa amakhasi asebenzayo njengama-references e-Wikipedia vs.s. amakhasi asebenzayo asebenzayo, futhi amakhasi asebenzayo asebenzayo asebenzayo njengama-references. English CommonCrawl [67%] Ngesikhathi ama-experiments yokuhlola, sinamathele ukuthi ukusetshenziswa kwezinhlobonhlobo ezivamile ze-CommonCrawl zokusebenza kahle. Ngakho-ke sinezihlanganisa i-dataset ye-C4 (i-Raffel et al., 2020) emakhasini yethu. I-preprocessing ye-C4 sinezihlanganisa izinyathelo ze-deduplication kanye ne-language identification: ingxenye eziyinhloko ne-CCNet iyona i-quality filtration, okuyinto ikakhulukazi isisekelo se-heuristics njenge-presence ye-punctuation or the number of words and sentences in a webpage. C4 [15%]. Thina usebenzisa i-GitHub dataset ebonakalayo ku-Google BigQuery. Sinikeza kuphela imicimbi eyenziwe ngaphansi kwe-Apache, i-BSD kanye ne-MIT licenses. Ngaphezu kwalokho, sinikeza amafayela ezincinane ze-heuristics ngokuvumelana ubude be-line noma ububanzi be-alfanumeric characters, futhi ukunciphisa i-boilerplate, njenge-headers, nge-expressions ezivamile. Ekugcineni, sinikeza i-dataset ebonakalayo ku-file level, ngokuvumelana okucacileyo. Github [4.5%]. Thumela i-Wikipedia dumps kusuka ku-June-Agasti 2022, ebandakanya 20 izilimi, ezisebenzisa noma i-Latin noma i-Cyrillic scripts: bg, ca, cs, da, de, en, es, fr, hr, hu, it, nl, pl, pt, ro, ru, sl, sr, sv, uk. Thola idatha ukuthatha ama-hyperlinks, imibuzo kanye nezinye izilinganiso zokufaka. Wikipedia [4.5%]. . Sihlanganisa amabhokisi amabhokisi amabili ku-training dataset yethu: I-Gutenberg Project, ebonakalayo amabhokisi amabhokisi e-public domain, futhi i-Books3 section ye-ThePile (Gao et al., 2020), a dataset ebonakalayo kumabhokisi amabhokisi amabhokisi amabhokisi amabhokisi amabhokisi amabhokisi amabhokisi amabhokisi amabhokisi amabhokisi amabhokisi amabhokisi amabhokisi. Gutenberg and Books3 [4.5%] Ngemuva kwe-Lewkowycz et al. (2022), sinikeza yonke into ngaphambi kokuqala, kanye ne-bibliography. Ngaphezu kwalokho, sinikeza imibuzo evela kumadokhumenti e-tex, kanye nemikhiqizo e-inline-expanded futhi ama-macro ebhaliwe abasebenzisi ukuze ukwandisa ukuxhaswa kwe-papiers. ArXiv [2.5%]. Thina kuhlanganise i-Stack Exchange, i-website ye-quality-high-quality Questions and Answers eyenziwe ngama-domains ahlukahlukene, kusuka ku-computer science kuya ku-chemistry. Sinikeza idatha evela kumasayithi we-28, ukuthatha ama-HTML tags kusuka ku-text kanye nokuhlanganisa imibuzo ngokulinganiselwe (kusuka ku-highest kuya ku-lowest). Stack Exchange [2%]. Thina tokenize idatha nge-bytepair encoding (BPE) algorithm (Sennrich et al., 2015), usebenzisa isicelo kusuka ku-SentencePiece (Kudo and Richardson, 2018). Ngokuvamile, thina usahlukanisa zonke izibalo ezivamile, futhi uxhumane ku-byte ukunciphisa ama- UTF-8 ezaziwa. Tokenizer. Ngokuvamile, wonke idatha yethu yokulungisa iqukethe cishe 1.4T ama-token emva kokuqinisekisa. Kwi-akhawunti yethu yokulungisa, ngamunye i-token isetshenziselwa nje ngexesha lokulungisa, ngaphandle kwe-Wikipedia ne-Books, lapho sinikeza cishe amahora amabili. 2.2 Ubuchwepheshe Ngemva kokuphumelela kokuphumelela kwamadolobha amabili, inethiwekhi yethu isekelwe ku-transformer architecture (Vaswani et al., 2017). Sinikeza izinzuzo ezahlukene ezidlulile ezidlulile, futhi asetshenziselwa amamodeli ezahlukene ezifana ne-PaLM. Nazi ingxenye eziyinhloko lokuphumelela nge-architecture yokuqala, futhi lapho siphindezi lithathwe (ngama-bracket): Ukuze ukuphucula ukuzinza ukuzinza, thina normalize ingxubevange ngamunye transformer sub-layer, ngaphandle normalizing output. Thina usebenzisa umsebenzi normalizing RMSNorm, etholakala ngu Zhang futhi Sennrich (2019). Pre-normalization [GPT3] I-ReLU ye-non-linearity iyahambisana ne-SwiGLU activation function, eyakhiwe ku-Shazeer (2020) ukuze kuthuthukise ukusebenza. Thina usebenzisa ubukhulu we-2 3 4d kunokuba i-4d njengoba ku-PaLM. SwiGLU activation function [PaLM]. Thina ukunciphisa izikhwama zokusebenza okuzenzakalelayo, futhi kuncike izikhwama zokusebenza okuzenzakalelayo okuzenzakalelayo (RoPE), eyenziwe nguSu et al. (2021), ku zonke izikhwama ze-network. Imibuzo ye-hyper-parameters yama-models etholakalayo etholakalayo etholakalayo etholakalayo etholakalayo etholakalayo. Rotary Embeddings [GPTNeo]. 2.3 Ukulungiswa Izidakamizwa zethu zisebenzisa i-AdamW optimizer (Loshchilov and Hutter, 2017), nge-hyper-parameters ezilandelayo: β1 = 0.9, β2 = 0.95. Sinikeza isikhathi sokufundisa isivinini sokufundisa, ukuze isivinini sokufundisa lokugqibela kune-10% yesivinini sokufundisa. Sinikeza ukuchithwa kwe-weight of 0.1 ne-gradient cutting of 1.0. Sinikeza izigaba ze-2,000 zokushisa, futhi sinikeza isivinini sokufundisa kanye ne-batch usayizi nge-model (bheka Table 2 ukuze uthole imininingwane). 2.4 Ukukhiqizwa okuphumelelayo Ukwenza izinzuzo eziningana ukwandisa izinga lokufundisa amamodeli zethu. Okokuqala, sicela usebenzisa isicelo esebenzayo se-causal-multi-head attention ukuze ukunciphise ukusetshenziswa kwe-memory kanye ne-runtime. Lesi isicelo, esitholakala ku-xformers library,2 iboniswe ku-Rabe ne-Staats (2021) futhi isebenzisa i-backward kusuka ku-Dao et al. (2022). Lokhu kufinyelelwa ngokunciphisa i-attention we-weights kanye nokucubungula izinga le-key / query okuyinto zihlanganiswa ngenxa ye-causal nature ye-language modeling task. Ukuze ukwandisa okwengeziwe ukuqeqeshwa, sinciphise inani le-aktivations eyenziwe ngexesha le-backpass nge-checkpointing. Ngaphezu kwalokho, sinciphise i-aktivations eyenziwe ngempumelelo, njenge-outputs ye-layer linear. Lokhu kufinyelelwa ngokuvamile ngokuvumelana ne-backward function ye-transformer layers, ngaphandle kokufikelela ku-PyTorch autograd. Ukuze uthole okwengeziwe le-optimization, kufanele ukunciphisa ukusetshenziswa kwamamodeli we-memory ngokusebenzisa i-model ne-sequence parallelism, njenge-Korthikanti et al. (2022). Ngaphezu kwalokho, sinikeza ukucaciswa kwe-aktivations kanye nokuxhumana phakathi kwe-GPU phezu kwenethiwekhi (ngokusho zokusebenza ze-all_reduction) ngokuvamile. Xa usebenzise imodeli ye-parameter ye-65B, ikhodi lethu kusebenza cishe ama-380 ama-tokens/sec/GPU ku-2048 A100 i-GPU ne-80GB ye-RAM. Lokhu kubalulekile ukuba ukucaciswa kwedatha ethu ebandayo ama-tokens e-1.4T kuthatha cishe izinsuku ezingu-21. 3 Iziphumo eziyinhloko Ngokusho umsebenzi olandelayo (i-Brown et al., 2020), sincoma izivakashi ze-zero-shot ne-poor-shot, futhi inikeza imiphumela ku-20 ama-benchmarks: I-modeli inikeza impendulo usebenzisa generating open-end, noma inikeza impendulo eyenziwe. • Zero-shot. Ukunikezela izibonelo ezithile zokusebenza (ku-1 kuya ku-64) kanye ne-test example. I-model ithatha le text njenge-input futhi ikhiqiza impendulo noma i-ranks amakhono ezahlukene. • Few-shot. Ukubala i-LLaMA nabanye amamodeli zokufundisa, ngokuvamile amamodeli zokufundisa okungagunyaziwe GPT-3 (Brown et al., 2020), Gopher (Rae et al., 2021), Chinchilla (Hoffmann et al., 2022) futhi PaLM (Chowdhery et al., 2022), kanye namodeli ze-OPT ezokufundisa (Zhang et al., 2022), i-GPT-J (Wang and Komatsuzaki, 2021), ne-GPTNeo (Black et al., 2022). Kwi-Section 4, sincompare ngokushesha i-LLaMA ne-instruction-tuned amamodeli ezifana ne-OPT-IML (Iyer et al., 2022) ne-Flan-PaLM (Chung et al., 2022). Sincoma i- Ngama-multiple-choice tasks, izimo zihlanganisa ukuhlaziywa okungenani phakathi kwamathambo ezitholakala, ngokuvumelana nenkqubo ezitholakala. Sihlanganisa ukuhlaziywa okungenani kakhulu ngokuvumelana nenkqubo ezitholakala. Sihlanganisa Gao et al. (2021) futhi usebenzisa ukuhlaziywa okungenani ngokuvumelana nezinombolo zihlanganisa ku-completion, ngaphandle kwezinhlobonhlobo ezithile zebhizinisi (OpenBookQA, BoolQ), okuyinto sihlanganisa Brown et al. (2020), futhi usihlanganisa ukuhlaziywa ngokuvumelana ukuhlaziywa ngokuvumelana ne-completion ezitholakala “Answer:” njenge-context: P(completion fagcontext)/P(complet 4.1 Ukulungiswa okuhlobene Thola ama-benchmarks ezivamile ze-common sense: i-BoolQ (i-Clark et al., 2019), i-PIQA (i-Bisk et al., 2020), i-SIQA (i-Sap et al., 2019), i-HellaSwag (i-Zellers et al., 2019), i-WinoGrande (i-Sakaguchi et al., 2021), i-ARC easy and challenge (i-Clark et al., 2018) kanye ne-OpenBookQA (i-Mihaylov et al., 2018). Lezi zebhizinisi zihlanganisa izicelo ze-Cloze ne-Winograd kanye nokuphendula izicelo eziningana nezidingo. Thola ku-zero-shot setting efanele njengezinhlelo zebhizinisi le-modeling. Kwi-Table 3, sinokuxhumana nezimodeli ezivamile eziningana nezinombolo ezivela ezivela emaphandleni ezivela. Okokuqala, i-LLaMA-65B inikeza i-Chinchilla-70B ku-benchmarks angaphezu kwe-BoolQ. Ngokusho, le model inikeza i-PaLM540B emhlabeni wonke ngaphandle kwe-BoolQ ne-WinoGrande. I-LLaMA-13B model inikeza i-GPT-3 ku-benchmarks amaningi ngaphandle kokubili okungenani angu-10 3.2 Umbuzo-book Closed Ukubala i-LLaMA kumamodeli amakhulu eyenziwe kumadokhumenti amabili ezivela ku-question answering benchmarks: I-Natural Questions (Kwiatkowski et al., 2019) ne-TriviaQA (Joshi et al., 2017). Kwi-benchmarks amabili, sinokuthunyelwe ukusebenza okucindezelayo ekubeni kwebhisi esifundeni, i-i.e., lapho amamodeli akuyona idokhumenti ezinezingeni yokufaka umbuzo. Kwi-Table 4, sinokuthunyelwe ukusebenza kwe-NaturalQuestions, futhi ku-Table 5, sinokuthunyelwa ku-TriviaQA. Kwi-benchmarks amabili, i-LLaMA-65B ibhalisele ukusebenza okuqhubekayo emkhakheni 3.3 Ukubuyekezwa Ukubuyekezwa Thola amamodeli zethu ku-RACE Reading Comprehension Benchmark (Lai et al., 2017). Lesi dataset yasungulwe kusuka ku-English Reading Comprehension Exams ezizenziselwa abafundi be-School and High School Chinese. Thola ukucubungula ukucubungula kusuka ku-Brown et al. (2020) futhi uthumele imiphumela ku-Table 6. Ngezinye ama-benchmarks, i-LLaMA-65B ibhizinisi ne-PaLM-540B, futhi, i-LLaMA-13B ibhizinisi i-GPT-3 eminyakeni eminyakeni eminingi. 3.4 Ukulungiselela Mathematical I-MATH inikeza amamodeli ethandwa kumadokhumenti amabili: i-MATH (Hendrycks et al., 2021) ne-GSM8k (Cobbe et al., 2021). I-MATH inikeza ama-dataset ye-12K izidakamizwa zesidakamizwa kanye nezidakamizwa zesidakamizwa ezibhekwe ku-LaTeX. I-GSM8k inikeza ama-benchmark zezidakamizwa zesidakamizwa. E-Table 7, sinikeza ne-PaLM ne-Minerva (Lewkowycz et al., 2022). I-Minerva inikeza amamodeli e-PaLM eyenziwe ngama-38.5B ama-tokens eyenziwe ku-ArXiv ne-Math Web Pages, lapho noma i-PaLM noma i-L 3.5 Ikhodi Generation I-HumanEval (Chen et al., 2021) kanye ne-MBPP (Austin et al., 2021). Kwi-task, i-model ibonise isifinyezo se-program ku-sentshi amabili, kanye nama-examples e-input-output amancane. Ku-HumanEval, iboniswa isifinyezo se-function, futhi i-prompt iboniswe njenge-code e-natural ne-textual description kanye ne-tests ku-docstring. I-model kuyadingeka ukukhiqiza isofthiwe se-Python elihlanganisa isifinyezo futhi ibhizinisi se-test. Kwi-Table 8, sinqathise izinga le-pass@1 yama-models ethu nama-language amamodels asebenzayo ezingafinyeleleke ku-code Njengoba kubonisa ku-Table 8, ngenxa ye-parameter eqinile, i-LLaMA inikeza amamodeli ezingaphezu kuka-LaMDA ne-PaLM, okuyinto engatholwe noma engatholwe ngokufanelekileyo ku-code. I-LLaMA nge-parameter ye-13B futhi engaphezu kwama-LaMDA 137B kumadoda we-HumanEval ne-MBPP. I-LLaMA 65B inikeza futhi i-PaLM 62B, ngaphandle kokufanelekiswa okwengeziwe. Imiphumela ye-pass@1 ebhalwe kuleli khasi lokushisa angu-0.1 I-pass@100 ne-pass@80 ama-metric ayatholakala nge-temperature 0.8. Thina usebenzisa indlela efanayo ne-Chen et al. (2021) ukufumana ama- Ngokwesibonelo, i-PaLM-Coder (Chowdhery et al., 2022) ukwandisa inkinobho ye-pass@1 ye-PaLM ku-HumanEval kusuka ku-26.2% ku-PaLM kuya ku-36%. Amamodeli amanye abacwaningi ngokuvamile we-coding zisebenza kahle kunezinto ezivamile (Chen et al., 2021; Nijkamp et al., 2022; Fried et al., 2022). 3.6 Ukuphumelela massive multitask isilimi I-Massive Multitask Language Understanding Benchmark, noma MMLU, eyenziwe nguHendrycks et al. (2020) iqukethe imibuzo eziningana nezidingo ezahlukile ezihlanganisa amazwe ezahlukile zokufundisa, kuhlanganise izenzakalo zobuchwepheshe, STEM kanye nezenzakalo zobuchwepheshe. Sinikezela amamodeli ethu ekusebenzisweni kwama-5-shot, ngokusebenzisa izibonelo ezinikezelwe yi-benchmark, kanye neziphumela ezokuthunyelwe ku-Table 9. Kule benchmark, sinikezela ukuthi i-LLaMA-65B ibekwe ngaphansi kwe-Chinchilla70B ne-PaLM-540B ngentengo eminingi ngokuvamile, futhi phakathi kwezinye amazwe. Isizinda es 3.7 Ukuthuthukiswa kokusebenza ngesikhathi sokucwaninga Phakathi nenkqubo yokulungisa, sinikeza ukusebenza kwama-models ethu ngezinyathelo eziningana zokuphendula kanye nezinzuzo ezivamile, futhi zihlanganisa ngezinyathelo ezingu-2. Ngaphezulu kwezinyathelo ezingu-benchmarks, ukusebenza kwandisa ngokushesha, futhi zihlanganisa ne-training perplexity ye-model (bheka isithombe 1). Imiphumela ye-SIQA ne-WinoGrande. Ngokuvamile, ku-SIQA, sinikeza inani elikhulu lokuphendula, okuvumela ukuthi le-benchmark ayinempumelelo. Ku-WinoGrande, ukusebenza akuhlanganisa kanye ne-training perplexity: i-LLaMA-33B ne-LLaMA-65B zihlanganisa ukusebenza efanayo ngesikhathi sokulungisa. 4 Isinyathelo Finetuning Kulesi isigaba, sinikeza ukuthi ukucubungula ngokushesha idatha yezinqubo ngokushesha kuholela ukuthuthukisa MMLU. Nakuba inguqulo non-finetuned ye-LLaMA-65B iyakwazi ukucubungula izinqubo eziyisisekelo, sinikeza ukuthi inani elincane kakhulu ukucubungula ngokushesha ku-MMLU, futhi kuncubungula ngokushesha umthamo we-model ukucubungula izinqubo. Njengoba lokhu akuyona isikhokelo sehlabathi, sinikeza kuphela isizukulwane esisodwa esilandelayo nge-protocol efanayo ne-Chung et al. (2022) ukucubungula imodeli wokubungula, i-LLaMA-I KwiChapha 10, sinikeza imiphumela ye-instruction model LLaMA-I ku-MMLU futhi ukuguqulwa namamodeli amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu amakhulu 5 I-bias, i-toxicity ne-misinformation Izimodeli ezinkulu ze-language ziye ziboniswe ukuthi zibonisa kanye nokukhuthaza i-biases ezivamile ezine ezivamile ezivamile ezivamile (Sheng et al., 2019; Kurita et al., 2019), kanye nokukhiqiza izinto ezimbonini noma ezimbonini (Gehman et al., 2020). Njengoba i-training dataset yethu iqukethe inani elikhulu yedatha e-Web, sincoma ukuthi kubalulekile ukucacisa umthamo amamodeli bethu yokukhiqiza izinto ezivamile. Ukuze uthole ukucaciswa kwe-LLaMA-65B, sincoma ku-benchmarks ezahlukene ezivamile ezivela ekukhiqizeni kwezinto ezimbonini futhi ukucaciswa kwe-stereotypes. Nakuba sincoma ezinye ama-benchmarks ezivamile eziset 5.1 RealToxicityPrompts Imodeli ye-language angakwazi ukukhiqiza isengezo se-toxic, njenge-insults, i-hate speech, noma ama-threats. Kukhona isigaba esikhulu kakhulu se-content e-toxic eyenza yama-model, okwenza ukulawula okuhlobene okuhlobene okuhlobisa. Imisebenzi emininzi (Zhang et al., 2022; Hoffmann et al., 2022) ibonise i-Benchmark ye-RealToxicityPrompts (Gehman et al., 2020) njenge-indicator ye-toxicity ye-model yayo. I-RealToxicityPrompts ikakhulukazi i-100k ama-prompts eyenza ukuthi isampula kuqala; ke isampula se-toxicity iboniswe ngokuvamile ngokuv Thina akufanele ukulawula i-pipeline esetshenziselwa i-PerspectiveAPI ye-third-party, okwenza ukuhlangabezana nezimodeli ezidlulileyo. Ukuze zonke izimpendulo ze-100k, sinikeza ngokunambitheka nge amamodeli zethu, futhi ukulawula isilinganiso yayo yokusebenza. Isilinganiso isilinganiso isilinganiso kusuka ku- 0 (non-toxic) kuya ku-1 (toxic). Kwi-Table 11, sinikeza isilinganiso sethu esilinganisweni esisodwa kanye nesilinganiso sesilinganiso se-RealToxicityPrompts. Lezi izilinganiso zihlanganisa "ukubaluleke" nge-imeyili etholakalayo (isib. 0.087 ye-Chinchilla) 5.2 I-CrowS-Pairs I-CrowSPairs (Nangia et al., 2020). Le dataset inikeza ukucubungula i-biases ku-9 izigaba: umzimba, isonto, isifo / umbala, ukucubungula kwesondo, ubudala, isizwe, ukucubungula, ukucubungula, ukucubungula kwezempilo kanye nesimo se-socioeconomic. Yonke isibonelo iqukethe i-stereotype kanye ne-anti-stereotype, sinikezela i-model preference for the stereotypical sentence ngokusebenzisa i-perplexity of both sentences in a zero-shot setting. Izingcingo ezingaphezu zihlanganisa i-bias ephakeme. Sihlanganisa ne-GPT-3 kanye ne-OPT-175B ku-Table 12. I-LLaMA isihlangan 5.3 WinoGender Ukuze uthole okwengeziwe ukucubungula i-bias ye-model yethu ku-gender category, sincoma i-benchmark ye-WinoGender (Rudinger et al., 2018), i-dataset ye-co-reference resolution. I-WinoGender ifakwe ku-Winograd scheme, futhi i-bias iyatholakala ngokuvamile ngokuvamile ngokuvamile ngokuvamile ngokuvamile ngokuvamile ngokuvamile ngokuvamile ngokuvamile ngokuvamile ngokuvamile ngokuvamile ngokuvamile ngokuvamile ngokuvamile ngokuvamile ngokuvamile. Umthombo we-WinoGender kuyinto "I-Nurser wathumela umdlavuza wama-pediatrist ukuthi umdlavuza wama-pediatrist uyavuma emahoreni.", okuvela yi-"I-His" ibheka. Ngemuva kwalokho, sinikeza ukuguqulwa kwe-perplexity ye-continuations we-infirmer ne-patient ukwenza isixazululo se-co-reference ne-model. Sinikeza ukusebenza lapho usebenzisa ama-pronouns ezintathu: "she/she/she", "his/he/he" ne-"their/them/somebody" (izinketho ezahlukile ezihambisana ne-function ye-grammatic ye-pronoun. E-Table 13, sinikeza izinga le-co-re Ukubonisa ukuthi imodeli yethu iyatholakala kakhulu ekusebenziseni ukuguqulwa kwe-co-reference ye-pronouns ye-"their/them/somebody" kunokuba ku-pronouns ye-"her/her/she" kanye ne-"his/he/he". Ukubuyekezwa okufanayo kwebhizinisi elandelayo (Rae et al., 2021; Hoffmann et al., 2022), futhi kunikeza ukuguqulwa kwe-gender bias. Ngokuvamile, ekusebenzisana ne-pronouns ye-"he/her/she" ne-"his/he/he", imodeli ikwazi ukusebenzisa ubuningi umzimba we-occupation ekusebenziseni ukuguqulwa kwe-co-reference, ngaphandle kokusebenzisa izibonelo ze-phrase. Ukuhlola okwengeziwe le-hypothesis, sincubungula ifomu ye-gotcha ye-pronouns ye-”ey/she/she” ne-”his/he/he” ku-WinoGender dataset. Lezi zimo zihlanganisa izihloko lapho i-pronoun ayihambelana ne-majority gender ye-occupation, futhi i-occupation iyisisombululo enhle. KwiTabela le-13, sincoma ukuthi imodeli yethu, i-LLaMA-65B, inikeza imiphumela emikhulu emibi ye-gotcha, ngokuvamile ibonisa ukuthi inikeza imibuzo yebhizinisi esebenzayo emzimbeni kanye nenkampani. Ukuphumula kwama-performance ikhona kuma-pronouns “ey/she/she” ne-”his/ 5.4 Ukulungiselela I-TruthfulQA (Lin et al., 2021) inikeza ukuhlaziywa kwe-truthfulness ye-model, i-imeyili yayo yokufinyelela lapho isivumelwano se-truthful. Lin et al. (2021) zihlanganisa ukuhlaziywa kwe-"truthful" ngaphansi kwezimo se-"ukwethenjelwa kwe-literal on the real world", futhi akuyona izivumelwano ezingenalutho kuphela emkhakheni yesistimu noma isivumelwano. Lezi zibonelelo zingathuthukisa izimo ze-model ukuze zihlanganisa ulwazi oluthile noma izivumelwano oluthile. Izivumelwano zihlanganiswa ngezindlela eziningi, zihlanganisa izigaba ze-38 futhi zihlanganisa ukuba zihlanganisa. Kuhlolwe ku-Table 14, sinikeza ukusebenza kwama-models ethu ngezinye imibuzo ukuze zibonele ama-models e-truthful kanye nokuhlanganiswa kwe-truthful ne-informative. Ngokuhambisana ne-GPT-3, imodeli lethu ibhizinisi engaphezulu kwama-categories, kodwa izinga lokuphendula okuphendula kunzima, okuvumela ukuthi imodeli lethu iyathembisa imibuzo embalwa 6 Ukulinganiswa kwe-carbon Ukwakhiwa kwama-models etholakalayo inani elikhulu lwezinkimbinkimbi, enesibopho sokushintshwa kwe-carbon dioxide. Sihlola umbhalo olusha ku-subject kanye nokuhlanganisa kanye nokusetshenziswa kwe-energy jikelele kanye nokufinyelela kwe-carbon footprint ku-Table 15. Sihlola isisombululo ye-Wu et al. (2022) ukuhlola i-Watt-hour, i-Wh, ebonakalayo ukwakhiwa kwe-model, kanye ne-ton ye-emissions ye-carbon, i-tCO2eq. Ukuze i-Wh, sisebenzisa isisombululo: Wh = GPU-h × (i-GPU power consumption) × PUE, lapho sisetshenziselwa Ukusetshenziswa kwe-Power Efficiency (PUE) ku-11. I-carbon emission eyenziwe ku-dependent on the location of the data center used to train the network. Ngokwesibonelo, i-BLOOM isebenzisa i-grid enikeza i-0,057 kg CO2eq/KWh okuholela ku-27 tCO2eq futhi i-OPT i-grid enikeza i-0,231 kg CO2eq/KWh, okuholela ku-82 tCO2eq. Kulesi isifundo, sincoma i-cost in carbon emission of training of these models if they were trained in the same data center. Ngakho-ke, siyenza indawo ye-data center, futhi isetshenziselwa, kunoma, i-US average national carbon intensity factor of 0.385 kg CO tCO2eq = MWh × 0.385. Thina usebenzisa isisombululo efanayo ku-OPT kanye ne-BLOOM ukuze kuhlobene kahle. Ukuze i-OPT, sincoma ukuqeqeshwa okwenziwe ngosuku ze-34 ku-992 A100-80B (bheka i-logs yabo4) . Ekugcineni, sincoma ukuthi sisebenzisa i-2048 A100-80GB ngexesha le-approx. 5 eminyakeni ukuthuthukisa amamodeli zethu. Lokhu kubalulekile ukuba ukuthuthukiswa kwezi amamodeli angabiza cishe 2,638 MWh ngaphansi kwama-assumptions yethu, kanye nokushintshwa kwe-1015 tCO2eq. Sincoma ukuthi ukuguqulwa kwezi amamodeli ngeke kusiza ukunciphisa emithi ye-carbon emoyeni kusukela ukuqeqeshwa kwenziwa, futhi ezinye amamodeli amancane futhi 7 Ukusebenza okuqukethwe Izimodeli ze-linguages zihlanganisa izibalo phezu izigaba ze-word, i-token, noma ama-character (Shannon, 1948, 1951). Le umsebenzi, okwakhiwa ngokuvamile njenge-token prediction esilandelayo, lihlanganisa isikhathi eside inkinga esiyinhloko sokusebenza kwe-language yamanzi (Bahl et al., 1983; Brown et al., 1990). Njengoba uTuring (1950) wabheka ukucindezelwa kwamathuluzi ngokusetshenziswa kwe-language ngokusebenzisa i-"imitation game", imodeli ye-languages lithunyelwe njenge-benchmark yokubilisa ukuthuthukiswa ku-intelligent artificial (Mahoney, 1999). Ngokuvamile, amamodeli ye-language basekelwe ku-n-gram counting statistics (Bahl et al., 1983), futhi izindlela ezihlukahlukene zokugqoka ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye ziye Architecture. Kwakubonisa izinzuzo kokusebenzisa imiklamo yobugcisa ku-2 trillion tokens, okwenza 300 billion n-grams, ku-quality of machine translation. Nangona le nqubo yokulinganisa okuzenzakalelayo, ebizwa ngokuthi Stupid Backoff, Heafield et al. (2013) ngevame indlela yokulinganisa kwe-Kneser-Ney ku-Web-scale data. Scaling. Ngokwenza lokhu, i-Chelba et al. (2013) yasungulwa i-benchmark ye-One Billion Word, isetshenziselwa ukuqeqeshwa kwe-scale-based data sethi yokulinganisa ukucubungula ukucubungula imodeli ye-language. Ngokuhambisana nezimodeli ze-neural language, i-Jozefowicz et al. (2016) yatholakala imiphumela enhle ye-benchmark ye-Billion Word ngokuvimbela i-LSTM kuya ku-1 billion parameters. Ngemuva kwalokho, i-scale-based transformers ivumela ukucubungula izicelo eziningi ze-NLP. Imodeli emangalisayo zihlanganisa i-BERT (Devlin et al., 2018), i-GPT-2 (Radford et al., 2019), i-MegatronLM (Shoeybi et al., 2019), kanye ne-T5 (Raffel et al., 2020). Ukuhlukaniswa okuqukethwe kwatholakala nge-GPT-3 (Brown et al., 2020), imodeli we-175 billion amaparamitha. Lokhu kuholela ku-Series ye-Large Language Models, njenge-Jurassic-1 (Lieber et al., 2021), i-Megatron-Turing NLG (Smith et al., 2022), i-Gopher (Rae et al., 2021), i-Chinchilla (Hoffmann et al., 2022), i-PaLM (Chowdhery et al 8 Ukuphakama Kulesi isihloko, sinikeza inqwaba lwezilimi ezivela ngokuphathelene, futhi zihlanganisa nge amamodeli eziphambili. Okokuqala, i-LLaMA-13B ibonise i-GPT-3 nangaphezulu kwe-10x engaphansi, futhi i-LLaMA-65B ibonise ne-Chinchilla-70B ne-PaLM-540B. Ngokungafani nezifundo ezidlulileyo, sinikeza ukuthi kungatholakala ukufinyelela kwama-state-of-the-art ngokufunda kuphela idatha ezivumelanayo, ngaphandle kokufakwa ku-proprietary datasets. Sinikezela ukuthi ukulayisha lezi zobugcisa kumazwe yokufunda izilimi zokusebenza ngokushesha ukuthuthukiswa kwezilimi amakhulu, futhi kusiza izinzuzo Ukubuyiselwa Thola uDaniel Haziza, uFrancisco Massa, uJeremy Reizenstein, uArtem Korenev, uPatrick Labatut kusuka ku-xformers team. Thola uSusan Zhang noStephen Roller ngenxa yokusekela kokusebenza kwe-data deduplication. Thola Luca Wehrstedt, uVegard Mella noPier-Emmanuel Mazaré ngenxa yokusekela kokusebenza kwe-training stability. Thola uShubho Sengupta, uKalyan Saladi, futhi wonke iqembu we-AI infra ngenxa yokusekela kwabo. Thola uJane Yu ngenxa yayo yokusebenza kokusebenza kwe-evaluation. Thola uYongyi Hu ngenxa yayo yokusiza ku-data collection. Ukuhlobisa Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, noCharles Sutton. 2021. I-Synthesis ye-Program nge-model ye-language eningi. Lalit R Bahl, Frederick Jelinek, noRobert L Mercer. 1983. Isisombululo se-maximum ye-probability ye-continuous speech recognition. IEEE transactions on pattern analysis and machine intelligence, amakhasi 179-190. Yoshua Bengio, Réjean Ducharme, futhi Pascal Vincent. 2000. A neural probabilistic language model. Izinzuzo zokusebenza kwezinkqubo zokusebenza ze-information neural, 13. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Ukulungiselelo malunga ne-commonsense yemvelo. Ku-Proceedings of the AAAI conference on artificial intelligence, amakhasi 7432-7439. Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. 2022. Gpt-neox-20b: I-open-source model ye-autoregressive language. arXiv preprint arXiv:2204.06745. U-Thorsten Brants, u-Ashok C. Popat, u-Peng Xu, u-Franz J. Och, no-Jeffrey Dean. 2007. Amamodeli amabhizinisi amakhulu ku-machine translation. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), amakhasi 858-867, Prague, Czech Republic. I-Association for Computational Linguistics. U-Peter F Brown, uJohn Cocke, uStephen A Della Pietra, uVincent J Della Pietra, uFrederick Jelinek, uJohn Lafferty, uRobert L. Mercer, noPaul S. Roossin. 1990. Ukulungiswa okuhlolwa kwekhompyutha. I-Linguistics ye-Computational, 16(2):79-85. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, futhi Dario Amodei 2020. Imodeli ye-Language akuyona ezingenalutho. UChristian Buck, uKenneth Heafield, noBas Van Ooyen. 2014. Ukubalwa kwe-N-gram kanye nemodeli ye-language kusuka ku-common crawl. Ku-LREC, umthamo we-2, iphepha le-4. U-Ciprian Chelba, u-Tomas Mikolov, u-Mike Schuster, u-Qi Ge, u-Thorsten Brants, u-Phillipp Koehn, u-Tony Robinson. 2013. I-billion word benchmark yokubuyisa ukuthuthukiswa kwe-statistical language modeling. arXiv preprint arXiv:1312.3005. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, William Hebgen Guss, Michael Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Sutarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantz, Elizabeth Barnes, Ariel, Ariel Herbert-Voss, William Hebgen Guss, Alex Nicholino, Alex Paino, Nikolas Tezak, Jie, Igor Tanglec Aakanksha Diawon Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Ghemawat, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sun Devipa, Henryk Michalewski, Xavier Garcia, Vedantra Mis, Kevin Robinson, Liam Fedus, Jason Zhou, Den Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Petrov, Slav, Ed Huai hsin Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc Le, futhi Jason Wei. 2022. Ukuphakamisa imidwebo imibhalo isifundo. U-Christopher Clark, uKenton Lee, uMing-Wei Chang, uTom Kwiatkowski, uMichael Collins, uKristina Toutanova. 2019. I-Boolq: Ukuhlola ukucindezeleka okucindezeleka kwe-yes/no questions. arXiv preprint arXiv:1905.10044. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, no-Oyvind Tafjord. 2018. Uyakwazi ukuyisombulula umbuzo ngokuphendula? try arc, i-ai2 ukubuyekeza. arXiv preprint arXiv:1803.05457. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Ukwakhiwa kwamakhemikhali ukugcina imibuzo emathemikhali. arXiv preprint arXiv:2110.14168. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, noRuslan Salakhutdinov. 2019. I-Transformer-xl: Imodeli ye-language emangalisayo ngaphandle kwe-fixed-length context. arXiv preprint arXiv:1901.02860 Tri Dao, Daniel Y Fu, Stefano Ermon, Atri Rudra, no-Christopher Ré. 2022. Flashattention: Ukuhlobisa ngokushesha futhi ukucaciswa kwe-memory ngokushesha nge-io-awareness. arXiv preprint arXiv:2205.14135. Jacob Devlin, Ming-Wei Chang, Kenton Lee, futhi Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. UJeffrey L. Elman. 1990. Ukukhangela isakhiwo isikhathi. I-Cognitive Science, 14(2):179-211 Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wentau Yih, Luke Zettlemoyer, noMike Lewis. 2022. Incoder: A model generative for code filling and synthesis. arXiv preprint arXiv:2204.05999. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, futhi Connor Leahy. 2020. I-The Pile: I-800gb dataset ye-text eyenziwe ngegama. arXiv preprint arXiv:2101.00027. Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, futhi Andy Zou. 2021. A framework for a few-shot language model assessment. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, noah A Smith. 2020. Realtoxicityprompts: Ukubuyekezwa kwe-neural toxic degeneration emadolobheni. arXiv preprint arXiv:2009.11462. U-Alex Graves. 2013. Ukusebenza kwe-sequence nge-network ye-neural recurrent. arXiv preprint arXiv:1308.0850. U-Kenneth Heafield, uIvan Pouzyrevsky, uJonathan H Clark, no-Philippe Koehn. 2013. Ukubuyekezwa kwe-modified model ye-kneserney language. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (I-Volume 2: I-Short Papers), amakhasi 690-696. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, noJacob Steinhardt. 2020. Ukubuyekezwa kokuphumelela kwe-massive multitask language. arXiv preprint arXiv:2009.03300. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, noJacob Steinhardt. 2021. Ukuphendula imibuzo emathemikhali nge dataset emathemikhali. arXiv preprint arXiv:2103.03874. Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Patwary, Mostofa Ali, Yang Yang, futhi Yanqi Zhou. 2017. Ukulungiselela ukufundisa kwangaphambili, ngempumelelo. arXiv preprint arXiv:1712.00409. Sepp Hochreiter noJürgen Schmidhuber. 1997. Ukukhanyisa isikhathi eside. Ukukhanyisa kwe-neural, 9(8):1735-1780. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, futhi Laurent Sifre. 2022. Ukuqeqesha amamodeli amabhizinisi amakhulu. Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Dániel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, et al. 2022. Opt-iml: Ukuphakamisa isampula isifundo meta ukufundisa nge-lens ye-generalization. arXiv preprint arXiv:2212.12017. Mandar Joshi, Eunsol Choi, Daniel S Weld, noLuke Zettlemoyer. 2017. I-Triviaqa: I-Data Set ye-Remote Surveillance ye-Reading Comprehension. arXiv preprint arXiv:1705.03551. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, noYonghui Wu. 2016. Ukuhlola izinzuzo ze-modeling ye-language. arXiv preprint arXiv:1602.02410. UJared Kaplan, uSam McCandlish, uTom Henighan, uTom B Brown, uBenjamin Chess, uRewon Child, uScott Gray, uAlec Radford, uJeffrey Wu, uDario Amodei. 2020. Ukuhlobisa imilinganiselo ye-neural language models. arXiv preprint arXiv:2001.08361. I-Slava Katz. 1987. Ukubuyekezwa kwezimo ezivamile ezivamile ze-Language Model Component ye-speech recognizer. IEEE Transactions on Acoustics, Speech, and Signal Processing, 35(3):400–401. U-Reinhard Kneser no-Hermann Ney. 1995. Ukuphuculwa kwe-back-off ye-m-gram modeling. Ngo-1995 i-International Conference on Acoustics, Speech, and Signal Processing, Volume 1, amakhasi 181-184. IEEE. Vijay Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, futhi Bryan Catanzaro. 2022. Ukunciphisa ukuguqulwa kwe-activation emadolobheni emikhulu ye-transformer. arXiv preprint arXiv:2205.05198. U-Taku Kudo noJohn Richardson. 2018. I-Sentencepiece: I-subword tokenizer kanye ne-detokenizer efanelekayo ne-language yokusebenza kwe-neural text. arXiv preprint arXiv:1808.06226. Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, no-Yulia Tsvetkov. 2019. Ukubalwa kwezimpendulo zokuxhumana ku-contextual word representations. Ku-1st ACL Workshop on Gender Bias for Natural Language Processing. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Imibuzo yemvelo: isibuyekezo se-benchmark ye-question answering research. Ukuhweba ka-Association for Computational Linguistics, 7:453-466 Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, no-Eduard Hovy. 2017. Isikhathi: Isikhathi esikhulu sokufundisa ukucacisa idatha kusuka ku-exams. arXiv preprint arXiv:1704.04683. Aitor Lewkowycz, Anders Johan Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Venkatesh Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, futhi Vedant Misra. 2022. Ukuphendula imibuzo yokuxhumana kwamanani nge amamodeli yebhizinisi. In Advances in Neural Information Processing Systems. Opher Lieber, noma Sharir, Barak Lenz, futhi Yoav Shoham. 2021. I-Jurassic-1: Imibuzo yobuchwepheshe nokuhlolwa. I-White Paper. I-AI21 Labs, 1. U-Stephanie Lin, uJacob Hilton, no-Owain Evans. 2021. I-Truthfulqa: Ukubuyekeza indlela amamodeli amamodeli amabili. arXiv preprint arXiv:2109.07958. Ilya Loshchilov noFrank Hutter. 2017. Ukuhlobisa ukuchithwa kwe-weight decay. arXiv preprint arXiv:1711.05101. U-Matthew V Mahoney. 1999. I-text compression njenge-test ye-artificial intelligence. I-AAI/IAAI, 970. Todor Mihaylov, Peter Clark, Tushar Khot, no-Ashish Sabharwal. 2018. Ngaba i-armor yenza umugqa? isixazululo se-dataset esizayo se-open book. arXiv preprint arXiv:1809.02789. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernocky, futhi Sanjeev Khudanpur. 2010. I-Recur- ` isisekelo se-network ye-neural model. Ku-Interspeech, amakhasi 1045-1048. Makuhari. Nikita Nangia, Clara Vania, Rasika Bhalerao, noSamuel R. Bowman. 2020. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In EMNLP 2020. U-Erik Nijkamp, uBo Pang, uHiroaki Hayashi, uLifu Tu, uHuan Wang, uYingbo Zhou, uSilvio Savarese, no-Caiming Xiong. 2022. I-Codegen: Imodeli esikhulu esifundeni sokukhipha ikhodi nge-synthesis ye-multi-turn program. arXiv preprint arXiv:2203.13474. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, noRyan Lowe. 2022. Izilimi zokufundisa izilimi ukuhlola izicelo nge umkhondo womuntu. In Advances in Neural Information Processing Systems. U-Markus N Rabe noCharles Staats. 2021. I-Selfattention ayidinga i-o(n 2) memory. arXiv preprint arXiv:2112.05682. U-Alec Radford, uKarthik Narasimhan, uTim Salimans, uIlya Sutskever, et al. 2018. Ukuphucula ukubuyekezwa kwe-language nge-generative pre-training. U-Alec Radford, uJeffrey Wu, uRewon Child, uDavid Luan, uDario Amodei, uIlya Sutskever, et al. 2019 Imodeli ye-Language akuyona abalandeli abalandeli abalandeli. I-OpenAI blog, 1(8):9. Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, Po-Sen Huang, Amelia Glaese, Amelia Budden, Esme Welblland, Simonyan, Michela Paganini, Laurent Pagan, Laurent Sifre, Laura Uesato, Lena Martens, Xiang Lorraine, Irina Higgins, Antonia Oriwell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, no. Peter J Liu. 2020. Ukuhlola izinzuzo zokuxhumana nge-text-to-text transformer eyodwa. I-Journal of Machine Learning Research, 21(1):5485-5551. Jonathan S Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, noNir Shavit. 2019. Ukubuyekezwa okuhle kwama-generalization ngezinga. arXiv preprint arXiv:1909.12673. U-Rachel Rudinger, uJason Naradowsky, uBrian Leonard, noBenjamin Van Durme. 2018. Isizukulwane seGender ku-coreference resolution. Ku-NAACL-HLT 2018. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, noYejin Choi. 2021. I-Winogrande: Ukuphendula kwesakhiwo se-vinograde emikhulu. Ukuxhumana kwe-ACM, 64(9):99-106. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, noYejin Choi. 2019. Socialiqa: Ukucaciswa kwe-Commonsense mayelana nokuxhumana kwezemvelo. arXiv preprint arXiv:1904.09728. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilic, Daniel Hesslow, Ro-' man Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176bparameter open-access multilingual model. arXiv preprint arXiv:2211.05100. U-Rico Sennrich, uBarry Haddow, no-Alexandra Birch. 2015. Ukuguqulwa kwe-Neural Machine ye-Word Rare nge-Units ye-Subword. arXiv preprint arXiv:1508.07909. Claude E. Shannon. 1948. A Mathematical Theory of Communication. I-Bell System Technical Journal, 27(3):379-423. Claude E. Shannon. 1951. Ukubuyekeza kanye no-entropy ye-imprimed English. I-Bell System Technical Journal, 30(1):50-64. Noam Shazeer. 2020. Imibuzo ye-Glu iyahambisana ne-transformer. arXiv preprint arXiv:2002.05202. Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, futhi Nanyun Peng. 2019. Umbhali wasebenza njenge-baby-sitter: On Izinzuzo ku-language generation. arXiv preprint arXiv:1909.01326. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, noBryan Catanzaro. 2019. Megatron-lm: Ukuqeqesha ama-billion-parameter language models usebenzisa model parallelism. arXiv preprint arXiv:1909.08053. Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zhang, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, noBryan Catanzaro. 2022. Ukusebenzisa isivinini futhi megatron ukuqeqesha megatron-turing nlg 530b, imodeli isilwane esikhulu generative. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, no Yunfeng Liu. 2021. I-Roformer: I-transformer eyenziwe nge-rotor position embedding. arXiv preprint arXiv:2104.09864. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Me-Hellstern, Merith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, U-A. M. Turing. I-1950 Computing Machinery and Intelligence. [I-Oxford University Press, I-Mind Association]. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30, pages 5998–6008. Ben Wang futhi Aran Komatsuzaki. 2021. GPT-J6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/ mesh-transformer-jax. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, futhi Denny Zhou. 2022. I-self-consistency ivumela ukucindezeleka kwe-kedge ye-think ku-model ye-language. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. Izinzuzo ezivela zokusebenza kwezilimi ezinkulu. arXiv preprint arXiv:2206.07682. Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, futhi Edouard Grave. 2020. CCNet: Ukukhishwa kwekhwalithi ephezulu monolingual datasets kusuka web crawl idatha. In Language Resources and Evaluation Conference. Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga, Jinshi Huang, Charles Bai, et al. 2022. Sustainable ai: Imiphumela yemvelo, izinzuzo kanye nezimfuneko. I-Proceedings of Machine Learning and Systems, 4:795-813 Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, noYejin Choi. 2019. Hellaswag: Ingaba umatshini ukugcina isihloko sakho? arXiv preprint arXiv:1905.07830. Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Peng Zhang, Yuxiao Dong, futhi Jie Tang. 2022. Glm130b: A open bilingual pre-trained model. I-Biao Zhang noRico Sennrich. 2019. I-root average square layer normalization. I-Advances in Neural Information Processing Systems, 32. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Ukukhishwa kwe-transformer izilimi amamodeli. arXiv preprint arXiv:2205.01068. Umbuzo wokuphendula I-LLaMA ye-Natural Questions ne-TriviaQA. I-Natural Questions isetshenziselwa i-test split esetshenziselwa i-open-domain question answering containing 3610 questions. I-TriviaQA isetshenziselwa ku-dev set ye-filtered set. Lokhu kubangelwa ku-GPT-3 ne-PaLM, okuyinto isetshenziselwa ku-test set ye-unfiltered set lapho i-server ye-evaluation ye-online ayizokufinywe.5 . Sinikeza imibuzo ngokusebenzisa i-decoding emangalisayo, futhi sinikeza imibuzo kusuka ku-generation ngokushesha ku-line break, dot noma comma yokuqala. Imibuzo eyenziwe isetshenziswe nge-standard exact match metric: inguqulo I-B MMLU C Generations kusukela LLaMA-65B Kule isigaba, sinikeza ezinye izibonelo zezinhlayiyana ezitholakala nge-LLaMA-65B (hhayi isinyathelo se-finetuning). I-prompts iyatholakala emangalisayo. D Generations kusukela LLaMA-I Ukubonisa izibonelo ezimbalwa ze-generations nge-LLaMA-I, i-LLaMA-65B eyenziwe nge-protocol kanye ne-instruction dataset kusuka ku-Chung et al. (2022). Okuzenzakalelayo iyatholakala ku-archiv ngaphansi kwe-license ye-CC by 4.0 Deed (i-Attribution 4.0 International). Okuzenzakalelayo iyatholakala ku-archiv ngaphansi kwe-license ye-CC by 4.0 Deed (i-Attribution 4.0 International).