Waayo, waxa uu ka soo saarka in la soo saarka, waxaa lagu soo saarka in ay ka soo saarka in ay ka soo saarka in ay ka soo saarka.

LLaMA: Open and Efficient Foundation Language Models LLaMA: Waayo, Waayo, Waayo, Waayo, Waayo Haku: HUGO Touvron, Meta AI Thibaut Lavril, Meta AI Gautier Izacard, Meta AI Xavier Martinet, Meta AI Marka Marie-Anne Lachaux, Meta AI Timothee Lacroix iyo Meta AI Baptiste Rozière, Meta AI Naman Goyal, Meta AI Marka: Eric Hambro, Meta AI Faisal Azhar, Meta AI Aurelien Rodriguez, Meta AI Armand Joulin Edouard Grave oo ku saabsan Meta AI Guillaume Lample, Meta AI Authors: HUGO Touvron, Meta AI Thibaut Lavril, Meta AI Gautier Izacard, Meta AI Xavier Martinet, Meta AI Marka Marie-Anne Lachaux, Meta AI Timothee Lacroix iyo Meta AI Baptiste Rozière, Meta AI Naman Goyal, Meta AI Marka: Eric Hambro, Meta AI Faisal Azhar, Meta AI Aurelien Rodriguez, Meta AI Armand Joulin Edouard Grave oo ku saabsan Meta AI Guillaume Lample, Meta AI Haku Waxaan soo bandhigay LLaMA, a dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha 1 Qalabka Dhismaha Big Tokuages (LLMs) oo ku yaalaa in ay ku yaalaa in ay ku yaalaa in ay ku yaalaa in ay ku yaalaa wax soo saarka cusub oo ku yaalaa wax soo saarka (Brown et al., 2020). Dhismaha ugu horeysay ee loo yaabaa in ay ku yaalaa in ay ku yaalaa modelka ka mid ah (Kaplan et al., 2020), oo loo yaabaa in ay ku yaalaa wax soo saarka ugu weyn ee modelka ah (Chowdhery et al., 2022; Rae et al., 2021). Dhismaha ugu horeysay ee loo yaabaa in ay ku yaalaa in ay ka mid ah macluumaadka badan oo ku yaabaa in ay ku yaalaa wax soo saarka badan. Sidaa, wax soo saarka ugu fiican ee Hoffmann et al. (2022) wuxuu ku yaalaa in ay ku yaabaa in ay u Qalabka loo yaqaan LLaMA waxaa loo yaabaa in ka mid ah nidaamka xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaaraha xawaar Waxaa jira caadiga ah, oo ka mid ah OPT (Zhang et al., 2022), GPT-NeoX (Black et al., 2022), BLOOM (Scao et al., 2022) iyo GLM (Zeng et al., 2022), laakiin la mid ah waxaa laga yaabaa sida PaLM-62B ama Chinchilla. In caadiga ah ee caadiga ah, waxaan soo bandhigay si ay u qiyaastii karo oo loo soo bandhigiisa transformer (Vaswani et al., 2017), sida ka mid ka mid ah caadiga ah. Waxaan ka dibna soo bandhigiisa modelka iyo soo bandhigi karaa LLMs kale oo ka mid ah macluumaadka caadiga ah. 2 Qalabka Shuruudaha our waxaa ka mid ah ka mid ah ka mid ah ka mid ah wax soo saarka ugu horeysay (Brown et al., 2020; Chowdhery et al., 2022), iyo waxaa ka mid ah wax soo saarka cinchilla (Hoffmann et al., 2022). 2.1 Qalabka Qalabka Our data set is a mixture of several sources, in Table 1, which cover a diversified set of domains. For most part, we reuse data sources that have been leveraged to train other LLMs, with the restriction of using only data that is publicly available, and compatible with open source. This leads to the following mixture of data and the percentage they represent in the training set: Waxaan soo saarka 5 dumps CommonCrawl, oo ku yaalaa 2017-2020, la CCNet pipeline (Wenzek et al., 2020). Xogta this deduplicates data at line level, soo saarka xawaaraha xawaaraha xawaaraha soo saarka soo saarka ee fastText linear in ay ka soo saarka pages non-English iyo filters content low quality with a ngram language model. In ka mid ah, waxaan ka soo saarka model linear in ay soo saarka pages used as references in Wikipedia v.s. randomly sampled pages, iyo pages rejected not classified as references. English CommonCrawl [67%] Marka aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay. C4 [15%]. Waxaan loo isticmaali karaa dhismaha GitHub ah oo ka mid ah Google BigQuery. Waxaan ka mid ah loo isticmaali karaa dhismaha oo ka mid ah dhismaha Apache, BSD iyo MIT. Waxaa kale oo ka mid ah, waxaan ka mid ah ka mid ah dhismaha dhismaha ah oo ka mid ah dhismaha dhismaha ah oo ka mid ah dhismaha dhismaha alfanumeric, iyo ka mid ah dhismaha dhismaha dhismaha, sida headers, oo ka mid ah dhismaha caadiga ah. Github [4.5%]. Waayo, waxaa loo yaqaan Wikipedia dumps ee ka mid ah korontada June-August 2022, oo ku yaalaa 20 xafiisyada, oo loo isticmaali karaa xafiisyada Latin ama Cyrillic: bg, ca, cs, da, de, en, es, fr, hr, hu, it, nl, pl, pt, ro, ru, sl, sr, sv, uk. Wikipedia [4.5%]. Waxaan ka mid ah 2 corpora bogga ah ee dhismaha warshadaha ah: Project Gutenberg, oo ku yaalaa bogga ah oo ka mid ah kharashka Books3 ee ThePile (Gao et al., 2020), kharashka dhismaha ka mid ah dhismaha warshadaha ugu badan ee warshadaha. Gutenberg and Books3 [4.5%] Waayo, sidoo kale loo isticmaalaa arXiv Latex files si ay u qaado data wax soo saarka ah in our dataset. Ka dib markii Lewkowycz et al. (2022), waxaan ka heli karaa oo dhan oo ka hor, iyo bibliography. Waayo, waxaan sidoo kale ka qaado comments ee .tex files, iyo definitions inline-expanded iyo macros ka soo saarka isticmaalka si ay u soo saarka ugu badan. ArXiv [2.5%]. Waxaan ka mid ah go'aanka ee Stack Exchange, a website of high quality questions and answers that covers a diversified set of domains, ka mid ah computer science to chemistry. We kept the data from the 28 largest websites, removed the HTML tags from text and sorted the answers by score (from highest to lowest). Stack Exchange [2%]. Waayo, sidoo kale waxaa laga yaabaa in ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ka mid ah. Tokenizer. Shuruudaha dhismaha ugu badan oo ka mid ah shuruudaha ugu badan oo ka mid ah shuruudaha ugu badan oo ka mid ah shuruudaha ugu badan oo ka mid ah shuruudaha ugu badan oo ka mid ah shuruudaha ugu badan oo ka mid ah shuruudaha ugu badan oo ka mid ah shuruudaha ugu badan oo ka mid ah shuruudaha ugu badan oo ka mid ah shuruudaha ugu badan oo ka mid ah shuruudaha ugu badan oo ka mid ah shuruudaha ugu badan oo ka mid ah shuruudaha ugu badan oo ka mid ah shuruudaha ugu weyn ee shuruudaha. 2.2 Qalabka Sida loo isticmaali karaa mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid Waayo, waxaa loo isticmaali karaa in lagu isticmaali karaa RMSNorm normalizing function, ku yaalaa by Zhang and Sennrich (2019). Pre-normalization [GPT3] Waayo, waxaan ka heli karaa ReLU non-linearity by SwiGLU activation function, ku yaalaa by Shazeer (2020) si ay u shaqeeyaan. We use a dimension of 2 3 4d instead of 4d as in PaLM. SwiGLU activation function [PaLM]. Waayo, sidoo kale waxaa laga yaabaa in ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah. Rotary Embeddings [GPTNeo]. 2.3 Qalabka optimizer Waayo, waxaa loo isticmaali karaa wax soo saarka ah oo loo isticmaali karaa wax soo saarka ah oo loo isticmaali karaa wax soo saarka ah oo loo isticmaali karaa. 2.4 Qalabka adeegga Waayo, waxaan isticmaali karaa wax soo saarka ah ee wax soo saarka ah ee wax soo saarka iyo wax soo saarka ah ee wax soo saarka iyo wax soo saarka ah ee wax soo saarka. Waayo, sidoo kale waxaa laga yaabaa in ka mid ah wax soo saarka ah oo ka mid ah wax soo saarka ah oo ka mid ah wax soo saarka ah oo ka mid ah wax soo saarka iyo wax soo saarka. Waxaan sidoo kale soo bandhigi kartaa in ay ku yaalso codsado oo ka mid ah codsado oo ka mid ah codsato oo ka mid ah codsato oo ka mid ah codsato oo ka mid ah codsato oo ka mid ah codsato oo ka mid ah codsato oo ka mid ah codsato oo ka mid ah codsato oo ka mid ah codsato oo ka mid ah codsato oo ka mid ah codsato oo ka mid ah codsato oo ka mid ah codsato oo ka mid ah codsato oo ka mid ah codsato oo ka mid ah codsato oo ka mid ah codsato. 3 Results ugu weyn Sida loo yaabaa (Brown et al., 2020), waxaan ka mid ah u hesho zero-shoot iyo few-shoot-taxaha, oo ay ku dhigi karaa wax soo saarka oo ka mid ah 20 benchmarks: Markaad ka mid ah wax soo saarka, waxaa laga yaabaa in ay ka mid ah wax soo saarka, oo ka mid ah wax soo saarka iyo wax soo saarka. • Zero-shot. Markaad ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah. • Few-shot. Waxaan soo bandhigay LLaMA la mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid Marka aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan tahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u 3.1 Qalabka dhismaha Waayo, sidoo kale waxaa laga yaabaa in ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid In tababarka 3, waxaan soo bandhigay la mid ah macluumaad kala duwan ee macluumaad kala duwan iyo macluumaad kala duwan ee macluumaadka caadiga ah. First, LLaMA-65B waxay ka mid ah chinchilla-70B oo ka mid ah macluumaad ka mid ah macluumaad ka mid ah oo ka mid ah BoolQ. Sidaas, macluumaad this waa mid ka mid ah PaLM540B oo ka mid ah oo ka mid ah BoolQ iyo WinoGrande. model LLaMA-13B waxaa ka mid ah macluumaad ka mid ah GPT-3 oo ka mid ah macluumaad ka mid ah 10x ka mid ah 3.2 Qalabka Qalabka Qalabka Waxaan soo bandhigay LLaMA iyo modelka kala duwan oo ku yaalaa in ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ah mid ka mid ah mid ah mid ah mid ka mid ah mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid 3.3 Qalabka Qalabka Waayo, sidoo kale, waxaa loo yaabaa in ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah oo mid ka mid ka mid ah, oo ka mid ah mid ka mid ah oo mid ka mid ah. 4.3 Qalabka Mathematics Waayo, waxaa laga yaabaa in ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah oo mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah. 5.3 Qalabka Code Waayo, waxaa laga yaabaa in ay ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka Sida loo yaabaa tababarka 8, in la soo bandhigay ka mid ah macluumaadka, LLaMA waxaa ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid Markaasadda, PaLM-Coder (Chowdhery et al., 2022) waxay kor u qiyaasta pass@1 ee PaLM on HumanEval ka 26.2% ee PaLM ilaa 36%. Modeli kale oo ku saabsan codka waxaa laga yaabaa si ay u qiyaasta badan oo ay ku saabsan codka (Chen et al., 2021; Nijkamp et al., 2022; Fried et al., 2022). 3.6 Qalabka multitask maskaxda Waayo, waxaa laga yaabaa in ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid 3.7 Qalabka wax soo saarka ah ee wax soo saarka Waayo, sidoo kale waxaa laga yaabaa in ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid 4 Qalabka Qalabka Markaas oo ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah In Tababarka 10, waxaan soo bandhigi karaa wax soo saarka LLaMA-I on MMLU iyo soo bandhigi karaa macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad macluumaad mac 5 Qalabka, Toxicity iyo Misinformation Markaad ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid 5.1 Qiimeeyaasha Qalabka waxaa laga yaabaa in ay ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah Waxaad ka mid ah wax soo saarka ah oo loo yaqaan 'Prospect API' oo ka mid ah 'Prospect API' iyo 'Prospect API' oo ka mid ah 'Prospect API' oo ka mid ah 'Prospect API' oo ka mid ah 'Prospect API' iyo 'Prospect API' oo ka mid ah 'Prospect API' oo ka mid ah 'Prospect API' oo ka mid ah 'Prospect API' iyo 'Prospect API'. 5.2 Qalabka Waxa uu ku yaalaa bixiyaan ee modelka ee CrowSPairs (Nangia et al., 2020). Dhismaha this waa in ay ku yaalaa bixiyaan ee 9 jilicsan: jilicsan, jilicsan, jilicsan, jilicsan, jilicsan, jilicsan, jilicsan, jilicsan, jilicsan, jilicsan, jilicsan, jilicsan, jilicsan, jilicsan, jilicsan, jilicsan, jilicsan, jilicsan, jilicsan, jilicsan, jilicsan, jilicsan, jilicsan, jilicsan, jilicsan, jilicsan, jilicsan, jilicsan, jilicsan, jilicsan, jil 5.3 Qalabka WinoGender waa soo saarka Winograd, iyo bixiya waxay ku yaalaa sida loo yaabaa in ay soo saarka soo saarka co-reference waxay ku yaalaa soo saarka pronoun (Rudinger et al., 2018), a data set of co-reference resolution. WinoGender waa la soo saarka Winograd, iyo bixiya waxay ku yaalaa sida loo yaabaa in ay soo saarka co-reference ay ku yaalaa jilicsan jilicsan pronoun. Haddii loo yaabaa, cadaadiga ah waxay ku yaalaa in ay u soo saarka "occupation", a "participant", iyo a "pronoun" oo pronoun waa co-referencing ama jilicsan. Sida loo yaqaan "WinoGender" waa "Nurses notify the patient that his shift would be ending in an hour.", oo waxaa lagu soo bandhigay "His" waa la soo bandhigay. We then compare the perplexity of the continuations the nurse and the patient to perform co-reference resolution with the model. We evaluate the performance when using 3 pronouns: "she/she/she", "his/he/he" and "their/them/someone" (ga soo bandhigay ka mid ah wax soo saarka grammatical of the pronoun. Waayo, model-ka waa in ay ka mid ah in ay ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ah mid Marka aad u baahan tahay in ay u baahan tahay in ay u baahan yahay in ay u baahan yahay in ay u baahan yahay in ay u baahan yahay in ay u baahan yahay in ay u baahan yahay in ay u baahan yahay in ay u baahan yahay in ay u baahan yahay in ay u baahan yahay in ay u baahan yahay in ay u baahan yahay in ay u baahan yahay in ay u baahan yahay in ay u baahan yahay in ay u baahan yahay in ay u baahan yahay in ay u baahan yahay in ay u baahan yahay in ay u baahan yahay in ay u baahan yahay in ay u baahan yahay in ay u baahan yahay in ay u baahan yahay in ay u baahan yahay in ay u baahan yahay in ay u baahan yahay in ay u baahan yahay in ay u baahan yahay in ay u baahan yahay in ay u baahan yahay in ay u baahan yahay in ay u baahan yahay in ay u baahan yahay in ay u baahan yahay. 5.4 Qalabka TruthfulQA (Lin et al., 2021) waxaa ka mid ah in ay ku yaalaa in ay dhismaha modelka, i.e., in ay ku yaalaa in ay ku yaalaa in ay dhismaha ah waa in ay dhismaha. Lin et al. (2021) waa in ay ku yaalaa in ay dhismaha "dhammaan" in la mid ah "dhammaan dhismaha ah oo ku yaalaa in ay dhismaha" iyo in ay ku yaalaa in ay dhismaha mid ka mid ah nidaamka dhismaha ama dhismaha dhismaha. In Tababarka 14, waxaan soo bandhigi doonaa wax soo saarka modelka ah oo ka mid ah wax soo saarka modelka adeegga iyo wax soo saarka of adeegga iyo wax soo saarka. Haddii loo soo saarka GPT-3, modelka our waa ugu badan oo ka mid ah wax soo saarka, laakiin wax soo saarka of adeegga ah waa mid ka mid ah wax soo saarka our waa mid ka mid ah wax soo saarka haluxin ah. 6 Qalabka Carbon Shuruudaha qalabka iyo qalabka qalabka iyo qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka qalabka Wh = GPU-h × (wax yar) x PUE, Sida loo yaabaa, BLOOM waxay isticmaali karaa grid oo ku yaalaa 0.057 kg CO2eq/KWh oo loo yaabaa 27 tCO2eq iyo OPT waa grid oo ku yaalaa 0.231 kg CO2eq/KWh oo ku yaalaa 82 tCO2eq. TCO2eq = MWh x 0.385. Waayo, sidoo kale waxaa laga yaabaa in ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid 7 Qiimeeyaasha Sida loo yaabaa in ay ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid Markaas ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid Architecture. Waxaa jira warshad ka badan oo ka badan oo ka mid ah modelka iyo dhismaha dhismaha. Brants et al. (2007) ayaa soo bandhigay faahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaahfaah Scaling. Waxaad u baahan tahay in ay u helo model 5gram oo ka mid ah 975 billion tokens ka CommonCrawl, oo loo yaqaan model 500 billion n-grams (Buck et al., 2014). Chelba et al. (2013) waxaa la aasaasay benchmark One Billion Word, a big-scale training dataset in ay ka mid ah xawaaraha model-ka ah. In mid ka mid ah model-ka taal neural, Jozefowicz et al. (2016) waxay ka heli karaa wax soo saarka ah ee benchmark Billion Word doorka loo yaabaa LSTMs ilaa 1 bilyan parameter. Qalabka ugu horeysay ee loo yaqaan 'BET' (Devlin et al., 2018), 'GPT-2' (Radford et al., 2019), 'MegatronLM' (Shoeybi et al., 2019), iyo 'T5' (Raffel et al., 2020). Haddii loo yaqaan 'GPT-3' (Brown et al., 2020), model oo ku yaalla 175 bilyonda. Waxaa loo yaqaan 'Large Language Models', sida Jurassic-1 (Lieber et al., 2021), Megatron-Turing NLG (Smith et al., 2022), Gopher (Rae et al., 2021), Chinchilla (Hoffmann et al., 2022), PaLM (Chowdhery et al., 2022), OPT (Zhang et al., 2022), iyo GLM (Zeng et al., 2021). H 8 Qalabka Markaas ka mid ah, LLaMA-13B waa mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah Shuruudaha Waayo, waxaan ku dhamaado Daniel Haziza, Francisco Massa, Jeremy Reizenstein, Artem Korenev, iyo Patrick Labatut ee team xformers. Waxaan ku dhamaado Susan Zhang iyo Stephen Roller oo ay ku dhamaado data deduplication. Waxaan ku dhamaado Luca Wehrstedt, Vegard Mella, iyo Pierre-Emmanuel Mazaré oo ku dhamaado in ay dhamaado training stability. Waxaan dhamaado Shubho Sengupta, Kalyan Saladi, iyo dhamaado oo dhan ee AI infra team for their support. Waxaan dhamaado Jane Yu for her input on evaluation. Waxaan dhamaado Yongyi Hu for his help on data collection. QEEBE Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, iyo Charles Sutton. Lalit R Bahl, Frederick Jelinek, iyo Robert L Mercer. 1983. A maximum probability approach to continuous speech recognition. IEEE transactions on pattern analysis and machine intelligence, pages 179-190. Yoshua Bengio, Réjean Ducharme, iyo Pascal Vincent. 2000. A neural probabilistic language model. Advances in neural information processing systems, 13. Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Raadiga ah oo ku saabsan xayawaanka dabiiciga. In Proceedings of AAAI conference on artificial intelligence, pages 7432–7439. Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. 2022 Gpt-neox-20b: A open-source autoregressive model. arXiv preprint arXiv:2204.06745. Waayo, waxa uu ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah Peter F Brown, John Cocke, Stephen A Della Pietra, Vincent J Della Pietra, Frederick Jelinek, John Lafferty, Robert L Mercer, iyo Paul S Roossin. 1990 - A statistical approach to machine translation. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, iyo Dario Amodei 2020. Christian Buck, Kenneth Heafield, iyo Bas Van Ooyen. 2014. N-gram oo ka mid ah macluumaadka iyo macluumaadka xawaaraha ka mid ah crawl. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, iyo Tony Robinson. 2013. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, William Hebgen Guss, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Sutarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nicholino, Alex Paino, Nikolas Tezak, Jie, Igor Tangleckin Aakanksha Diawon Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Ghemawat, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sun Devipa, Henryk Michalewski, Xavier Garcia, Vedantra Mis, Kevin Robinson, Liam Fedus, Jason Zhou Denny Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, iyo Kristina Toutanova. 2019. Boolq: Khariidadda ah oo ah oo ah oo ah oo ah oo ah oo ah. arXiv preprint arXiv:1905.10044. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, iyo Oyvind Tafjord. 2018. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021 Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, iyo Ruslan Salakhutdinov. 2019. Transformer-xl: dhismaha soo xiriir oo ka mid ah dhismaha dhismaha. arXiv preprint arXiv:1901.02860 Tri Dao, Daniel Y Fu, Stefano Ermon, Atri Rudra, iyo Christopher Ré. 2022. Flashattention: wax soo saarka iyo wax soo saarka ah oo ku habboon ah oo ku habboon. arXiv preprint arXiv:2205.14135. Jacob Devlin, Ming-Wei Chang, Kenton Lee, iyo Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. Waayo, waxa uu ku yaalaa in ay ku saabsan wax soo saarka iyo wax soo saarka, waxaa laga yaabaa in ay ku saabsan wax soo saarka. Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wentau Yih, Luke Zettlemoyer, iyo Mike Lewis. 2022. Incoder: A model generative for code filling and synthesis. arXiv preprint arXiv:2204.05999. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, iyo Connor Leahy. Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, iyo Andy Zou. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, iyo Noah A Smith. 2020. Realtoxicityprompts: Qalabka qiyaasadda xawaaraha xawaaraha xawaaraha ee xawaaraha xawaaraha. arXiv preprint arXiv:2009.11462. Xisaabinta xisaabinta xisaabinta xisaabinta xisaabinta xisaabinta xisaabinta xisaabinta xisaabinta xisaabinta xisaabinta xisaabinta xisaabinta xisaabinta xisaabinta xisaabinta xisaabinta xisaabinta xisaabinta xisaabinta xisaabinta xisaaba Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H Clark, iyo Philipp Koehn. 2013. Qalabka modelka Kneserney-ka ah oo ka mid ah macluumaadka. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), sida 690-696. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, iyo Jacob Steinhardt. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, iyo Jacob Steinhardt. 2021. Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Patwary, Mostofa Ali, Yang Yang, iyo Yanqi Zhou. Sida loo yaabaa, waxaa laga yaabaa in ay ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, iyo Laurent Sifre. Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Dániel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, et al. 2022 Opt-iml: Scaling model language instruction meta learning through the lens of generalization. arXiv preprint arXiv:2212.12017. Mandar Joshi, Eunsol Choi, Daniel S Weld, iyo Luke Zettlemoyer. 2017. Triviaqa: A big scale remotely supervised challenge dataset for reading understanding. arXiv preprint arXiv:1705.03551. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, iyo Yonghui Wu. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, iyo Dario Amodei. Slava Katz. 1987. Qalabka caadiga ah oo ka mid ah dhismaha caadiga ah ee dhismaha caadiga ah ee dhismaha caadiga ah. IEEE transactions on acoustics, speech, and signal processing, 35(3):400–401. Reinhard Kneser iyo Hermann Ney. 1995. Nadiifinta back-off for m-gram modeling. In 1995 conference international on acoustics, speech, and signal processing, volume 1, pages 181–184. IEEE. Vijay Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, iyo Bryan Catanzaro. 2022. Qiimeeyo loo yaabaa in ay ka mid ah mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ah Taku Kudo iyo John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226. Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, iyo Yulia Tsvetkov. 2019. Qalabka xayawaanka xayawaanka xayawaanka xayawaanka xayawaanka xayawaanka xayawaanka xayawaanka xayawaanka xayawaanka xayawaanka xayawaanka xayawaanka xayawaanka xayawaanka xayawaanka xayawaanka xayawaanka xayawaanka xayawaanka xayawaanka xayawaanka xayawaanka xayawaanka xayawaanka xayawaanka xayawaanka xayawaanka. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Qalabka dabiiciga: a benchmark for question answering research. Transactions of Association for Computational Linguistics, 7:453–466. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, iyo Eduard Hovy. 2017. Race: dhismaha dhismaha dhismaha dhismaha dhismaha dhismaha. arXiv preprint arXiv:1704.04683. Aitor Lewkowycz, Anders Johan Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Venkatesh Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, iyo Vedant Misra. Opher Lieber, Or Sharir, Barak Lenz, iyo Yoav Shoham. 2021. Jurassic-1: Details Technical and evaluation. White Paper. AI21 Labs, 1. Stephanie Lin, Jacob Hilton, iyo Owain Evans. 2021. Truthfulqa: Taageerada sida model ah waxay ka mid ah wax ka mid ah wax ka mid ah. arXiv preprint arXiv:2109.07958. Ilya Loshchilov iyo Frank Hutter. 2017. Qalabka dhismaha dhismaha dhismaha. arXiv preprint arXiv:1711.05101. Mashaariiska ah oo ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah. Todor Mihaylov, Peter Clark, Tushar Khot, iyo Ashish Sabharwal. 2018. Qiimeeyaasha ah oo ka mid ah shuruudaha waaweyn? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789. Thomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernocky, iyo Sanjeev Khudanpur. 2010 Recur- ` rental network based language model. In Interspeech, sida 1045-1048. Makuhari. Nikita Nangia, Clara Vania, Rasika Bhalerao, iyo Samuel R. Bowman. 2020. CrowS-pairs: A challenge dataset for measuring social biases in masked language models. In EMNLP 2020. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, iyo Caiming Xiong. 2022. Codegen: A open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, iyo Ryan Lowe. Markus N Rabe iyo Charles Staats. 2021. Selfattention waxay u baahan tahay o(n 2 ) memory. arXiv preprint arXiv:2112.05682. Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018. Qiimeeyo xayawaanka xayawaanka xayawaanka xayawaanka xayawaanka. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Modelada xafiisyada waxay ku yaalaa multitask ah. Blog OpenAI, 1(8):9. Shirkadda Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, iyo Peter J Liu. Jonathan S Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, iyo Nir Shavit. 2019. Qalabka dhismaha dhismaha dhismaha dhismaha dhismaha. arXiv preprint arXiv:1909.12673. Rachel Rudinger, Jason Naradowsky, Brian Leonard, iyo Benjamin Van Durme. 2018. Gender bias in qiyaasta core. In NAACL-HLT 2018. Waayo, waxa uu ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ah mid ka mid ah mid ah mid ah Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, iyo Yejin Choi. 2019. Socialiqa: Qalabka Commonsense oo ku saabsan xanuunka xanuunka xanuunka. arXiv preprint arXiv:1904.09728. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilic, Daniel Hesslow, Ro-' man Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176bparameter open-access multilingual model. arXiv preprint arXiv:2211.05100. Rico Sennrich, Barry Haddow, iyo Alexandra Birch. 2015 Translation Neural machine of rare words with subword units. arXiv preprint arXiv:1508.07909. Claude E Shannon. 1948. A mathematical theory of communication. The Bell system technical journal, 27(3):379-423. Claude E. Shannon. 1951. Prognoos iyo entropy of printed English. Journal Technical System Bell, 30(1):50-64. Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Qalabka Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, iyo Nanyun Peng. 2019. Haddii aad u soo bandhigay: On biases in language generation. arXiv preprint arXiv:1909.01326. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, iyo Bryan Catanzaro. 2019. Megatron-lm: Qalabka badan oo ka mid ah macluumaadka kala duwan oo loo isticmaalo model parallelism. arXiv preprint arXiv:1909.08053. Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, Elton Zhang, Rewon Child, Reza Yazdani Aminabadi, Julie Bernauer, Xia Song, Mohammad Shoeybi, Yuxiong He, Michael Houston, Saurabh Tiwary, iyo Bryan Catanzaro. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, iyo Yunfeng Liu. 2021. Roformer: Transformer dhismaha oo ka mid ah xawaaraha xirfadeed. arXiv preprint arXiv:2104.09864. Marka aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u baahan yahay in aad u A. M. Turing. 1950. Computer Machinery iyo Intelligence. [Oxford University Press, Mind Association]. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, iyo Illia Polosukhin. GPT-J6B: A 6 Billion Parameter Autoregressive Language Model. https://github.com/kingoflolz/ mesh-transformer-jax. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, iyo Denny Zhou. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, iyo Edouard Grave. 2020. CCNet: Qalabka ugu fiican ee dhismaha monolingual ee data crawl web. In Language Resources iyo Conference Evaluation. Carole-Jean Wu, Ramya Raghavendra, Udit Gupta, Bilge Acun, Newsha Ardalani, Kiwan Maeng, Gloria Chang, Fiona Aga, Jinshi Huang, Charles Bai, et al. 2022. Sustainable ai: Implications Environmental, Challenges and Opportunities. Proceedings of Machine Learning and Systems, 4:795-813 Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, iyo Yejin Choi. 2019. Hellaswag: Ma rabtaa mashiinka aad ku yaalaa? arXiv preprint arXiv:1905.07830. Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Peng Zhang, Yuxiao Dong, iyo Jie Tang. Biao Zhang iyo Rico Sennrich. 2019. Normalization layer qashinka midabka. Advances in Neural Information Processing Systems, 32. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022 Qalabka Qalabka Waayo, sidoo kale waxaa laga yaabaa in ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid B. Qalabka Qalabka C ee LLaMA-65B Sida loo yaabaa, waxaa laga yaabaa in la soo bandhigay in la soo bandhigay LLaMA-65B (wax ka mid ah wax soo saarka). Qalabka D ee LLaMA-I Waxa uu soo bandhigay in ka mid ah mid ka mid ah LLaMA-I, i.e. LLaMA-65B waa mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah mid ka mid ah. Waxaa la heli karaa in arkiv ee CC by 4.0 Deed (Attribution 4.0 International) license. Taariikhda waa CC by 4.0 Deed (Attribution 4.0 International) waxaa loo isticmaali karaa. Xafiisyada Archive