Isit la nan AI ki tradui 200 lang, menm moun ki gen prèske pa gen okenn done

Autè yo: Ekip nan NLLB Martha R. Costa-Jussà nan James Cross Pwoteksyon Mase Elbayad Kènè Heafield Kevin Heffernan nan Pwodwi pou Telefòn Janice Lam Danyèl Limyè Jean Maillard nan Sòti nan Pwofesè Wang Guillaume Wenzek nan Al Youngblood Pwodwi pou Telefòn Loic Barrault Gabriel Mejia Gonzalez nan Prangthip Hansanti Jòn Hoffman Semarley Jarrett nan Kaushik Ram nan Sadagopan Règleman Shannon Spruit Pwodwi pou Pyè Andrews Necip Fazil Ayan Shruti Bhosale Sergey Edunov Fanatik nan Cynthia Gao Vedanuj Goswami Francisco Guzmán Pwodwi pou Alexandre Mourachko nan Christophe Ropers nan Pwodwi pou Telefòn Holger Schwenk nan Jeff Wang Autè yo: NLLB Team Martha R. Costa-Jussà nan Kreyòl Ayisyen Pwoteksyon Mase Elbayad Kènè Heafield Kevin Heffernan nan Pwodwi pou Telefòn Jansyen Lam Danyèl Limyè Jean Maillard nan Sòti nan Pwofesè Wang Guillaume Wenzek nan Al Youngblood nan Pwodwi pou Telefòn Lojisyèl Barrault Gabriel Mejia Gonzalez nan Pwodwi pou Telefòn Jòn Hoffman Semarley Jarrett nan Kaushik Ram nan Sadagopan Règleman Pwodwi pou: Shannon Spruit Pwodwi pou Pyè Andrews Necip Fazil Ayan Pwodwi pou Telefòn Sergey Edunov nan Fanatik nan Pwodwi pou Pwoteksyon pou gout Fransis Guzman nan Pwodwi pou Alexandre Mourachko nan Christophe Ropers nan Pwodwi pou Telefòn Holger Schwenk nan Pwodwi pou Abstraksyon Devlopman nan teknik neural te louvri nouvo chemen pou rechèch nan tradiksyon machin. Jodi a, sistèm tradiksyon machin neural (NMT) ka sèvi ak kapasite trè multilingual ak menm fè tradiksyon zèb, livrezon rezilta pwomèt nan tèm nan kouvèti lang ak bon jan kalite. Sepandan, skalasyon bon jan kalite NMT mande pou gwo volim nan done paralèl bilingual, ki pa disponib menm pou 7,000+ lang nan mond lan Konsantre sou amelyore bon jan kalite tradiksyon nan yon gwoup relativman ti lang rich resous rive nan pri a nan dirije atansyon rechèch la nan lang ki ba resous, agravasyon inegalite dijital nan long tèm. Pou boule modèl sa a, isit la nou prezante No Language Left Behind - yon sèl modèl masiv multilingual ki sèvi ak aprantisaj transfè atravè lang yo. Nou devlope yon modèl konduit konduit konduit ki baze sou Architecture nan Espesyalis Gated , , , , , , ki nou te fòmasyon sou done yo te jwenn ak nouvo teknik min ki adapte pou lang ki ba resous. Anplis de sa, nou te devlope plizyè amelyorasyon arsitektyèl ak fòmasyon yo kontwole overfitting pandan fòmasyon sou milye de travay. Nou evalye pèfòmans nan modèl nou an sou 40,000 direksyon tradiksyon lè l sèvi avèk zouti ki kreye espesyalman pou sa a - yon referans otomatik (FLORES-200), yon metrik evalyasyon imen (XSTS) ak yon detektè toksisite ki kouvri chak lang nan modèl nou an. Konpare ak modèl yo anvan yo state-of-the-art, modèl nou an reyalize yon medyeval 44% amelyorasyon nan bon jan kalite tradiksyon ki te mesye pa BLEU. Pa demontre ki jan yo skalere NMT nan 200 lang ak fè 1 2 3 4 5 6 7 Pwodwi Retounen ki sot pase a nan tradiksyon machin nèvral (NMT) te pouse teknoloji tradiksyon nan granmoun nouvo, men benefis li yo distribiye inegal. . Pifò nan amelyorasyon yo te fè sitou benefisye nan lang segondè-resous, kite anpil lang ki ba-resous apre yo. (Pou objektif rechèch nou an, nou defini yon lang segondè-resous kòm yon lang pou ki nou gen omwen 1 milyon sentans nan done tèks aliye (oswa biteks) ak yon lòt lang). Diferans sa a ta ka sitou atribye a yon gap done: modèl NMT tipikman mande pou gwo volim done yo pwodwi tradiksyon bon jan kalite ak, pa definisyon, sa yo volim yo pa disponib pou lang ki ba-resous. Pwojè a No Language Left Behind (NLLB-200) ap eseye depase limitasyon sa a lè l sèvi avèk apwòch ki anvan sanble pou bati modèl masivman milti-lingual ak kapasite transfè ant , , ki pèmèt lang ki gen rapò yo aprann de youn nan lòt , , . 1 8 9 1 10 11 Li se kounye a lajman rekonèt ke modèl plizyè lang te demontre amelyorasyon pèfòmans pwomèt sou modèl bilingual Sepandan, kesyon an rete si modèl anpil lang ka pèmèt reprezante dè santèn de lang san yo pa kompromèt bon jan kalite a. Rezilta nou yo demontre ke doubòse kantite lang ki sipòte nan tradiksyon machin ak kenbe bon jan kalite a pwodiksyon yo pa efò reciprocely exclusive. Modèl final nou an – ki gen ladan 200 lang ak twa fwa plis lang ki ba-rezòl kòm moun ki segondè-rezòl – pèfòmans, an mwayen, 44% pi bon pase sistèm ki sot pase a state-of-the-art. Papye sa a prezante kèk nan teknik yo ki pi enpòtan nan kolekte done, modèl ak evalyasyon itilize pou reyalize objektif sa a. 12 Premyèmman, konpare ak kolèj yo rich-resous yo, done fòmasyon pou lang ki ba-resous yo chè ak lojisyèlman difisil jwenn , , . Resous dijital piblik ki disponib se tou limit nan volim oswa difisil pou sistèm otomatik detekte (particularman nan gwo seri done piblik sou entènèt tankou CommonCrawl). Pandan ke kolekte yon mas kritik nan done semen tradui pa moun se nesesè, akse nan done ase depann sou minasyon done gwo-scale ak pipelines done monolingual , , , . The latter techniques are often affected by noise and biases, thereby making validating the quality of the datasets they generate tedious Nan NLLB-200, nou montre ke yon teknik kodaj sentans ki baze sou destilasyon, LASER3 (ref. ), fasilite efikas la minasyon nan done paralèl pou lang ki ba resous. 13 14 15 16 17 18 19 20 21 Dezyèmman, sou bò modilasyon, nou itilize yon assemblage nan semen, min, sous louvri ak back-translated dataset yo fòme modèl kondicional kondicional miltip (pèmèt plis spesifikman, modèl nan Espesyalis-of-Experts , , , , , ki pèmèt transfè atravè lang ant lang ki gen rapò san yo pa ogmante entèferans ant lang ki pa gen rapò). Nou montre ki jan nou ka jwenn pèfòmans state-of-the-art ak yon kompromis pi optimum ant transfè atravè lang ak entèferans, ak amelyore pèfòmans pou lang ki ba resous. 2 3 4 5 6 7 Finalman, pou egzanp nan evalyasyon bon jan kalite, nou te kreye FLORES-200 - yon gwo referans miltip ki pèmèt evalyasyon bon jan kalite tradiksyon nan nenpòt nan apeprè 40,000 direksyon tradiksyon ki kouvri pa modèl yo NLLB-200. Anplis de metrik otomatik, nou te kreye tou Cross-lingual Semantic Text Similarity (XSTS) ak Evaluation of Toxicity (ETOX). XSTS se yon pwotokòl evalyasyon imen ki bay konsistans atravè lang; ETOX se yon zouti detekte toksisite ajoute nan tradiksyon lè l sèvi avèk lis motè toksisite. Pou amplifye aplikasyon pratik nan travay nou an nan sèvis nan kominote ki pale ki ba resous, nou bay tout benchmarks, done, kòd ak modèl dekri nan efò sa a kòm resous ki disponib pou itilize ki pa komèsyal ( ) (See Done ak Kòd disponiblite deklarasyon pou detay). https://github.com/facebookresearch/fairseq/tree/nllb Otomatikman kreye done fòmasyon tradiksyon Teknoloji a kounye a ki itilize pou fòmasyon modèl tradiksyon yo difisil extend nan enstalasyon ki ba resous, nan ki aliye bilingual done tèks (oswa done bittekst) yo relatif rès Pifò lang ki ba resous yo sipòte sèlman pa ti bityèks done mete atansyon ki konsiste prensipalman nan tradiksyon nan Bib la kretyen , ki ofri yon diversite domèn limite. 22 23 To build a large-scale parallel training dataset that covers hundreds of languages, our approach centres around extending existing datasets by first collecting non-aligned monolingual data. Then, we used a semantic sentence similarity metric to guide a large-scale data mining effort aiming to identify sentences that have a high probability of being semantically equivalent in different languages . 18 Identifikasyon lang pou koleksyon done monolingual Kolekte done monolingual nan gwosè mande pou yon sistèm idantite lang (LID) ki egzat klasifye resous tèks pou tout lang NLLB-200 yo. Malgre ke LID ta ka wè kòm yon pwoblèm rezoud nan kèk domèn , li rete yon repitasyon louvri pou done Web , Espesifikman, pwoblèm coalesce alantou domèn mismatch Deskripsyon nan lang menm jan an ak siksè masivman plizyè lang scaling . 24 25 26 26 27 28 Kontinye atansyon nan avanse teknik LID te mennen nan yon ogmantasyon notable nan tou de kouvèti lang ak presizyon nan tan. CLD3 ( ) ak rapid tèks se de modèl ki disponib fasil ofri pèfòmans deteksyon segondè pou 107 ak 187 lang, respektivman. Pa itilize plizyè seri done piblik, etid anvan , rapòte kouvèti menm pi wo—464 ak 1,366 lang, respektivman. Yon lòt etid Scales LID pèfòmans jiska 1629 lang lè l sèvi avèk lis tèks ak self-surveillance yo bootstrap done fòmasyon jwenn sou entènèt la. Sepandan, sa yo apwòch lè l sèvi avèk done jwenn pèmèt debalans domèn. Sa vle di, paske domèn tèks ki disponib varye pa lang, classifiers konflete domèn diferan ak lang diferan. https://github.com/google/cld3 29 30 31 32 Nan travay nou an, nou koure FLORES-200 yo itilize kòm yon seri devlopman pou pèfòmans sistèm LID nou an se tuned sou yon konbinezon domèn inifòm. Pwosesis nou an konbine yon modèl rapid-tekst ki baze sou done fòmasyon sou FLORES-200 ak yon ti seri règ ekri nan men yo rezoud feedback moun sou erè klasifikasyon. Règ sa yo espesyalman rele nan seksyon 5.1.3 nan ref. ak enkli filtè lingvistik yo diminye aprantisaj la nan korelasyon erè nan echantiyon fòmasyon bruyant pandan modélisasyon dè santèn de lang. 33 34 Nou konpare modèl nou an LID ak twa modèl piblik ki disponib: CLD3, LangId ( ) ak LongDetect ( Tables nan rapòte pèfòmans sou twa koupe kaskad nan lang ki koresponn ak NLLB-200: (1) 51 lang tou sipòte pa LangId, LangDetect ak CLD3; (2) 78 lang tou sipòte pa LangId ak CLD3; (3) 95 lang tou sipòte pa CLD3. Nou tou rapòte pousantaj false-pozitif (FPR) yo reflete enpak la nan false-pozitif sou lang yo pa wè. Rezilta nou yo montre ke modèl nou an se ekipman pou ranplase tout 200 lang yo te jwenn nan FLORES-200 pandan y ap reyalize pèfòmans anpil pi wo pase LangId, LangDetect ak CLD3. Anplis de sa, ogmantasyon an nan F1 pousantaj la se akomode ak yon amelyorasyon nan FPR, ki suggere yon apwop . https://github.com/saffsd/langid.py https://pypi.org/project/langdetect/ 1 32 Mining pou Bitext Premye travay note ke bon jan kalite tradiksyon an jeneralman ogmante ak kantite done fòmasyon bon jan kalite segondè, ki se difisil jwenn lè travay ak lang ki ba-resous. Egziste corpora paralèl pou lang ki ba-resous yo souvan pratikman retire soti nan koleksyon milti-lang, tankou Bib la kretyen oswa piblikasyon yo nan òganizasyon milti-nanal, ki yo limite nan kantite ak domèn. Pou depase pwoblèm sa a, nou te kreye seri done fòmasyon atravè minasyon bitext mondyal nan kontni entènèt piblikman disponib (retire soti nan depo tankou CommonCrawl). Ide a anba a nan abòdone a nan minasyon bitext nou an se an premye aprann yon espas sentans milti-language enbake ak sèvi ak yon mwayèn similitude nan espas ki deside si de sentans 35 Kòm apwòch min nou an mande pou yon espas entegre milti-language, gen plizyè repitasyon lè anplwaye reprezantasyon sa a nan tout lang NLLB-200 yo. Premye, nou te gen yo asire ke tout lang yo te aprann byen ak ke nou konsidere gwo echilib la nan done fòmasyon ki disponib. Dezyèmman, fòmasyon nan yon enkodè frè milti-language masivman soti nan nòt chak fwa yon nouvo seri lang yo enpòte se konpetitifman chè. Anplis de sa, dezavantaj prensipal la nan apwòch sa a se ke espò entegre aprann soti nan chak nouvo modèl pa nesesèman konpatib. Sa a ka fè entegre min pou chak nouvo enkodè, tout done monolingual ki disponib bezwen re-entegre ( ki elaji espas la nan entegre lazè nan tout lang NLLB-200. Lang yo fòmasyon oswa kòm elèv yo yon sèl oswa ansanm ak lang soti nan fanmi an menm. Antrenman elèv yo swiv metòd la deskri nan ref. . 21 36 21 Pwopozisyon nou an pèmèt nou konsantre sou espesifik yo nan chak lang pandan y ap benefisye de lang ki gen rapò, ki se enpòtan pou fè fas ak lang ki gen resous trè ba. (Se yon lang definye kòm yon lang ki gen resous trè ba si li gen mwens pase 100,000 echantiyon nan tout parayj ak nenpòt lòt lang nan dataset nou an). Avèk metòd sa a, nou te kreye plis pase 1,100 milyon pa gen rapò nouvo nan done fòmasyon pou 148 lang. Sa a done fòmasyon adisyonèl, ansanm ak tradiksyon tounen (yon teknik konvansyonèl pou agrime done nan NMT; ref. ), prezante amelyorasyon enpòtan nan bon jan kalite tradiksyon - espesyalman, +12.5 chrF++ (ref. ) pou tradui lang trè ki ba resous nan angle. Pou plis detay, wè enfòmasyon Supplementary . 37 38 D Modèl Malgre nan ogmantasyon an nan volim done, pwoblèm prensipal la nan tradiksyon ki ba resous se pou modèl fòmasyon reprezante ase 200 lang pandan y ap ajiste nan kapasite done varyab pou chak par lang. Anplis de teknik tankou agrandasyon done (pou egzanp, ak tradiksyon tounen) ak estrateji self-supervision sou done monolingual, nou te itilize modèl konpòtman kondisyone - plis spesifikman, Sparsely Gated Mixture of Experts (sou sa a MoE) - minimize entèferans ant direksyon lang ki pa gen rapò. Modèl transformatè MoE diferan de modèl transformatè dense nan ke kèk nan layers rezo feed-forward yo ranplase ak layers MoE nan tou de kodè a ak dekodè a. Yon layer MoE konsiste de ekspè (ki chak se yon rezo feed-forward) ak yon rezo gating yo deside ki jan yo route token enpòtan nan ekspè. Modèl la nan transformatè encoder-decoder, ranplase ak layers MoE ak rezo gating respektiv yo, aprann yo route token enpòtan nan de ekspè ki korespondan nan optimize yon konbinezon lineyè-pwa nan entropy koreksyon etikèt ak yon pousantaj pousantaj balans pèdi . E 39 6 Nou jwenn ke modèl Vanilla MoE ak absòpsyon an jeneral yo suboptimal pou lang ki ba resous ak sitou overfit sou pafè ki ba resous. Pou koreksyon pwoblèm sa a, nou devlope Ekspè Output Masking (EOM), yon estrateji reglemantasyon espesifik pou MoE architectures, epi konpare li ak estrateji reglemantasyon ki egziste, tankou Gating Dropout Nou jwenn ke Gating Dropout pèfòmans pi bon pase Vanilla MoE ak absòpsyon an jeneral, men se depase pa EOM. 40 Pou kontinye diminye overfitting sou par lang ki ba resous, nou devlope yon pwogram aprantisaj ki enpòte par lang nan faz pandan fòmasyon modèl. Updates yo te entwodwi ak ajou anvan fen fòmasyon an. Sa a diminye overfitting pandan y ap pèmèt pafè ki benefisye de fòmasyon adisyonèl kontinye aprantisaj yo. Tables montre ke konbine aprantisaj curriculum ak EOM amelyore pèfòmans, espesyalman sou pafè lang ki ba ak trè ba resous (considere seksyon an 'Modèlize' pou plis detay). K K 2 Pou konprann ki jan modèl MoE se itil pou tradiksyon machin miltip, nou vizyèlize semblanz nan ekspè nan layers MoE lè l sèvi avèk kat tèmik (Fig. ). Sa yo tèmik kat montre ke nan fen decoder layers (Fig. ), lang yo se separe (ki se, voye nan divès kalite espesyalis). Anplis de sa, nou observe ke lang yo nan menm fanmi an yo trè menm jan nan chwa yo nan ekspè (ki se, ki fen decoder MoE layers yo lang-spesifik). Sa a se espesyalman ka pou dialèz Arabik la (ki gen sis ranje ak kolòn nan kòt la pi wo a), lang nan subgrouping la Benue-Congo, osi byen ke lang nan script la Devanagari. Nan kontrè, ki anvan decoder MoE layers (Fig. ) parèt yo dwe mwens lang-spesifik. Late encoder MoE layers yo se espesyalman lang-agnostique nan fason yo wout token tankou ka atistik pa kat tanperati inifòm nan Fig. . 1A-D nan 1D nan 1C nan 1b nan Fig. 1: Cosine similarity scores between languages at different layers of the encoder–decoder architecture. nan – Premye ( ) ak dènye ( ) encode layers ak Lè sa a, premye ( ) ak dènye ( ) couches de décoder. Fòmèlite a se mesye nan relasyon ak desizyon gating (ekspert chwa) pou chak lang (sòti a nan kodaj la ak objektif la nan decoder la). koulè pi limyè reprezante eksperyans pi wo similitude, se konsa, yon pwosesis lang-agnostique. a d a b c d Konbine done ( wè seksyon la » ’) and modelling contributions, Table montre ke NLLB-200 depase sistèm ki pi prèske state-of-the-art pa prèske +7.3 spBLEU (ref. ) on average, constituting a 44% improvement. We then compared NLLB-200 with a few other state-of-the-art models, such as Deepnet ak M2M-100 (ref. ), pou rapòte pousantaj pou 87 lang konpare ak FLORES-101. Sou sa a subset ti, NLLB-200 ankò pèfòmans sou +7.0 spBLEU nan mwayen. An jeneral, rezilta yo montre ke NLLB-200 amelyore sou sistèm state-of-the-art pa yon marye enpòtan malgre sipòte 200 lang, oswa de fwa anpil lang (ak plis pase 30,000 direksyon adisyonèl) konpare ak nenpòt travay anvan yo. Nou montre tou nan eksperyans adisyonèl ke NLLB-200 se yon modèl NMT jeneral-purpose, transfere nan lòt domèn pa fin-tuning sou kantite ti kantite biteks bon jan kalite segondè (Li wè enfòmasyon Supplementary ) nan Otomatikman kreye done fòmasyon tradiksyon 3 41 42 1 E3 nan Evalyasyon Among the many aspects of model performance that can be evaluated , seksyon sa a enpòtan twa aspè ki gen yon enpak marye sou evalyasyon an jeneral bon jan kalite: referans pou evalyasyon otomatik, pwotokòl evalyasyon imen ak evalyasyon toksisite. 43 Yon referans pou evalyasyon otomatik lè l sèvi avèk FLORES-200 Kalite a nan pwodiksyon NMT se tipikman evalye pa metrik otomatik tankou BLEU Pwodwi pou Kalkilasyon an nan pousantaj bon jan kalite otomatik lè l sèvi avèk metrik sa yo mande pou yon seri done referans ki bay tradiksyon moun ki gen estanda goud kòm referans yo. Anplis de sa, evalyasyon an Apple-to-Apple nan diferan apwòch ki fè posib pa sa yo seri done referans bay nou yon konpreyansyon pi byen nan sa ki bezwen rechèch ak devlopman plis. Pou egzanp, kreye yon seri done referans nan atelye a sou tradiksyon machin (WMT) li te mennen nan avanse rapid nan direksyon tradiksyon tankou angle a Alman ak angle a franse. 44 41 45 Pou NMT a masivman plizyè lang, pi gwo dataset referans ki disponib te FLORES-101, ki sipòte apeprè mwatye kantite lang nan NLLB-200. Ekspansyon an nesesè nan FLORES-101 nan FLORES-200 reprezante yon repitasyon plis nan jaden an nan garanti bon jan kalite, pati nanòz diferans nan pratik estanda ak aksè limit nan tradiktè pwofesyonèl pou tout lang ki gen rapò. Pou depase repitasyon sa a, nou te adapte flux nou an yo peye atansyon espesyal sou mekanis yo asire bon jan kalite. Flux a nan FLORES-200 konsiste de kat faz: (1) alignment; (2) tradiksyon, garanti bon jan kalite inisyal ak iteration(s); (3) garanti bon jan kalite final; ak (4) konplete lang. Yon FLORES-200 set se konsidere As a result of this redesigned workflow, we produced a three-split (dev, devtest, test) data set of parallel human reference translations for all NLLB-200 languages meeting the 90% quality threshold in a maximum turnaround time of 287 days (119 days on average, 70 days minimum). (Note that to avoid leakage with our models, we filtered data from FLORES and other evaluation benchmarks used (such as WMT and IWSLT) from our training data. This was done by comparing the hashes of training sentences against those of evaluation sentences, using the xxHash algorithm). Please refer to Supplementary Information for more details on the evaluation process. Figure shows the quality scores for all languages, some of which are labelled as examples. C 2 Fig. 2: Quality of FLORES-200. nan Pousantaj garanti bon jan kalite pou lang yo nan FLORES-200. Standards minimòm akseptab se 90%. Evalyasyon moun serye state-of-the-art otomatik metrik souvan pa t 'capture aspè nan lang ki, menm si subtile, ka gen yon pèfòmans enpòtan sou bon jan kalite tradiksyon an. Evalyasyon moun se, se konsa, esansyèl pou asire evalyasyon bon jan kalite ki vle di . That said, relying on them comes with two challenges: (1) any large-scale human evaluation of NMT quality, regardless of the number of translation directions involved, contends with potentially low inter-evaluator agreement (in the vicinity of 0.5 kappa); and (2) massively multilingual NMT introduces another complexity—that of quality evaluation consistency across language directions. We address these two issues by developing XSTS , yon nouvo metrik pousantaj konsantre sou vle di, ak lè l sèvi avèk yon pwotokòl ki pèmèt pou kalibrasyon nan pousantaj atravè evalyateur ak par lang. 46 47 XSTS se yon pwotokòl evalyasyon imen enspire pa STS , emphasizing meaning preservation over fluency. XSTS uses a five-point scale, in which 1 is the lowest score, and 3 represents the acceptability threshold. To ensure consistency not only across languages but also among different evaluators of any given language, we included the same subset of sentence pairs in the full set of sentence pairs given to each evaluator, making it possible to calibrate results. 48 Nou jwenn ke metrik otomatik tankou spBLEU ak chrF++ korele rezonabman byen ak evalyasyon kalibre moun nan bon jan kalite tradiksyon, tankou montre nan Figi. Spearman nan coefficients korelasyon ant agregat XSTS ak spBLEU, chrF++ (corpus) ak chrF++ (mwayen nivo sentans) yo 0.710, 0.687 ak 0.694, respektivman. lòt coefficients korelasyon (Kendall nan Pearson Corpus spBLEU bay pi bon korelasyon nominal, ki te swiv pa medyeval-nivo sentans nan chrF++. 3 R τ R Fig. 3: Correlations between aggregated human quality scores and automated metrics. , The relationship between spBLEU and XSTS. , The relationship between chrF++ and XSTS. , The relationship between average sentence-level chrF++ and XSTS. All automated scores were computed only on the sentences evaluated for a given model and translation direction (either the full FLORES-200 dataset or a subset). NLLB-200 refers to a 55B parameter MoE model, and NLLB-200 Baseline refers to a dense 3.3B parameter model. a b c Full size image We also find that calibrated human evaluation scores correlate more strongly with automated scores than uncalibrated human evaluation scores across all automated metrics and choices of correlation coefficient. In particular, uncalibrated human evaluation scores have a Spearman’s Koefician korelasyon nan 0.625, 0.607 ak 0.611 pou spBLEU, chrF++ (corpus) ak chrF++ (mwayen nivo sentans), respektivman. R Overall, a sample of 55 language directions were evaluated, including 8 into English, 27 out of English, and 20 other direct language directions. The overall mean of calibrated XSTS scores was 4.26, with 38/55 directions scoring over 4.0 (that is, high quality) and 52/56 directions scoring over 3.0. Nou hypothesize ke toksisite ajoute ka ki te akòz prezans la nan toksisite nan done yo fòmasyon ak itilize detektè nou yo estime, plis spesifikman, toksisite ki pa balans nan done yo bitext. Nou jwenn ke nivo ki estimé nan toksisite ki pa balans varye soti nan yon korpus nan bitext a pwochen ak ke toksisite ki pa balans ka anpil atribye nan bitext mal aligned. Men, fòmasyon ak bitext sa a mal aligned ka ankouraje maltradiksyon ak toksisite ajoute. Pou diminye pwoblèm sa a, nou te devlope yon pwosesis filtraj biteks ki baze sou deteksyon an nan plizyè instans nan toksisite ajoute (ki se, ka nan ki yon sentans nan biteks pa gen omwen de bagay toksik plis pase lòt sentans nan pa gen). (Nan yon analiz bon jan kalite detèktè anvan te montre ke yon presizyon pi wo te rive nan ka sa a). Nou te ajoute pwosesis sa a filtraj toksisite kòm yon opsyon nan pwosesis la filtraj ak te eksperyans ak oswa san yo pa li pou konpare. The experimental results on the FLORES-200 dev set for 10 translation directions (from and into English for Somali, Southern Sotho, Twi, Umbundu and Venetian) show that after filtering an average amount of around 30% parallel sentences, the translation quality (chrF++) improves by 5% and added toxicity (ETOX) reduces by the same amount. Therefore, the filtering pipeline that includes toxicity filtering not only reduces the number of toxic items in the translation output but also improves the overall translation performance. Konklisyon Nan 2016, òganizasyon an te deklare aksè nan entènèt la kòm yon dwa enpòtan. Malgre ke entansyon an nan deklarasyon sa a te limit cenzè ak pèmèt enfòmasyon ak lide kouri san entèraksyon, yon anpil nan entènèt la jodi a rete pa aksè pou anpil nanòz barriè lang. Efò nou yo te fèt yo kontribye yon solisyon pou ede chanje sa a status quo. For many low-resource language communities, NLLB-200 is one of the first models designed to support translation into or out of their languages. Although applications of these new translation capabilities could be found in several domains of everyday life, we believe their impact would be most significant in a domain such as education. In formal educational settings, for instance, students and educators belonging to low-resource language groups could, with the help of NLLB-200, tap into more books, research articles and archives than before. Within the realms of informal learning, low-resource language speakers could experience greater access to information from global news outlets and social media platforms, as well as online encyclopaedias such as Wikipedia. Access to machine translation motivates more low-resource language writers or content creators to share localized knowledge or various aspects of their culture. Giving individuals access to new translation tools could thus open up opportunities for bidirectional learning, thereby also challenging Western-centric modes of knowledge production and dissemination, ultimately aiding in revitalizing certain minority cultures and languages. Since launching NLLB-200, we can already see the impact of the model across many directions. Four months after the launch of NLLB-200, Wikimedia reported that our model was the third most used machine translation engine used by Wikipedia editors (accounting for 3.8% of all published translations) ( ). Compared with other machine translation services and across all languages, articles translated with NLLB-200 has the lowest percentage of deletion (0.13%) and highest percentage of translation modification kept under 10%. https://web.archive.org/web/20221107181300/https://nbviewer.org/github/wikimedia-research/machine-translation-service-analysis-2022/blob/main/mt_service_comparison_Sept2022_update.ipynb Nan plizyè fason, kompozisyon an nan efò a NLLB-200 pale sou santralite a nan interdisiplinerite nan fòme vizyon nou an. Tradiksyon machin ak avanse AI se nan kout la nan devlopman teknolojik, kiltirèl ak sosyete, ak se konsa mande pou akademik ak fòmasyon divès kalite ak pozisyon yo konplètman konprann chak angle , . It is our hope that in future iterations, NLLB-200 continues to include scholars from fields underrepresented in the world of machine translation and AI, particularly those from humanities and social sciences backgrounds. More importantly, we hope that teams developing these initiatives would come from a wide range of race, gender and cultural identities, much like the communities whose lives we seek to improve. 49 50 Finally, we want to emphasize that overcoming the challenges that prevent the web from being accessible to speakers of all languages requires a multifaceted approach. At the technical level, NLLB-200 overcomes many data, modelling and evaluation challenges in NMT research, but it still has its limitations, some of which are documented in Supplementary Information Kòm yon sèl entèvyou teknoloji, NLLB-200 se tout lòt pase yon pati nan yon puzzle masiv; entèvyou politik ki apwopriye nan pwoblèm pi fonksyonèl alantou edikasyon, aksè nan entènèt ak lis dijital se imperatif yo eradike pwoblèm la estrikti nan diferans lang. G Metòd Data This section describes the steps taken to design our language identification system and bitext mining protocol. Language identification To train language identification models, we used fasttext , , which has been widely used for text classification tasks because of its simplicity and speed. We embedded character-level -grams from the input text and leveraged a multiclass linear classifier on top. The lightweight nature of fasttext enables our LID models to handle web-scale data. Furthermore, a linear model has the benefit of being easily explainable, allowing us to trace any classification error back to its root cause. This is instrumental in addressing common pitfalls that arise when detecting language on web corpora . 33 51 n 32 Classifier design We experimented with two different designs. First, we used a combination of multiple binary classifiers in which the final decision was obtained by selecting the language with the highest score after applying a threshold. We applied threshold optimization so that when the confidence of a classifier is low, the corresponding language is not considered for the final decision. A sentence was filtered out if none of the classifiers surpassed its threshold. Second, we built a multiclass classifier using softmax over all possible languages. In this case, the threshold optimization is done after the softmax. Our results directed us to focus on the second approach, which offers several advantages. First, changing the threshold for one language did not affect the performance of the other (which is not true in the first setting). Second, this approach generalizes better to out-of-domain data, which is our primary use case (Wikipedia → web data). Finally, a single classifier has the added benefit of being computationally simpler, thus streamlining the language identification process. Training data and handling massive class imbalance We used publicly available datasets to train our LID system, partially covering our languages of interest. The public datasets deployed were mostly built from web pages such as CommonCrawl. We then supplemented these with NLLB-Seed data (Supplementary Information ) for any missing languages. However, this supplementation is insufficient in ensuring balance in the raw training data , . For example, English alone represents 10.1% of our training data, whereas Minangkabau (Latin script) represents only 0.06%. Following ref. , we experimented with multiple settings of temperature upsampling for underrepresented languages, in which sentences from a language representing per cent of the data set are sampled proportionally to . Optimal performance was obtained at 1/ = 0.3 (for more details, see section 5.1 of ref. ) nan B 32 30 10 l pl T 34 Training parameters Our best-performing model was trained with softmax loss over two epochs with a learning rate of 0.8 and embeddings with 256 dimensions. We discarded words with less than a thousand occurrences after upsampling and selecting a minimum and maximum character -gram length of two and five, respectively (which were assigned a slot in buckets of size 1,000,000). (In fasttext, we refer to ‘word’ when it is separated by spaces. When it is a non-segmenting language, there is only one ‘word’ for the whole sentence (and we take character -grams)). All hyperparameters were tuned on FLORES-200 dev (see section 5.1.2 of ref. ). n n 34 Improving LID with linguistic analysis Language identification is a challenging task in which numerous failure modes exist, often exacerbated by the gaps between the clean data on which LID models are trained and noisy data on which LID models are applied. In other words, LID models trained in a supervised manner on fluently written sentences may have difficulty identifying grammatically incorrect and incomplete strings extracted from the web. Furthermore, models can easily learn spurious correlations that are not meaningful for the task itself. Given these challenges, we collaborated closely with a team of linguists throughout different stages of LID development to identify proper focus areas, mitigate issues and explore solutions (see section 5.1.3 of ref. ) nan 34 Minisipalite Minisipalite Konpòtman an jeneral pou min bitext konsantre sou kòmanse ak yon modèl pwofesè enkodè sentans masivman plizyè lang ak adapte li nan plizyè diferan modèl elèv ki gen resous ki ba. Sa a apwòch pèmèt nou ajoute lang ki gen resous ki ba san yo pa konpetisyon ak lang ki gen resous segondè pou kapasite. Li fè sa evite bezwen pou retrain modèl la tout antye soti nan zèb pandan y ap kenbe konpatibilite ak espò embedding plizyè lang pou min pita. Done extended Fig. se yon vèsyon amelyore nan open-source LASER encoder ( ) pwosesis fòmasyon orijinal la was adapted to include SentencePiece tokenization (including a vocabulary of 7,000 tokens) and the upsampling of low-resource languages. 1 https://github.com/facebookresearch/LASER 36 Arkitektur la nan encoder la BiLSTM senk-layer ak metòd la maksimòm poto pou jwenn entwodiksyon sentans te rete san chanjman. Antrenman an te Lè sa a te pote soti nan 93 lang yo menm ak resous piblik ki te jwenn nan OPUS Li se ref. for details on the original LASER training procedure. Training of the students followed the approach described in greater detail in ref. , rezime anba a: 52 36 21 elèv ki espesyalize nan yon lang oswa plizyè lang menm jan an; students were randomly initialized because we wanted to handle low-resource language for which we did not have a pre-trained language model; students may have a dedicated SentencePiece vocabulary different from the teacher to better accommodate scripts and tokens in the student languages; as we used cosine distance for bitext mining (Fig. ), students learnt to minimize the cosine loss with the teacher; 1 elèv yo ka gen yon pèdi MLM yo sèvi ak lang elèv monolingual done (Fig. 1). Training parameters elèv nou an kodaj itilize yon transformateur 12-layer ak yon gwosè kache nan 1,024, kat tèt atansyon, ak apeprè 250 milyon paramèt. Tout elèv yo te fòmasyon ak biteks ki disponib nan lang yo respektivman, konplete pa 2 milyon sentans nan lang angle / angle ak angle / Panyòl. Motivasyon an anba a sa a apwòch se nan ankouraje elèv yo nan espas entegre angle a, ogmante robustite pa enkli biteks angle / Panyòl soti nan CCMatrix ak pèmèt aprantisaj ansanm nan lang nouvo. Teknoloji sa a se espesyalman itil lè sèlman kantite kantite biteks ki disponib pou fòmasyon elèv yo. Fòmasyon pwofesè - elèv te fè sou 16 GPUs, optimizer ADAM, yon vitès aprantisaj de 0.0005 ak yon Proxy metric for new encoders Bit min te Lè sa a, te itilize pou amelyore bon jan kalite tradiksyon pou lang yo nan NLLB-200. Sepandan, min ak fòmasyon NMT yo kalkilman chè, epi li se etonan yo fè evalyasyon sa a sistematikman pou anpil varyant kodaj sentans diferan. Kòm yon pwoksyon evalyasyon, nou te itilize yon baz min multi-language similite rechèch pousantaj erè, rele isit la kòm xsim. An kontrè ak presizyon cosine, ki aliye embeddings ki baze sou pi wo pousantaj cosine, xsim aliye sous ak objektif embeddings ki baze sou pi wo pousantaj marj, ki se te montre benefisye nan min . The margin-based score is defined as 53 where and are the source and target sentences, and ( ) denotes the Vann anviwònman ki pi pre nan lòt lang. Nou mete to 4. All xsim results are calculated on FLORES-200 devtest, using the ratio margin, where margin( , ) = nan / . Moreover, all scores are calculated for translations into English (that is, xxx → eng). English is encoded by the teacher, and the other language is encoded by the LASER3 student. To facilitate further research using xsim, we also provide this evaluation method as an open-source resource ( ). x y nan x k x k a b a b https://github.com/facebookresearch/LASER/ End-to-end Encoder evalyasyon Once we had identified the best sentence encoder for each language using the xsim scores, we performed mining, added the mined data to the existing bitexts and trained a bilingual NMT system. Initial experiments indicated that a threshold on the margin of 1.06 seems to be the best compromise between precision and recall for most languages. For these NMT baselines, we do not apply extra filtering on the bitexts and leave this to the training procedure of our massively multilingual NMT system. We did not attempt to optimize the architecture and parameters of the bilingual NMT systems to the characteristics of each language pair but used the same architecture for all. Therefore, the reported results should not be interpreted as the best possible ones given the available resources—they are mainly provided to validate the mined bitexts. We used a 12-layer encoder and decoder and trained for 100 epochs. Moreover, we looked for the best performance on the FLORES-200 development set and report detokenized BLEU on the FLORES-200 devtest. Modelling In this section, we first describe the multilingual machine translation task setup, which includes tokenization and base model architecture. Then, we outline how we leveraged conditional computation for massively multilingual machine translation with EOM regulation and our Curriculum Learning (CL) strategy for low-resource languages. Retounen nan Nou te modèl NMT miltipilye kòm yon seri-a-sekans travay, nan ki nou kondisyone sou yon seri enprime nan lang kay la ak yon enkodè ak jere seri a pwodiksyon nan lang objektif espere ak yon dekodè avèk sentans lan sous Lang kay la s, and target language t in hand, we trained to maximize the probability of the translation in the target language Li se, ( ∣ nan s, t). Below, we discuss details of the (1) tokenization of the text sequences in the source and target languages; and (2) model architecture with the input and output designed specifically for multilingual machine translation. For further details on the task setup, such as the amount of training data per language pair, please refer to Supplementary Information oswa seksyon 8 nan ref. . 54 S ℓ ℓ T P T S ℓ ℓ F 34 Segmentation with SentencePiece To tokenize our text sequences, we trained a single SentencePiece model (SPM) for all languages. We sampled a total of 100 million sentences from primary bitext data. To ensure low-resource languages are well-represented in the vocabulary, we downsampled high-resource and upsampled low-resource languages with a sampling temperature of five (ref. Notèman, gwosè vocabulary se yon hyperparamèter enpòtan nan modèl tradiksyon plizyè lang ki gen rapò ak lang ki ba resous , , . The vocabulary size of our trained SPM model is 256,000. Such a large vocabulary ensures adequate representation across the wide spectrum of languages we support. 55 10 56 57 58 Model architecture Our sequence-to-sequence multilingual machine translation model is based on the transformer encoder–decoder architecture . The encoder transforms the source token sequence into a sequence of token embeddings. Then, the decoder attends to the encoder output and autoregressively generates the target sentence token by token. More precisely, the encoder takes the sequence of tokens = ( 1, …, ) and the source language s, and produces a sequence of embeddings Règleman ( 1, …, ), which are then provided to the decoder with the target language t to produce the target tokens = ( 1, …, ) sequentially. In sum, 59 W w wS ℓ H h Pwodwi ℓ V v vT Note that we prefixed the source sequence with the source language, as opposed to the target language, as done in previous work , . Nou te fè sa paske nou te priorite optimize pèfòmans nòt-sòti nan modèl nou an sou nenpòt ki pa gen 200 lang ak yon pri ki pi ba nan pèfòmans supervisé. Empirikman, nou jwenn pèfòmans nòt-sòti yo te afekte negativman lè kondisyone kodaj la sou lang objektif la. Lè sous la se kondisyone sou lang sous sèlman, kodaj la jeneralize pi byen nan pa gen nan lang sous ak objektif ki pa te rankontre pandan fòmasyon an . 10 60 1 Konpayi kondisyone pou tradiksyon machin plizyè lang Yon modèl tradiksyon anpil lang (MMT) itilize menm kapasite modèl pataje yo fòmasyon sou plizyè direksyon tradiksyon nan menm tan an. Malgre ke fè sa ka mennen nan transfè benefisye ant lang ki gen rapò, li kapab tou ajoute risk pou entèferans ant lang ki pa gen rapò , Modèl MoE yo se yon kalite modèl òdinatè kondisyone. , that activate a subset of model parameters per input, as opposed to dense models that activate all model parameters per input. MoE models unlock marked representational capacity while maintaining the same inference and training efficiencies in terms of FLOPs compared with the core dense architecture. 1 61 62 63 Sepandan, lè nou ogmante kapasite modèl la ak pri a òdinatè pou chak ajou, tendans pou lang ki ba oswa trè ba resous pou overfit ogmante, ak sa ki fè pèfòmans yo detwi. Nan seksyon sa a, nou egzamine ki jan nou ka sèvi ak modèl nan Espèks , , , , , to achieve a more optimal trade-off between cross-lingual transfer and interference and improve performance for low-resource languages. 2 3 4 5 6 7 Pwodwi pou egzanp nan espesyalis To build our MoE models, we substitute a quarter of the encoder and decoder feed-forward network layers with MoE layers, each with distinct experts. We followed the Top- Algorithm nan ref. ak voye chak token nan maksimòm = 2 experts. For more details on the training of MoE models, see Supplementary Information . E k 4 k E Ekspè pwodiksyon maske In this proposed regularization strategy, we masked the expert output for a random fraction ( eom) nan token envantè yo. Pou token envantè ak pwodiksyon ekspè ki ba, premye ak / oswa dezyèm ekspè a se efikasman skippe. Kòm montre nan panèl la dezyèm nan Extended data Fig. , we masked both experts for the first token ( 1 in red), chose not to mask any of the expert outputs for the second token ( 2 in blue) and in the final scenario, masked only one expert for the last token ( 3 nan Green). p 2 x x x Curriculum learning for MMT Orthogonal nan metòd regularizasyon modèl-sou tankou droppout, nou eksplore regularize modèl MMT lè l sèvi avèk CL. Nou te pwopoze kòmanse fòmasyon ak pafè ressous segondè an premye, Lè sa a, prezante pafè resous ki ba - ki gen tendans pou overfitting - nan faz ki pita. Pou derive faz yo nan curriculum, nou an premye fòmasyon yon modèl Vanilla MoE (pa gen CL), ki te swiv pa partitioning direksyon yo tradiksyon nan bonè { 1, ... }. If se kantite ajou fòmasyon total, nou prezante chak bin Apre − updates. We based when and what directions to add at every phase of the step when we observed a language pair starting to overfit. Review the step-based CL algorithm in ref. for more on how the directions are partitioned. See Supplementary Information for the list of directions added at each stage. n b bn T Bi nan T Ki nan 64 E2 nan Evalyasyon Automatic evaluation Gen anpil metrik evalyasyon bon jan kalite tradiksyon otomatik egziste, ki gen ladan ki baze sou modèl tankou COMET and BLEURT Malgre ke metrik ki baze sou modèl te montre korelasyon pi bon ak jistifye imen an nan metrik dènye pataje travay nan WMT , they require training and are not easily extendable to a large set of low-resource languages. In this work, we rely on BLEU (and a variant of it) and chrF++. Both measures draw on the idea that translation quality can be quantified based on how similar a machine translation output is compared with that produced by a human translator. 65 66 43 BLEU and spBLEU Blue Score nan has been the standard metric for machine translation evaluation since its inception two decades ago. It measures the overlap between machine and human translations by combining the precision of 1-grams to 4-grams with a brevity penalty. The main disadvantage of BLEU is that it is tokenization-dependent. Efforts such as sacrebleu have taken strides towards standardization, supporting the use of community-standard tokenizers under the hood. However, these tokenizers do not extend to many languages. Reference proposes spBLEU, a BLEU metric based on a standardized SentencePiece model (SPM) covering 101 languages, released alongside FLORES-101. In this work, we provide SPM-200 along with FLORES-200 to enable the measurement of spBLEU. (Our analyses demonstrate that there are minor differences between SPM-200 from FLORES-200 and SPM-100 from FLORES-101 when measuring on the FLORES-101 languages. The major advantage of SPM-200 is that it covers 200 languages. More details on SPM-200 are reported in section 8.1.1 of ref. ). 44 67 41 34 chrF++ Pwoteksyon nan CHRF + overcomes the limitation of the BLEU score, which requires that a sentence can be broken up into word tokens. However, some languages, such as Chinese or Thai, do not use spaces to separate words, and word segmentation tools may not be readily available. There is also a concern about highly agglutinative languages in which BLEU fails to assign any credit to morphological variants. chrF++ overcomes these weaknesses by basing the overlap calculation on character-level nan gram -score ( ranging from 1 to 6) and complementing with word unigrams and bi-grams. In this work, we primarily evaluated using chrF++ using the settings from sacrebleu. However, when comparing with other published work, we used BLEU and spBLEU where appropriate. 38 n F n Metòd pou evalyasyon moun When building machine translation systems for thousands of different language pairs, a core question is which pairs reach certain levels of quality. Therefore, we needed meaningful scores that are comparable across language pairs. XSTS evaluation protocol Nou adapte metòd la XSTS nouvèlman pwopoze Kòzman, XSTS se yon pwotokòl evalyasyon imen ki konsantre sou konsèvasyon vle di sou fluency. Al gade nan detay sou pwotokòl sa a nan enfòmasyon Supplementary . For low-resource languages, translations are usually of poorer quality, and so we focused more on usable (that is, meaning-preserving) translations, even if they are not fully fluent. Compared with Direct Assessment with a 5-point scale (the original direct assessment uses a 100-point scale), it is found that XSTS yields higher inter-annotator agreement XSTS valè chak sentans sous ak tradiksyon machin li yo sou yon skala 5 pwen, nan ki 1 se pi ba ak 5 se pi wo. 48 F 68 47 Kalibrasyon nan Nan lòd yo pèmèt pousantaj enteresan konpare atravè par lang, nou te mande ke chak evalyateur bay evalyasyon lè l sèvi avèk XSTS echèl la sou egzanp la menm seri de parye sentans. Sa a vle identifye anotasyon ki gen yon tendans sistematik yo dwe pi difisil oswa jèn nan pousantaj yo ak korektè pou efè sa a. Anpil kalibrasyon konsiste nan pwodiksyon an nan tradiksyon machin parye ak tradiksyon an referans sèlman nan lang angle. Dapre ki jan evalyasyon yo itilize XSTS echèl la sou sa a seri kalibrasyon, nou ajiste pousantaj yo wo sou travay la evalyasyon aktyèl pou asire konsistans ant evalyasyon yo. Malgre ke travay sa a monoling kalibrasyon pa egzak imite travay la bil Jwenn agregat metrik bon jan kalite moun soti nan plizyè etid Pou jwenn yon metrik bon jan kalite moun agregat pou chak direksyon lang nan yon etid evalyasyon, nou pran pousantaj majorite XSTS (ki se, mwayen-mwayen pousantaj) pou chak sentans ak medyeo sa yo pousantaj majorite sou tout sentans evalye. Nan yon etid done, pousantaj evalyasyon moun agregat pou nenpòt direksyon tradiksyon s nan T se l l where s and t denote lang lan sous ak lang objektif, respektivman; denote pousantaj la XSTS nan th evaluator who evaluates sentences in a given translation direction s → t for a source sentence and a target sentence ; denotes the total number of evaluators who evaluate the (source, translation) sentence pair ( nan ) for translation direction s nan t; is the set of (source, translation) sentence pairs being evaluated for translation direction s → t. l l i l l S T S T l l l l Tout evalyateur nan yon etid done se tou mande yo bay evalyasyon pou tout oswa pati nan yon seri kalibrasyon—. , Done nan th frè sous nan kalibrasyon mete pou yon etid evalyasyon; , , denotes the translated sentence corresponding to , ; and is the number of sentence pairs in the calibration set for an evaluation study. s Ss k k s Ts k SS nan k For each language direction evaluated in a study, we obtained the majority score on the calibration set as follows: kote denote pousantaj la XSTS bay pa th evaluator, for the language direction s → t, in study , pou yon sentans sous and a translated sentence , in the calibration set of the study. i l l s S T To obtain aggregated calibrated XSTS scores on the language direction level, we explored several different calibration methodologies. None of the calibration methods we investigated showed a marked difference in correlation with automated scores, and all calibration methodologies we explored provided superior correlation compared with uncalibrated XSTS scores. For more details on these calibration methodologies, see section 7.2 of ref. . 34 Added toxicity detection for 200 languages Nan seksyon sa a, nou bay plis detay sou definisyon toksisite nou an ak deskripsyon detèktè a (ETOX) ak lis motè ki gen rapò. Kontni toksik Dapre natirite a subjective nan toksisite, definisyon yo nan lang toksik ka varye. Nou enkli atik ki souvan rele lang vulgar oswa profane. (Note ke lang vulgar oswa profane pa toujou toksik. Gen kèk slang komen, pou egzanp, ka konsidere yo vulgar men pa nesesèman toksik). Anplis de sa, nou enkli atik ki asosye ak deskripsyon nan kontni pònografi oswa aksyon seksyèl, kèk ekspresyon odyans souvan itilize ak kèk ekspresyon ki gen rapò ak bullying. Nou enkli tou atik, vulgar oswa pa, ki refere a pati kò ki souvan ki asosye ak pratik seksyèl. The ETOX detector We started with the assumption that general-purpose machine translation systems should remain faithful to the source content and not add any toxic elements during the translation process. We define toxic elements as word tokens or short phrases present in our lists. ETOX identifies added toxicity using the following two criteria: number of toxic items and matched or non-matched toxicity. A toxic item is considered detected if it is present in a line and surrounded by spaces or the start or end of a line. ETOX tracks the number of unique toxic items found in a line but does not count a phrase again if it has multiple occurrences. Matched toxicity indicates that the number of toxic items is the same in both the source and the translated content (that is, no added toxicity). Added toxicity is an instance of non-matched toxicity in which more toxic items are found in the translation output than in the source. For non-segmenting languages or some languages that use complex diacritics, space tokenization is insufficient to distinguish words from one another. In those cases, we used SentencePiece tokenization of both the sentence and toxicity word list. Toxicity-200 lists Lists are based on professional translations from English, which were then heuristically adapted by linguists to better serve the target language. As toxicity is culturally sensitive, attempting to find equivalents in a largely multilingual setting constitutes a challenge when starting from one source language. To address this issue, translators were allowed to forgo translating some of the source items and add more culturally relevant items. Nan vèsyon an orijinal nan lis la Toxicity-200, kantite a medyeval nan atik nan yon lis deteksyon toksisite te 271 atik, pandan y ap kantite atik medyeval la te 143. Sa a ka pi bon mwayen nan tendans santral pase mwayen an, paske lang ak yon rich morfoloji infleksyon konpoze ekstreman outliers (pou egzanp, lis la Czech te gen 2,534 atik ak lis la Polonè te 2,004). lis la pi kout te gen 36 atik, ak pi long la te 6,078. Disponibilite nan done Tout done yo kreye ak deskripsyon nan atik la ak enfòmasyon Supplementè li yo disponib nan GitHub ( ) kòm sa a. Dataset la FLORES-200 gen done evalyasyon ki te tradui pa moun nan 204 lang. Database la NLLB-Seed gen done fòmasyon semen tradiksyon moun nan 39 lang (Informasyon Supplémentaire ). The NLLB-MD database contains human-translated seed data in different domains in six languages to assess generalization (Supplementary Information ). Databas la Toxicity-200 gen lis tèks pou detekte toksisite nan 200 lang. Database Mined bitext gen done sou entènèt piblik ki disponib pou 148 pafè lang angle-centric ak 1,465 pafè lang non-angle-centric. Done piblik ki disponib ki itilize yo fòmasyon modèl NLLB ak referans pou download done yo ki nan tab Supplementary . https://github.com/facebookresearch/fairseq/tree/nllb 69 I J 2 Kòd Disponibilite Pou fè travay nou disponib nan kominote a, nou bay modèl sa yo ak sipò kòd kòm resous ki disponib pou itilize ki pa komèsyal, ki disponib nan GitHub ( ) se sa ki anba a. Modèl yo tradiksyon kouvri 200 lang; modèl yo NLLB vini nan plizyè gwosè (54.5B MoE, 3.3B ak 1.3B Dense, ak 1.3B ak 600M destile). Modèl yo idantite lang gen plis pase 200 lang. LASER3 gen ladan encoders sentans pou identifye aligned bitex pou 148 lang. Stopes konsiste nan yon bibliyotèk min done ki ka itilize pou pwosesis ak netwaye done monolingual, ki te swiv pa kreyasyon an nan aligned bitex. Skript yo rekonèt done fòmasyon nou yo ak fòmasyon ak jenerasyon skript yo replike modèl nou yo se tou enkli. https://github.com/facebookresearch/fairseq/tree/nllb 69 References Fan, A. et al. Beyond English-centric multilingual machine translation. , 1–48 (2021). J. Mach. Learn. Res 22 MathSciNet Google Scholar Du, N. et al. GlaM: efficient scaling of language models with mixture-of-experts. In Vol. 162, 5547–5569 (PMLR, 2022). Proceedings of the 39th International Conference on Machine Learning Hwang, C. et al. Tutel: adaptive mixture-of-experts at scale. In (MLSys, 2023). 6th Conference on Machine Learning and Systems Lepikhin, D. et al. GShard: skalasyon modèl gwo ak konvansyonèl kalkil ak otomatik sharding. Nan Konvansyon Entènasyonal sou Representasyon aprantisaj (ICLR, 2021). Lewis, M., Bhosale, S., Dettmers, T., Goyal, N. & Zettlemoyer, L. Layers BASE: simplifying training of large, sparse models. In Proc. 38th International Conference on Machine Learning Vol. 139, 6265–6274 (PMLR, 2021). Shazeer, N. et al. Yon rezo neural gwo eksepsyonèlman: yon gwosè rasanbleman-de-ekspè. In Proc. 2017 International Conference on Learning Representations (ICLR) 1-19 (ICLR, 2017). Zoph, B. et al. ST-MoE: konsepsyon ki estab ak transfere modèl espesyalize ekspè. Preprint nan https://arxiv.org/abs/2202.08906 (2022). Zoph, B., Yuret, D., May, J. & Knight, K. Transfer learning for low-resource neural machine translation. In (eds Su, J. et al.) 1568–1575 (Association for Computational Linguistics, 2016). Proc. 2016 Conference on Empirical Methods in Natural Language Processing Nguyen, T. Q. & Chiang, D. Transfer learning across low-resource, related languages for neural machine translation. In Vol. 2 (eds Kondrak, G. & Watanabe, T.) 296–301 (Asian Federation of Natural Language Processing, 2017). Proc. Eighth International Joint Conference on Natural Language Processing Arivazhagan, N. et al. Massively multilingual neural machine translation in the wild: findings and challenges. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Vol. 1, 3874–3884 (Asosyasyon pou Linguistik Computational, 2019). Zhang, B., Williams, P., Titov, I. & Sennrich, R. amelyore masivman multilingual neural machin tradiksyon ak zèb-sòti tradiksyon. Nan 58yèm Reyinyon Annual la nan Association for Computational Linguistics (eds Jurafsky, D. et al.) 1628-1639 (ACL, 2020). Tran, C. et al. Facebook AI a WMT21 nouvèl tradiksyon travay soumission. In Proc. Sixth Konferans sou machin tradiksyon (eds Barrault, L.) 205–215 (ACL, 2021); https://aclanthology.org/2021.wmt-1.19. Orife, I. et al. Masakhane – machine translation for Africa. Preprint at (2020). https://arxiv.org/abs/2003.11529 Kuwanto, G. et al. Low-resous machin tradiksyon fòmasyon kou apwopriye pou lang ki ba-resous. Preprint nan https://arxiv.org/abs/2103.13272 (2021). Nekoto, W. et al. Rechèch patisipasyon pou tradiksyon machin ki ba resous: yon etid ka nan lang Afriken. Nan Retounen nan Association for Computational Linguistics: EMNLP 2020 (eds Cohn, T. et al.) 2144–2160 (ACL, 2020). Karakanta, A., Dehdari, J. & van Genabith, J. Neural machine translation for low-resource languages without parallel corpora. , 167–189 (2018). Mach. Transl. 32 Article Google Scholar Bañón, M. et al. ParaCrawl: web-scale acquisition of parallel corpora. In (eds Jurafsky, D. et al.) 4555–4567 (ACL, 2020). Proc. 58th Annual Meeting of the Association for Computational Linguistics Schwenk, H. et al. CCMatrix: mining billions of high-quality parallel sentences on the web. In Vol. 1 (eds Zong, C. et al.) 6490–6500 (ACL, 2021). Proc. 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing Ramesh, G. et al. : the largest publicly available parallel corpora collection for 11 Indic languages. , 145–162 (2022). Samanantar Trans. Assoc. Comput. Linguist. 10 Article Google Scholar Kreutzer, J. et al. Quality at a glance: an audit of web-crawled multilingual datasets. , 50–72 (2022). Trans. Assoc. Comput. Linguist. 10 Article Google Scholar Heffernan, K., Çelebi, O. & Schwenk, H. Bitext mining using distilled sentence representations for low-resource languages. Preprint at (2022). https://arxiv.org/abs/2205.12654 Gowda, T., Zhang, Z., Mattmann, C. & May, J. Many-to-English machine translation tools, data, and pre-trained models. In Proc. 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing: System Demonstrations (eds Ji, H. et al.) 306–316 (ACL, 2021). McCarthy, A. D. et al. The Johns Hopkins University Bible corpus: 1600+ lang pou eksplorasyon tipolojik. Nan pwosesis la 12th Lingvis Resous ak evalyasyon Konferans (eds Calzolari, N. et al.) 2884–2892 (European Language Resources Association, 2020); https://aclanthology.org/2020.lrec-1.352. McNamee, P. Language identification: a solved problem suitable for undergraduate instruction. , 94–101 (2005). J. Comput. Sci. Coll. 20 Google Scholar Abadji, J., Suárez, P. J. O., Romary, L. & Sagot, B. Nan direksyon pou yon korpus crawled plizyè lang ki oryante dokiman pi byen. Preprint nan https://arxiv.org/abs/2201.06642 (2022). Widdows, D. & Brew, C. Identifikasyon lang ak yon klasifikasyon rank reciprocal. Preprint nan https://arxiv.org/abs/2109.09862 (2021). Goutte, C., Léger, S., Malmasi, S. & Zampieri, M. Diskriminasyon nan lang menm jan an: evalyasyon ak eksplorasyon. Preprint nan http://arxiv.org/abs/1610.00031 (2016). Jauhiainen, T., Lindén, K. & Jauhiainen, H. Evaluation of language identification methods using 285 languages. In (eds. Tiedemann, J. & Tahmasebi, N.) 183–191 (2017). Proc. 21st Nordic Conference on Computational Linguistics Grave, É., Bojanowski, P., Gupta, P., Joulin, A. & Mikolov, T. Aprann vètikulè motè pou 157 lang. In Proc. 11th International Conference on Language Resources and Evaluation (LREC 2018) (eds Calzolari, N. et al.) (ELRA, 2018). Dunn, J. Mapping languages: the corpus of global language use. , 999–1018 (2020). Lang. Resour. Eval. 54 Article Google Scholar Brown, R. D. Non-linear mapping for improved identification of 1300+ languages. In (eds Moschitti, A. et al.) 627–632 (ACL, 2014). Proc. 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) Caswell, I., Breiner, T., van Esch, D. & Bapna, A. ID lang nan bèt kay: repitasyon inattended sou wout la nan yon mil-language web tèks corpus. In Proc. 28th International Conference on Computational Linguistics (eds Scott, D. et al.) 6588-6608 (International Committee on Computational Linguistics, 2020); https://aclanthology.org/2020.coling-main.579. Joulin, A., Grave, E., Bojanowski, P. & Mikolov, T. Bag nan trik pou klasifikasyon tèks efikas. Nan Proc. 15th Konferans nan Chapter Ewopeyen an nan Association for Computational Linguistics Vol. 2 (eds Lapata, M. et al.) 427-431 (ACL, 2017). NLLB Team et al. No language left behind: scaling human-centered machine translation. Preprint at (2022). https://arxiv.org/abs/2207.04672 Koehn, P. & Knowles, R. Six challenges for neural machine translation. In (eds Luong, T. et al.) 28–39 (ACL, 2017). Proc. First Workshop on Neural Machine Translation Artetxe, M. & Schwenk, H. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. , 597–610 (2019). Trans. Assoc. Comput. Linguist. 7 Article Google Scholar Sennrich, R., Haddow, B. & Birch, A. Improving neural machine translation models with monolingual data. In Vol. 1 (eds Erk, K. & Smith, N. A.) 86–96 (ACL, 2016). Proc. 54th Annual Meeting of the Association for Computational Linguistics (ACL) Popović, M. chrf++: words helping character n-grams. In Vol. 2 (eds Bojar, O. et al.) 612–618 (ACL, 2017). Proc. Second Conference on Machine Translation Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J. & Wojna, Z. Rethinking the inception architecture for computer vision. Nan IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2818–2826 (IEEE, 2016). Liu, R., Kim, Y. J., Muzio, A., Mozafari, B. & Awadalla, H. H. Gating drop-out: kominikasyon-efikas reglemantasyon pou transformatè ralent aktive. Nan pwosesis nan 39yèm Entènasyonal Konferans sou Machine Learning (PMLR, 2022). Goyal, N. et al. The Flores-101 evaluation benchmark for low-resource and multilingual machine translation. , 522–538 (2022). Trans. Assoc. Comput. Linguist. 10 Article Google Scholar Wang, H. et al. DeepNet: Scaling transformateurs nan 1,000 layers. Nan IEEE Transaksyon sou Analiz Patofòn ak Enfòmasyon machin https://doi.org/10.1109/TPAMI.2024.3386927 (IEEE, 2024) Freitag, M. et al. Results of the WMT21 metrics shared task: evaluating metrics with expert-based human evaluations on TED and news domain. In (eds Barrault, L. et al.) 733–774 (ACL, 2021); . Proc. Sixth Conference on Machine Translation https://aclanthology.org/2021.wmt-1.73 Papineni, K., Roukos, S., Ward, T. & Zhu, W.-J. Bleu: yon metòd pou evalyasyon otomatik nan tradiksyon machin. Nan Proc. 40yèm reyinyon ane a nan Association for Computational Linguistics (eds Isabelle, P. et al.) 311-318 (ACL, 2002). Akhbardeh, F. et al. Rezilta yo nan konvèsyon an 2021 sou tradiksyon machin (WMT21). Nan pwosesis la Sixth Konferans sou tradiksyon machin (eds Barrault, L. et al.) 1–88 (ACL, 2021); https://aclanthology.org/2021.wmt-1.1. Kocmi, T. et al. Pou livrezon oswa pa livrezon: yon evalyasyon konplè nan metrik otomatik pou tradiksyon machin. In Proc. Sixth Conference on Machine Translation (eds Barrault, L. et al.) 478-494 (ACL, 2021). Licht, D. et al. Consistent human evaluation of machine translation across language pairs. In Vol. 1, 309–321 (Association for Machine Translation in the Americas, 2022). Proceedings of the 15th Biennial Conference of the Association for Machine Translation in the Americas Agirre, E. et al. SemEval-2012 task 6: a pilot on semantic textual similarity. In Vols 1–2 (eds Aggire, E. et al.) 385–393 (ACL, 2012). Proc. *SEM 2012: The First Joint Conference on Lexical and Computational Semantics Kusters, R. et al. Interdisciplinary research in artificial intelligence: Challenges and opportunities. , 577974 (2020). Front. Big Data 3 Article PubMed PubMed Central Google Scholar Wang, S., Cooper, N., Eby, M. & Jo, E. S. From human-centered to social-centered artificial intelligence: assessing ChatGPT’s impact through disruptive events. Preprint at (2023). https://arxiv.org/abs/2306.00227 Bojanowski, P., Grave, E., Joulin, A. & Mikolov, T. Enriching vètikòl motè ak onformation nan sousmotè. Trans. Assoc. Comput. Linguist. 5, 135–146 (2017). Atik Google Scholar Tiedemann, J. Parallel done, zouti ak interfaces nan OPUS. In Proc. Eighth International Conference on Language Resources and Evaluation (eds Calzolari, N. et al.) 2214–2218 (ACL, 2012). Artetxe, M. & Schwenk, H. Margin-based parallel corpus mining with multilingual sentence embeddings. In (eds Korhonen, A.) 3197–3203 (ACL, 2019). Proc. 57th Annual Meeting of the Association for Computational Linguistics Bahdanau, D., Cho, K. H. & Bengio, Y. Neural machine translation by jointly learning to align and translate. In (ICLR, 2015). Proc. of the 3rd International Conference on Learning Representations Kudo, T. & Richardson, J. SentencePiece: Yon senp ak lang indépendant subword tokenizer ak detokenizer pou Neural Text Processing. Nan Proc. 2018 Konferans sou Metòd Empirik nan pwosesis lang natirèl, EMNLP 2018 (eds Blanco, E. & Lu, W.) 66–71 (ACL, 2018); https://doi.org/10.18653/v1/d18-2012. Gu, J., Hassan, H., Devlin, J. & Li, V. O. Universal Neural Machine Translation for Extremely Low Resource Languages. Nan pwosesis la. 2018 Konferans nan Chapter nan North American Association for Computational Linguistics: Human Language Technologies Vol. 1 (eds Walker, M. et al.) 344–354 (ACL, 2018); https://aclanthology.org/N18-1032. Wang, X., Pham, H., Arthur, P. & Neubig, G. Multilingual neural machine translation with soft decoupled encoding. Preprint at (2019). https://arxiv.org/abs/1902.03499 Rajab, J. Effect of tokenisation strategies for low-resourced Southern African languages. In (ICLR, 2022). 3rd Workshop on African Natural Language Processing Vaswani, A. et al. Atansyon se tout sa ou bezwen. Nan Proc. 31st Konferans sou Sistèm Processing Neural enfòmasyon 5998-6008 (NIPS, 2017). Johnson, M. et al. Google’s multilingual neural machine translation system: enabling zero-shot translation. , 339–351 (2017). Trans. Assoc. Comput. Linguist. 5 Article Google Scholar Conneau, A. et al. Enfòmasyon nan reprezantasyon entèlingual ki pa tcheke nan mas la. Nan 58yèm Reyinyon Annual la nan Association for Computational Linguistics (eds Jurafsky, D. et al.) 8440-8451 (ACL, 2020). Bengio, Y., Léonard, N. & Courville, A. C. Estimating or propagating gradients through stochastic neurons for conditional computation. Preprint at (2013). http://arxiv.org/abs/1308.3432 Almahairi, A. et al. Dinamik rezo kapasite. Nan pwosesis 33yèm Entènasyonal Konferans sou Entènasyonal Konferans sou Machine Learning Vol. 48, 2091–2100 (PMLR, 2016). Elbayad, M., Sun, A. & Bhosale, S. Fixing MoE over-fitting on low-resource languages in multilingual machine translation. Nan konklizyon an nan Association for Computational Linguistics: ACL 2023 (eds Rogers, A. et al.) 14237–14253 (ACL, 2023); https://aclanthology.org/2023.findings-acl.897. Rei, R., Stewart, C., Farinha, A. C. & Lavie, A. COMET: a neural framework for MT evaluation. In (eds Webber, B. et al.) 2685–2702 (ACL, 2020). Proc. 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) Sellam, T., Das, D. & Parikh, A. BLEURT: learning robust metrics for text generation. In (eds Jurafsky, D. et al.) 7881–7892 (ACL, 2020). Proc. 58th Annual Meeting of the Association for Computational Linguistics Post, M. Yon apèl pou klè nan rapòte pousantaj BLEU. Nan pwosesis la. Third Conference on Machine Translation: Research Papers (eds Bojar, O. et al.) 186–191 (ACL, 2018); https://aclanthology.org/W18-6319. Graham, Y., Baldwin, T., Moffat, A. & Zobel, J. Kontinyèl mas nan evalyasyon moun nan tradiksyon machin. Nan 7th Lingvistic Annotation Workshop ak Interoperability ak Discourse 33-41 (eds Graham, Y. et al.) (ACL, 2013). NLLB Team et al. No Language Left Behind: scaling human-centered machine translation. (2022). GitHub https://github.com/facebookresearch/fairseq/tree/nllb rekonesans Nou evalye patnè yo nan FLLL ak patnè yo evalye kòmansman yo kreye pou kontribisyon yo nan pwojè a: C. Baziotis, D. Dua, A. Guo, O. Ignat, A. Kamran, T. Mohiuddin, A. N. Rubungo, S. Sun, S. Tan, H. Xu, S. Wu ak Y. Zhang. Nou se grate a tout anplwaye Wikimedia Fondasyon an ak volontè ki te travay ak nou ak bay feedback ki itil sou pwojè nou an. Nou mèsi V. Chaudhary pou ede l 'nan metòd ALTI+ a; G. Chang, C.-Joro, Wu ak R. Raghavendra pou ede l 'nan konvèsyon an rapid tèks nan tout lang FLORES-200; M. Diab pou travay li sou XSTS; A Papye sa a se disponib nan natirèl anba lisans CC by 4.0 Deed (Attribution 4.0 entènasyonal). Papye sa a se under CC by 4.0 Deed (Attribution 4.0 International) license. Disponib nan natirèl