Jinsi Microsoft ilichangia AI ya 270M-Pair ili kuwezesha utafutaji wa akili

Waandishi wa: Liang Wang (Microsoft Corporation) Nan Yang (Microsoft Corporation) Xiaolong Huang (Microsoft Corporation) Binxing Jiao (Microsoft Corporation) Linjun Yang (Microsoft Corporation) Daxin Jiang (Microsoft Corporation) Rangan Majumder (Microsoft Corporation) Furu Wei (Microsoft Corporation) Waandishi wa: Liang Wang (mtaalamu wa Microsoft) Kwa mujibu wa kampuni ya Microsoft (Microsoft Corporation) Xiaolong Huang (Ushirika wa Microsoft) Binxing Jiao (Ushirika wa Microsoft) Linjun Yang (Ushirika wa Microsoft) Daxin Jiang (Ushirika wa Microsoft) Rangan Majumder (Ushirikiano wa Microsoft) Furu Wei (Ushirika wa Microsoft) Abstract ya Makala hii inahusu E5 E5 inaweza kutumika kwa urahisi kama mfano wa kuingizwa kwa madhumuni makubwa ya maandishi ambayo huchukua kazi nyingi. Mfano huu unajitolea kwa njia tofauti na ishara za ufuatiliaji dhaifu kutoka kwa seti yetu ya data ya kipekee ya kipekee ya maandishi (yaani CCPairs). E5 inaweza kutumika kwa urahisi kama mfano wa kuingizwa kwa madhumuni ya jumla kwa kazi yoyote ambayo inahitaji utambulisho wa vektor moja wa maandishi kama vile kutafuta, kukusanya, na kuagiza, na kufikia utendaji mkubwa katika mipangilio ya null-shot na fine-tuned. Tunachukua tathmini nyingi juu ya seti 56 ya data kutoka kwa benchmarks ya BEIR na MTEB. Kwa mipangilio ya null-shot, E5 1 1 Maelezo ya Uingizaji wa maandishi ni maonyesho ya vektor ya kiwango cha chini kwa maandishi ya urefu wowote na hucheza jukumu muhimu katika kazi nyingi za NLP kama vile upatikanaji wa kiwango kikubwa. ikilinganishwa na maonyesho ya kiwango cha juu na cha kutosha kama TF-IDF, uingizaji wa maandishi una uwezo wa kushinda tatizo la upungufu wa sauti na kuwezesha upatikanaji wa ufanisi na ufanisi kati ya maandishi. Pia hutoa interface ya kila aina ambayo inaweza kutumika kwa urahisi na maombi ya baadaye. Wakati mifano ya lugha ya awali ya mafunzo kama BERT [ (Kwa mujibu wa TFF) ] inaweza kuzalisha maonyesho ya maandishi yanayoweza kuhamishwa, sio bora kwa kazi kama vile kutafuta na kufanana na maandishi ambapo kuingizwa kwa vektor moja ya maandishi ni zaidi ya kutaka kutokana na ufanisi na usahihi wake. Ili kupata kuingizwa kwa maandishi bora, kujifunza kinyume ni mara nyingi mfumo wa kwenda kuimarisha maonyesho ya ngazi ya mfululizo kutoka kwa wanandoa wa maandishi. Kwenye mstari huu wa utafiti, baadhi ya kazi ni kuelekezwa kufundisha kuingizwa kwa kazi maalum. Kwa mfano, GTR [ Mstari wa 5 ( ] Fine-tune mifano kabla ya mafunzo na datasets kusimamiwa kujifunza embeddings customized kwa ajili ya kupokea njia na semantic maandishi sawa, kwa mujibu. kazi nyingine kujifunza embeddings usimamizi kutoka pairs maandishi kujengwa moja kwa moja. mbinu ya kawaida ya kujenga pairs maandishi ni pamoja na Inverse Close Task (ICT) [ (Kwa mfano, uchafuzi wa baridi) ] na maandishi ya jirani ya spans [ Ingawa data hiyo ya synthetic ni ya idadi isiyo na kikomo, mara nyingi ni mbaya katika ubora na uingizaji uliotokana hauwezi kulinganisha na utendaji wa baseline classic BM25 bila kurekebisha zaidi 17 7 43 44 9 28 41 [Tafakari ya 40] Katika kazi hii, tunajifunza ubora wa ubora wa maandishi ya maana ya jumla inayoitwa E5, Mb ya Dings kutoka Bidir Kijani Msimamizi wa R Maonyesho. E5 ina lengo la kutoa uingizaji wa maandishi wenye nguvu nje ya shamba unaofaa kwa kazi yoyote ambayo inahitaji maonyesho ya vektor moja katika mipangilio ya null-shot au fine-tuned. Ili kufikia lengo hili, badala ya kutegemea data zilizo na alama mdogo au viungo vya maandishi vya ubora wa chini, tunajifunza uingizaji wa E5 kutoka CCPairs, seti ya data ya viungo vya maandishi ya kiwango cha wavuti iliyohifadhi ishara za mafunzo tofauti. Tunajenga seti ya data ya CCPairs kwa kuunganisha vyanzo mbalimbali vya data vya nusu kama vile CommunityQA, Crawl ya kawaida na makala za kisayansi, na kutekeleza uchapishaji wa kikatili na filters ya ufuatiliaji [ ] kuboresha ubora wa data. Tunachagua mapishi rahisi ya kujifunza kinyume na kutumia uchafuzi wa batch na ukubwa wa batch kubwa ili kufundisha mfano wetu. majaribio makubwa juu ya benchmarks zote za BEIR na MTEB yanaonyesha ufanisi wa mbinu iliyopendekezwa. ], E5 ni mfano wa kwanza wa kushinda kiwango cha juu cha msingi cha BM25 bila kutumia data yoyote iliyotajwa. Wakati wa kurekebishwa kwa datasets zilizotajwa, utendaji unaweza kuboreshwa zaidi. Matokeo juu ya datasets 56 kutoka benchmark iliyotolewa hivi karibuni ya MTEB [ kuonyesha kwamba E5base yetu ni ushindani dhidi ya GTRxxl na Sentence-T5xxl, ambayo ina vigezo 40x zaidi. E E E E E 15 53 40 2 Kazi zinazohusiana Kulikuwa na maslahi ya muda mrefu katika kubadilisha maandishi katika kuingizwa kwa kiwango cha chini. Kazi ya awali ni pamoja na Indexing ya Semantic ya Latent (LSA) [ [Tafakari ya Mwisho wa Mwisho wa Mwisho wa Mwisho] ]. LSA utilizes the decomposition of a word-document co-occurrence matrix to generate document embeddings, while LDA adopts probabilistic graphical models to learn topic distributions. kuonyesha kwamba wastani wa wastani wa maneno ya vektor [ ] inaweza kuwa msingi mzuri kwa uingiliaji wa sentence. 16 3 ya Arora ya al. 38 Kutokana na maendeleo ya mifano ya lugha zilizoandaliwa kabla [ ya ya (Kwa mfano, viwango vya juu vya viwango vya juu vya viwango vya juu vya viwango vya juu vya viwango vya juu vya viwango vya juu vya viwango vya juu vya viwango vya juu vya viwango vya juu vya viwango vya juu vya viwango vya juu vya viwango vya juu vya viwango vya juu vya viwango vya juu vya viwango vya juu vya viwango vya juu). (Kwa mfano wa Marmaray) Mifano kama ilivyoelezwa katika Ibara ndogo ya [ Kwa mfano, kwa mfano [ Mstari wa 5 [ (Kwa mfano wa SPSS) ] moja kwa moja fine-tune lugha mifano ya kutolewa uingizaji endelevu. utafiti wengi ni kuzingatia maandishi fupi na hivyo hutumia neno "kuingizwa kwa kauli." Kwa nyaraka ndefu, ni bado swali la utafiti wazi kama uingizaji kwa urefu imara inaweza coding habari zote. kupoteza kinyume maarufu na SimCLR [ ] inaonekana kuwa na ufanisi zaidi kuliko kupoteza kwa misingi ya usimamizi [ ya ] for embeddings. LaBSE [ Kwa upande wa laser [ (Kwa mfano wa kioo) Kuendeleza zaidi kwa matukio mengi ya lugha na multi-modal kwa kutumia maneno ya parallel na pande za picha na maandishi. 17 35 48 6 8 49 22 44 39 10 49 14 20 2 47 Njia nyingine ni kubuni majukumu ya mafunzo ya awali ya kujitegemea kwa ajili ya kuchanganya na kupata maandishi. [ ] inapendekeza kazi inayojulikana ya reverse cloze (ICT), ambapo sentensi ya random ndani ya sehemu inachaguliwa kama swali la uongo na wengine hutumiwa kama sampuli chanya. ] inaonyesha kwamba uingizaji wa random kwa kuongeza data ni ufanisi zaidi kuliko ICT katika mfululizo wa kazi za kupokea habari za shots. Tumia maandiko ya jirani kama chanya na kuongeza ukubwa wa mfano hadi 175B. Oguz et al. Kuanzisha mlinzi juu ya mlinzi juu ya mlinzi juu ya mlinzi juu ya mlinzi juu ya mlinzi. [ ] hufundisha retriever ya kutosha kwa kutibu BM25 kama mfano wa mwalimu. Ingawa mbinu zilizotajwa hapo juu zinaweza kupata ishara nyingi za usimamizi kwa urahisi, data hiyo ya synthetic huwa na ubora mdogo. Matokeo juu ya benchmark ya BEIR [ ] kuonyesha kwamba wanakabiliana na utendaji wa BM25 ikiwa hawajatengenezwa zaidi kwenye seti za data zilizotajwa. 9 28 41 [Tafakari ya 45] 11 53 Tathmini na tafsiri ya uingizaji wa maandishi pia si ya kipekee. viwango vingi vinathamini ubora wa uingizaji kupitia utendaji wa kazi ya chini. Kwa mfano, SentEval [ ] inatumia uchunguzi wa linear na mkusanyiko wa data ya semantic textual similarity (STS), wakati benchmark ya BEIR [ ] inazingatia matukio ya upatikanaji wa habari ya shots. Benchmark ya MTEB iliyotolewa hivi karibuni [ ] inachanganya mitambo ya data ya 56 ambayo hufunika majukumu 8 na lugha 112. Majaribio yanaonyesha kwamba hakuna mfano anayeweza kufikia matokeo ya kisasa juu ya majukumu yote ya kuingiza. Katika makala hii, hatuwezi kutumia SentEval Toolkit kwa sababu mipangilio yake ya kutafuta linear inategemea hyperparameters ya uboreshaji. 13 53 40 zaidi yanayohusiana na kazi yetu ni mfululizo wa juhudi za jumuiya kufundisha kuingizwa kwa mkusanyiko wa data zilizotajwa na zilizokusanywa moja kwa moja. Katika makala hii, tunathibitisha kuwa inawezekana kufundisha kuingizwa kwa ubora kwa kutumia tu mafunzo ya awali ya kujitegemea. Katika suala la matokeo ya benchmark, mfano wetu anaweza kufikia utendaji bora wakati wa kurekebisha data chini ya alama. Mabadiliko ya hukumu 2 3 CCPairs: Mkusanyiko mkubwa wa Dataset ya Pair ya maandishi Ubora na utofauti wa data ni muhimu kwa mafunzo ya kuingizwa kwa maandishi kwa madhumuni ya jumla. Katika kazi hii, tunazindua na kukusanya CCPairs, mkusanyiko mkubwa wa data ya jumla ya maandishi ya ubora kutoka vyanzo vya wavuti ambavyo hutoa ishara mbalimbali za mafunzo zinazoweza kuhamishwa vizuri kwa idadi kubwa ya kazi. Mipango ya data kubwa ya ubora kama vile C4 [ (Kwa upande wa CCM) ] ni muhimu kwa mafanikio ya lugha mfano pre-kufundisha na mashine tafsiri. Kwa kujifunza maandishi embeddings, kazi zilizopo au kutumia mdogo kiwango cha binadamu anotated data kama NLI [ (Kwa mfano wa Marmaray) au kuchukua heuristics kama vile random cropping [ kutafuta ishara za ufuatiliaji wa kiwango kikubwa lakini kubwa sana. Harvesting semi-structured data sources 48 51 22 8 28 Kwa upande mwingine, sisi kuratibu orodha ya viungo vya data CCPairs ( olossal Maelezo ya Lean (Kwa mujibu wa kifungu cha 4 cha Sheria ya Kodi ya Ongezeko la Biashara na Uwekezaji wa Biashara) ( ya ) kuelezea mfululizo wa maandishi uliojumuisha query and a passage Hapa tunatumia » ” kuelezea mfululizo wa maneno ya urefu wowote, ambayo inaweza kuwa sentensi fupi, paragraph, au hati ndefu. , (maswali, majibu ya juu) wanandoa kutoka Stackexchange , (jina la kiungo + kichwa cha sehemu, kifungu) wanandoa kutoka Wikipedia ya Kiingereza, (jina, uchapishaji) na wanandoa wa maagizo kutoka makala za kisayansi [ [Hii ni moja ya mbinu zilizotumika kwa ajili ya kuzuia maumivu ya nywele. Tovuti na vyanzo mbalimbali vya habari. C C Pairs q p q p Mlango wa 3 4 36 5 Sisi tu kuingiza vyanzo data ambazo zinaweza kupatikana moja kwa moja, na baadhi ya subset ni moja kwa moja kutumika tena kutoka datasets zilizopo. sheria rahisi heuristic ni kutumika kwa ajili ya filters data kutoka Reddit na Common Crawl. Kwa mfano, sisi kuondoa maoni Reddit ambayo ni muda mrefu sana ( 4096 wahusika) au kupokea alama chini ya 1, na kuondoa sehemu kutoka kwa kurasa za wavuti na upumbavu mkubwa [ ]. Baada ya uchapishaji wa awali, sisi hatimaye na ∼ 1 *.*3 bilioni ya wanandoa wa maandishi, wengi wao ni kutoka Reddit na Common Crawl. Kwa maelezo zaidi na mifano, tafadhali angalia Kiambatisho > 60 A. Ili kuboresha zaidi ubora wa data na kufanya gharama za mafunzo zinaweza kusimamiwa, tunapendekeza mbinu ya uchapishaji wa data kulingana na ufuatiliaji: mfano unafundishwa kwanza juu ya wanandoa wa maandishi ya 1 * * 3B, na kisha hutumiwa kuweka kila wanandoa dhidi ya pool ya sehemu za random ya milioni 1. Kwa maneno mengine, utabiri wa mfano unapaswa kuwa sawa na alama za mafunzo. = 2 kulingana na uchunguzi wa mikono wa ubora wa data. Baada ya hatua hii, sisi hatimaye na ∼ 270M pairs ya maandishi kwa ajili ya mapema ya mafunzo. Consistency-based filter k k Intuition kwa mbinu hii inatokana na tabia ya kumbukumbu ya mitandao ya neural [ ]: wakati wa mafunzo juu ya datasets gumu, mitandao ya neural huwa kumbukumbu labels safi kwanza na kisha hatua kwa hatua juu ya labels gumu. mbinu sawa [ ya ya Pia inawezekana kutumia filter hii iteratively, tutaacha kwa ajili ya kazi ya baadaye. 19 42 15 23 4 Method Uingizaji wetu unaweza kufundishwa na wanandoa tu wa maandishi wasio na alama kutoka kwa CCPairs na mafunzo ya awali ya kupinga. Tathmini ya pili ya hatua juu ya mikutano madogo ya data yenye alama ya juu inaweza kufanywa ili kuongeza zaidi ubora wa uingizaji uliofanywa. kwa ajili ya kutafakari. 1 4.1 Mafunzo ya awali ya kinyume na data zisizojulikana Kwa mujibu wa jarida la tiba ya sayansi, sensa zilizopo katika mkono huo bandia hutumiwa kutuma ishara moja kwa moja katika mwili. ( ) ya , tunashirikisha orodha ya sehemu mbaya { ya =1 for the -th mfano. Kisha InfoNCE kupoteza kinyume cha Ni kama ifuatavyo: kwa ajili ya Pi n p M ya i [Tafakari ya 10] ambapo ( ) ni kazi ya kugharamia kati ya query Kwa njia ya parameterized by Kufuatia usanifu maarufu wa biencoder, tunatumia encoder ya Transformer iliyohitimu na jumla ya jumla juu ya kiwango cha output ili kupata uingizaji wa maandishi wa ukubwa wa fikra na ya Kiwango ni sawa na cosine kupima kwa hyperparameter ya joto : wa ya P, P q p θ E q E p τ θ θ ambapo imewekwa kwa 0.01 katika majaribio yetu kwa default. Tunatumia encoder iliyoshirikiwa kwa maandishi yote ya kuingia na kuvunja utambulisho kwa kuongeza alama mbili za prefix na ya ya na ya Kwa baadhi ya vyanzo vya data, kama vile wanandoa wa quote, haijulikani ni upande gani unapaswa kuwa swali, sisi randomly kuchagua moja kwa urahisi. Mipango hii asymmetric inaonekana kuwa muhimu kwa baadhi ya kazi ya kupata ambapo kuna paraphrases ya swali katika corpus ya lengo. τ “query:” Kwa njia ya: q d Jambo jingine muhimu kwa mafunzo ya kinyuklia ni jinsi ya kuchagua sampuli mbaya. Hapa tunachagua kutumia mabadiliko ya batch [ ], ambapo mistari kutoka kwa wanandoa wengine katika mfululizo hufanya kazi kama sampuli hasi. Tunaona kwamba mkakati huu rahisi huwezesha mafunzo ya imara zaidi na hufanya mbinu kama vile MoCo wakati ukubwa wa batch ni kubwa ya kutosha. 10 [25] 4.2 Fine-tuning na Data zilizotajwa Ingawa mafunzo ya mapema ya CCPairs hutoa msingi imara kwa kuingizwa kwa madhumuni ya jumla, mafunzo zaidi juu ya data iliyotajwa yanaweza kuingiza maarifa ya kibinadamu katika mfano ili kuongeza utendaji. Ingawa seti hizi za data ni ndogo, kazi zilizopo zinaweza kuingizwa. ya ] wamegundua kuwa udhibiti wa juu unaoongozwa unaongoza kwa faida ya utendaji. Katika makala hii, sisi kuchagua mafunzo zaidi na mchanganyiko wa datasets 3: NLI (Natural Language Inference), MS-MARCO passage ranking dataset [ (Hii ni kwa mujibu wa NQ (Natural Questions) data set. , Empirically, kazi kama STS (Semantic Textual Similarity) na mtihani linear faida kutoka data NLI, wakati MS-MARCO na NQ datasets kuhamisha vizuri kwa kazi za kupata. 43 44 6 8 30 32 Kujenga juu ya mazoezi ya mafunzo ya state-of-the-art dense retrievers [ ya ], tunatumia negatives ngumu zilizopigwa na destillation ya ujuzi kutoka kwa mfano wa mwalimu wa cross-encoder (CE) kwa datasets ya MS-MARCO na NQ. Kwa datasets ya NLI, maneno ya upinzani yanahesabiwa kama negatives ngumu. kazi ya kupoteza ni interpolation linear kati ya kupoteza ya kinyume Tazama kwa makala ngumu na divergent ya KL KL kwa destillation soft labels kutoka mfano wa mwalimu. 50 58 L D ambapo Mimi na stu ni uwezekano wa mfano wa mwalimu wa mstari wa encoder na mfano wetu wa mwanafunzi. ni hyperparameter kwa usawa wa kazi mbili za kupoteza. Kumbuka ni sawa na katika usawa p p α L 1. 4.3 Maombi kwa kazi za kuingiza maandishi After the above two steps, we obtain high-quality text embeddings transferring well to a wide range of tasks without fine-tuning the model parameters. Combined with techniques like approximate nearest neighbor search, embeddings provide a scalable and efficient solution for applications like web search. Here we briefly illustrate several use cases of our text embeddings. Kwanza, ufungaji wa mpito kwa corpus ya lengo hutegemea na kutafakari nje ya mtandao. Kisha kwa kila swali, tunakadiria ufungaji wake wa swali na kurudi juu- Orodha ya orodha kutoka corpus kulingana na usawa wa cosine. Zero-shot Retrieval k A linear classifier is trained on top of the frozen embeddings with a few labeled examples. Different tasks only need to train and save the parameters of the classification heads. It can be seen as a particular form of parameter-efficient learning Few-shot Text Classification [27]. Mifano ya kuingia na alama hutengenezwa kwa maneno yanayohusiana na templates za barua pepe zilizoandikwa kwa mikono. Mifano ya kutabiri ni moja ambayo iko karibu zaidi na maandishi ya kuingia katika nafasi ya kuingiza. ”, maandishi ya alama ni “ " na maandishi ya kuingia itakuwa " ”. Zero-shot Text Classification Ninafurahia kutazama ni mfano wa tathmini ya filamu ya kutisha / kubwa Maoni ya filamu: Ninafurahia kutazama Given two text embeddings, we use the cosine function to measure their semantic similarity. Since the absolute similarity scores do not enable an easy interpretation, the evaluation is usually based on rank correlation coefficients. Semantic Textual Similarity Algorithms ya kawaida ya kikundi kama vile k-maana zinaweza kutumika kwa urahisi. maandiko yanayohusiana na jamii hiyo inatarajiwa kuwa karibu katika nafasi ya kuingiza. Text Clustering For tasks other than zero-shot text classification and retrieval, we use the query embeddings by default. 5 Utafiti wa 5.1 Mafunzo ya awali na Configurations za Fine-Tuning Tuna mafunzo ya awali juu ya seti yetu iliyopendekezwa ya maandishi ya wanandoa kwa ukubwa wa mifano mitatu: E5small, E5base na E5large iliyoanzishwa kutoka MiniLM [59], bert-base-uncased, na bert-large-uncased-whole-wordmasking, kulingana. Ukubwa wa mfululizo unaweka kwa thamani kubwa ya 32, 768 ili kuongeza idadi ya madhara. Kiwango cha kujifunza ni {3, 2, 1}×10−4 kwa mifano ya {small, base, large}, na uharibifu wa linear na hatua za kwanza za 1, 000 kwa ajili ya joto. Tuna mafunzo ya awali kwa hatua 20k kwa jumla na AdamW optimizer, ambayo ni epochs karibu 2.5 juu ya seti ya data. Inachukua {16, 32, Pre-training is performed on the concatenation of 3 datasets: MS-MARCO passage ranking [ ], NQ [ , ], and NLI [ ] datasets. We reuse the mined hard negatives and re-ranker scores from SimLM [ ] for the first two datasets. Models are fine-tuned for 3 epochs with batch size 256 on 8 GPUs. Learning rate is {3*,* 2*,* 1}×10−5 for the {small, base, large} models with 400 steps warmup. For each example, we use 7 hard negatives. Since the NLI dataset only has 1 hard negative for each example, 6 sentences are randomly sampled from the entire corpus. Fine-tuning 8 32 30 22 58 Tunatumia E5-PT kuelezea mifano yenye msimamo wa mapema tu. maelezo zaidi ya utekelezaji yanaweza kupatikana katika Kiambatisho B. 5.2 Evaluation Datasets ni mkusanyiko wa mikutano ya data ya 19 ya upatikanaji wa habari, kutoka kwa utafutaji wa wavuti wa ad-hoc, majibu ya maswali, ukaguzi wa ukweli na upatikanaji wa maswali ya duplicate, nk Tunatathmini mikutano ya data ya 15 ambayo hutoa kupakuliwa kwa umma. BEIR Benchmark [ ] 53 53 is recently proposed for benchmarking massive text embedding tasks. Though MTEB is multilingual due to the inclusion of bitext mining datasets, most datasets are still only available in English. In this paper, we evaluate the English subsets, which have 56 datasets spanning across 6 categories: Classification (Class.), Clustering (Clust.), Pair Classification (PairClass.), Rerank, Retrieval (Retr.), STS, and Summarization (Summ.). The evaluation metrics are accuracy, v-measure, average precision, MAP, nDCG@10, and Spearman coefficients, respectively. Please refer to the MTEB paper for details. MTEB Benchmark [ ] 40 40 5.3 Results on BEIR benchmark In Table we show model results that do not use any labeled data. When averaged over all 15 datasets, E5-PTbase outperforms the classic BM25 algorithm by 1*.*2 points. To the best of our knowledge, this is the first reported result that an unsupervised model can beat BM25 on the BEIR benchmark. When scaling up to E5-PTlarge, we see further benefits from42. * 2 ya Results with Unsupervised Methods 1, 9 to 44 Katika suala la majukumu ya kabla ya mafunzo, Contriever inachukua mapambo ya random, wakati LaPraDor inachanganya ICT na kesi ya kupoteza kama chanya kutoka SimCSE. Mbinu zinaweza kupata data ya mafunzo ya kiwango kikubwa kwa urahisi, wakati mbinu yetu inahitaji juhudi zaidi katika kutibu datasets. Jitihada hizo zinafanywa na matokeo mazuri. Uchunguzi wa hivi karibuni [ ya , ] also show that improving data quality is a vital step for training large language models. 34 60 21 In Table tunapendekeza mifano yetu kwenye datasets zilizosimamiwa na kisha kuhamisha kwenye benchmark ya BEIR. Kwa kuwa datasets zetu za fine-tuning zinajumuisha MS-MARCO na NQ, namba zinazohusiana ni matokeo ya kikoa. Kwa datasets nyingine, hizi ni matokeo ya uhamisho wa shots. Mfano wetu wa E5base unapata wastani wa nDCG@10 ya 48*.*7, tayari kupita njia zilizopo na vigezo vingine kama GTRlarge [ ]. Most datasets benefit from supervised fine-tuning, but there are also a few exceptions such as FiQA, Scidocs, and Fever, etc. This is likely due to the lack of enough domain diversity for the fine-tuning datasets. Results with Supervised Fine-tuning 2, 43 5.4 Results on MTEB benchmark In Table E5 models not only substantially outperform existing ones with similar sizes, but also match the results of much larger models. The top-2 models on MTEB leaderboard GTRxxl and Sentence-T5xxl have 4*.*8B parameters, while our E5large model is more than 10× smaller with 300M parameters. We expect that our model will benefit from continual scaling up. 3, 7 Since the difference between BERT-FTbase and E5base is that BERT-FTbase only has fine-tuning stage, their performance gap demonstrates the usefulness of contrastive pre-training on our proposed CCPairs dataset. For most task categories except Clustering, performance improves after supervised fine-tuning. Consistent with prior works [ , ], hii tena inaonyesha umuhimu wa kuingiza maarifa ya kibinadamu kwa kujifunza embeddings bora ya maandishi. Ni swali la wazi kama embeddings ya kisasa inaweza kupatikana kwa njia ya kujitegemea tu. 43 44 Table inaonyesha matokeo ya usambazaji wa maandishi ya zero-shot kwenye seti ya maendeleo ya seti ya data ya SST-2 [ ]. By formulating text classification as embedding matching between input and label texts, our model can be much better than the “majority” baseline in a zero-shot setting. We use the prompt template from Section 4 52 4.3 ya 5.5 Analysis In this section, we conduct a series of analyses to examine various design choices. All the numbers in this section are from base-size models. For the BEIR benchmark, we choose 6 datasets with more stable results across different runs. Some negative results are also listed in Appendix C. Since we use in-batch negatives for contrastive pre-training, larger batch size will provide more negatives and therefore improve the quality of the learned text embeddings. In Table increasing batch size from 1K to 32K leads to consistent gains across all 6 datasets. It is also possible to train with smaller batch sizes by adding hard negatives [ ]. However, the engineering efforts of mining hard negatives for large datasets (>100M) are non-trivial. Impacts of Batch Size 5 ya 50 GTR models are fine-tuned with “MS-MARCO + NQ”, while Sentence-T5 models use NLI instead. In Table Tunaweza kuona kwamba mipangilio ya "MS-MARCO + NQ" inafanya kazi bora kwenye kazi za kupata, na data ya NLI ni faida kwa STS na usambazaji wa linear. . Combining all of them leads to the best overall scores on the MTEB benchmark. This also illustrates the importance of dataset diversity for learning text embeddings. Fine-tuning Datasets 6, [40] One crucial step in our dataset curation pipeline is filtering out low-quality text pairs. In Table when training with 1M pairs, using filtered data has a nearly 6 points advantage. When all the text pairs are used, the “w/o filter” setting has about 4× more data but is still behind by 1*.*6 points. Though recent studies [ ya Kuonyesha kwamba mifano ya kujifunza kina ni imara sana kwa sauti ya dataset, uchapishaji wa data bado una faida katika kuboresha ufanisi wa mafunzo na ubora wa mifano. Data Filtering 7, 29 47 Tunatafuta mbinu mbili za kuboresha idadi ya madhara: madhara ya pre-batch [ ] reuse embeddings from previous batches as additional negatives, while MoCo Negative Sampling 33 [ ] inazindua encoder ya kasi na hutumia mstari wa FIFO kuhifadhi negatives. Kwa mbinu zote mbili, ukubwa wa negatives unaweza kupanua kwa urahisi bila kuathiri mengi ya kumbukumbu ya GPU. Upungufu ni kwamba negatives wengi hutengenezwa na toleo la zamani la vigezo vya mfano. in-batch negatives still perform favorably. Empirically, we find that MoCo is more sensitive to certain hyperparameters such as temperature, better results are possible with more tuning. 25 8, With the rapid development of dense retrieval models, can we replace the long-standing BM25 algorithm from now on? The answer is likely “ BM25 bado ina faida za dhahiri katika suala la urahisi, ufanisi, na ufafanuzi. Kwa maeneo ya nywele ndefu kama vile Trec-Covid [ ] and retrieval tasks that involve long documents (Touche-2020) [ au kutegemea kwa kiasi kikubwa juu ya mechi halisi ya leksikali (Ujinga) [ ], further research efforts are still necessary to improve current dense retrievers. BM25 vs Dense Retrieval not yet 55 4 54 6 Conclusion In this work, we train a general-purpose text embedding model E5 from weak supervision signals. We adopt a simple contrastive training framework with in-batch negatives and learn from a large-scale text pair dataset we harvest from heterogeneous data sources across the web. E5 offers strong off-the-shelf performance for a wide range of tasks requiring single-vector text representations such as retrieval, semantic textual similarity, and text matching. When further customized for downstream tasks, E5 achieves superior fine-tuned performance compared to existing embedding models with 40× more parameters on the large, 56-task MTEB benchmark datasets. References [1] Sanjeev Arora, Yingyu Liang, and Tengyu Ma. A simple but tough-to-beat baseline for sentence embeddings. In . OpenReview.net, 2017. URL . 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings https://openreview.net/forum?id=SyK00v5xx [2] Mikel Artetxe and Holger Schwenk. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. , 7:597–610, 2019. doi: 10.1162/tacl_a_00288. URL . . Mkataba wa Chama cha Lugha ya Kompyuta https://aclanthology Kifungu cha Q19-1038 [3] David M. Blei, Andrew Y. Ng, and Michael I. Jordan. Latent dirichlet allocation. In Thomas G. Dietterich, Suzanna Becker, and Zoubin Ghahramani, editors, , pages 601–608. MIT Press, 2001. URL . Advances in Neural Information Processing Systems 14 [Neural Information Processing Systems: Natural and Synthetic, NIPS 2001, December 3-8, 2001, Vancouver, British Columbia, Canada] https://proceedings.neurips.cc/paper/2001/hash/ 296472c9542ad4d4788d543508116cbc-Abstract.html [4] Alexander Bondarenko, Maik Fröbe, Johannes Kiesel, Shahbaz Syed, Timon Gurcke, Meriem Beloucif, Alexander Panchenko, Chris Biemann, Benno Stein, Henning Wachsmuth, et al. Overview of touché 2022: argument retrieval. In , pages 311–336. Springer, 2022. International Conference of the Cross-Language Evaluation Forum for European Languages [5] Vera Boteva, Demian Gholipour, Artem Sokolov, and Stefan Riezler. A full-text learning to rank dataset for medical information retrieval. In , ukurasa wa 716-722 Springer, 2016. European Conference on Information Retrieval [6] Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In , ukurasa wa 632-642, Lisbon, Portugal, 2015. Association for Computational Linguistics. doi: 10.18653/v1/D15-1075. URL Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing https: . // aclanthology.org/D15-1075 [7] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari-wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learn-ers. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, , 2020. URL . Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual https://proceedings.neurips.cc/paper/2020/hash/ 1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html [8] Daniel Fernando Campos, Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, Li Deng, and Bhaskar Mitra. Ms marco: A human generated machine reading comprehension dataset. , abs/1611.09268, 2016. Maelezo ya [9] Wei-Cheng Chang, Felix X. Yu, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar. Pre-training tasks for embedding-based large-scale retrieval. In . OpenReview.net, 2020. URL . Mkutano wa Kimataifa wa 8 juu ya Uwakilishi wa Elimu, ICLR 2020, Addis Ababa, Ethiopia, Aprili 26-30, 2020 https://openreview.net/forum?id=rkg-mA4FDr [10] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In , volume 119 of , pages 1597–1607. PMLR, 2020. URL Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event Utafiti wa Machine Learning http: . //proceedings.mlr.press/v119/chen20j.html [11] Xilun Chen, Kushal Lakhotia, Barlas Og˘uz, Anchit Gupta, Patrick Lewis, Stan Peshterliev, Yashar Mehdad, Sonal Gupta, and Wen-tau Yih. Salient phrase aware dense retrieval: Can a dense retriever imitate a sparse one? , 2021. arXiv mapitio ya awali arXiv:2110.06918 [12] Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S Weld. Specter: Document-level representation learning using citation-informed transformers. In , ukurasa wa 2270-2282, 2020. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics [13] Alexis Conneau and Douwe Kiela. SentEval: An evaluation toolkit for universal sentence representations. In , Miyazaki, Japan, 2018. European Language Resources Association (ELRA). URL . Proceedings of the Eleventh International Conference on Language Re-sources and Evaluation (LREC 2018) https://aclanthology.org/L18-1269 [14] Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. Super-vised learning of universal sentence representations from natural language inference data. In , ukurasa wa 670-680, Copenhagen, Denmark, 2017. Association for Computational Linguistics. doi: 10.18653/v1/D17-1070. URL . Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing https://aclanthology.org/D17-1070 [15] Zhuyun Dai, Vincent Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith B. Hall, and Ming-Wei Chang. Promptagator: Few-shot dense retrieval from 8 examples. , abs/2209.11755, 2022. ArXiv [16] Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman. Indexing by latent semantic analysis. , 41(6):391–407, mwaka wa 1990. Journal of the American society for information science [17] Jacob Devlin, Ming-Wei Chang, Kenton Lee, na Kristina Toutanova. BERT: Mafunzo ya awali ya mabadiliko ya kina ya mwelekeo kwa uelewa wa lugha. , pages 4171–4186, Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1423. URL . Proceedings of the 2019 Confer-ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) https://aclanthology.org/N19-1423 [18] Thomas Diggelmann, Jordan Boyd-Graber, Jannis Bulian, Massimiliano Ciaramita, and Markus Leippold. Climate-fever: A dataset for verification of real-world climate claims. , 2020. arXiv preprint arXiv:2012.00614 [19] Vitaly Feldman na Chiyuan Zhang. Nini mitandao ya neural kukumbuka na kwa nini: Kupata nywele ndefu kupitia tathmini ya ushawishi. Katika Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, na Hsuan-Tien Lin, waandishi, , 2020. URL . Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual https://proceedings.neurips.cc/ karatasi/2020/hash/1e14bfe2714193e7af5abc64ecbd6b46-Abstract.html [20] Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. Language-agnostic bert sentence embedding. In , pages 878–891, 2022. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) [21] Leo Gao, Stella Rose Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. , abs/2101.00027, 2021. ArXiv [22] Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. In , pages 6894–6910, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.552. URL . Mkutano wa 2021 juu ya mbinu za empirical katika usindikaji wa lugha ya asili https://aclanthology.org/2021.emnlp-main.552 [23] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor W. Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kris-ten Grauman, Nicolò Cesa-Bianchi, and Roman Garnett, editors, , Maendeleo katika Mfumo wa Usindikaji wa Habari wa Neu-ral 31: Mkutano wa Mwaka juu ya Mfumo wa Usindikaji wa Habari wa Neural 2018, NeurIPS 2018, Desemba 3-8, 2018, Montréal, Canada Ukurasa wa 8536–8546, 2018 URL . https://proceedings.neurips.cc/paper/2018/hash/ a19744e268754fb0148b017647355b7b-Abstract.html [24] Faegheh Hasibi, Fedor Nikolaev, Chenyan Xiong, Krisztian Balog, Svein Erik Bratsberg, Alexander Kotov, and Jamie Callan. Dbpedia-entity v2: a test collection for entity search. In , pages 1265–1268, 2017. Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval [25] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In , pages 9726–9735. IEEE, 2020. doi: 10.1109/CVPR42600.2020.00975. URL . 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020 https://doi.org/10.1109/ CVPR42600.2020.00975 [26] Doris Hoogeveen, Karin M Verspoor, and Timothy Baldwin. Cqadupstack: A benchmark data set for community question-answering research. In , pages 1–8, 2015. Proceedings of the 20th Australasian document computing symposium [27] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, , volume 97 of , ukurasa wa 2790–2799. PMLR, 2019 URL . Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA Proceedings of Machine Learning Research http://proceedings.mlr.press/v97/houlsby19a.html [28] Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Towards unsupervised dense information retrieval with contrastive learning. , abs/2112.09118, 2021. ArXiv [29] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Marina Meila and Tong Zhang, editors, , volume 139 of , pages 4904–4916. PMLR, 2021. URL . Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event Proceedings of Machine Learning Research http://proceedings.mlr.press/v139/jia21b.html [30] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In , pages 6769–6781, Online, 2020. Association for Computational Linguistics. doi: 10. 18653/v1/2020.emnlp-main.550. URL . . Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) https://aclanthology.org/2020.emnlp-main 550 [31] Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contex-tualized late interaction over BERT. In Jimmy Huang, Yi Chang, Xueqi Cheng, Jaap Kamps, Vanessa Murdock, Ji-Rong Wen, and Yiqun Liu, editors, , pages 39–48. ACM, 2020. doi: 10.1145/3397271.3401075. URL . Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Vir-tual Event, China, July 25-30, 2020 https://doi.org/10.1145/3397271.3401075 [32] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural questions: A benchmark for question answering research. , 7:452–466, 2019. doi: 10.1162/tacl_a_00276. URL . Mkataba wa Chama cha Lugha ya Kompyuta https://aclanthology.org/Q19-1026 [33] Jinhyuk Lee, Mujeen Sung, Jaewoo Kang, na Danqi Chen. Kujifunza maonyesho mafupi ya maneno kwa kiwango. , pages 6634–6647, Online, 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.518. URL . . Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) https://aclanthology.org/2021 acl-long.518 [34] Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, na Nicholas Carlini. Idadi ya data ya mafunzo hufanya mifano ya lugha kuwa bora. , 2022. ACL [35] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. , abs/1907.11692, 2019. ArXiv [36] Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. S2ORC: The semantic scholar open research corpus. In , pages 4969–4983, Online, 2020. Associ-ation for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.447. URL . Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics https://aclanthology.org/2020.acl-main.447 [37] Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. Www’18 open challenge: financial opinion mining and question answering. In , pages 1941–1942, 2018. Companion proceedings of the the web conference 2018 [38] Tomas Mikolov, Kai Chen, Gregory S. Corrado, na Jeffrey Dean. Tathmini ya ufanisi wa uwakilishi wa neno katika nafasi ya vektor. , 2013. ICLR [39] Niklas Muennighoff. Sgpt: Gpt sentence embeddings for semantic search. , ABS/2202.08904, mwaka wa 2022. ArXiv [40] Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. Mteb: Massive text embedding benchmark. , abs/2210.07316, mwaka 2022. ArXiv [41] Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas A. Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David P. Schnurr, Felipe Petroski Such, Kenny Sai-Kin Hsu, Madeleine Thompson, Tabarak Khan, Toki Sherbakov, Joanne Jang, Peter Welinder, na Lilian Weng. maandishi na coding kuingizwa kwa mapema kabla ya mafunzo. , abs/2201.10005, 2022. ArXiv [42] Duc Tam Nguyen, Chaithanya Kumar Mummadi, Thi-Phuong-Nhung Ngo, Thi Hoai Phuong Nguyen, Laura Beggel, na Thomas Brox. Mimi mwenyewe: kujifunza kufunika alama za giza kwa kujenga mwenyewe. OpenReview.net, URL ya mwaka 2020 . . 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 https://openreview net/forum?id=HkgsPhNYPS [43] Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hern’andez ’Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith B. Hall, Ming-Wei Chang, and Yinfei Yang. Large dual encoders are generalizable retrievers. , abs/2112.07899, 2021. Maelezo ya [44] Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, na Yinfei Yang. Maneno-t5: Encoders maneno kupanua kutoka mifano kabla ya mafunzo ya maandishi kwa maandishi. , ukurasa wa 1864-1874, 2022. Matokeo ya Chama cha Lugha ya Kompyuta: ACL 2022 [45] Barlas Oguz, Kushal Lakhotia, Anchit Gupta, Patrick Lewis, Vladimir Karpukhin, Aleksandra Piktus, Xilun Chen, Sebastian Riedel, Scott Yih, Sonal Gupta, and Yashar Mehdad. Domain-matched pre-training tasks for dense retrieval. In , ukurasa wa 1524–1534. shirika la lugha za kompyuta, 2022. doi: 10.18653/v1/2022.findings-naacl.114. URL . Findings of the Association for Computational Linguistics: NAACL 2022, Seattle, WA, United States, July 10-15, 2022 https://doi.org/10.18653/v1/2022.findings-naacl.114 [46] Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vassilis Plachouras, Tim Rocktaschel, na Sebastian Riedel. Kilt: kiwango cha ujuzi kwa kazi za lugha. Katika , 2020. North American Chapter of the Association for Computational Linguistics [47] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, na Ilya Sutskever. Kujifunza mifano ya maonyesho kutoka kwa lugha ya asili. Katika Marina Meila na Tong Zhang, waandishi, Mstari wa 139 , pages 8748–8763. PMLR, 2021. URL . Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event Proceedings of Machine Learning Research http://proceedings.mlr.press/v139/radford21a.html [48] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. , 21:1–67, 2020. Journal of Machine Learning Research [49] Nils Reimers na Iryna Gurevych. sentence-BERT: sentence embeddings kutumia Siamese BERT-networks. katika , pages 3982–3992, Hong Kong, China, 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1410. URL . Proceedings of the 2019 Conference on Empirical Methods in Natural Lan-guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) https://aclanthology.org/D19-1410 [50] Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, QiaoQiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. RocketQAv2: A joint training method for dense passage retrieval and passage re-ranking. In , ukurasa 2825–2835, Online na Punta Cana, Jamhuri ya Dominika, 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.emnlp-main.224. URL . Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing https://aclanthology.org/2021.emnlp-main.224 Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, Armand Joulin, na Angela Fan. CCMatrix: Kutengeneza mabilioni ya maneno yenye ubora wa juu kwenye mtandao. , pages 6490–6500, Online, 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.acl-long.507. URL . Utaratibu wa Mkutano wa 59 wa Mwaka wa Chama cha Lugha za Kompyuta na Mkutano wa Kimataifa wa 11 juu ya Utaratibu wa Lugha ya asili (Tomo 1: Makala ndefu) https://aclanthology.org/2021.acl-long.507 [52] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, A. Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In ya mwaka 2013. Mkutano juu ya mbinu za empirical katika usindikaji wa lugha ya asili [53] Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, na Iryna Gurevych. Beir: kiwango cha tofauti cha tathmini ya shutuma ya mifano ya upatikanaji wa habari. , 2021. Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) [54] James Thorne, Andreas Vlachos, Christos Christodoulopoulos, na Arpit Mittal. FEVER: Dataset kubwa kwa uchambuzi wa ukweli na Verification. katika , ukurasa wa 809-819, New Orleans, Louisiana, 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-1074. URL Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) https: . // aclanthology.org/N18-1074 Ellen Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, William R Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, na Lucy Lu Wang. , volume 54, pages 1–12. ACM New York, NY, USA, 2021. ACM SIGIR Forum [56] Henning Wachsmuth, Shahbaz Syed, and Benno Stein. Retrieval of the best counterargument without prior topic knowledge. In , kurasa 241–251, 2018. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) [57] David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, na Hannaneh Hajishirzi. Ukweli au fikra: kuthibitisha madai ya kisayansi. Katika , pages 7534–7550, 2020. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) [58] Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, na Furu Wei. Simlm: Mafunzo ya awali na shinikizo la kuwakilisha kwa ajili ya upatikanaji wa mpito mkali. , abs/2207.02578, mwaka 2022. ArXiv [59] Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, na Furu Wei. Minilmv2: Usambazaji wa uhusiano wa kujitegemea wa viungo vingi kwa kushinikiza mabadiliko ya kabla ya mafunzo. , pages 2140–2151, 2021. Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 [60] Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, na Edouard Grave. CCNet: Kuondoa viwango vya juu vya data ya monolingual kutoka kwa data ya wavuti. , ukurasa wa 4003–4012, Marseille, Ufaransa, 2020 European Language Resources Association. ISBN 979-10-95546-34-4. URL . Utaratibu wa Mkutano wa 12 wa Rasilimali za Lugha na Tathmini https://aclanthology.org/2020.lrec-1.494 [61] Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, na Arnold Overwijk. Kuhusu jirani ya karibu ya kujifunza kinyume cha ubaguzi kwa ajili ya upatikanaji wa maandishi. Katika OpenReview.net, mwaka wa 2021. . . Mkutano wa Kimataifa wa 9 juu ya Uwakilishi wa Kujifunza, ICLR 2021, Mkutano wa Virtual, Austria, Mei 3-7, 2021 https://openreview net/forum?id=zeFrfgyZln [62] Canwen Xu, Daya Guo, Nan Duan, na Julian McAuley. Laprador: Unsurveiled pre-trained dense retriever kwa zero-shot text retrieval. , ukurasa wa 3557–3569, 2022. Findings of the Association for Computational Linguistics: ACL 2022 [63] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In , ukurasa wa 2369–2380, 2018. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing Maelezo ya data Kwa Crawl ya kawaida, tunashusha snapshot ya 2022-33 na cc_net inatumika kwa ajili ya preprocessing ikiwa ni pamoja na utambulisho wa lugha, de-duplication, filtering lugha mfano, nk kurasa za wavuti kutoka MS-MARCO makala ratings corpus pia ni pamoja. Kwa hatua ya data filtering, sisi kuangalia kila pande mbili ya sehemu ndani ya ukurasa wa wavuti badala ya tu kutumia kichwa kama swali. Kwa Wikipedia, sisi kutumia toleo iliyotolewa na Petroni et al. . To avoid possible data contamination, we remove text pairs that occur in the evaluation datasets based on exact string match. 8 [46] Reddit data is collected from the year 2018 to August 2022. For the S2ORC data, we use a sample weight of 0*.*3 during training to avoid over-fitting the scientific domains. For the BEIR benchmark, we use the 15 datasets that provide public downloads: MS MARCO [ Kwa mfano, kwa ajili ya kuhifadhi [ Kwa mujibu wa NFCorpus [ Kwa mujibu wa NCC [ Kwa mfano, mimea ya mimea [ Kwa mfano, kwa mfano [ ], ArguAna [ Mkataba wa Mkataba wa 2020 [ ], CQADupStack [ ], Quora, DBPedia [ Kwa mfano, kwa ajili ya kuondoa Kuongezeka kwa joto ( ], Climate-Fever [ ], and Scifact 8 55 5 32 63 37 56 4 26 24 12 54 18 [57]. B Implementation Details We list the hyperparameters in Table Since some evaluation datasets have long texts, we freeze the position embeddings during both pre-training and fine-tuning and set the maximum text length to 512 for evaluation. 11. For the Quora duplicate retrieval task in the BEIR benchmark, we add prefix “ ” to all the questions. For other retrieval tasks, we use “ ” and “ “Tumepewa kipaumbele kwa mujibu wa Sheria. Maombi ya: Maombi ya: Kwa njia ya: Matokeo ya MS-MARCO katika Meza use document titles provided by RocketQA [ ]. Utaratibu huu wa tathmini unatumika kulingana na wateja wengi wa hali ya juu. Hata hivyo, data ya MS-MARCO kutoka benchmark ya BEIR haina majina, hivyo matokeo yanatarajiwa kuwa ya chini. 12 50 Tunaarifiwa matokeo kwa datasets ya ndani ya uwanja katika Meza These results can help illustrate the benefits brought by contrastive pre-training when abundant in-domain labeled data are available. For MS-MARCO passage ranking, MRR@10 and Recall@1k are reported. For the NQ dataset, Recall@20 and Recall@100 are the main metrics. In-domain Evaluation 12. C. Matokeo ya Hapa ni baadhi ya majaribio ambayo hatimaye kuacha: Kama ilivyo kwa Dk. ], tunaongeza moja BM25 ngumu hasi kwa kila pande chanya wakati wa mafunzo. Wakati wa kutumia data ya 15M, mkakati huu unaboresha matokeo ya jumla kwa ~ pointi 0.5 juu ya benchmark ya BEIR. Hata hivyo, kuendesha algorithm ya BM25 juu ya dataset ya 250M+ ni ya muda mrefu hata na parallelism ya multi-node na multi-process. Adding BM25 hard negatives 30 Ingawa RoBERTa inaonyesha faida ya kudumu katika kazi nyingi za NLP, tunaona kuwa RoBERTa inafanya kazi mbaya zaidi kuliko uanzishaji wa BERT kwenye seti nyingi za data za benchmark za BEIR. Using RoBERTa instead of BERT for initialization Tunaongeza kupoteza lugha masked kwa 25% ya wanandoa wa maandishi ya mafunzo. Idadi ni sawa na kuondoa lengo hili la msaada, lakini gharama ya mafunzo inaongezeka. Auxiliary MLM objective Makala hii inapatikana kwenye archiv chini ya leseni ya CC by 4.0 Deed (Attribution 4.0 International). Makala hii inapatikana kwenye archiv chini ya leseni ya CC by 4.0 Deed (Attribution 4.0 International).