Qorayaasha:  (1) Hanoona Rasheed, Mohamed bin Zayed University of AI oo si isku mid ah uga qayb qaatay qorayaasha koowaad;  (2) Muhammad Maaz, Mohamed bin Zayed University of AI oo si isku mid ah uga qayb qaatay qorayaasha koowaad;  (3) Sahal Shaaji, Mohamed bin Zayed University of AI;  (4) Abdelrahman Shaker, Mohamed bin Zayed University of AI;  (5) Salman Khan, Mohamed bin Zayed University of AI iyo Jaamacadda Qaranka ee Australia;  (6) Hisham Cholakkal, Mohamed bin Zayed University of AI;  (7) Rao M. Anwer, Mohamed bin Zayed University of AI iyo Jaamacadda Aalto;  (8) Eric Xing, Mohamed bin Zayed University of AI iyo Carnegie Mellon University;  (9) Ming-Hsuan Yang, Jaamacadda California - Merced iyo Google Research;  (10) Fahad S. Khan, Mohamed bin Zayed University of AI iyo Linköping University.   Xusuusin Tifaftiraha: Tani waa qaybta 1 ee 10 ee daraasadda si faahfaahsan uga hadlaysa horumarinta qaabka AI kaas oo loogu talagalay in lagu sifeeyo sawirrada isticmaaleyaasha. Hoos ka akhri inta kale.  Shaxda Xiriirinta   Abstract iyo 1 Hordhac   2. Shaqada la xidhiidha   3. Habka   4. Dhuumaha Faafinta Xogta   5. Tijaabooyin   6. Gabagabo iyo Tix-raac   Qalabka Kaabayaasha ah (Qaybta 1)   A. Faahfaahinta Hirgelinta Dheeraadka ah   B. Howlaha Hoose ee Dheeraadka ah   C. Natiijooyin Tayo Dheeraad ah   Qalabka Kaabayaasha ah (Qaybta 2)   D. Sawirka Xogta   E. Xaddidaad iyo Shaqada Mustaqbalka   F. Anshaxa iyo Saamaynta Bulshada  Abstract   Moodooyinka Multimodal-ka ee Waaweyn (LMMs) waxay ku fidiyaan Hababka Luuqadaha Waaweyn xagga aragtida. LMM-yadii ugu horreeyay waxay adeegsadeen sawirro iyo qoraallo qoraal ah si ay u dhaliyaan jawaabo qoraal ah oo aan sal lahayn. Dhawaan, LMM-yada heer gobol ayaa la isticmaalay si ay u dhaliyaan jawaabo ku salaysan muuqaal. Si kastaba ha ahaatee, waxay ku xaddidan yihiin kaliya inay tixraacayaan hal shay markiiba, waxay u baahan yihiin isticmaalayaasha inay qeexaan gobollada, ama ma bixin karaan shay cufan oo cufan leh. Shaqadan, waxaan ku soo bandhigaynaa Grounding LMM (GLaMM), qaabka ugu horreeya ee dhalin kara jawaabaha luqadda dabiiciga ah si aan kala go 'lahayn oo ay ula socdaan maaskaro kala qaybinta shayga. GLaMM ma aha oo kaliya in ay sababto shayyada ka soo muuqda wada sheekeysiga balse waa dabacsanaan ku filan si ay u aqbasho qoraal ahaan iyo soo jeedinta muuqaalka ikhtiyaariga ah labadaba (gobolka xiisaha) si ahaan. Tani waxay awood u siinaysaa isticmaalayaasha inay la falgalaan moodalka heerar kala duwan oo granularity ah, labadaba xagga qoraalka iyo muuqaalka. Sababtoo ah la'aanta halbeegyada caadiga ah ee goobta cusub ee Jiilka Wadahadalka Aragga Ku Salaysan (GCG), waxaanu ku soo bandhigaynaa hab-maamuus qiimayneed oo dhamaystiran oo lala yeesho wada sheekaysigeena salka ku haya. Hawsheenna GCG waxay u baahan tahay fikrado cufan oo ku salaysan muuqaallada dabiiciga ah oo baaxad weyn. Si taas loo gaaro, waxaan soo jeedinaynaa Grounding-wax kasta oo Dataset (GranD) si cufan loo sharraxay anagoo adeegsanayna soo jeedinta dhuumaha tafatirka tooska ah ee ka kooban 7.5M fikrado gaar ah oo ku saleysan wadar ahaan 810M gobol oo la heli karo waji-gashiyo. GCG ka sokow, GLaMM sidoo kale waxay si wax ku ool ah u qabataa dhowr hawlood oo hoose, tusaale, tixraaca kala qaybinta odhaahda, sawirka iyo qoraalka heerka-gobolka iyo wada sheekaysiga luqadda aragga.  1. Hordhac  Waxaa shiday mowjadda AI ee curiyaha ah, Moodooyinka Multimodal Models Large (LMMs) waxay u soo baxeen horumar muhim ah, oo isku xiraya farqiga u dhexeeya aragtida iyo hawlaha luqadda [2]. Dadaallada ugu horreeya sida [6, 8, 22, 29, 52, 61] waxay muujinayaan jawaabo qoraal ah oo wax ku ool ah oo ku salaysan sawirada la geliyo. In kasta oo moodooyinkani ay yihiin kuwo casri ah, haddana ma aysan sal dhigi karin jawaabahooda macnaha guud. Dejinta noocaan oo kale ah ayaa muhiim u ah codsiyada horumarsan sida faham muuqaal ah oo faahfaahsan, wakiilo ka kooban is dhexgal, iyo khalkhalgelinta macluumaadka gudaha. Dadaaladii dhawaa waxa ay bilaabeen in ay wax ka qabtaan xaddidaaddan iyaga oo awood u siinaya moodooyinka in ay ka baaraandegaan isticmaale-qeexay gobollo ku qeexan sanduuqyada xadka [5, 31, 35, 36, 57].   Dhawr shaqo oo dhowaan la sameeyay ayaa sahamiyay jiilka jawaabta qoraalka salka ku haya [5, 21, 35, 59] laakiin ma bixiyaan saldhigyo heer pixel ah oo faahfaahsan. Si la mid ah kuwan, dadaal ayaa lagu sameeyay suugaanta qaybinta ee tilmaamaysa qoraallada dhulka ee sawirada dabiiciga ah [21]. Si kastaba ha ahaatee, waxay ku xaddidan yihiin inay dhulka dhigaan hal shay mana geli karaan sheeko dabiici ah, wada sheekaysi isku dhafan, si ay u xaddidaan ku-dhaqankooda wax-ku-oolka ah ee hawlaha is-dhex-galka ah ee u baahan faham qoto-dheer ee nuxurka muuqaalka iyo qoraalka labadaba. Si wax looga qabto xaddidaadyadan shaqooyinka jira, waxaan soo bandhigeynaa Grounding LMM (GLaMM), taas oo si isku mar ah u bixisa faham qoto-dheer oo gobolka ah, aasaaska heerka pixel, iyo awoodaha wada hadalka iyada oo loo marayo habka tababarka dhamaadka-ilaa-dhamaadka (eeg sawirka 1 iyo Tab. 1).  Si wax looga qabto la'aanta halbeegyada wada sheekaysiga aragtida salka ku haya, waxaanu soo bandhigaynaa hawsha cusub ee Jiilka Wadahadalka Grounded (GCG). Hawsha GCG waxa ay rabta in ay soo saarto jawaabaha luqadda dabiiciga ah oo ay ku dhex milmeen waji-gashiyada kala qaybinta shay. Hawshan adagi waxay midaysaa dhawr hawlood oo ka jira aragga kombayutarka kuwaas oo sida caadiga ah loola dhaqmo si gooni-gooni ah, tusaale ahaan, tixraacaya qaybinta odhaahda, sawirka iyo xaraf-u-dhigista heerka-gobolka, soo-dejinta weedha, iyo wada-hadallada afka-aragga ah. Sidaas awgeed, qaabkayaga midaysan iyo xogta tababarka kahor ee la soo jeediyay ayaa si wax ku ool ah ugu wareejin kara dhowr hawlood oo hoose (oo tixraacaya qaybinta hadalka, xaraf-u-dhigista heerka-gobolka, qoraalka sawirka, iyo qaabka wada hadalka QA). Waxaan u soo bandhignay GLaMM qaabkii ugu horreeyay ee si gaar ah loogu talagalay hawshan adag. Si ka duwan shaqooyinkii hore, GLaMM waxay la shaqayn kartaa qoraal iyo muuqaal labadaba waxayna soo saari kartaa wax soo saar muuqaal ah, sidaas darteed waxay bixinaysaa khibrad isticmaale oo badan.  Fahamka heerka-goboleed ee faahfaahsan wuxuu u baahan yahay habka hawl-karnimada leh ee ururinta tafsiirrada baaxadda leh ee gobollada sawirka. Waxaan soo jeedineynaa dhuumo iswada si loo qeexo baaxadda weyn ee Grounding- Anything Dataset (GranD) si loo yareeyo dadaalka calaamadinta gacanta. Ka faa'iidaysiga dhuumaha otomaatiga ah ee leh tillaabooyin xaqiijineed oo go'an, GrandD waxa ay ka kooban tahay 7.5M fikrado gaar ah oo ku qotoma 810M gobol, mid walbana leh maaskaro qaybin. Isticmaalka aragtida casriga ah iyo moodooyinka luqadda, xog-ururinta ayaa sharraxaysa sawirrada SAM [18] iyada oo loo marayo nidaam heerar kala duwan leh oo kor u qaadaya tayada sharraxaadda. Iyada oo leh sawirro 11M ah, 84M oo tibaaxo tixraacaya, iyo 33M qoraallo salka ku haya, GrandD waxa ay dejisaa halbeeg cusub oo dhammaystiran. Marka lagu daro xogta si toos ah loo soo saaray ee GCG, waxaanu bixinaa xogta tayada sare leh ee ugu horeysa ee wada sheekaysiga salka ku haya ee la helay anagoo dib u habayn ku samaynayna xogta gacanta lagu sharraxay ee jira [16, 37, 49] ee GCG iyadoo la isticmaalayo GPT-4 [34] barasho aan macno lahayn. Waxaan u tixraacnaa xogta tayada sare leh sida GranDf, taasoo muujineysa ku habboonaanta hagaajinta.  Shaqadayadu waxay leedahay saddex wax ku darso oo waaweyn:  • Waxaan soo bandhignay GLaMM, moodeelkii ugu horreeyay ee awood u leh inuu soo saaro jawaabaha luqadda dabiiciga ah si aan kala go 'lahayn loogu dhex daray maaskaro kala-saarid shay. Si ka duwan moodooyinka jira, GLaMM waxa ay dejisaa soo jeedin qoraal iyo muuqaal ah, iyada oo fududaynaysa isdhexgalka isticmaale hab-dhaqameedyo badan.  • Annaga oo aqoonsanaya la'aanta halbeegyada jaangooyooyinka ee wada-hadallada salka ku haya aragga, waxaan soo jeedinaynaa hawsha cusub ee Jiilka Wadahadalka Salka ah (GCG). Waxaan sidoo kale soo bandhigeynaa hab-maamuus qiimeyn oo dhameystiran si loo cabbiro waxtarka moodooyinka GCG ee mideeya hawlo badan oo go'doonsan, buuxinaya farqiga weyn ee suugaanta.  • Si loo fududeeyo tababbarka moodeelka iyo qiimaynta, waxaanu abuurnaa Grounding- Anything Dataset (GranD), xog-ururin si cufan loo sharraxay. Waxaa la sameeyay iyadoo la adeegsanayo dhuumaha tafatirka tooska ah iyo shuruudaha xaqiijinta, waxay ka kooban tahay 7.5M fikrado gaar ah oo salka ku haya 810M gobol. Intaa waxa dheer, waxaanu soo jeedinaynaa GranDf, xog tayo sare leh oo si cad loogu qaabeeyey hagaajinta hawsha GCG, anagoo dib u soo celinayna xogta kaydka furan ee jira.  Warqadan waxaa   iyadoo la raacayo shatiga CC BY 4.0 DEED. laga heli karaa arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

AutoEncoder.tech

Research & publications on Auto Encoders, revolutionizing data compression and feature learning techniques.

Auto Encoder's blog

Codkan waxa lagu soo saaray afkii asalka ahaa ee sheekada!

Cilmi-baarayaasha UAE waxay abuuraan AI Kaas oo ku sifayn kara muuqaallada si faahfaahsan

About Author

FAALLO

HANG TAGS

MAQAALKAN AYAA LAGU SOO BANDHIGAY

Related Stories

Startups Winners, Turn Your Win into a Story Worth Sharing

10 Stories To Learn About Medium

LayerZero's Hype Train: A Deeper Look Into DeFi's Cross-Chain Tech

Creating NFTs in Arbitrum

Startups Winners, Turn Your Win into a Story Worth Sharing

10 Stories To Learn About Medium

LayerZero's Hype Train: A Deeper Look Into DeFi's Cross-Chain Tech

Creating NFTs in Arbitrum

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps