Vatsari: .  (1) An Yan, UC San Diego, eka ayan@ucsd.edu;  (2) Zhengyuan Yang, Microsoft Corporation, zhengyang@microsoft.com hi ku hoxa xandla loku ringanaka;  (3) Wanrong Zhu, wa le UC Santa Barbara, loyi a nga eka unrongzhu@ucsb.edu;  (4) Kevin Lin, Xiyenge xa Microsoft, keli@microsoft.com;  (5) Linjie Li, Nhlangano wa Microsoft, lindsey.li@mocrosoft.com;  (6) Jianfeng Wang, Nhlangano wa Microsoft, jianfw@mocrosoft.com;  (7) Jianwei Yang, Nhlangano wa Microsoft, jianwei.yang@mocrosoft.com;  (8) Yiwu Zhong, Yunivhesiti ya Wisconsin-Madison, yzhong52@wisc.edu;  (9) Julian McAuley, wa le UC San Diego, eka jmcauley@ucsd.edu;  (10) Jianfeng Gao, Nhlangano wa Microsoft, jfgao@mocrosoft.com;  (11) Zicheng Liu, Nhlangano wa Microsoft, zliu@mocrosoft.com;  (12) Lijuan Wang, Xiyenge xa Microsoft, lijuanw@mocrosoft.com.   Xitsundzuxo xa muhleri: Lexi i xiphemu xa 1 xa phepha leri kambelaka ku tirhisiwa ka AI yo tumbuluxa ku fambisa ti-smartphone. U nga hlaya phepha hinkwaro hi ku tirhisa tafula ra swihlanganisi leswi nga laha hansi.  Tafula ra Swihlanganisi   Xitlhokovetselo na 1 Xingheniso   2 Ntirho lowu Fambelanaka na wona  3 MM-Xitirhisiwa xo famba-famba   3.1 Ku Vumbiwa ka Swiphiqo na 3.2 Ku veka ehansi ka xikirini na ku fambafamba hi ku tirhisa Seti ya Marka   3.3 Ku tumbuluxiwa ka Matimu hi ku tirhisa Ku Tihlanganisa ka Tindlela to Tala  4 Xikambelo xa ku Famba-famba ka Xikirini xa iOS   4.1 Ku Hlela ka Swikambelo   4.2 Nhlamuselo ya Xiendlo lexi Kunguhatiweke   4.3 Ku Endliwa ka Goza ra Ndhawu na 4.4 Xiyimo xa sweswi na GPT-4V  5 Xikambelo xa ku Famba-famba ka Xikirini xa Android   5.1 Ku Hlela ka Swikambelo   5.2 Ku pimanisiwa ka Matirhelo   5.3 Tidyondzo ta ku susa swirho   5.4 Nxopaxopo wa Swihoxo   6 Bulo   7 Mahetelelo na Swikombo  Xianakanyiwa  Hi humesa MM-Navigator, muyimeri loyi a simekiweke eka GPT-4V wa ntirho wa ku fambafamba wa vuhlanganisi bya mutirhisi wa swifaniso swa smartphone (GUI). MM-Navigator yi nga tirhisana na xikirini xa smartphone tanihi vatirhisi va vanhu, na ku kumisisa swiendlo leswi landzelaka ku hetisisa swiletelo leswi nyikiweke. Swikumiwa swa hina swi kombisa leswaku timodeli letikulu ta multimodal (LMMs), ngopfungopfu GPT-4V, ti humelela eka ku fambafamba ka GUI ka zero-shot hi ku tirhisa nhlamuselo ya yona ya xikirini ya xiyimo xa le henhla, ku anakanya ka xiendlo, na vuswikoti bya ku veka swiendlo swa ndhawu hi ku kongoma. Hi sungula hi pimanisa MM-Navigator eka dataset ya hina ya xikirini xa iOS leyi hlengeletiweke. Hi ku ya hi swikambelo swa vanhu, sisiteme yi kombisile mpimo wa ku pakanisa wa 91% eku tumbuluxeni ka tinhlamuselo ta swiendlo leswi twalaka na mpimo wa ku pakanisa wa 75% eku hetisiseni ka swiendlo leswinene swa swiletelo swa goza rin’we eka iOS. Ku engetela kwalaho, hi kambisisa modele eka ntlawa lowutsongo wa dataset ya ku fambafamba ka xikirini xa Android, laha modele wu tlulaka vafambisi va GUI va khale hi ndlela ya zero-shot. Nxopaxopo wa hina wa xiyimo xa le henhla na vuxokoxoko swi kongomisa ku veka masungulo lama tiyeke ya ndzavisiso wa nkarhi lowu taka eka ntirho wa ku fambafamba wa GUI. Tluka ra phurojeke ri le ka https: //github.com/zzxslp/MM-Navigator.  1 Xingheniso  Ku aka vayimeri lava tiyimeleke lava kotaka ku tirhisana na switirhisiwa swa khomphyuta na ku landzelela swileriso swa vanhu ku vile nhlokomhaka ya khale eka vaaki va dyondzo ya michini (Bolt, 1980; Lieberman et al., 1995). Ku sukela loko ku humelele ti-smartphone, ku vile na xilaveko lexi tirhaka xo tumbuluxa vapfuni va xiviri, ku fana na Siri, Cortana, na Google Assistant, lava nga na vuswikoti byo ndlandlamuxa swinene ntokoto wa vatirhisi na ku pfuna vanhu lava nga na swiphiqo swa miri kumbe xiyimo. Hi ndlela leyinene, vapfuni lava a va ta endla mintirho ya siku na siku hi vuswikoti leyi simekiweke eka swiletelo swa ririmi ra ntumbuluko, ku suka eka swiendlo swo olova swo fana na ku veka nkarhi ku ya eka mintirho yo tika swinene yo fana na ku kuma hodela leyinene ya ku wisa ka ndyangu.  Tidyondzo ta sweswinyana ti sungurile ku lavisisa vulawuri bya switirhisiwa swa tiselfoni na ku otomatiki ka ntirho wa smartphone hi ku landzelela swiletelo swa vanhu (Rawles et al., 2023; Wen et al., 2023; Zhan na Zhang, 2023; Wang et al., 2023). Maendlelo yo yimela ya katsa ku hlamusela swifaniso swa xikirini hi tsalwa na ku lulamisa tsalwa leri hundzuriweke hi timodeli ta ririmi letikulu (LLMs) (Rawles et al., 2023; Wen et al., 2023), kumbe ku letela modele wa ririmi ra xivono ku tumbuluxa swiendlo hi ndlela leyi languteriweke ( Rawles na van’wana, 2023; Hambiswiritano, timodeli leti ti langutiwaka, loko ti leteriwile eka tinxaka to karhi ta swikirini na swiletelo (Rawles et al., 2023), ti kombisa ku humelela loku nga nyawuriki eka ku andlala eka swiyimo swa misava ya xiviri. Hi hala tlhelo, maendlelo lama simekiweke eka LLM ya andlala ku antswa, kambe goza ra le xikarhi ro hundzuluxa swifaniso swa xikirini ku va tsalwa ri endla leswaku ku lahlekeriwa hi mahungu naswona hikwalaho ka sweswo ri vavisa matirhelo. Hi ku hlohloteriwa hi ku tirha kahle na ku tirhisiwa hi ku anama ka timodeli letikulu ta sweswinyana ta multimodal (LMMs), hi lavisisa hi ku tirhisa LMM, GPT-4V (OpenAI, 2023a,b,c; gpt, 2023; Yang et al., 2023c), eka zeroshot smartphone GUI ku famba-famba, hi xikongomelo xo veka xisekelo lexintshwa xo tiya xa ntirho lowu wo tsakisa.  Hi kuma mintlhontlho yimbirhi ya nkoka eka ku fambafamba ka GUI na ti-LMM, ku nga nhlamuselo ya xiendlo lexi kunguhatiweke na ku hetisisiwa ka xiendlo xa laha kaya. Xo sungula, modele wu fanele ku twisisa xifaniso xa xikirini na ku nghenisiwa ka swiletelo swa tsalwa, na ku anakanya hi xivutiso ku kumisisa goza leri faneleke ku tekiwa, ku fana na ku nyika nhlamuselo ya ririmi ra ntumbuluko “ku tikhoma xifaniso xa Amazon eka layini ya vunharhu na kholomo ya vumune.” Xa vumbirhi, modele wu fanele ku hundzula ku twisisa ko tano ka xiyimo xa le henhla ku va xiendlo lexi fomatiweke lexi nga endliwaka hi ku olova hi ku ya hi milawu, ku fana na “{Xiendlo: Click, Location: (0.31, 0.57)}.” Eka endlelo ra hina, hi hlohlotela GPT-4V hi xifaniso na tsalwa ku pulana maendlelo, na ku veka tithegi ta set-of-mark (Yang et al., 2023b) ku anchor swikumiwa leswi tumbuluxiweke. Hi ku kongoma, hi hlanganisa timaraka leti na tindhawu ta ndhawu hi ku pfuniwa hi timodeli ta ku avanyisa kumbe ta OCR. Ku fikelela leswi, sisiteme ya hina leyi ringanyetiweke ya GPT-4Vbased, ku nga MM-Navigator, yi nga humesa swiendlo leswi tirhisiwaka leswi ringanisiweke eka xifaniso xa xikirini, xiletelo xa tsalwa na matimu ya xona ya vuhlanganisi.  Hi pimanisa MM-Navigator eka tidathaseti timbirhi. Hi sungula hi dataset ya ku fambafamba ya iOS GUI leyi nga na swifaniso swa xikirini na swiletelo swa mutirhisi leswi hi swi hlengeleteke hi voko. Dataset leyi yo basa ya nxopaxopo yi endleriwe ku lavisisa vutivi bya mintlhontlho yimbirhi eka ku fambafamba ka GUI: nhlamuselo ya xiendlo lexi kunguhatiweke na ku hetisisiwa ka xiendlo xa laha kaya. Ku kamberiwa ka vanhu ku tirhisiwa ku kambela GPT-4V eka mintirho leyi yimbirhi, hi mimpimo ya ku pakanisa ya 91% na 75%, hi ku landzelelana. Ku engetela kwalaho, hi kambela modele eka ntlawa lowutsongo wa xitshuketa ku suka eka mpimo wa ku fambafamba wa Android lowu humesiweke sweswinyana (Rawles et al., 2023). Hi landzelela phurotokholo ya nkambisiso leyi ringanyetiweke eka mpimo, kun’we na ku kamberiwa ka vanhu loku engetelekeke. Matirhelo yo tiya ya kombisa leswaku MM-Navigator i mufambisi wa GUI loyi a tirhaka kahle eka ti-smartphone, ku tlula swinene maendlelo ya khale lama simekiweke eka LLM. Hi nyika minxopaxopo yo enta ya milandzu ya ku humelela na ku tsandzeka ka vuyimeri. Hi kuma leswaku xiyimo xa sweswi xa GPT-4V xi nga ha va se xi tirha kahle eku pfuneni ka vanhu eka swiyimo swo hambana swa misava ya xiviri swa GUI swa ku fambafamba, tanihilaha swi kombisiweke hakona hi mimbuyelo ya swikirini swo tala eka Xifaniso 4. Hambiswiritano, ku antswisiwa loku yaka emahlweni ka ha ri ka nkoka ku ya emahlweni ku engetela ya sisiteme ku tshembheka, tanihilaha swi paluxiweke hakona eka minxopaxopo ya hina.  Minyikelo ya hina yi katsakanyiwile hi ndlela leyi landzelaka  • Hi humesa MM-Navigator, sisiteme ya muyimeri leyi akiweke eka GPT-4V ya ku fambafamba ka GUI ya smartphone. MM-Navigator hi ndlela leyinene yi katsa matimu ya swiendlo na tithegi ta set-of-mark ku humesa swiendlo leswi kongomeke leswi nga endliwaka.  • Hi hlengeleta dataset leyintshwa ya vuxopaxopi leyi nga na swikirini swo hambana swa iOS na swiletelo swa mutirhisi, leswi kambelaka mintlhontlho yimbirhi leyikulu eka ku fambafamba ka GUI na ti-LMM: nhlamuselo ya xiendlo lexi kunguhatiweke na ku hetisisiwa ka xiendlo xa laha kaya.  • Hi endla swikambelo swo anama, swa otomatiki na swa vanhu, eka tidathaseti timbirhi naswona hi nyika minxopaxopo ya vuxokoxoko. Mimbuyelo yo hlamarisa yi kombisa ku tirha kahle ka MMNavigator eka ku fambafamba ka GUI.  Phepha leri ri   ehansi ka layisense ya CC BY 4.0 DEED. kumeka eka arxiv

Part of HackerNoon's growing list of open-source research papers, promoting free access to academic material.

FEW SHOT .tech

Audio leyi yi humesiwile hi ririmi ro sungula ra xitori!

Valavisisi Va Microsoft Va Vula Leswaku Modele Leyintshwa Ya AI Yi Nga ‘Vona’ Xikirini Xa Foni Ya Wena

About Author

SWIVUTISO SWA SWIVUTISO

HANG TAGS

XITLHOKOVETSELO LEXI XI NYIKERIWE EKA

Related Stories

Digg Free Gear? – A Thank You Contest by HackerNoon

Digg Free Gear? – A Thank You Contest by HackerNoon

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps