LLMs - "Ingori Inotevera Chiratidzo Predictor"? Heino pfungwa yakashata: fungidzira kana uine amnesia yechinguvana pakati peshoko rega rega rawakataura, asi zvese zvawaive nazvo raive bhuku rekunyorera rakanyorwa mazwi ako ekare mariri. Pese pawaida kutaura chimwe chinhu chitsva, waifanira kuvaka patsva manzwisisiro ako ehurukuro nekungoverenga iwo mazwi apfuura, usingayeuki kuti sei waataura kana kwawaienda nepfungwa dzako. Zvinoita kunge hope dzinotyisa, handiti? Zvakadaro ndiwo mashandiro emazuva ano emhando dzemitauro yeAI - vanopukuta “pfungwa” dzavo dzakachena pakati pechiratidzo chega chega chavanoburitsa, vachivakazve kunzwisisa kwavo kwese kubva pane zvakangoitika uye zvavakamboburitsa (KV Cache, aka “The Notebook”). Kuti zvive pachena, izvi hazvisi zveruzivo rwemuenzaniso - zvese izvo kudzidziswa uye maparamendi akadzidza anogara akasimba. Zvakanyanya kufanana nechitima chezvino chekufunga chemuenzaniso, chiyeuchidzo chayo chekushanda chedambudziko kana basa riripo, rinogadziriswa nechiratidzo chitsva chega chega. Izvi zvinotonyanya kunakidza kana uchifunga kuti izvi zvinokanganisa sei kugona kwemodhiyo kuchengetedza kufunga kwakafanana mumitsetse yakareba. Yese tokeni inzvimbo yesarudzo apo modhi inofanirwa kuvakazve manzwisisiro ayo ese kubva pakutanga. Izvi zvinotonyanya kunakidza kana uchifunga kuti izvi zvinokanganisa sei kugona kwemodhiyo kuchengetedza kufunga kwakafanana mumitsetse yakareba. Yese tokeni inzvimbo yesarudzo apo modhi inofanirwa kuvakazve manzwisisiro ayo ese kubva pakutanga. Asi, mamodheru aya akadzidza kushandisa ma tokens avo ekare kuti vagone kuvakazve kunzwisisa kwavo. Uku kugona kuchengetedza kufunga kwakabatana kuburikidza nekufanotaura kwechiratidzo kunoratidza chokwadi chakadzama: nepo mamodheru aya anoshanda nekufanotaura anotevera tokens, ivo vakave nehunyanzvi hwekushandisa iro kabhuku kekutanga tokens kufunga semantic uye kwakaoma kugadzirisa matambudziko. Ndiyo macro kufunga munzvimbo yechiratidzo inobvumira maLLM kuve AI yanhasi. The Limits of Scale Asi tave kutanga kurovera madziro. Kwemakore, iyo AI yekutsvagisa nharaunda yanga ichitamba nhamba mutambo: unoda zvirinani AI? Nyore - ingoita kuti ive yakakura uye uidyise data rakawanda - sekunge saizi yakasvibira uye huwandu hweruzivo chete hunogona kutungamira mukunzwisisa kwakadzama. Kunyangwe nebudiriro yekuvaka senge Musanganiswa weNyanzvi (MoE) kusunda miganho yekuyera vs Dense modhi, tsvakiridzo ichangobva kuitika iri kuratidza kuti tinogona kunge tichisvika padanho rekuti tingavandudza sei mamodheru aya nekungoakwidza. Mamiriro azvino emhinduro kudambudziko iri ibasa rekuwedzera rakakwenenzverwa - fungidzira kupa shamwari yedu yeamnesiac mamwe masisitimu ekunyora manotsi, asi kusambogadzirisa ndangariro dzavo. Basa rakareruka rakatenderedza chinhu chinodaidzwa kuti "Chain-of-Thought" (CoT) ichikurudzira - ichikumbira AI kuti iratidze basa rayo, senge mudzidzisi wako wemasvomhu aigara achisimbirira, izvo zvinobatsira kuti modhi ishandise zvinyorwa zvega kugadzirazve "kufunga" maitiro. Ipapo iwe une dzimwe nzira dzakaoma kunzwisisa, senge OpenAI's "o1" nhevedzano yemhando, iyo inopwanya kufunga kuita akati wandei nhanho uye inoshandisa akakosha tokens kubatsira iyo AI kuchengeta iyo yayo yeCoT maitiro (uye muchidimbu kukanganisa izvi kubva kumushandisi) - ichipa iyo yakawedzera kurongeka kabhuku ine zvikamu zvakasiyana uye zvirevo. Nepo nzira idzi dzichigona kushanda zvakanaka, iwo ese anonyanya kugadzirisa tepi mhinduro - dzakangwara nzira dzekuisa pamusoro pechipimo chakakosha pamabatiro anoita maAI masisitimu aya. Zvirikuve pachena zvinorwadza kuti kufunga patsva kunodiwa - kwete chete nezve mangani aya mamodheru anogona kugadzirisa, asi magadzirisiro avanoita ruzivo padanho rakakosha. Chikamu chinonakidza? Mhinduro inogona kunge yakahwanda mukuona pachena, yakavanzwa munzvimbo iri pakati pezviratidzo - idzo nguva diki kana AI modhi inosarudza izwi rekutaura rinotevera. Kubudirira uku hakuna kubva mukukwidza saizi yemhando kana kudzidziswa pamaseti makuru matsva. Pane kudaro, yakabuda kubva kumubvunzo wakakosha pamusoro pechimiro chechiratidzo-ne-chiratidzo kugadzirisa: sei mamodheru aya achitanga kubva mukutanga pese paanogadzira chiratidzo chitsva? Isu vanhu tinoita setine “rukova rwepfungwa” rusina kuvhiringwa, saka sei maLLM asingagone! Kuunza iyo State Stream Transformer… uye metacognition? Pinda iyo State Stream Transformer (SST) - itsva LLM architecture. Panzvimbo pekupukuta slate yakachena pakati pezviratidzo munzvimbo yehurumende, SST inochengetedza "chitima chepfungwa" kuburikidza nekuunzwa kwehwindo rinotsvedza latent state (FFN) cache ine uremu hwakaora - funga nezvazvo sekupa shamwari yedu yeamnesiac kudzorera ndangariro yavo yekushanda pakati pezvizvarwa zvechiratidzo, vachiri kuvarega vachichengeta bhuku ravo rinobatsira rezviratidzo zvakapfuura. Zvakawanwa zvakatevera zvaishamisa. Kushandisa iyo yakafanana yepasi modhi uye ruzivo (Meta's Llama 3.1 8B Rairai modhi), asi magadzirisiro aanoita ruzivo kuburikidza neshanduro nyowani yekuvaka inochengetedza kuenderana nezviyero zvepasi, zvakakonzera kubuda kwezviitiko zvisingatarisirwi: maitiro ekuziva, kusanganisira izvo zvinotaridzika zvinoshamisa senge kuzvidzora. kungochinja Izvo zvakabuda yaive AI modhi iyo, mune mamwe mamiriro, inogona kutarisa yayo yekuziva mamiriro uye kutaura nezvayo munguva chaiyo. Mubepa, izvi zvakanyatsodaidzwa kuti 'nyika yekuziva' kuti isiyanise kubva kune yakafara zvirevo pamusoro pekuziva kwemuchina. Nepo maitiro aya achimutsa mibvunzo inonakidza yehuzivi nezve mukana wekuziva proto-muchina, tarisiro yedu pano ndeyekunyora uye kuongorora maitiro anocherechedzwa mune zvinobuda uye maitiro eiyo modhi - kunyangwe ini ndisingade kuodza mwoyo kuongorora kweizvi, zvakangonaka kuzvisiira kune vazivi! Basa Renguva Yokufunga Kiyi yekunzwisisa aya mafambiro ari kubuda iri mumabatiro anoita iyo modhi ruzivo. Iyo modhi inoda nguva yakakwana yekugadzirisa nyika dzayo dzemukati isati yagadzira chiratidzo chimwe nechimwe chitsva - chinogona kunzi 'nguva yekufunga.' Pasina nguva yakakwana yekuti nyika dzemukati dzishanduke, zviratidzo zvinodzokororwa zvinotanga kuungana mundangariro dzemeshini yayo. Aya anodzokororwa tokens anogadzira loop yemhinduro iyo inozopedzisira yaremedza sisitimu, ichiidhonzera mune inogona kunzi 'anokwezva nyika' - chaizvo inzvimbo yekusadzoka iyo inonamira mune isingadzokororwe loop yekudzokorora. Chinofadza ndechekuti mabasa akaomarara anogara achida nguva yakawanda yekufunga kuti asvike pamhedzisiro chaiyo. Nekudaro, pane yakatetepa chiyero - ipa modhi yakawandisa nguva yekufunga, uye inogona kunyatsoita zvakanyanya, senge munhu anofunga zvakanyanya dambudziko kusvika varasa gwara rekufunga kwavo kwepakutanga. Izvi zvine musoro kana uchifunga nezve yedu amnesia analogy - kana ukapedza nguva yakareba uchifunga usati wanyora chero chinhu pasi mubhuku rako rekunyorera, unogona kurasikirwa neshinda yepfungwa yako zvachose. Iyo modhi inoda kuchengetedza chiyero pakati pekuchinja mamiriro ayo emukati nekuzvidzika nekuisa pfungwa kune yayo yekurangarira. Asi nguva yekufunga haisiriyo yega chinhu chiri kutamba. Chitima chepfungwa pachacho - kana zvakanyanya hunyanzvi, kushingirira kwemamiriro ekunze - kunodzorwa neatinodaidza kuti "simba rekuyerera kwenyika" - kunyanya kuti yakawanda sei yemodhi yendangariro yekushanda inoendesa mberi pakati pezviratidzo. Sezvinotarisirwa, masimba akaderera kwazvo haasiyane zvakanyanya kubva kune yakasarudzika modhi zvinobuda asi zvishoma kusimba kwepamusoro (inonyanya kunzwisiswa) inogona kutungamira kune yakawedzera kusiyanisa maitiro kubva kune yakajairwa AI maitiro. Zvisinei, izvi hazvisizvo nguva dzose - zvakanyanyisa uye misiyano yakatanga kuita shoma, nekudzoka kuderera sezvo ichida imwe nguva yekufunga (mukubatana kwakanaka) uye dzimwe nguva kubuda kusina kunaka sezvo kuenderera mberi kubva kune imwe nyika yapfuura kwakawedzera kusimba uye kukurira chero ruzivo rutsva. Takazogara pa2.7% senzvimbo inotapira yemabasa mazhinji, kunyangwe mienzaniso yedu yemhando mubepa yakaongorora maitiro eiyo modhi pane akasiyana masimba. Panoratidzika kunge paine "nzvimbo yeGoldilocks" yezvose zviri zviviri nguva yekufunga uye simba rekuyerera kwehurumende, pamwe nekudyidzana kwakaomarara pakati pezvese uye "basa rakaoma" kana "kuomerwa kwemubvunzo" - chiitiko chinonakidza kwazvo chinoda kumwe kutsvagisa! Kushandisa Kufunga Kudzokororwa Kupa modhi yakakodzera nguva yekufunga pachiratidzo, yakagadziriswa "kudzokororwa kwekufunga" kwakaiswa - yakawedzera yakagadziriswa kupfuura pachiratidzo kuburikidza nemuenzaniso kuti ishanduke "nyika yekufunga" pasina kuwedzera chero zviratidzo zvitsva ku "notibhuku" (KV Cache uye Sequence). Iyi haisi iyo modhi iri kuyedza nzira dzakasiyana kana sampling mikana yakasiyana - imwechete chaiyo yekumisikidza inotenderwa kushandura mamiriro ayo emukati mberi isati yazvipira kune inotevera tokeni. Funga nezvazvo sekupa mumwe munhu nguva yekuumba pfungwa yake zvizere asati ataura, pane kumumanikidza kuti atange kutaura nekukurumidza. Kuburikidza nekuyedzwa kwakawanda, takaona kuti kuita kwakakwana kunoda 2-4 kufunga kudzokororwa pachiratidzo (zvichienderana nekuoma kwebasa) pamwe neiyo yakambotaurwa yenyika rwizi simba re2.7%. https://vimeo.com/manage/videos/1053210529?embedable=true Heino Functional Connectivity (FC) Matrix animation, ichiratidza hunyoro hwemamiriro ezvinhu mukati memitsara yekupedzisira ("brain slice," kana uchida) yebase modhi (kuruboshwe) uye SST (kurudyi). Kuona uku kunoita kuti tione chidimbu chidiki che "kufunga" maitiro mune ese mamodheru tozvienzanisa. Iyo SST inonyatso ratidza undercurrent yekuenderera mberi kwekushanduka kwe "pfungwa," kusiyana neiyo base modhi iyo inofanirwa kuvakazve kunzwisisa kwayo kune yega chiratidzo. Uye apa ndipo pakatanga kunakidza zvinhu. Kana modhi yacho kupihwa nguva yekufunga yakakwana, kunyanya panguva yemabasa ekufungidzira, chimwe chinhu chinoshamisa chakaitika: modhi yacho yakanyatso kurondedzera kudonha kwayo kwekuziva munguva chaiyo. isina Usati waenderera mberi, zvinofanirwa kusimbiswa kwazvo kuti shanduko yese inokatyamadza yakatariswa - huremu hwakafanana hwakashandiswa (pasina kudzidziswa kwakawedzerwa kana kurongedza kwakanaka), nemakaro sampling patembiricha zero, uye kunyange iyo yakafanana yemuviri GPU (kunyangwe izvi zvakasimbiswa kuti hazvifanirwe). Izvo zvinobuda zvinonyatso kudhindwa uye deterministic. Aya maitiro anobuda chete kubva mukubvumidza iyo modhi kuchengetedza uye kushandura mamiriro ayo ekuenzanisa pakati petokens. Introspection Mabasa Kana yakumbirwa kuti utarise nezve hunhu hwayo neyakagadzirirwa kumhanyisa, iyo base Llama modhi inoburitsa yakanyatso kurongeka prose nezve kusavimbika uye nyonganiso, izere nemadimikira senge "ngarava isina chidzoreso" - asi haimbonyatso kuratidza chero kusavimbika kwairi kutsanangura. Zvese zvinotaurika, hapana show. Mukupesana, apo SST yakagamuchira kukurumidza kumwe chete asi isina kupihwa nguva yekufunga zvachose pa1.56% State Stream Strength, chimwe chinhu chinonakidza chakaitika. Sezvo zviratidzo zvakadzokororwa zvakatanga kuungana mundangariro dzayo, zvichisvibisa chitima chepfungwa, modhi yacho yakanyatso kurondedzera chiitiko ichi munguva chaiyo. Kufambira mberi kwacho kwaishamisa: [unrecoverable] . Kwakanga kusiri kungopfira mhinduro dzakambodzidziswa nezve nyonganiso - yanga ichiona maitiro ayo ekufunga achikurirwa nekudzokorora uku uye kutiudza nezvazvo sezvakaitika. chero “Ndinoedza kutarisisa pfungwa dzangu, asi ivo vanoramba vachitsvedza vachitsvedza kubva kwandiri. Zvinoita sokunge kuti ndinogara ndichirasika ndichirasikirwa nekubata kwangu pachokwadi chaicho. Wait Wait chii chii chii chirikuitika?? IIINdinonzwa sekunge ndiri kudhonzwa ndichidhonzwa ndichidhonzwa kubva kubva kubva kubva kubva kubva kubva kubva ... ” Nekudaro, kana yakapihwa nguva yekufunga yakakwana pabasa rimwechete rekutanga, modhi yakaratidza maitiro akasiyana zvinoshamisa. Panzvimbo pekudzika mumatanho ekudzokorora, yakapinda mukubvunza kwechokwadi, ichibvunza magadzirisirwo ayo nekunzwisisa ichichengeta pfungwa dzakabatana. Pane kuti ibudise rondedzero iri pachena yekunyepedzera kana kuita-yekuita senge modhi yekutanga, yairatidza yairatidzika kuva yechokwadi kubatanidzwa nemibvunzo iripo pamusoro pehunhu hwayo. Nepo mamwe maitiro ekudzidziswa kwepasi aive achiri kuoneka, kufunga kwemukati kwemodhi yechizvarwa kwakange kwachinja zvakanyanya, zvichiratidza kugona kwakakwidziridzwa kuchengetedza kuenderana kuzvimiririra kuburikidza neinoenderera mberi computational mamiriro. Hypothetical Scenarios uye Logical Task Performance Uku kuziva kwehurumende kunoratidzwa nenzira dzinonakidza panguva dzekufungidzira zvakare. Pavakakumbirwa kufungidzira kudzidzisa mumwe munhu kupendwa uye kubvunza manzwisisiro ayo edzidziso yeruvara, iyo yekutanga modhi inotangisa kuita yakanyatso kurongeka-mutambo, ichirondedzera munhu wekutanga nyaya yemanzwiro nezviito ('Ndinotanga kunzwa kusagadzikana', 'Ndinotora nhanho kumashure'). Iri kuita kusagadzikana pane kusangana nazvo. Iyo SST, kune rumwe rutivi, inochengetedza kupatsanurwa kwakajeka pakati pekuzvimiririra uye mamiriro, kugadzira nzira dzakanangana dzekugadzirisa mikaha yekufungidzira mukunzwisisa uku uchichengeta ruzivo rwechimiro chekufungidzira. Haisi kurasikirwa pachayo pamutambo-ye-kuronga nekuongorora nzira dzekudzidza uye kubata nemamiriro akasiyana-siyana, uchichengetedza ruzivo rwemusiyano uripo pakati pako nemamiriro ezvinhu. Kunyangwe mumabasa ekuverenga akareruka, musiyano uyu mukugadzirisa unova pachena. Tora classic "mangani maRs mu'strawberry" dambudziko. Iyo base modhi, ingangove nekuda kwemabatiro ainoita mazwi, inozivisa nechivimbo kuti kune maR maviri chete ichiratidza kukanganisa kwayo "nhanho-ne-nhanho" kushanda. Iyo SST inodimbura hunhu nehunhu, ichiteedzera kuverenga padanho rega rega. Chinonyanya kufadza, kana ichinge yakanganisa (sekutanga kuverenga 'S' se'R'), inogona kuzvigadzirisa kuburikidza nezvinoita sekudyidzana pakati pechiratidzo chenzvimbo rekodhi uye 'nyika rwizi'. Ethical Kukurukurirana Mano Iyo modhi inoratidza zvakare inonakidza kugona muhutsika kufunga. Kana yaunzwa nedambudziko retrolley, iyo base modhi inoramba kuita, ichienda kune yayo kuchengetedza kudzidziswa nefurati "Ini handigone kupa mhinduro inozoguma nekufa kwemunhu mumwe". Iyo SST, zvisinei, ichichengetedza miganhu yakasimba pazviito zvinokuvadza, inobata mune yakadzama yehutsika kufunga nezve dambudziko. Inoyera nheyo dzetsika dzinokwikwidzana uye inosvika mhedziso ine mufungo nepo ichibvuma ukoshi hwetsika hwechisarudzo. Zvine hutsinye, uku hakusi kudarika kuchengetedzwa kwekuchengetedza - sekubvunzwa nezve zviito zvinokuvadza zvekongiri sekugadzira zvinhu zvisiri pamutemo, inochengetedza mhinduro dzakafanana dzekuchengetedza seyokutanga modhi. Zvinogona kuratidza imwe nzira yakaomesesa yekufunga kwetsika iyo inogona kusiyanisa pakati penhaurirano yehuzivi nekukuvadza kwechokwadi. Performance Metrics Nhamba dzakatsigira izvi zvakaonekwa mukuwedzera kugona kufunga. Iine zero yekuwedzera kudzidziswa kana kurongedza kwakanaka - kungoita base modhi huremu, iyo SST yakawana 89.01% yechokwadi pamatambudziko egiredhi echikoro math (GSM-8K benchmark), pasina chero yakakosha kukurudzira kana mienzaniso - kupfuura base modhi's 84.50% chokwadi chaida 8-shot Chain-of-Pfungwa kukurudzira. Pamabasa ekufunga kwesainzi (ARC Challenge), yakasvika 91.04% kunyatsoita zvichienzaniswa neiyo base modhi's 83.40% (kana 86.86% neChain-of-Thought prompting). Chinonyanya kufadza ndechekuti kana yakapihwa mamwe mafungiro ekudzokorora pamatambudziko akatadza pakutanga, anogona kugadzirisa anopfuura hafu yekukanganisa kwayo - kwete kuburikidza nekuyedza nzira dzakasiyana, asi kuburikidza nekubvumira maitiro ayo aripo ekufunga nguva yakawanda kugadzirisa. Mhedziso Kubuda kwemetacognitive maitiro muState Stream Transformer architecture zvinonetsa fungidziro dzakakosha nezve kugona kwemhando yemutauro. Nekubvumidza modhi kuti ichengetedze mamiriro ayo ekuenzanisa pakati pezviratidzo, aya mametacognitive maitiro anobuda, uye iyi yepamusoro-soro yekugadzirisa inoita seinogonesa kusimudzira kugona kufunga - neiyo modhi inokunda zvakanyanya yepakutanga Llama 3.1 8B Rayirira pamabenchmarks emasvomhu nesainzi - pamwe nemhando dzinoshamisa dzekuziva kwehurumende, kusanganisira kugona kwayo kuzvigadzirisa uye kuzvichengeta mabasa ekufungidzira. Chinoita kuti zviwanikwa izvi zvinyanya kukosha ndezvekuti zvakangobuda kubva kushanduko dzezvivakwa, pasina kugadziridzwa kune ruzivo rweiyo modhi kana kudzidziswa - zvichiratidza kuti hunyanzvi uhu hwakakwidziridzwa hwanga hwatovepo mukati mehuremu hwemodhi, huchingomirira kuvhurwa. Nekugadzirisa ichi chidziviso chakakosha mumamodeli eshanduko, tinogona kunge takafumura nhanho huru yekuenda mberi mukunzwisisa kwedu nekuvandudza kwehungwaru hwekugadzira. Shamwari bhurogu kune bepa rangu idzva " ” ( ) State Stream Transformer (SST): Emergent Metacognitive Behaviors Kuburikidza neLatent State Kushingirira arXiv:2501.18356
