Ikiwa huwezi kuelewa thamani yako mwenyewe, huwezi kuelewa AI Nini kama nilikuambia kwamba namba tatu - tu tatu - inaweza kutabiri kama AI yako itafanya kazi au kushindwa? Hakuna ujenzi mpya. Hakuna mafunzo mapya. Hakuna maboresho ya miundombinu. Mathematics ambayo imekuwa kukaa katika vitabu tangu 1950, kusubiri kwa mtu yeyote kuwa na wasiwasi kuangalia. Takwimu hizi tatu zilikuwa zinaweza kuchukuliwa: Tesla yako phantom kuzuia bila sababu. Tesla yako phantom kuzuia bila sababu. GPT-4 inahusu kesi ya mahakama ambayo haipo: iliyoandaliwa na mwanasheria ambaye alikamatwa. GPT-4 inahusu kesi ya mahakama ambayo haipo: iliyoandaliwa na mwanasheria ambaye alikamatwa. Mfano wako, mafunzo kwa wiki tatu, kurasa nzuri ya kupoteza, kuhamia prod, mara moja kuanguka juu ya uso wake. Mfano wako, mafunzo kwa wiki tatu, kurasa nzuri ya kupoteza, kuhamia prod, mara moja kuanguka juu ya uso wake. Kila mara kwa sababu moja. Nambari tatu ambazo kila mtu anajua Hakuna anayefahamu kikamilifu chanzo cha vifo hivyo, isipokuwa kwamba walikufa kwa kuungua moto uliolipuka ghafla. Ikiwa ufuatiliaji utakuwa imara au hallucinatory. Ikiwa kiwango chako cha chini ni halisi au mashaka. ingawa wanatabiri kama mafunzo yako yatafanya kazi au kuchomwa kwenye uso wako ingawa wanatabiri kama mafunzo yako yatafanya kazi au kuchomwa kwenye uso wako : Je, optimizer yako kupoteza kompyuta zig-zagging? idadi hii inathiri jinsi "kuenea" mazingira yako ya kupoteza ni. Wewe ni kuchoma 90% ya kompyuta yako kwenda upande badala ya chini (mzunguko kwa makosa ya chini) 1.  Idadi ya hali κ (λ_max / λ_min) Kiwango cha juu cha K ina maana ya Canyon. Idadi ya hali κ (λ_max / λ_min) Idadi ya hali κ (λ_max / λ_min) Kiwango cha juu cha K ina maana ya Canyon. : Je, mfano wako kweli kujifunza - au tu kukumbuka? kusoma zaidi, na utapata GIF hapa chini ambayo inaelezea bora kuliko mfululizo mzima wa usawa. 2.  Kiwango cha thamani ya mwenyewe ε (̧λ_max) nzuri ya Kiwango cha thamani ya mwenyewe ε (̧λ_max) Kiwango cha thamani ya mwenyewe ε (̧λ_max) 3.  Unaweza kuwa na uhakika kwamba gradient kutoka kwenye mfumo wako wa AI ni ya kutosha ili kukamilisha mafunzo ya mtandao wako mpya. : Idadi ya thamani ya thamani ya thamani ya thamani ya thamani ya thamani ya thamani ya thamani ya thamani ya thamani (#λ < 0) gradient inasema "tumefanya, hakuna mahali pa kwenda" kwa sababu mwelekeo ni mwepesi katika kila mwelekeo unavyopima. Idadi ya thamani ya thamani ya thamani ya thamani ya thamani ya thamani ya thamani ya thamani ya thamani ya thamani (#λ < 0) Idadi ya thamani ya thamani ya thamani ya thamani ya thamani ya thamani ya thamani ya thamani ya thamani ya thamani (#λ < 0) Lakini nini ikiwa mafunzo yako imefungwa kwenye kiwango cha chini cha uongo? Hiyo ndiyo sababu ya michezo ya kompyuta ya mbwa ajabu hii ni katika mafanikio makubwa kama hayo. at 
 
 
 Wakati δ = 0, wewe ni katika kiwango cha chini cha kweli - ngazi ya mlima. Wakati δ > 0, wewe ni juu ya mlima. Zaidi chini, utaona graphics animated ambayo itafafanua maswali unayoweza kuuliza mwenyewe sasa. And Yet — Believe It or Not — This   Old Math Is Is Tofauti hii imekuwa ikitokea tangu mwaka wa 1950, kabla ya mtu yeyote kufikiri juu ya mtandao wa neural. Miaka saba na tano ya wanasayansi wakipiga kelele ndani ya uchafu. Na bado - sasa, leo, kama wewe kusoma hii - si katika PyTorch. si katika TensorFlow. si katika JAX. si katika Keras. si katika pipeline yoyote ya kawaida ya mafunzo, kuchoma mamilioni ya dola. Kila kitu kinachoweza kukujulisha, kinachoonekana kwa macho. The biggest AI labs in the world? They’re watching loss curves. Crossing fingers. And calling it engineering. 
 
 OpenAI inasahau. Google inasahau. Anthropic hata haijui kwamba wanaishi. hakuna mtu anayeangalia. OpenAI inasahau. Google inasahau. Anthropic hata haijui kwamba wanaishi. hakuna mtu anayeangalia. Hebu tuone jinsi vile nyongeza za kazi na jinsi salama kwa kompyuta. 
 
 Je, unajua kitu kuhusu Kiwira kama historia yake? ☆☆☆☆☆ Je, unajua kitu kuhusu Kiwira kama historia yake? ☆☆☆☆☆ The Three Numbers Everyone Misses Hivyo umekutana na κ, ε, na δ. Nambari tatu ambazo zinatabiri kila kitu. Sasa swali mwenyewe: wapi wao ni? Si kwenye skrini yako, si katika kumbukumbu zako, si mahali popote katika sekta ya AI ya trilioni ya dola. Nambari hizi tatu sio nadharia; ni panel ya uchambuzi ambayo inapaswa kuwepo ... lakini sio. Badala yake, hapa ni nini mifumo maarufu zaidi ya AI - PyTorch, TensorFlow, JAX, Keras - kwa kweli hutoa: 
 
 
 
 Kupoteza thamani ya pamoja Kuanzisha mlinzi mwenyewe Kiwango cha kujifunza ✓ Haya yote ni ya dharura, hii ni ya dharura. Tatu - lakini si tatu sahihi. Si κ (mfano wa hali). Si ε (mfano wa thamani nafuu). Si δ (mfano wa thamani nafuu). κ κ ε ε δ δ Na sio mfumo wako tu. Zana za ndani za OpenAI? Nyuma ya moja kwa moja. Miundombinu ya Google? Upatikanaji wa moja kwa moja. Utaratibu wa mafunzo wa Anthropic? Paneli moja ambayo haipatikani. Viwanda vyote vimeendesha kikundi kimoja cha vifaa vilivyoanguka. 
 
 Mafunzo ya AI leo ni kama ndege ya 747 na kiwango cha kasi, busara, na vibanda. Mafunzo ya AI leo ni kama ndege ya 747 na kiwango cha kasi, busara, na vibanda. Mafunzo ya AI leo ni kama ndege ya 747 na kiwango cha kasi, busara, na vibanda. The loss is going down. Great — that means the error is going down, training looks good. Lakini Je, mfano unapata suluhisho halisi, au tu kukumbuka sauti na kuiba mwenyewe ndani ya shimo ambalo kamwe hakutaweza kuepuka? Kwa nini Kiwango cha kipekee kinakuambia. dashboard yako favorite ya AI framework haitakuwa. Figure 2b makes this obvious: Ambapo maneno haya matatu yanatoka kwa : ya Ni encoding kurudi kamili ya mazingira yako ya kupoteza. Kila canyon, kila ngome, kila ngome. Ni wote huko. Wanaishi ndani ya kitu kinachojulikana kama Hessian Matrix ya pili ya derivative ya kazi yako ya kupoteza 
 
 Na hapa ni sehemu ya ajabu: Hessian ni kamili ya kompyuta. Iko ameketi mahali popote, kuimba data ya uchunguzi, na hakuna mtu anayeangalia. Na hapa ni sehemu ya ajabu: Hessian ni kamili ya kompyuta. Iko ameketi mahali popote, kuimba data ya uchunguzi, na hakuna mtu anayeangalia. Hapa ni jinsi inavyofanya kazi: Look at the two boxes in the middle of the figure. On the left: what PyTorch shows you. Loss looks good. Gradient near zero. Learning rate set. Green light. Ship it. Kwa upande wa kulia: nini ya na ya  reveal. Condition number through the roof. Sharpness off the charts. Two negative eigenvalues. You’re not at a minimum:  kwa ajili ya κ ε δ you’re stuck at a saddle point pretending to be done κ ε δ 
 
 Same numbers. Same moment. Opposite conclusions. Mwisho wa siku.Mwisho wa siku.Mwisho wa siku.Mwisho wa siku.Mwisho wa siku. Mwisho wa siku.Mwisho wa siku.Mwisho wa siku.Mwisho wa siku.Mwisho wa siku. OpenAI doesn’t compute this. Google doesn’t compute this. 
 
 Hakuna mtu anayeendesha cluster ya GPU ya mamia ya maelfu ya bucks kwa mwezi anashughulikia thamani za kibinafsi. Hakuna mtu anayeendesha cluster ya GPU ya mamia ya maelfu ya bucks kwa mwezi anashughulikia thamani za kibinafsi. Wao ni kuangalia curves kupoteza na unadhani sahihi ... kuomba! But the Hessian is Too Heavy! Hiyo ni ambapo mtu kutoka Google anainua mkono wake. Hessian ni n×n. Kwa mfano wa mabilioni ya vigezo, hiyo ni bilioni ya mara bilioni. Huwezi kuhesabu hivyo. Hessian ni n×n. Kwa mfano wa mabilioni ya vigezo, hiyo ni bilioni ya mara bilioni. Huwezi kuhesabu hivyo. Correct. And completely irrelevant. You don’t need the full matrix. You never did. Mwanasayansi wa Kiholanzi aliyeitwa — nyuma wakati "kompyuta" ilimaanisha chumba kikamilifu cha tubes za udongo na ufisadi wa cross-fingering. mbinu yake inachukua thamani za kipekee kwa kutumia bidhaa za matrix-vector iterative. utata: . Basically free compared to a single forward pass. Cornelius Lanczos alijifunza hili katika 1950 O(n) per iteration Cornelius Lanczos alijifunza hili katika 1950 Miaka saba na tano ya maendeleo tangu wakati huo? Sasa tuna Hutchinson trace estimator, stochastic Lanczos quadrature, randomized SVD. Unaweza kupata makadirio ya umbali wa mzunguko, thamani ya juu, mipaka ya idadi ya hali - yote kwa gharama ya hesabu ya chini. Tools like   already prove this works at the ImageNet scale. Today. Right now. PyHessian PyHessian 
 
 So why isn’t this in PyTorch? Why isn’t it in TensorFlow? Why isn’t it in JAX? Kwa nini hii haipatikani katika PyTorch? Kwa nini haipatikani katika TensorFlow? So why isn’t this in PyTorch? Why isn’t it in TensorFlow? Why isn’t it in JAX? Kwa sababu watu ambao wanaelewa nadharia ya spectra ni katika idara ya kimantiki, kuandika makala hakuna mtu anasoma. Watu kujenga mifumo ni utoaji wa vipengele vinavyoonekana vizuri katika demo. Na watu mafunzo mifano ni busy sana babysitting kupoteza curves kuuliza kwa nini wao ni babysitting kupoteza curves. Hivi Iringa ya wakati ule ilionekanaje, unakumbuka? Seventy-five years. Still waiting… Nini Nambari hizi kwa kweli inakuambia Hebu tufanye hili la vitendo, laini, na - zaidi ya yote - laini, ili uweze kujenga hisia sahihi za kimwili. Hebu tufanye hili la vitendo, laini, na - zaidi ya yote - laini, ili uweze kujenga hisia sahihi za kimwili. Nambari ya hali ya ajali The condition number κ (kappa) = λ_max / λ_min controls how hard your optimization problem actually is. Here’s what nobody tells you: gradient descent convergence scales as  (κ-1)/(κ+1)  per iteration. Here’s what nobody tells you: gradient descent convergence scales as  (κ-1)/(κ+1)  (κ-1)/(κ+1)  kwa ajili ya iteration. Tafsiri ya: 
 
 
 X = 100? Takriban iterations 100 kupunguza makosa yako kwa nusu. 10k iterations kwa maendeleo sawa. Na κ = 10000 ni ya kawaida katika mazoezi. Si ya ugonjwa. Si ya kawaida. Jumanne. Nini cha kufanya kwa kiwango cha juu Jinsi ya? Kuwa na hisia Fikiria rolling marble chini ya mlima. nzuri round bowl? marble rolls moja kwa moja hadi chini. Lakini kiwango cha juu cha κ inamaanisha mlima wako ni mstari mdogo: ukuta una urefu wa maili, udongo hauwezi kuonekana. Marmaramu yako haifai chini. Inakua nje ya ukuta. Kwenye mkono wa kulia. Kwenye kulia. Kwenye kulia. Kwenye kulia. Nishati ya moto huenda upande badala ya chini. That’s your gradient. Pointing exactly where calculus says: steepest downhill. Problem is, “steepest downhill” aims at the  Sio chini ya ardhi kuelekea suluhisho. Miji ya Canyon Kupoteza imefungwa kwa 0.0847, kisha 0.0846, kisha 0.0847 tena? Unaweza kurekebisha kiwango cha kujifunza. Hakuna. Unahifadhi pato la rubber kwa miungu ya ML bila shaka. Hakuna. Those training runs that plateau for hours? Hiyo ni juu ya k. Your optimizer isn’t broken. It’s doing exactly what you asked. What you asked is just geometrically insane. You’re burning compute fighting geometry your framework refuses to acknowledge exists. The fix? Preconditioning. Second-order methods. Adaptive optimizers that implicitly estimate curvature. The math exists. It’s been in textbooks since the 1960s. Lakini kwanza, unahitaji kujua κ ni tatizo. Your dashboard won’t tell you. So you’ll never ask. And your compute bill keeps climbing. Picha hapa chini inaonyesha hasa kile kinachotokea. Kituo kimoja cha kuanza. Hatua sawa. Njia moja inachukua ~100 iterations. Jambo jingine inachukua ~10,000 - zigzag kutoka ukuta wa canyon while your GPU burns money. Sharp vs. Flat: Unabii wa Generalization Sasa tutazungumzia kuhusu — thamani ya thamani ya kipekee. ε (epsilon) 
 
 This number answers the question every ML engineer secretly dreads:  Did your model actually learn anything, or did it just memorize the test? Nambari hii inatoa jibu la swali ambalo kila mhandisi wa ML anaogopa kwa siri: Did your model actually learn anything, or did it just memorize the test? Fikiria eneo lako la kupoteza kama ardhi. a ni milima kubwa—unaweza kutembea na urefu hauwezi kubadilika. ni rangi ya mfupa - hatua moja mbaya na unakimbia ndani ya shimo. flat minimum sharp minimum Small ε = flat minimum = good news. When ε is small, your model found that wide valley. Production data comes in slightly different from training data: users type weird things, lighting changes, accents vary, and your model shrugs.  Close enough. I got this. Hiyo ni kawaida. Hiyo ni nini wewe ni kulipa kwa ajili ya. GIF hapa chini inaonyesha jinsi hii inaonekana kama geometrically: Now flip the script. What happens when ε is huge? When your eigenvalues are screaming large numbers? Mafunzo ya kupoteza inaonekana kamili. Uhakiki ni "kuendesha bora kamwe." Hivyo, wakati wa kubwa ya , your model has squeezed itself into a tiny, sharp crevice Hivyo, wakati wa large kubwa ya , your model has squeezed itself into a tiny, sharp crevice Kisha data halisi inakuja. maneno tofauti kidogo. picha tofauti kidogo. kitu tofauti kidogo. And the model doesn’t  ya Kufanya kazi mbaya kidogo It blows up: 
 
 
 
 Maoni ya uaminifu huwa ya kushangaza Predictions turn random Mfumo wako wa log ya tahadhari unashambuliwa saa 2 asubuhi. Hiyo ni kiwango cha chini cha hatua. Miji ni ngumu sana kwamba mabadiliko madogo zaidi hutoa kupoteza roketing. mfano wako haukuwa imara. . brittle and faking it The GIF below shows the difference. Same horizontal shift. Wildly different outcomes. Bonyeza au bonyeza kutazama picha kwa ukubwa kamili Kama unaweza kuona, hii sio falsafa. Hii ni geometry unaweza kupima. Your model that crushed the benchmark and died in production? It found a sharp minimum. The eigenvalues would have told you… But nobody checked. Saddle Points: The Silent Killer Now for   — the negative eigenvalue count, #(λ < 0). This one’s sneaky. This one lies to your face. δ (delta) Your gradient hits zero. Mipango yako ya kupoteza inaendelea. Mfumo wako unaandika: ** “kuunganishwa.” \ Wewe kujiondoa. But you’re not at an error minimum. You’re at a  Sehemu ya Saddle ! 💀 Wewe ni katika a Sehemu ya Saddle Sehemu ya Saddle Mlango wa milima ambapo ardhi inapiga juu katika baadhi ya mwelekeo na chini katika wengine. But you might say:  Sio, msomaji mzuri.No. Hiyo inamaanisha tu kwamba wewe ni usawa juu ya ukuta, si kwamba umepata kitu muhimu. PyTorch inaonyesha gradient yangu ni zero - hii haimaanishi kuwa uboreshaji umefanyika? Je, vitu vya saddle ni vya kawaida kiasi gani? Hebu tufanye kazi ya Math. At any critical point, you can think of each eigenvalue as having a coin-flip chance of being positive or negative. For a true minimum, you need  Miongoni mwao ni chanya. uwezekano mkubwa? nusu ya nusu iliongezeka kwa nguvu ya kiwango chako cha parameter. all Kwa mamilioni ya vigezo, hiyo ni 1/ 2^ (106). You have better odds of winning the lottery while being struck by lightning while a shark bites your leg. In high dimensions, almost every critical point is effectively a saddle.  True minima are statistical miracles. Habari njema: vitu vingi vya saddle ni vya kutisha. sauti ya ndani ya SGD mara nyingi huchukua wewe mwishowe. Habari mbaya: "mwishowe" inaweza kuwa wiki tatu za kompyuta zilizotumika. saddles ya uharibifu - ambapo thamani za kibinafsi hupanda karibu na nusu - hutoa plates ambapo gradient inapiga kelele badala ya kuzungumza. Kupoteza haina kwenda mahali popote. Wewe umefungwa, lakini hujui kama unashikiliwa au ni polepole tu. 
 
 δ > 0 itakuambia mara moja. Nambari moja. Saddle au si. Lakini mfumo wako hauhesabiwi.  would tell you instantly. One number. Saddle or not. But your framework doesn’t compute it. δ > 0 GIF hapa chini inaonyesha jinsi hii inaonekana kama: The Gradient Flow Time Bomb Je, unadhani kuwa tumefungwa? inazidi kuwa mbaya. Your neural network isn’t one function. It’s a chain of functions — layer after layer after layer. And gradients have to flow backward through every single one during training. Like a game of telephone, except each person might whisper too quietly or scream too loudly. Each layer has a Jacobian matrix — the matrix of partial derivatives that governs how signals propagate.  . The singular values of these Jacobians determine whether your gradients survive the journey or die along the way Maana ya kipekee ya Jacobians hizi huamua kama gradients yako kuishi safari au kufa pamoja na njia Gradient huongezeka kwa kila kiwango. Wakati inapofikia kiwango cha mapema, sio gradient tena - ni bomu. Gradients kuanguka. uzito wako unakwenda bila ya mwisho. Mafunzo ya kushuka. NaN mji. Singular values > 1: Uharibifu wa kinyume. Gradient inachukuliwa katika kila ngazi. Wakati inapofikia ngazi za awali, ni makosa ya kuongezeka. Uharibifu wa gradients. ngazi zako za awali huacha kujifunza. Wao ni baridi wakati wengine wa mtandao hufanya kama mafunzo. Singular values < 1: Goldilocks eneo. Gradients kuenea kwa usahihi kutoka mwisho hadi mwisho. Kila kiwango hujifunza. Hiyo ni kwa nini mwanzo wa orthogonal hufanya kazi. Hiyo ni kwa nini normalization ya spectra inaonekana. Singular values ≈ 1: Lakini hapa ni jambo: mbinu hizi zimegunduliwa kwa ajali na kutumika kama band-aids. Hakuna mtu wa kufuatilia spectra ya Jacobi wakati wa mafunzo. Hakuna mtu wa kufuatilia thamani za kipekee. Kwa kweli, haipo katika mfumo wowote wa kibiashara. Layer 47 ni karibu kuua mtiririko wako wa gradient Layer 47 ni karibu kuua mtiririko wako wa gradient Mtandao wako unaweza kuwa na damu nje ya ndani, na dashboard haifai kitu. See it for yourself: And Here You Go: The Real Dashboard Your AI Framework Is Missing Hii ni dashboard ambayo mfumo mzuri wa AI unapaswa kuonyesha. Njia ya mafunzo ya ujuzi wa kimantiki inaweza kuhesabu uchambuzi wa spectra wa mwanga katika vituo vya kudhibiti na kutenda juu yao: 
 
 
 
 Idadi ya hali ya Hessian inapita kiwango → kubadilisha kwa mbinu iliyopangwa.  → apply spectral normalization. Jacobian singular values drift away from 1  → you’re at a saddle, perturb to escape. Negative eigenvalues appear Hessian condition number exceeds a threshold Jacobian singular values drift away from 1 Malipo ya thamani ya moja kwa moja yanaonekana Hivi sasa, hakuna ramani ya AI inayopatikana kibiashara inakuwezesha kutabiri thamani halisi ya msingi. Badala yake, unatengeneza viwango vya kujifunza kwa mkono na matumaini ya ... Kwa nini hakuna mtu aliyeamua hili Kuna sababu tatu:  When throwing more compute at the problem works, nobody questions the foundations. This is temporary. Scaling laws plateau. When they do, the industry will suddenly need mathematics it never bothered to learn. Scaling obscures mathematical sins.  The people who understand spectral theory work in applied math departments solving inverse problems. The people building AI frameworks took optimization and statistics. The Venn diagram overlap is nearly empty. Disciplinary silos. Utekelezaji wa ufuatiliaji sahihi wa mzunguko unahitaji miundombinu ambayo hakuna mtu anataka kujenga. Kila mtu atafaidika. Abstraction debt. Mkataba huo unakuja kwa mara ya Here’s what’s missing in every production framework: Viwanda vimewekeza dola bilioni 100 katika kupanua AI. The mathematical foundations remain incomplete. Every unexplained training failure, every generalization anomaly, every model that worked in the lab and died in production — these are symptoms of mathematical pathologies that your tools cannot diagnose. What Can You Do Now To Go Even Beyond Those Big AI Companies? Kuanza kutoka hapa: 
 
 
 
 
 Jifunze nini Hessian eigenvalues maana kwa usanifu wako maalum  during training — even crude estimates help Monitor condition numbers  before you ship — PyHessian exists, use it Check for sharpness Maswali yote wakati kupoteza plateaus - unaweza kuwa juu ya sadla, si chini ya Mathematics inapatikana. imekuwa kusubiri tangu 1950. The only question is whether you’ll learn it before your next production failure teaches you the hard way. 
 
 Curious about the hidden side of AI? Discover more on my page here Curious about the hidden side of AI? Discover more on my page here

This story contains new, firsthand information uncovered by the writer.

This story contains AI-generated text. The author has used AI either for research, to generate outlines, or write the text itself. 

Keep Reading: The Real Tech Narrative continues here

Sauti hii imetolewa katika lugha asilia ya hadithi!

Nambari tatu. Hiyo ni kila kitu AI yako inahitaji kufanya kazi

About Author

MAONI

HANG TAGS

MAKALA HII ILIWASILISHWA NDANI

Related Stories

#ellenBrain Turns a Long Weekend Into: HackerNoon Swag Around the World

Sweet Security Unveils First Unified Detection And Response Platform

Details of the Aerial Voyage.—Kennedy silenced

Speed up Swift compile time

#ellenBrain Turns a Long Weekend Into: HackerNoon Swag Around the World

Sweet Security Unveils First Unified Detection And Response Platform

Details of the Aerial Voyage.—Kennedy silenced

Speed up Swift compile time

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps