Uyenza njani iKubernetes kwiMifanekiso yeDocker enkulu

Isishwankathelo sengxaki

Ngenye imini, ngexesha lohlaziyo olucwangcisiweyo lweqela le-k8s, safumanisa ukuba phantse zonke iiPOD zethu (malunga nama-500 kwi-1,000) kwiindawo ezintsha azikwazanga ukuqalisa, kwaye imizuzu yakhawuleza yajika yaba ziiyure. Sikhangelwa ngenkuthalo oyena nobangela, kodwa emva kweeyure ezintathu, ii-PODS zazisekwimeko ye ContainerCreating .

Okuvuyisayo kukuba, le ibingeyiyo imeko yemveliso kwaye ifestile yolondolozo yayicwangciselwe ngempelaveki. Saba nexesha lokuphanda ngalo mba ngaphandle koxinzelelo.

Kufuneka uqale phi ukukhangela oyena nobangela? Ngaba ungathanda ukufunda ngakumbi ngesicombululo esisifumeneyo? Xhuma kwaye wonwabe!

Iinkcukacha ezithe vetshe malunga nengxaki

Ingxaki yayikukuba sinenani elikhulu lemifanekiso ye-docker efuna ukutsalwa kwaye iqale kwindawo nganye kwiqela ngexesha elinye. Oku kungenxa yokuba i-docker ye-docker yemifanekiso emininzi itsalwa kwindawo enye inokukhokelela ekusebenziseni idisk ephezulu kunye namaxesha okuqala okubandayo.

Ixesha nexesha, inkqubo yeCD ithatha ukuya kwiiyure ezi-3 ukutsala imifanekiso. Nangona kunjalo, ngeli xesha lalibambekile ngokupheleleyo, kuba inani le-PODS ngexesha lokuphucula i-EKS (inline, xa sithatha indawo yazo zonke ii-nodes kwiqela) laliphezulu kakhulu.

• Zonke ii-apps zethu zihlala kwii-k8s ( EKS esekwe). Ukugcina iindleko zethu ze-DEV env, sisebenzisa iimeko zendawo.

• Sisebenzisa umfanekiso weAmazonLinux2 kwiinodi.

• Sinenani elikhulu lamasebe abonakalayo (ii-FBs) kwindawo yophuhliso esasazwa ngokuqhubekayo kwiqela lethu le-Kubernetes. I-FB nganye ineseti yayo yezicelo, kwaye isicelo ngasinye sineseti yaso yokuxhomekeka (ngaphakathi komfanekiso).

• Kwiprojekthi yethu, phantse ii -apps ezingama-200 kwaye eli nani liyakhula. I-app nganye isebenzisa enye ye-7 yemifanekiso ye-docker enobungakanani be ~ 2 GB. Ubungakanani obupheleleyo bomfanekiso ogciniweyo (kwi -ECR ) malunga ne-3 GB.

• Yonke imifanekiso igcinwe kwi-Amazon Elastic Container Registry (ECR).

• Sisebenzisa uhlobo lwevolyum ye-gp3 EBS engagqibekanga kwiinodi.

  
  

Imiba Ejongene Nayo

• Ixesha elongeziweyo lokuQala okuBanda: Ukuqala ipod entsha ngomfanekiso omtsha kunokuthatha ngaphezulu kweyure enye, ngakumbi xa imifanekiso emininzi itsalwa ngaxeshanye kwindawo enye.

• Iimpazamo ze-ErrImagePull: ErrImagePull rhoqo okanye unamathele kwi- ContainerCreating states, ebonisa imiba ngokutsalwa komfanekiso.

• Ukusetyenziswa kweDiski ePhezulu: Ukusetyenziswa kweDiski kuhlala kufutshane ne-100% ngexesha lenkqubo yokutsalwa komfanekiso, ngokuyinhloko ngenxa yedisk enzulu ye-I/O efunekayo ukuthotywa (umzekelo, "unpigz").

• Imiba yeSixokelelwano seDaemonSet: Enye inkqubo yeDaemonSets(efana ne aws-node okanye ebs-csi-node ) isiwe kwindawo "engekalungi" ngenxa yoxinzelelo lwediski, ichaphazela ukulungela indawo.

• Akukho cache yomfanekiso kwiindawo: Kuba sisebenzisa imizekelo yamabala, asinako ukusebenzisa idiski yendawo kwimifanekiso egciniweyo.

  
  

Oku kubangela ukusasazwa okuninzi okumisiweyo kumasebe ophawu, ngakumbi kuba iFB eyahlukileyo ineseti ezahlukeneyo zemifanekiso esisiseko.

Emva kophando olukhawulezayo, sifumene ukuba eyona nto iphambili yayiluxinzelelo lwediski kwiinodi ngenkqubo unpigz . Le nkqubo inoxanduva lokunciphisa imifanekiso ye-docker. Asizange sitshintshe izicwangciso ezingagqibekanga zohlobo lwevolumu ye-gp3 EBS, kuba ayifanelekanga kwimeko yethu.

Hotfix ukubuyisela iqela

Njengenyathelo lokuqala, sagqiba ekubeni sinciphise inani le-PODs kwiinodi.

1. Sihambisa ii-nodes ezintsha kwi-"Cordon" state
2. Susa zonke iiPODS ezibambekayo ukunciphisa uxinzelelo lwedisk
3. Qhuba nganye nganye iiPOD ukufudumeza iindawo
4. Emva koko, sihambisa iindawo ezifudumeleyo ukuya kwindawo eqhelekileyo ("unCordon")
5. Susa zonke iinodi kwindawo yokuxinga
6. Zonke ii-PODS zaqala ngempumelelo ukusebenzisa i-cache yomfanekiso we-Docker

Uyilo loqobo lweCI/CD

Ingcamango ephambili yesisombululo kukufudumala iindawo zokuhlala phambi kokuba inkqubo yeCD iqale ngeyona ndawo inkulu yomfanekiso we-docker (i-JS dependencies layer), esebenzisa njengomfanekiso weengcambu kuzo zonke ii-apps zethu. Ubuncinci sineentlobo ze-7 zeengcambu zemifanekiso exhomekeke kwi-JS, ehambelana nohlobo lwe-app. Ke, makhe sihlalutye uyilo loqobo lweCI/CD.

Kwimibhobho yethu ye-CI/CD, sineentsika ezi-3:

Umbhobho woqobo weCI/CD:

1. Kwinyathelo le Init : silungiselela imeko-bume / izinto eziguquguqukayo, sichaza iseti yemifanekiso eza kuphinda yakhiwe, njl.

2. Kwinqanaba Build : sakha imifanekiso kwaye siyityhalele kwi-ECR

3. Kwinqanaba Deploy : sihambisa imifanekiso kwiik8s (uhlaziyo lokusasazwa, njl..)

   
   

Iinkcukacha ezithe vetshe malunga noyilo lwe-CICD yoqobo:

• Amasebe ethu omsebenzi (FB) afowunelwe kwisebe main . Kwinkqubo yeCI, sihlala sihlalutya iseti yemifanekiso eyatshintshwa kwi-FB kwaye yakha kwakhona. Isebe main lihlala lizinzile, njengenkcazo, kufuneka kubekho inguqu yamva nje yemifanekiso yesiseko.
• Sakha ngokwahlukeneyo imifanekiso ye-docker exhomekeke kwi-JS (kwimeko-bume nganye) kwaye siyityhale kwi-ECR ukuze iphinde isetyenziswe njengengcambu (isiseko) umfanekiso kwiDockerfile. Sine malunga ne-5-10 iintlobo ze-JS yokuxhomekeka komfanekiso wedocker.
• I-FB isasazwe kwiqela le-k8s kwindawo yamagama eyahlukileyo, kodwa kwiindawo eziqhelekileyo ze-FB. I-FB ingaba ne ~ 200 apps, kunye nobukhulu bomfanekiso ukuya kwi-3 GB.
• Sinenkqubo ye-autoscaling ye-cluster, eyenza i-nodes kwi-cluster esekelwe kumthwalo okanye i-PODS elindileyo kunye ne-nodeSelector kunye nokunyamezela.
• Sisebenzisa imizekelo yamabala kwiinodi.

Ukuphunyezwa kwenkqubo yokufudumala

Kukho iimfuno zenkqubo yokufudumala.

Isinyanzelo:

1. Usombululo loMba : Idilesi kwaye isombulule ContainerCreating imiba.
2. Ukuphuculwa kokuSebenza : Linciphisa ngokuphawulekayo ixesha lokuqalisa ngokusebenzisa imifanekiso esisiseko efudunyeziweyo (ukuxhomekeka kweJS).

Kumnandi ukuba nophuculo:

1. Ukuguquguquka : Ivumela utshintsho olulula kuhlobo lwe-node kunye nobomi bayo (umzekelo, i-SLA ephezulu okanye ixesha elongeziweyo lokuphila).
2. Ukungafihli : Ibonelela ngeemetriki ezicacileyo ngokusetyenziswa kunye nokusebenza.
3. Iindleko eziSebenzayo : Konga iindleko ngokucima iVNG ngoko nangoko emva kokuba icandelo elinxulumeneyo licinyiwe.
4. Ukwahlukaniswa : Le ndlela iqinisekisa ukuba ezinye iindawo azichaphazeleki.

Isisombululo

Emva kokuhlalutya iimfuno kunye nezithintelo, sagqiba ekubeni siphumeze inkqubo yokufudumala eya kutshisa i-nodes kunye nesiseko semifanekiso ye-cache ye-JS. Le nkqubo iya kuqaliswa ngaphambi kokuba inkqubo yeCD iqale, ukuqinisekisa ukuba iindawo zilungele ukuthunyelwa kwe-FB, kwaye sinethuba elikhulu lokubetha i-cache.

Olu phuculo sahlulahlula kumthi amanyathelo amakhulu:

1. Yenza iiseti zeendawo (iQela leeNodi eziNgcolileyo) kwi-FB nganye

2. Yongeza imifanekiso esisiseko kwi-cloud-init script kwiinodi ezintsha

3. Yongeza inyathelo lokusasaza kwangaphambili ukuqhuba i-DaemonSet kunye necandelo le initContainers ukukhuphela imifanekiso efunekayo yedokhi kwiindawo zokuhlala phambi kokuba inkqubo yeCD iqale.

   
   

Umbhobho ohlaziyiweyo weCI/CD unokujongeka ngolu hlobo:

Umbhobho ohlaziyiweyo weCI/CD:

1. Inyathelo lokuqala
   1.1.(inyathelo elitsha) Init deploy : Ukuba sisiqalo sokuqala se-FB, ngoko yenza iseti entsha yobuqu yemizekelo ye-node (ngokwemiqathango yethu yi-Virtual Node Group okanye i-VNG) kwaye ukhuphele yonke imifanekiso yesiseko se-JS (imifanekiso emi-5–10 ) ukusuka kwisebe eliphambili. Kulunge ngokwaneleyo ukuba siyenze, kuba siyifolosile i-FB kwisebe eliphambili. Inqaku elibalulekileyo, ayingomsebenzi wokuthintela.
2. Yakha inyathelo
3. Inyathelo lokusasaza kwangaphambili Khuphela imifanekiso emitsha ebhakiweyo yesiseko se-JS enethegi ethile ye-FB esuka kwi-ECR.
   3.1.(inyathelo elitsha) Amanqaku abalulekileyo : Kumsebenzi wokuthintela, kuba kufuneka sinciphise uxinzelelo lwedisk. Ngomnye, sikhuphela imifanekiso esisiseko yenode nganye ehambelanayo.
   Btw, enkosi ngenyathelo elithi " init deploy" , sele sinayo imifanekiso ye-docker esisiseko evela kwisebe eliphambili, elisinika ithuba elikhulu lokubetha i-cache ekuqaleni kokuqala.
4. **Fakela
   **Akukho lutshintsho kweli nyathelo. Kodwa enkosi kwinyathelo langaphambili, sele sinabo bonke oomaleko bemifanekiso ye-docker kwiindawo eziyimfuneko.

Init deploy step

Yenza iseti entsha yeenodi kwi-FB nganye ngokusebenzisa umnxeba we-API (kwinkqubo ye-3rd party autoscaling) ukusuka kumbhobho wethu weCI.

Imiba esonjululweyo:

1. Ukwahlukaniswa : I-FB nganye ineseti yayo yeenodi, eqinisekisa ukuba okusingqongileyo akuchatshazelwa zezinye ii-FB.

2. Ukuguquguquka : Sinokulutshintsha ngokulula uhlobo lwe-node kunye nobomi bayo.

3. Ukusebenza kweendleko : Sinokucima ii-nodes ngokukhawuleza emva kokuba i-FB icinyiwe.

4. Ukungafihli : Siyakwazi ukulandelela ngokulula ukusetyenziswa kunye nokusebenza kweenodi (i-node nganye inethegi ehambelana ne-FB).

5. Ukusetyenziswa okusebenzayo kweziganeko zendawo : Umzekelo webala uqala ngemifanekiso esele ichazwe kwangaphambili, oko kuthetha ukuba, emva kokuba indawo yendawo iqalile, sele kukho imifanekiso esisiseko kwi-node (ukusuka kwisebe eliphambili).

   
   

Khuphela yonke imifanekiso yesiseko se-JS ukusuka kwisebe eliphambili ukuya kwiindawo ezintsha nge cloud-init script.

Ngelixa imifanekiso ikhutshelwa ngasemva, inkqubo yeCD ingaqhubeka nokwakha imifanekiso emitsha ngaphandle kwemiba. Ngaphezu koko, ii-nodes ezilandelayo (eziya kudalwa yinkqubo ye-autoscaling) ukusuka kweli qela liya kudalwa kunye nedatha ehlaziyiweyo cloud-init , esele inemiyalelo yokukhuphela imifanekiso ngaphambi kokuqala.

Imiba esonjululweyo:

1. Isisombululo soMba : Uxinzelelo lweDiski luhambile, kuba sihlaziye iskripthi cloud-init ngokongeza ukukhuphela kwemifanekiso yesiseko kwisebe eliphambili. Oku kusivumela ukuba sibethe i-cache ekuqaleni kokuqala kwe-FB.

2. Usebenziso olusebenzayo lweziganeko zendawo : Umzekelo webala uqala ngedatha ehlaziyiweyo cloud-init . Kuthetha ukuba, emva kokuba i-node yendawo iqala, sele sele ikhona imifanekiso esisiseko kwi-node (ukusuka kwisebe eliphambili).

3. Ukusebenza okuphuculweyo : Inkqubo yeCD ingaqhubeka nokwakha imifanekiso emitsha ngaphandle kwemiba.

   
   

Esi senzo songeza imizuzwana eyi-17 (umnxeba we-API) kumbhobho wethu we-CI/CD.

Esi senzo siyavakala kuphela okokuqala xa siqala i-FB. Kwixesha elizayo, sihambisa ii-apps zethu kwiindawo esele zikhona, esele zinemifanekiso esisiseko, esithe sayihambisa kuthumelo lwangaphambili.

Inyathelo lokusasaza kwangaphambili

Sidinga eli nyathelo, kuba imifanekiso ye-FB ihluke kwimifanekiso yesebe eliphambili. Kufuneka sikhuphele isiseko semifanekiso ye-FB kwiindawo ngaphambi kokuba inkqubo yeCD iqale. Oku kuya kunceda ukunciphisa amaxesha okuqalisa okubandayo kunye nokusetyenziswa kwedisk ephezulu okunokuthi kwenzeke xa kutsalwa imifanekiso emininzi enzima ngaxeshanye.

Iinjongo zeNyathelo yokusasazwa kwangaphambili

1. Thintela Uxinzelelo lweDiski : Khuphela ngokulandelelana idocker eyona mifanekiso inzima. Emva kwesinyathelo se-init-deploy, sele sinemifanekiso esisiseko kwii-nodes, oku kuthetha ukuba sinethuba elikhulu kwi-cache hit.

2. Phucula iNdlela yokuSebenza ngokuFanelekileyo : Qinisekisa ukuba iindawo zokuhlala zifudunyezwa kwangaphambili ngemifanekiso ye-docker ebalulekileyo, ekhokelela ngokukhawuleza (phantse ngoko nangoko) amaxesha okuqalisa kwePOD.

3. Ukuphucula Uzinzo : Nciphisa amathuba okudibana neempazamo ErrImagePull / ContainerCreating kwaye uqinisekise ukuba iiseti ze-daemon zenkqubo zihlala "zilungile".

   
   

Ngaphantsi kweli nyathelo, songeza imizuzu eyi-10-15 kwinkqubo yeCD.

Inyathelo lokusasaza kwangaphambili Iinkcukacha:

• Kwi-CD senza i-DaemonSet kunye necandelo le initContainers .
• Icandelo le initContainers lenziwa phambi kokuba isikhongozeli esiphambili siqale, siqinisekisa ukuba imifanekiso eyimfuneko ikhutshelwe ngaphandle phambi kokuba isikhongozeli esikhulu siqale.
• KwiCD sijonga rhoqo ubume bedaemonSet. Ukuba i-daemonSet ikwimeko "elungile", siyaqhubeka nokuthunyelwa. Kungenjalo, silinda i-daemonSet ukuba ilungile.

Ukuthelekisa

Ukuthelekiswa kwamanyathelo okuqala kunye nokuhlaziywa kunye nenkqubo yokutshisa.

Into ephambili, ixesha elithi "Deploy" litshintshile (ukusuka kumyalelo wokuqala wokusebenza kwi-Running state of pods) ukusuka kwi-11m 21s ukuya kwi-25 imizuzwana. Ixesha elipheleleyo litshintshile ukusuka kwi-11m 21s ukuya kwi-1m 31s.

Iphuzu elibalulekileyo, ukuba akukho mifanekiso yesiseko evela kwisebe eliphambili, ngoko ixesha elithi "Deploy" liya kufana nexesha lokuqala okanye lincinci. Kodwa nangona kunjalo, sisombulule umba ngoxinzelelo lwediski kunye nexesha lokuqala elibandayo.

Ukuqukumbela

Umba ophambili ContainerCreating wasonjululwa yinkqubo yokufudumala. Njengenzuzo, sinciphise kakhulu ixesha lokuqala elibandayo le-POD.
Uxinzelelo lwedisk lwaluphelile, kuba sele sinemifanekiso yesiseko kwiinodes. Inkqubo ye-daemonSets ikwimeko "elungile" kwaye "isempilweni" (kuba akukho xinzelelo lwediski), kwaye asikhange sidibane nazo naziphi na iimpazamo ErrImagePull ezinxulumene nalo mba.

Izisombululo ezinokwenzeka kunye namakhonkco

• Sebenzisa kwiimeko zemfuno kwiindawo zokuhlala endaweni yeemeko zamabala
  Asinakusebenzisa ngale ndlela, kuba ingaphandle kohlahlo lwabiwo-mali lwethu lweendawo ezingezizo imveliso.
• Sebenzisa iAmazon EBS gp3 (okanye engcono) uhlobo lwevolumu ngokunyuka kwe-IOPS
  Asinakusebenzisa ngolu hlobo, kuba eli nqaku likwaphumela kumda wohlahlo lwabiwo-mali lweendawo ezingezizo imveliso. Ngaphezu koko, i-AWS inemida ye-IOPS yeakhawunti yakho ngokwengingqi.
• Nciphisa ixesha lokuqalisa isikhongozeli kwi-Amazon EKS ngevolumu yedatha yeBottlerocket
  Ngokwenyani asikwazi ukuhamba ngale ndlela, kuba inempembelelo ephezulu kakhulu kwimveliso nakwezinye iindawo, kodwa sisisombululo esihle kumcimbi wethu.
• Troubleshoot Kubernetes Cluster Autoscaler ithatha 1 hr ukukala 600 pods phezulu