Ngithole i-challenge: ukwakha i-real-time YOLOv8 i-video pipeline usebenzisa i-vanilla ONNX Runtime. Akukho i-frameworks ebuthile. Akukho i-Python bottlenecks. Just raw C++ grit. Kodwa uma ufuna ukudlulisela ividiyo live H.264 ngokusebenzisa inethiwekhi ye-neural ku-scale on edge hardware? I-Global Interpreter Lock ye-Python (GIL) ne-obsession yayo yokuphathelene ne-memory copying zihlanganisa izibopho. Ngemuva kokufika ngempumelelo elula: ukwakha ukuhlaziywa okusheshayo ye-video stream usebenzisa isikhathi sokuhamba se-ONNX ye-vanilla kanye ne-YOLOv8 imodeli yokuhlanganisa. Kuboniswa kulula ku-paper. Grab FFmpeg, ukulawula i-frames, futhi i-coding. Ngiyavuza kanjani i-prototype ye-10 FPS enhle e-rock-solid i-29 FPS beast, kanye ne-bugs ye-"final boss" ebonakalayo ngenkathi. In reality, it was a journey through engineering hell. (I-source code ephelele ye-masochists: i-video-yolo-dash-processor) Video-yolo-dash-ukudluliselwa I-FogAI Sandbox: Ukuvalwa ngaphambi kokusebenza Le repository ayikho umdlalo okuzenzakalelayo - kuyinto a Ngisebenzisa le mkhakha ukucindezeleka ngokugqithisileyo amamodeli ezithile ye-computer vision, ama-engine builds, kanye nama-optimization amamodeli ngaphambi kokuthuthukiswa ku-computer vision. . dedicated testbed FogAI core Uma isinyathelo (njenge-Zero-Copy hardware mapping) asikwazi ukufinyelela lapha ku-29 FPS, kungekho ibhizinisi ukuthi ngaphakathi uhlelo lwe-nerves autonomous zezimboni. Previous Chapters in the FogAI Saga: I-Manifesto: I-Prompts I-Overrated. Ngiyazi kanjani i-Node ye-Zero-Copy Fog AI ngaphandle kwe-Python I-Career Story: I-Prompts I-Overrated: I-I Built A Zero-Copy Fog AI Node ngaphandle kwePython (And It Hurt) Umthombo weSifo: GitHub: NickZt/FogAi I-"Memory Copy Tax" I-Trap I-prototype ye-computer vision iyatholakala ngokushesha ngenxa yokusebenza kwamemori njengoba umdlalo we-Hot Potato. I-architecture yami yokuqala yaba "umgangatho": I-FFmpeg ihlolwe i-H.264 ku-YUV hardware formats, i-converted ku-OpenCV (BGR) ukugcina imodeli, isicelo masks ku RGB umfanekiso, ukuguqulwa YUV, futhi ekugqibeleni i-encoder. cv::Mat Waze On a ARM CPU processing 4K frames, okuyinto overhead ivula kuze kube Ukushintsha bits kuphela. That's three unnecessary memory copies and two heavy pixel-format conversions. 30% of your cycles Ngiyaxolisa lokhu ngokusebenzisa Ngaphandle kokuguqulwa kwe-frame, i-mapped the Hardware Y-plane (Luminance) ngqo ku-OpenCV Ukuhlobisa Zero-Copy Hardware Mapping AVFrame cv::Mat C ++ // Mapping the hardware Y-plane natively - zero memcpy, zero overhead. cv::Mat y_plane(yuvFrame->height, yuvFrame->width, CV_8UC1, yuvFrame->data, yuvFrame->linesize); // YOLO segmentation masks now inject binary modifications directly // onto the hardware Y sequence. y_plane(bbox).setTo(0, valid_mask); Ngokuvimbela i-overhead ye-conversion, ngithole i-CPU ye-bottleneck ngokugcwele. Kodwa ngithole i-23 FPS. Why? I-Mutability ne-Asynchronous Reorder I-Profiling ibonise ukuthi ama-threads yami zihlanganisa ku-sequential death grip. Ukuphakamisa kusekelwe ku-mutation ye-shared internal buffers. Uma ngithole i-threads ezininzi ku-model eyodwa, zihlanganisa ngamunye, futhi uhlelo lashukumisa. YOLO Ngiyaxolisa i-concurrent pool of models--one unique ONNX model isithombe ngamunye worker thread. The Fix: std::unique_ptr<YOLO_Segment> Kodwa kukhona catch: Njengoba abacwaningi wahlala isikhathi ezahlukene, Frame 2 ingakhululwa ngaphambi kweFrame 1, okwenza ividiyo ukujabulela njenge-90s jump-cut. Ngiye ngimisa i-reorder buffer usebenzisa i-frame. H.264 ukunakekelwa ngokushesha. DASH video requires strict frame order. std::map C ++ // Reorder buffer logic to keep the stream sequential std::map<int64_t, FramePayload> reorderBuffer; int64_t expected_pts = 0; while (true) { auto payload = inferenceQueue.pop(); // Workers drop processed frames here reorderBuffer[payload.pts] = payload; // Emit frames only when the sequential timestamp flags align while (!reorderBuffer.empty() && reorderBuffer.begin()->first == expected_pts) { auto it = reorderBuffer.begin(); encoder.writeFrame(it->second.yuvFrame, it->second.pts); reorderBuffer.erase(it); expected_pts++; } } I-The Final Boss: Thread Cache Thrashing Ngaphepha, logic iyona ephelele. Ngokuvamile, FPS yami wahlala . I-Time-To-Inference (TTI) Imininingwane yamahhala kusuka ku-43ms kuya ku-890ms emangalisayo. 10 FPS I was a victim of CPU Cache Thrashing. Nangona ngifaka amabhodlela yami, izibuyekezo ze-ML (i-OpenCV kanye ne-ONNX) ziye "ngokusiza" ngokuvimbela ama-threads zabo zangaphakathi. I-ONNX Runtime: I-defaults ku-hardware_concurrency() / i-2 i-threads ngosuku. Nge-10 abasebenzi, ivela i-100+ i-threads e-CPU yam-20-core. OpenCV: Okuzenzakalelayo ukunikela abasebenzi ngezinsizakalo ezifana .setTo(). Abasebenzi yami abacebile abacebisa ama-threads e-ONNX, okuyinto abacebisa ama-threads e-OpenCV. Thousands of context switches were destroying my L1/L2 caches every second. Ukuguqulwa kwalo "Ngo" ebomvu ku-implacable concurrency. Ngitholise amabhayisikili izidingo zabo ukukhula ama-threads: C ++ int main() { // Globally disable implicit OpenCV threading cv::setNumThreads(1); // Cap ONNX Runtime to a single thread per op Ort::SessionOptions session_options; session_options.SetIntraOpNumThreads(1); session_options.SetInterOpNumThreads(1); } I-context-switching noocingo asuswe. I-cache yayo ye-CPU is a synchronized. I-pipeline ngokushesha iye yenza i-flawless nge TTI isixazululo ~329ms. 29 FPS I-Maintenance Over Ego: I-Strategy ye-Vanilla Umbuzo oluvame: "Uyaziwa kakhulu ku-performance, ngakho-ke akuyona i-fork ye-engine kanye nokuphucula ama-kernels ngokuvamile?" Ukusabela Technical Debt avoidance. Uma uxhumane i-internal ye-engine, uxhumane isilinganiso se-unending. Ngemuva kwalokho, isilinganiso esisha esuka nge-support for fresh hardware---like (57% yokushintshwa kwe-prefix) noma --- kufanele ukuguqulwa kwebhizinisi yakho eyakhelwe ngamanani. Ngokuvala nge-a , Ngingathola "ukushintshwa" lezi zokusebenza ezamahala ngokushesha nje ukusuka inombolo ye-version. ARM KleidiAI Intel DL Boost (VNNI) Vanilla Inference Engine Ngaphezu kwalokho, ngifuna ukuncintisana i-coding / decoding pipeline. Ngoba? Ngenxa yokuthengiswa kwe-hardware. Noma kungenzeka ukuthi noma i , lezi chips babe silicon-level acceleration for H.264. Qhagamshelana ikhodi ku ; wabelane ama-codecs kuma-metal yayo yasungulwa. Intel QuickSync Rockchip VPU Zero-Copy Bridge Umhlahlandlela: Hlola Ukuhlobisa, Hlola Ukuhlobisa Ukuphakama kwe-AI kumhlaba real kufuneka ukwehlise izindandatho ze-abstraction etholakalayo. I-Python ivimbela lezi zebhizinisi ze-latency kuze kube ukugcina uhlelo ekukhiqizeni. Uma unemibuzo enkulu ye-tensor payloads ku-video: Kill i-pixel conversions --- ukusebenza kwi-plane ye-hardware ngqo. Isolate amamodeli yakho--one isibonelo ngamunye abasebenzi. Reorder outputs sequential---ukunciphisa async isikhathi lokugqibela ukuchitha stream yakho. Ungakuthumela ibhizinisi akho ukuvelisa ama-threads zabo. Stay Vanilla---optimize isakhiwo yakho, akuyona injini, ukuze ukugcina ingozi tech encane. Next for the FogAI node? Thina prepping Ukuze zero-copy run ... Grounding DINO
