Ukusebenza kwe-LLM nge-LM Cache: I-Architectures, i-Strategy, ne-Real-World Applications

Abstract abstract Kulesi sihloko inikeza ukubuyekeza ubuchwepheshe okuhlobene nezimo zokusebenza kwe-LM Cache futhi indlela yokusebenza kwe-caching machinery ekuphuculeni ukusebenza, ukucindezeleka kanye nokunciphisa izindleko zokusebenza kwe-Large Language Model (LLM). Sihlola izinhlobo ezihlukahlukene zokusebenza ze-caching kanye namakhemikhali, indlela yokuxhumana nge-infrastructure entsha ye-AI kanye nokuhlolwa ukusebenza. Izibonelo ezisekelwe emkhakheni zibonisa indlela ezinye amakhasimende ethu amakhulu zokusebenza i-LM Caches ekuphuculeni. Ekupheleni, siphindezile ngokuvumela ezinye izinzuzo, izinzuzo kanye izindlela zokuqala kule ndawo esebenzayo ngokushesha. Introduction Ukuhlolwa Thola umphathi we-AI owenziwe ukunikezela imibuzo efanayo ngosuku zonke ngosuku eminingi! Lokhu akuyona imiphumela, kufanele ukuguqulwa yonke imibuzo ukusuka kwangaphakathi, ngokuvamile njalo? Nangona i-LLMs (ngena i-GPT-4, i-PaLM njll) zibonisa konke kusuka ku-support ye-customer kuya ku-codegeneration yayo Ukukhiqizwa, zonke izivakashi amamodeli amakhulu kungabangela imizuzu futhi ukucubungula kwe-GPU emininzi. Njengoba ububanzi amamodeli kanye nezidingo kwamakhasimende amakhulu, ukunemba okusheshayo kubaluleke kakhulu. deployment at scale is stymied by latency requirements, computational cost and memory footprint Ngokusho zonke lezi zihlanganisa, i-cache ye-LM ibonakalisa njengomphumela enhle kakhulu kumemfuno ezivela ku-cache kanye nokusetshenziselwa imiphumela eyenziwe kakade. I-cache yenza inkqubo yethu ukujabulela yonke into etholakalayo ngaphambi. Umqondo jikelele we-concept ayikho entsha emayunithi yekhompyutha, kodwa i-cache ye-LLM inesibopho enhle kakhulu nokunciphisa isikhathi sokuphendula kakhulu. Njengoba i-model distillation (ukwakha amamodeli amancane, amaphakhefu), ukukhiqizwa kwe-recovery-enhanjiswa (ukuvimbela ku-external knowledge bases), ne-vector databases for semantic search, i-LM Cache ibonakalisa nezinye iz Foundations of LM Cache Izinzuzo ze-LM Cache Umqondo we I-LLM is a reuse computing for not overwork. In the case of LLM inference, caching kungase exist at various levels of granularity: LM cache Cache the entire output to a given input prompt. If our model receives the same prompt again, it returns the cached output without running the model. This is effectively memorizing the model's response to entire queries, at the highest-level cache. It is particularly powerful for tasks that involve the reapplication of the same query many times (e.g. Q&A model with a single question asked repeatedly) This could be implemented, for example (sample implementation in the functions promptify and getPossibleResults) as a simple dictionary that maps prompts to possible results e.g. Prompt-level cache: Pseudocode: prompt-level cache integration prompt_cache = {} # A dictionary to store prompt->response mappings def generate_response(prompt): if prompt in prompt_cache: return prompt_cache[prompt] # return cached answer result = model.generate(prompt) # expensive LLM call prompt_cache[prompt] = result # store for next time return result We cache intermediate results for each token in a sequence instead of caching whole responses. Autoregressive LLMs generate text one token after the other. This is because without caching, the entire previous context will have to generated again from start for every new token to be created. Token level caching is the process of stores intermediate state of model tokens like the activations or hidden states in a way that, when producing next token, the model can skip recomputing what it already did for prior tokens. This is highly connected to the in transformer models, more on this below. Token specific caching: key–value cache LLMs based on transformers use self-attention, which means that each token pays attention to all of the ones that came before it. Frameworks cache the "key" and "value" projections for previous tokens and only add new ones to avoid having to compute attention over the whole sequence again. . This KV cache speeds up autoregressive decoding, which is an important GPT-style optimization. It cuts the work per step from O(n²) to O(n) without changing the outputs (Yazdani Aminabadi et al., 2022; Pol, 2025) arxiv.org. By default, libraries like Hugging Face Transformers do this with during generation, which means that only new tokens are processed. A simple pseudocode loop shows how grows with each iteration. KV cache (Key-Value cache) : neptune.ai use_cache=True past_kv Pseudocode: using KV cache in an autoregressive decoder loop past_kv = None for token in prompt_tokens: output, past_kv = model.decode_next(token, past_key_values=past_kv) # past_kv now contains keys/values for all tokens up to current one The above loop accumulates the key-value pairs to be carried forward as such as that model. Then can use the cached context and need not recompute it. The KV caching makes the quadratic slowdown now avoidable, and transforming models to work with this form of storage is crucial for the continued efficiency of transformer inference, particularly if GPT-4 like models are able to generate much longer continuations far more efficiently. past_kv model.decode_next LLM follows the same procedure with all text input as firstly it converts tokens into vector embeddings. Calculating these embeddings (especially a large contextual embedding model) can cost nearly as much as running the full LLM itself, so why do it again for some common tokens/phrases? Embedding Cache — a cache where the vector representations of common inputs are stored. (Paul, 2025) . For example, we can cache those embeddings instead of computing them from scratch if application encodes the same sentences or document chunks many times (in a retrieval-augmented setup) This will save time and guide the style of such representations. . Embedding caching is a natural fit for retrieval pipelines and semantic search, as it caches the “input features” before they hit the big model. Embedding cache: rohan-paul.com rohan-paul.com Multi-Query Attention (MQA) is a type of Transformer attention in which all heads use the same keys and values as input to their attention function (Shazeer, 2019) . Typically, with 12 or 24 heads, each head has its own keys and values. This means that the cache stores 12x or 24x vectors for each token. Instead, MQA caches just one set, and heads keep their query projections while looking up a shared key-value store. In one benchmark (Shazeer, 2019) ( ; ), this cut memory and bandwidth overhead by a huge amount, from about 15 GB to less than 400 MB on a 16k-token context. There is only one shared cache instead of 100% memory overhead per head. Multi-query attention caching: arxiv.org arxiv.org huggingface.co Noam Shazeer first suggested MQA (and its grouped-query versions) in One Write-Head is all we Need ( ). It is used in PaLM, Falcon, MPT, and BLOOM to make longer contexts work better. It also makes inference faster because it needs to fetch less data from memory, which means it can do attention computations faster ( ). In theory, MQA is a smart way to cache data for the architecture, so that the number of trainable parameters doesn't go up as the number of heads or the length of the context grows. huggingface.co huggingface.co us_cache=Ukuhlezi ekupheleni_kv ekupheleni_kv isicelo.decode_next Zonke izinga ezidlulileyo ze-caching zisebenzisa isisombululo esifanele ukuthi Ngenxa ye-self-attention mechanism, ukucubungula ngamunye i-token entsha e-transformer-based LLM ngokuvamile kusekelwe ama-token ezidlulileyo. Noma ku-prompt level noma ebhokisini, ukucubungula imiphumela yama-calculations ezidlulileyo ivumela imisebenzi ezidlulayo ukujabulela ngqo ku-resultat. Izinzuzo eziningi zokushesha kanye nokusetshenziswa kwamakhemikhali zihlanganisa lokhu. system shouldn't re-compute what you have already done LM Cache Architectures and Strategies I-LM Cache Architectures kanye ne-Strategy Njengoba akuyona zonke izimpahla zokusebenza zihlukile ku-LLM izinhlelo, inani elikhulu lwezakhiwo kanye nezinhlangano ziye ziye ziye ziboniswe ngu-engineers ukwandisa i-cache hits kanye nokunciphisa i-overhead. Nazi ezinye izindlela eziyinhloko: I-Static vs. Dynamic Caches: I-Caches ye-Static ikunye ne-pre-filled input / output pairs, okwenza kubo ezinhle ukuze zokusebenza kwezimpendulo ezokufanele. Lezi zixazululo ze-caching yezimpendulo ezivamile noma izivakashi ezivakashi ezivakashi, njenge-FAQ noma izicelo zokusebenza. I-Caches ze-Dynamic, ngokuvamile, zihlanganisa njengokusebenza kwe-system, ukuguqulwa ukuhlangabezana nezimpendulo zokuhamba, ukuguqulwa kwezilinganiso ze-query, kanye nezinsizakalo zokusebenzisa. Lezi zihlanganisa "ukufunda" kusuka ku-user traffic ukuthuthukisa izinga lokuphumula. Kwi-real life, izinhlelo z I-Distributed and Federated Caching: Ngo-Distributed Caching: I-Distributed Caching: I-Distributed Caching yi-Distributed Caching yi-Distributed Caching yi-Distributed Caching yi-Distributed Caching yi-Distributed Caching yi-Distributed Caching yi-Distributed Caching yi-Distributed Caching yi-Distributed Caching yi-Distributed Caching yi-Distributed Caching yi-Distributed Caching yi-Distributed Caching yi-Distributed Caching yi-Distributed Caching yi-Distributed Caching yi-Distributed Caching yi-Distributed Caching yi-Distributed Caching yi-Distributed Caching I-caching ye-layer-wise kanye ne-GPU/TPU memory optimization: I-caching ayidingi ku-output; ingasetshenziselwa kuma-layers ezithile ze-LLM. Ukuze ukunceda ukucubungula i-intermediate layer ephakeme ngokushesha ku-cache hit, ingatholakala, isibonelo, ukucubungula i-outputs ye-layer yayo ngenxa ye-inputs ezithile ezingenalutho. Ngaphezu kwalokho, kungase kunzima ukucubungula kuphela ama-activations ezingenalutho noma ukusetshenziswa kwe-on-chip memory ku-cache ku-hardware accelerators (GPUs/TPUs), lapho isithombe yinto enhle. I-inference analogs yobuchwepheshe afana ne-gradient checkpointing, I-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye-Cache ye- I-Cache Size and Eviction Policies: Ngenxa yokuxhumana kwe-cache ayinambuzane, i-LLM isetshenziselwa izinsizakalo ezifana ne-LRU noma i-LFU ukuchaza ukuthi izihloko zihlanganisa ne-eviction. Lokhu kususa indawo ye-data entsha. Ngaphandle kokuxhumana kwezinto ngokuvamile noma ngokuvamile, ama-cache ye-LLM kufanele ukulawula ukuchithwa kwe-relevance kanye ne-model noma ukuhlaziywa kwedatha ngokuvimbela izinto ezindala. Ngo-2024, izifundo zibonise ukuthi ukuchithwa kwe-learning-driven isebenza: ama-cache eyenziwe ukufumana nokukhuthaza ama-keys nge-future value eningi (Xiong et al., 2024). Lezi zinsizakalo Ngokuvamile, ukwakhiwa kwe-cache ye-LM kuyinto i-balance enhle: Sithanda ukwandisa i-hit rate nangokunciphisa i-overhead efana nokusetshenziswa kwe-memory kanye ne-volatility. Uma lokhu kubalulekile ukuba usebenza izinhlelo zokudluliselwa kwama-machine emangalisayo kakhulu, bese uye nomkhosi yami! Lezi zindlela (i-static vs. dynamic, i-architectures e-distributed, ukwandisa i-cache e-hit ku-terminology ngokuvimbela ku-client time nge-example languages) zihlanganisa zonke izici ezinikezele ukwenza ngaphezulu kwe-cache ngaphandle kokuphumula ama-cache-related amaphakheji. Izicelo ezahlukahlukene zingahlanganisa lezi zindlela futhi I-Architecture Diagram ye-LM Cache ku-LLM Inference Pipeline Ngezansi, isakhiwo se-reference esibonisa ukuhlanganiswa kwe-LM Cache ku-LLM ye-typical inference flow: I-Workflow Diagram I-Cached vs. I-Non-Cached Inference Requests Ukuze uthole okwengeziwe ukubuyekeza izinzuzo, bheka isibuyekezo se-workflow phakathi kwezicelo ze-cache ne-non-cache ye-inference: Ukubambisana ne-AI modern infrastructure Ukucaciswa kwe-LM akuyona nje - kuya kuholele kusuka ku-application e-ecosystem e-AI. Ngiyazi ukuthi i-caching iyahambisana ne-infrastructure kanye ne-frameworks eziningana: The good news is that KV caching is integrated into the majority of contemporary AI frameworks. To avoid redundant computation, Hugging Face Transformers, for example, allows you to set (the default in generate()) to automatically return and reuse previous key/value states. KV caching is used for incremental decoding in Microsoft's DeepSpeed-Inference, which is crucial for scaling large autoregressive models. Kernels are tuned appropriately. (Yazdani Aminabadi et al., 2022) . Cache decoding is also accelerated internally by NVIDIA's TensorRT and ONNX Runtime. In order to maximize speed, model architectures now assume caching in addition to software stacks. UC Berkeley's vLLM presents PagedAttention to manage GPU KV cache, paging it similarly to virtual memory and sharing identical prompt prefixes. (Kwon et al., 2023) . FlashAttention and xFormers optimize attention only for new tokens. This block-based approach reduces memory fragmentation, than standard Transformers. . Additionally, you can anticipate even wider caching support across popular frameworks as these research advances develop. Framework Support (Hugging Face, DeepSpeed, etc.): use_cache=True huggingface.co. arxiv.org blog.vllm.ai lowers overhead to under 4%, and delivers up to 24× higher throughput blog.vllm.ai Caching can greatly improve performance in retrieval-assisted LLM pipelines, such as question answering over a vector database. You can cache retrieved passages, final answers, or embeddings for semantically similar queries rather than using similarity search to retrieve new documents and calling the LLM each time. For example, the 2025 Proximity layer by Malladi et al. reduces retrieval latency by up to 59% with negligible accuracy loss by reusing passages for near-duplicate queries. . Caching retrieval results reduces the load on vector engines and scales systems because many user queries overlap (Chaidez, 2025). Caching query outputs is advised by providers such as Redis and Pinecone. In the end, caching saves duplicate embedding or retrieval effort for frequently asked or comparable queries. Retrieval-Augmented Systems (RAG and Vector DBs): rohan-paul.com These are addressed by caching in the case of serverless LLM deployments (a.k.a. AWS Lambda) and edge scenarios (on-device or in-browser LLM inference) that need to mitigate expensive cold-starts and tight resource limits. For serverless setups—in which models do not persist between invocations—an external store (like Redis or DynamoDB) enables the completion of each function call to check previous outcomes, so that redundant work can be skipped. There is potential to cache recent inferences or embeddings (e.g., user names, common commands ) on things like smartphones or small IoT type edge devices, to reduce latency and save battery. While limited memory and privacy will be a consideration, optimized should start to appear soon for ultra-efficient edge inference. Serverless and Edge Deployments: compressed caches Production-scale LLMs exceed single-GPU or machine capacity, so distributed inference requires caching. Typically, each model shard caches the key/value states for tokens it processes. When a new token arrives, parallel shards coordinate to fetch and update their cached states, avoiding redundant recomputation and memory overhead. Libraries like DeepSpeed and FasterTransformer implement this: with pipeline parallelism, each stage caches its outputs to avoid recomputing all layers per token. Tensor parallelism shards KV , to synchronize new keys/values and maintain consistency across devices. Caching and Model Parallelism: caches across GPUs using cross-GPU protocols Distributed KV caching by limiting work to new computations. One whitepaper noted that without caching, shipping the full context for each inference during distributed execution incurs prohibitive network hops. Caching keeps context local, making large-model inference across multiple machines practical. dramatically reduces inter-node communication and idle time us_cache=Ukuhlezi Ukuphathelene, izakhiwo ezintsha ze-AI zihlanganisa kakhulu. Izinhlelo ezingaphezulu zihlanganisa ukuvuselelwa oku, kusukela izibuyekezo zokusiza imishini yakho kuya ku-systems architecture elihlanganisa ama-caches phakathi nezicelo nezinfrastructure. I-cache iyiphi na inkonzo ezininzi ze-LLM zokusebenzisa, njengoko lokho abacwaningi ezinkulu ze-AI (i-OpenAI, i-Google ne-Microsoft isibonelo) zihlanganisa noma zenziwe ngezicelo zayo zokusebenza ze-LLM. Izinhlelo ezithile zaziwa kakade njenge-OpenAI's ChatGPT (hhayi-publicly-documented, kodwa le maphulo ikakhulukazi ibhaliswe) okwenza ukusetshenziswa kwe-context caching em Performance Metrics and Evaluation I-Performance Metrics ne-Evaluation Okay, ku-theory caching kungase kubonakala emangalisayo kodwa indlela ungayifaka? Kukho ezinye izinga lokusebenza ezivamile ezinikezela ukulawula ukusebenza kwe-LM caching I-Lower Latency: Lower Latency: I-advantage ikakhulukazi i-latency engaphansi, okuyinto i-time it takes to get an answer. I-caching i-caching i-performance curve: i-cache primed ngokuvamile inikeza i-high throughput kanye ne-low latency. Senza lokhu ngokuchofoza amahora etholakalayo yokwenza imibuzo nge-cache futhi ngaphandle kwe-cache. I-chatbot eyenza imizuzu angu-2 ukuchofoza imibuzo iyahora angu-0.2 uma ivela ku-cache, okungenani angu-90% ngokushesha. I-speedup efanele ku-hit rate. Ngokwesibonelo, uma i-80% yama-hits iyahora angu-1000, i-latency Ukuphuculwa kwe-Throughput: Nge-cache, i-throughput -- requests per second (RPS) noma i-tokens per second (TPS) - ingatholakala nge-roof. Lokhu kuncike ukuthi i-cache ibhokisi ukucubungula ngokunciphisa ukucubungula okufanayo. I-DenseNet benchmark ye-NVIDIA Triton wahluka kusuka ku-80 kuya ku-329 inf/sec (okusho kwe-4×) futhi amamodeli amancane babona amayunithi angama-20% nge-caching yesisombululo. rohan-paul.com. Uma kuxhomekeke ku-LLM serving, i-PageAttention ye-vLLM isetshenziselwa ukuphuculwa kwe-KV cache ngokufanelek I-Compute/Cost Savings: Kuyinto imiphumela ye-dollar-and-cents ye-caching. Njengoba i-caching iyakwazi ukunciphisa ukucubungula okuzenzakalelayo, kunikeza i-FLOPs ezingenalutho (isib. Uma ushiye i-GPU ye-hour noma i-API call). Ukucubungula okuqukethwe okuqukethwe kwe-GPU futhi ngisho isikhathi sokucubungula okuqukethwe. Ngokwesibonelo, i-Anthropic ibizwa ukuthi ngezikhathi ezinguqulo ezinguqulo ezinguqulo, i-caching ye-prompt angakwazi ukunciphisa ukusetshenziswa kwe-token (ngokuthi izindleko) nge-90% (i-Anthropic, 2024), i-anthropic.com. Lokhu kwenziwa futhi ku-pre I-Hit Rate ne-Miss Penalty: Lezi zindlela ezizodwa ze-cache. I-Hit Rate: I-percentage ye-requests (noma ama-tokens) eyenziwe nge-cache I-Hit Rates engcono kakhulu ngokuvamile. Lezi zinto ezisebenzayo ukuhlaziywa, njenge-”istimu yethu iye iye iye iye iye iye iye iye iye iye iye iye iye iye iye iye iye iye iye iye iye iye iye iye iye iye iye iye iye iye iye (X ngokuvamile X vs Y ngaphandle kwe-cache). I-miss penalty kuyinto okungenani okungenani iye iye iye iye iye iye iye iye iye I-Memory ne-Overhead Trade-offs: I-Caching ibhalisele imisindo yama-model noma izimo ezingu-intermediate, okuyinto isetshenziselwa i-RAM ye-GPU ngaphezulu. Ngokwesibonelo, i-Cache ye-KV ifakwe eminyakeni eluhlaza. Ngokwesibonelo, i-Cache ye-model 405 B ye-16k tokens ingasebenzisa cache kusuka ku-15GB (okusho eminyakeni yama-model). I-GitHub isetshenziselwa ama-metricals efana ne-"memory overhead per additional token" noma "i-cache-size ratios" ukuze ufunde i-compromises. Ngokusebenzisa imishini efana ne-multi-query attention, ungakwazi ukunciphisa i-cache kusuka kumayib Ukunemba / Quality Impact: Ukukhiqizwa kwe-model kufanele akuyona (ngaphandle kokusebenza okungenani okungenani okungenani) ngenxa ye-cache. Kodwa-ke, eminye amamodeli ye-cache (isib. Amamodeli okungenani noma semantic), ukuguqulwa kungenzeka ukuthi akuyona okungcono ukusuka kwebhizinisi entsha. Ngakho-ke, singakwazi ukubona ukuthi i-cache ikhiqiza ukunciphisa ikhwalithi noma akuyona. Ngokuvamile, uma siphindeza imibuzo esebenzayo kwebhizinisi entsha futhi impendulo ye-cache entsha kufanele akuyona ngokufanelekileyo, ngakho-ke akuyona njenge-accuracy noma ukubuyekezwa kwe-relevance. Kwi-academic community, ukuthi isilinganiso se Ukuqhathanisa ngeBaselines: Abacwaningi ngokuvamile uhlanganyela imodeli yokukhishwa kwe-cache nge-base-line ngaphandle kwe-cache ukuze ukhawuleze ukufinyelela. Kutholwe futhi ukuqhathanisa ukuqhathanisa izindlela ezahlukene ze-cache. Isibonelo, caching-match-exact vs. caching-based-embedding-based vs. akukho caching ku-dataset ye-queries, nge-latency ne-accuracy ngoba. vLLM vs. vanilla HF server Throughput & Memory. Lezi zifundo zokuhlanganisa zihlanganisa izici ukuze kuqinisekisa ukuthi caching njengoba umugqa we-tomb is a good idea. Umphumela isibonelo ingaba: caching iyahambisana nge-2x kanye ne-latency nge-40%, nge- Uma thola i-LM caching, kubalulekile ukuchithwa ama-tradeoffs lapho ungakwazi ukucubungula ku-computing kodwa ngokushesha kwe-memory; ukunciphisa i-latency lokugqibela, kodwa kufuneka ukuhlangabezana nezimpendulo ezingenalutho. Ngakho-ke, kubalulekile ukubuyekeza ama-metrics efana ne-one emaphaketheni. Isibonelo, i-5% ingozi yokuphendula okungenani-out-of-date ingatholakala ukuthola imiphumela yakho i-10x ngokukhawuleza - noma kungekho (ukuxhumana ne-medical vs. i-casual chit-chat zihlanganisa ezahlukile!). Zonke izibuyekezo zemfundo kanye nezibuyekezo zebhizinisi zihlanganisa i-cache ku-benchmarks zayo, okuyinto ibonakalayo ngokuvamile ngokuvumelana nokuqukethwe kwe-cache. Kule isifundo esilandelayo ka-2024, kuboniswa ukuthi cishe zonke izivakashi eziphambili zokusebenza kwe-LLM ezingenalutho zihlanganisa ngezindlela ezithile zokusebenza kwe-cache noma ukucindezeleka ukuhlangabezana ngokushesha ne-context length (Chen et al., 2024). Izibalo zihlanganisa ngokuvamile izixazululo ezikhulu. Ngokuphathelene: i-cache ukunciphisa i-latency, ukwandisa i-throughput kanye nokunciphisa i-cost ngalinye ngokuvumelana neziv Ngaphandle kwe-optimizations, noma iyiphi isilinganiso se-LLM inikezela ku-complete. ikhaya Real-World Case Studies I-Real-World Case Study Konke singazi ukuthi caching akuyona kuphela ngenxa yobugcisa yenzululwazi, futhi lezi izici zithunyelwe umthamo emangalisayo emzimbeni. Ngiyazi ezinye izibonelo ezahlukile, nje ukubonisa ukuthi caching usizo : I-AI ye-Conversational (Chatbots/Assistants): Ngaphandle kokukhipha idatha ephelele ngalinye, i-multi-turn chat LLMs i-cache izivumelwano ezidlulile. Ngokwesibonelo, zihlanganisa isivumelwano se-user yokuqala kanye nesivumelwano se-assistant ukuze kuphela isivumelwano esisha kusetshenziselwa izivumelwano ezidlulileyo. huggingface.co. Lokhu kuholela izinhlelo ezifana ne-ChatGPT ukuguqulwa ngokushesha ngaphandle kokwenza yonke imathematiki. Nge-cache ebizwa, i-Anthropic's Claude 3 ivumela abathengisi ukuchaza izimo ezide, okunciphisa i-latency nge-79% nezindleko nge-90% ku-100k-token prompt. (Anthropic, I-Code Completion and Assistance: I-GitHub Copilot ne-TabNine usebenzisa i-caching ukuze ijubane ukufinyelela kwe-code. I-model kufanele inqubo inkqubo ye-context ephelele ngokushesha lapho uxhumene i-key, okwenza kube lula kakhulu. Ngaphandle kwalokho, zihlanganisa ukubonisa kwe-model ye-prefix ku-cache ukuze kuphela ama-characters ehlanganisiwe zihlanganiswe. Lokhu kubizwa ngokuthi i-caching ye-prefix. Iningi kwezimpendulo zokuphrinta zihlanganisa (njenge-Fibonacci), ngakho-ke i-caching ye-prompt-level inikeza izimpendulo efanayo ngokushesha. I-AI Caching Assistants ingathola izimpendulo ezivamile ze-developer ngamazwe I-Enterprise “Copilot” Assistants: Izinkampani ezininzi zithunyelwe ama-AI assistants ezinsizakalo ezinsizakalo ezinsizakalo ezinsizakalo ezinsizakalo ezinsizakalo ezinsizakalo ezinsizakalo ezinsizakalo ezinsizakalo ezinsizakalo ezinsizakalo ezinsizakalo ezinsizakalo ezinsizakalo ezinsizakalo ezinsizakalo ezinsizakalo ezinsizakalo ezinsizakalo ezinsizakalo ezinsakalo ezinsakalo ezinsakalo ezinsakalo ezinsakalo ezinsakalo ezinsakalo ezinsakalo ezinsakalo ezinsakalo ezinsakalo ezinsakalo ezinsakalo ezinsakalo ezinsakalo ezinsakalo I-Multi-turn Tools/Agents: I-Caching ingathola i-short-circuit i-cycle e-AI agents izinhlelo ezinzima (njenge-agent eyenza umsebenzi nge-multi-LLM call, futhi isetshenziselwa izixhobo). Ngokuquka i-agent enikeza imibuzo ngokuchofoza inkonzo ye-web kanye nokufumana imiphumela ukuze isetshenziswe njenge-input ye-LLM. Ngokuvamile, uma imibuzo efanayo isetshenziselwa okokuqala, i-agent ingathola imibuzo edlule kusuka ku-cache yayo ngqo (ngaphandle kwe-re-invocating izixhobo) noma ivame ukusetshenziswa okungagunyaziwe noma iyiphi imiphumela yokufaka (iziphumela ze-cache izixhobo kanye ne-LLM). Izinsizakalo ze-Internet: Lokhu kubalulekile ikakhulukazi ngama-search engama-caching engama-LLM. Ngokwesibonelo, i-Bing Chat inikeza amamilioni abasebenzisi futhi kukhona izicelo eziningana eziningi. Uma i-LLM backend ye-Bing (e-GPT-4) inikeza imibuzo yemibuzo ezivamile ("What's the capital of X", "How to make pancakes"), kunokukwazi ukuguqulwa ngokushesha kunomphumela ukusuka kwangaphambili ngosuku zonke. Ngokuvamile, i-non-LLM search ye-Bing isebenzisa i-query caching iminyaka eminyakeni; nge-LLM imibuzo eyenza manje sikhona isampula esifanayo esifanayo. Abacwaningi wabheka uhlelo esifanayo ukuthi uma Ngo-August 2024 I-LLM ye-Cache ye-Anthropic ingasiza ukuvikelwa kwekhwalithi eningi, engatholakali, njenge-book (100,000 tokens). Ukubhalisa ku-cache kunezinto ezingaphezu kuka-25% ngalinye ye-input token, kodwa ukubhalisa kwekhwalithi e-cache kunezinsiza kuphela i-10% ye-base rate. . Umhlahlandlela wokuqala ukugcina idatha; Umhlahlandlela ezilandelayo usebenzisa i-Prompt ID ukufumana futhi ukunikela kuphela impahla entsha. I-Clients wabhala ukuthi imibuzo yaba ngempumelelo angaphezu kwama-4 ngesivinini (i-latency wahlala kusuka ku-11.5 amaminithi kuya ku-24.4 amaminithi) futhi ukuthi futhi izindleko ze-10,000-token prompts zangaphantsi nge-86%. . Izesekeli zokuxhumana, ama-coding assistants, kanye ne-big-document Q&A zihlanganisa zonke izicelo ezivamile, futhi umklamo oqinile iyona engcono ukulanda kwe-cache. I-Anthropic iye yenza i-cache yokusebenza okuqinile ngokubandakanya ukuthuthukisa ukusebenza nge-incentive zezimali. Akungabonakali ukuthi ama-concurrents ezifana ne-OpenAI ne-Google zihlanganisa izici ezivamile ukuze uthole izindleko ezifanayo nezinzuzo. Anthropic's Claude 3 added Prompt Caching I-antropic cost of 100,000-token prompts went down by 90% I-antropic Isimo esinye: . I-OpenAI ayikwazi ukucubungula okuhle ngokubanzi, kodwa ingathanda ukuthi ama-developer babonise futhi / noma babonise imithombo embalwa embalwa ngalinye isicelo, ukuthi ingxenye ingatholakala ku-server-side cache ukuze imodeli ayidinga ukulayisha ama-examples kusuka ekupheleni kwelanga. Kuyinto efana ne-Anthropic's approach. isib. isib. Uma ungathumela i-2000 token system prompt + Instruction on zonke i-API call (eyenza ezininzi), i-OpenAI ingathayela i-model embedded yayo ngaphakathi futhi kuphela i-concatenate ne-query yakho processed representation. It uya kuba ngokuphelele kakhulu kubasebenzisi kodwa (ngakwazi ukucubungula) inani elikhulu le- OpenAI's system messages and few-shot examples caching Ngokuvamile, cishe zonke amazwe ezinkulu ze-AI zisebenzisa i-caching ye-LM emoyeni enhle. Nakuba i-OpenAI, i-Anthropic, i-Microsoft, i-Google ne-Hugging Face zihlanganisa noma zihlanganisa izindlela ze-caching nge-learning ezahlukile njengesiphumela, amaningi zihlanganisa efanayo: i-caching inikeza ukukhuthaza kwamandla enhle, kodwa kubaluleke ukulawula i-cache freshness ngaphandle kokuphumelela emoyeni nokuphumelela. Izindaba ze-"ukukhuthaza i-cache futhi bakwazi ukunciphisa i-30% ukusetshenziswa kwe-GPU yethu ngenkathi ukugcina i-QPS enhle" akuyona. Ekupheleni kwelanga, lezi izifundo zihlanganisa ink caching is the ticket to turbocharging LLM applications. Challenges and Limitations Izinzuzo futhi Izinzuzo Ngokuvamile, ukucindezelwa kwama-cache akuyona i-silver bullet. Lokhu, ngokuvamile, kuyinto enhle - uma kulula, wonke umuntu uyaziqhathanisa yonke into! Kukhona izinzuzo eziningi nezimfuneko ezisebenzayo ukuthi ama-engineers abakwazi ukuyisebenzisa i-cache ye-LM: I-Cache Invalidation (i-Stale Data): Uma idatha esifundeni noma i-LLM update ukuguqulwa okungenani lokuphendula, imiphumela ye-cache iyahlekile, njenge-web cache invalidation imiphumela. Isisombululo enhle kuyinto ukunciphisa zonke ama-cache lapho imodeli iyahlaziywa. Lokhu kulula kodwa kuyimfuneko ngenxa yokususa ama-entries ezingenalutho. Abanye abasebenzisa isicelo se-"ifakwe, catch fresh", kodwa lokhu kungase kuholele ukuba zithole idatha enhle. Kubalulekile ukunikeza imiphumela i-time-to-live (TTL) izindleko ukuze zithole ngemuva kwesikhathi esifundeni esifundeni, noma ukuze i-version cache key yi-ID ye-model noma ama Ukukhishwa kwe-Context ku-Dialogue: Uma i-context iyahlaziywa ngemva kwe-multi-turn chat, imibuzo e-cache ingangena. Ngokwesibonelo, uma ushiye impendulo ku-question A bese umsebenzisi ukongeza ulwazi oluthile, ukusetshenziswa kwama-response efanayo ungayifaka ama-changes ebalulekile. Ukuze ukugcina lokhu, ama-cache isitoreji isitoreji id-conversation ephelele, akuyona kuphela inguqulo esidlulile, noma i-cache kuphela izigaba ezincinane ze-dialogue. Ngaphandle kokugcina imibuzo ephelele, ungayifaka ama-summary ezincinciphisa. Lokhu kuncubungula ingozi yokuqinisekisa okuphakeme. Ingxaki ukulawula i-cache kahle. I-Prompt Variability kanye ne-Unicity: I-LLM isebenza ngokuvamile nge-inputs eziningana kakhulu, ngakho-ke ama-cache e-naive akusebenza kahle kakhulu. Ngokwesibonelo, izicelo ze-creative-generation ezivela ukuba "bheka i-poem mayelana ne-X ngesisindo se-Y nge-twist Z" akufanele ngokuvamile ukuthola iziphumo ngaphandle kwe-semantic caching. Ama-cache ze-semantic asebenza ngokuvamile ngokubambisana ne-embeddings, kodwa zisebenza kuphela lapho izicelo zihlanganisa kakhulu. i-rohan-paul.com. Izinzuzo ze-cache zihlanganisa ukusabalalisa umsebenzi. I-KV cache iyasiza izisombululo eziningi, kodwa i-ad-hoc full-response I-Memory Footprint: I-Caches isebenzisa inani elikhulu le-memory, noma i-RAM noma i-GPU, ngoba i-KV cache ye-context ye-long can be as large as the model itself. huggingface.co. I-Cache ye-Response ingabangela kakhulu, okungenani kufuneka ifakwe ku-CPU noma i-disk. Lokhu kubalulekile ukuba ukhethe phakathi kwama-capacity ne-model size noma i-batch throughput. Ngokwesibonelo, uma ukhethe i-GB 20 ye-GPU ye-40 GB ye-cache, akukho indawo kakhulu amamodeli amakhulu noma ukwelashwa izicelo ezininzi. Ukuze uthole ukwelashwa okuhle, abacwaningi ama-KV tensors (Shutova et al., 2025) argiv.org noma ama-qu I-Complexity and Engineering Overhead: I-Caching inikeza izinto ngokukhawuleza, kodwa inikeza izinto emangalisayo. Ufuna ukwakha izindlela ze-trade-safe, ezingenalutho zokuhamba, ukubuyekeza, ukuhlaziywa, nokugqwala ama-entry, okungenani kubalulekile ukusetshenziswa kwamahhala ezingenalutho ezifana ne-Redis ukuze i-cache eyenziwe, okuyinto ingangena. Ukuze sicele ukuthi izinto zihlanganisa, ungenza ukwenza izinsizakalo ezinzima, njenge-usebenzisa ama-blocking async calls noma izakhiwo ezingenalutho. I-Debugging iyatholakala kakhulu ngoba imiphumela emangalisayo ingangena kusuka kumadatha ye-cache ebusuku noma eyenziwe emzimbeni. Ngakho- I-Security kunye ne-Privacy: Uma akunakekelwa ngokufanelekileyo, i-caching ye-sensitive prompts ne-outputs ingakwazi ukwenza idatha ye-private ngokuvumelana. Kwi-multi-tender systems, uxhumane ngokuzenzakalelayo ukufinyelela kwe-cache ku-requests ezivumisiwe kanye nokufaka ama-entries e-user noma isizinda eyodwa. Ama-caches zihlola ingozi ye-side-channel attacks, okuyinto ama-hackers zibonisa izivakashi kwezinye abantu. Ukuze ukunciphise ingozi, noma ukunciphisa impahla ye-cache sensitive. Nokho, ukucubungula kungenziwa ukuxhumana kwe-semantic. Kwakhona, izinhlelo ezifana ne-GDPR z I-Cache Coherence ku-Distributed Systems: I-Cache e-Distributed inokukwazi ukwenza amaphepha idatha, kodwa inesibopho ngokuhambisana. Ngokwesibonelo, izivakashi noma ama-invalidations eminyakeni eyodwa akuyona bonke, futhi uma amaphepha amabili abalandeli i-cache elifanayo ngexesha elifanayo, kungase kufuneka ukuguqulwa nokuthumela isicelo esifanayo. Ukuqhathanisa ama-duplicates, kufuneka izinsizakalo ze-athomu noma ama-lockers. Ngokwesibonelo, izivakashi ze-athomu ze-Redis zisekela lokhu. Medium. Ukuze ukwakha i-cache e-distributed, kufanele ukhethe i-protocols ye-coherence enhle noma uk Nokho, i Ngakho-ke, akugqiba kakhulu ukuthi cishe wonke inkonzo ye-Enterprise-spanning LLM kuhlanganise caching noma enye ifomu noma ezinye, futhi ingxaki kuyinto wonke umntu wokubeka le design umsebenzi ukuxhumana nezimo. Ukunciphisa izimo: I-Cache izinto okuyinto engaziwayo kakhulu (noma amahora ngokushesha) + Usebenzisa izindlela eziningi zokugqoka kwe-cache + Hlola ukusetshenziswa kwe-hit rate / ram kanye Nika izimpendulo (Uma yini okufakiwe nge-cache yakho, uhlelo uya ukusebenza, kuya kuba kulula). Sishayele ukubona izixhobo ezinhle nezibuyekezo ukuxhumana kakhulu okuzenzakalelayo (isibonelo, cache encrypting-level encrypting noma i-auto-validate hooks ku-model updates). overall performance gains of caching generally significantly exceed LLM kuyinto enhle kakhulu, kodwa akuyona mahhala njengasehlisa futhi inikeza ingxubevange eyengeziwe ye-LLMs - i-cache. Enye ye-experiment esaziwa kakhulu ye-informatics yenzulululwazi yenza lokhu: . Abacademics LLM manje uyazi yokuqala! Kodwa nge design enhle, i-compromise ingasetshenziselwa, futhi izinzuzo eziningi zingatholakala. Kwezinye izinto ezimbili ezinzima ku-Computer Science: ukuguqulwa kwe-cache ne-naming izinto Future Directions and Research Opportunities Izinzuzo zokuqala kanye nezidingo zokufundisa I-LLM cache field kuyinto enhle kakhulu, nge izindlela ezininzi ezinzima ukuthi abacwaningi kanye nezobuchwepheshe zihlole ukuthuthukiswa okwengeziwe izindlela zayo: I-Adaptive and Intelligent Caching: Machine learning ingasiza nge-caching kanye ne-eviction amaphrojekthi. Ngokwesibonelo, ama-agents e-RL sincubungula ukufumana izinzuzo eziningana kakhulu (i-hit rate × i-correctness). Baya i-prefetch idatha ukuthi bafuna, njenge-OS-read-ahead, i-cache imiphumela emikhulu ngesikhathi esifushane, futhi ukunciphisa ama-entries engabizi kakhulu. Ukusebenza kwe-speculative kusebenza lapho ama-cache zihlanganisa ngokuvamile futhi zitholele idatha ezinokwenzeka. I-LLM-based meta-learning ingathola ukuthi izibalo ezingu-intermediate izindleko futhi zitholele ezingenalutho ku-cache. Ukufundiswa I-Cache Compression ne-Efficient Storage: I-Quantization ne-scratching zindlela ezimbili ezisetshenziselwa i-cache slendering. I-AQUA-KV ivimbele ama-tensors ze-KV kuya kuma-bits amancane ngokunciphisa okungenani (Shutova et al., 2025). I-Pruning, njenge-drop-stopword KVs, ivimbele ama-entries ezidingekayo. Kwiimeko amakhulu, i-Anchor Attention ivimbela kuphela ama-token "anchor" KVs futhi ivimbela okungenani (Sun et al., 2024). Ama-cache "umthwalo" ezijwayelekile zihlanganisa ama-vectors ezivamile futhi zihlanganisa ku-one. I-high-bandwidth memory I-Cross-Modal ne-Cross-Domain Caching: I-LLM ye-multimodal ne-video translation kufuneka i-cache eyodwa. Ngokwesibonelo, ukugcinwa kwe-image embeddings inokukwazi ukulayisha izithombe ezivame, futhi i-cache ye-audio snippets eyenza i-speech assistants ukusebenza ngokushesha. Ukuze zonke izinhlobo zebhizinisi, ufuna isinyathelo esahlukile. Ngokwesibonelo, i-hashes ye-perceptual for images ne-audio fingerprints for audio. I-cache ye-federated nge-edge devices ingathenga imiphumela embi, okuyinto ngokuvumelana nezimfuneko ze-federated learning. Njengoba i-cache ye-browser ye-global ye-AI assistants, ama I-Caching ku-Training ne-Fine-tuning: I-Caching ingasetshenziselwa i-Training ne-Fine-tuning LLMs, futhi akuyona kuphela ukulungiselela. Ukukhangisa i-Disk I/O, izivakashi ezingu-scale zihlanganisa idatha e-tokenized ku-memory. I-Training-time caches zihlanganisa izimo ezingu-intermediate ngezindlela ezintsha kakhulu. Ngokwesibonelo, amamodeli ze-multi-task angasebenzisa ukubonisa inqwaba phakathi kwezimfuneko ngaphandle kokuthumela. Ukukhangisa i-biases noma i-gradients ezisetshenziselwa ngokuvamile ku-cache futhi ukunciphisa ukucubungula okungagunyaziwe. It ikakhulukazi inikeza ukucubungula Ukulawulwa kwe-cache kanye nokulawula ikhwalithi: Umhlahlandlela Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo Olandelayo I-Standardization kanye ne-Cache API Layers: Akukwazi ukuthatha izigaba ezivamile ezivamile ze-machine learning stack ezivela ngokuvamile ku-cache. Kwangoku zonke i-framework noma i-application zihlanganisa ukukhiqiza i-cache ye-logic yayo. Mhlawumbi siphinde, kodwa ingathanda ezinye i-API ye-caching ezivela ezivela kumadivayisi ye-model noma i-serving frameworks futhi ivumela abathengisi ukuba zihlanganise i-logic yayo ye-custom (ngaphansi kwe-cache plugins nge-web servers). Ngokwesibonelo, i-OpenAI noma amanye ama-suppliers angakwazi ukunikeza ama-API yokusebenza kwe-cache yokusebenza, ngokuvamile zihlanganisa iz Ukubuyekezwa - Scratch Ngakho-ke, kubona elandelayo ye-LM caching kuyinto emangalisayo kakhulu futhi iqukethe izixazululo ezintsha. Ngokuvamile, umphumela iyatholakala ngokuvamile: Njengoba sikhula amamodeli amakhulu kubasebenzisi amakhulu, i-strategy caching kuyinto indawo efanelekayo yokufaka i-slack. I-iterations ezilandelayo izinzuzo zithuthukisa caching (okufundisa ukufundisa), ukunciphisa isisindo yayo (ukushintshwa kanye nokushintshwa okuhle ku-memory), futhi kwenza kakhulu ngokuvamile phakathi kwezimo ezihlukahlukene kanye nokufundisa / ukufundisa *. Ngakho-ke, kuyinto isitimela se-Arms, hhayi kuphela ukwakha amamodeli amakhulu kodwa ukunikezele kakhulu._* Ngokwenza lokhu Ukusuka lapha, mhlawumbe abo ne-strategy cache enhle kakhulu angakwazi ukufumana i-title yokunikeza izinsizakalo zokushesha futhi ekugcineni i-AI ezingenalutho. Ngoba, ngemuva kwalokho, ukuthi ungakwazi cache, ngoko ke i-computing? Ukuhlobisa Ukungena ngemvume Anthropic. (2024, EyeThupha 14). Kusukela ku-Anthropic News: Ukulungiselela ngokushesha nge-Claude. Ukuhlobisa Ukuhlobisa Hinton, G., Vinyals, O., & Dean, J. (2015). I-NIPS Deep Learning kanye ne-Representation Learning Workshop. Ukulungiselela ulwazi ku-Neural Network. U-Lewis, P., et al. (2020) Izinzuzo ze-Neural Information Processing Systems. I-Retrieval-Augmented Generation ye-knowledge-intensive NLP. Paul, R. (2025 ngenyanga 20). I-Rohan's Bytes Blog I-Caching Strategies e-LLM Services ye-Training ne-Inference. ikhaya ikhaya Pol, S. (2025, 6 April) Ikhaya Blog Ukushesha i-Transformer Inference nge-KV Caching. izikhangibeli522.medium.com izikhangibeli522.medium.com Ukulungiselela, N. (2019) ekhaya:1911.02150 . I-Fast Transformer I-Decoding: I-One Write-Head I-All You Need. ikhaya Shutova, A., Malinovskii, V., Egiazarian, V., et al. (2025). arXiv:2501.19392 . I-Cache Me If You Must: I-Adaptive Key-Value Quantization ye-Language Models. ikhaya Yazdani Aminabadi, R., Rajbhandari, S., Zhang, M., et al. (2022). isifinyezo:2207.00032 . I-DeepSpeed Inference: Ukuvumela Ukuvuthwa Okusheshayo Kwama-Transformer Amamodeli Ngama-Scale Amasipala. ikhaya Kwon, W., Li, Z., Zhuang, S., et al. (2023). Ukusebenza kwe-SOSP 2023 . Ukuphathwa kwamemori efanelekayo ye-Language Model Serving with PagedAttention (vLLM). blog.vllm.ai blog.vllm.ai Hugging Face Transformers Team. (2023). I-Hugging Face Blog Ukuphucula i-LLM yakho ekukhiqizeni. Ukuhlobisa Ukuhlobisa Lorello, S., McCormick, R., & Partee, S. (2023). Nvidia Technical Blog. Indlela yokwakha I-Distributed Inference Cache nge-NVIDIA Triton ne-Redis. (Additional references for academic rigor and completeness kungenziwa ifakwe ngokuhambisana zonke izihloko ezidlulileyo.)