GenAI demos are impressive but the production systems are expensive. What most teams do not realize until it is too late is that the cost is not coming from the model, rather it is coming from everything you send to it. We ran into this the hard way. The system was working well, users were happy, and nothing major had changed. Yet the costs kept climbing. At first, we assumed it was the model. We tried switching models, tweaking parameters, and reducing output tokens, but nothing made a meaningful difference. The real issue was something far less obvious. It was the context. There is a common assumption that more context leads to better results. So over time, systems start accumulating information. Full chat histories get passed along, multiple retrieved documents are included, logs and metadata sneak into prompts, and instructions grow longer with every iteration. None of these decisions seem wrong on their own. In fact, they often improve quality in the short term. But together, they create a quiet and persistent problem. quiet and persistent When we finally measured what was being sent to the model, the results were surprising. A large portion of the tokens had little to do with the actual user question. We were not just sending context. We were sending noise, and paying for it every time. Where Context Bloat Actually Comes From Where Context Bloat Actually Comes From If you zoom out, most GenAI systems follow a simple flow, but the problem is what gets added along the way. At first glance, each step makes sense. You want memory, relevant documents, system awareness, and structured prompts. The problem is that none of these layers are designed to remove information. They only add to it. Chat history grows with every interaction, even when older turns are no longer relevant. Retrieval systems return multiple chunks that often overlap semantically, repeating the same idea in different ways. Logs and metadata get included “just in case,” even when they are never used. Prompts evolve over time, accumulating instructions that were added for edge cases but never removed. By the time the request reaches the model, it looks less like a focused question and more like a collection of everything the system has ever seen. What makes this worse is that modern GenAI systems are not single-step. A typical request may trigger multiple model calls for routing, reasoning, and tool usage. That means this bloated context is not just sent once. It is sent repeatedly, multiplying the cost across the entire workflow. Fixing Context Bloat (What Actually Worked) Fixing Context Bloat (What Actually Worked) The fix was not about tuning the model, it was about changing the shape of the system. We redesigned the flow to treat context as something that must be earned, not assumed. The biggest shift was introducing a filtering mindset early in the pipeline. Instead of passing everything forward, we asked what was truly necessary for this specific request. Chat history was no longer treated as a full transcript but as a source of relevant signals. Only the most recent or meaningful interactions were included. Older context was either dropped or summarized into short, high-value representations. Retrieval became more deliberate. Instead of pulling in multiple large chunks, we focused on bringing in only what added new information. Redundant or overlapping content was eliminated. In many cases, we found that fewer, more precise inputs produced better results. We also replaced raw context with compressed meaning. Long passages were summarized, logs were converted into structured signals, and prompts were simplified to remove legacy instructions. The goal was not to reduce information, but to remove unnecessary form. Once we made these changes, the impact was immediate. Token usage dropped significantly, costs came down, and response times improved. More importantly, the model performed better because it was no longer distracted by irrelevant inputs. What Changed in How We Think What Changed in How We Think This experience forced us to rethink a fundamental assumption. GenAI systems are not just about intelligence, they are about control. Every token sent to a model is a decision and most systems are making too many of them without realizing it. The most important lesson we took away from this is simple, the most expensive token is the one that did not need to be sent. the most expensive token is the one that did not need to be sent

This story contains AI-generated text. The author has used AI either for research, to generate outlines, or write the text itself. 

The is an opinion piece based on the author’s POV and does not necessarily reflect the views of HackerNoon.

Context Bloat: The Silent Killer of GenAI Budgets

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

The Multi-Agent AI Revolution: Why Your Next Enterprise System Should Be Serverless

Context and Provider Pattern Using Vue 3 Composition API

Context Setting FTW

How To Build Great React Components Easily And Effortlessly

[React] Using Context is easy (30s read)

Understanding Session Management Using React Router V6

The Multi-Agent AI Revolution: Why Your Next Enterprise System Should Be Serverless

Context and Provider Pattern Using Vue 3 Composition API

Context Setting FTW

How To Build Great React Components Easily And Effortlessly

[React] Using Context is easy (30s read)

Understanding Session Management Using React Router V6

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps