The Late-Night Pager Alert and The Reality of Context Rot Last month I woke up to a pager alert and a heated Slack thread regarding a critical production failure. The polished writing and perfect syntax of our AI agent hid a massive underlying flaw. The agent was hallucinating and making incorrect operational decisions. The large language model was not the problem. The issue was that the underlying knowledge graph the agent relied upon was completely out of sync with our live transactional databases. What I experienced is known in the industry as "Context Rot," a severe architectural failure that is currently breaking long AI sessions. Independent studies show that models exhaust their attention budget, and performance degrades significantly as context length increases. A July 2025 study by Chroma evaluating 18 advanced language models found that at the 32,000 token threshold, 11 out of 12 primary models dropped below 50% of their baseline accuracy. study by Chroma A 2023 Stanford University study, Lost in the Middle, proved that LLMs exhibit a U-shaped attention distribution; accuracy falls from 75% to 55% when facts are pushed to the middle of a prompt. Furthermore, an October 2025 arXiv study confirmed that context length alone imposes a severe "cognitive tax," dropping accuracy by 13.9% to 85% even with 100% perfect semantic retrieval. University study Lost in the Middle arXiv study Retrieval Augmented Generation (RAG) is an architectural pattern designed to enhance the accuracy of models by retrieving external data at the exact time a prompt is issued to prevent these hallucinations. However, when building autonomous agentic systems that run continuous execution loops, pulling from a stale vector database is catastrophic. I quickly realized that scaling enterprise AI is a deeply complex data engineering problem. RAG does not typically fail because LLMs hallucinate out of nowhere; it fails because data systems drift. Vector embeddings are effectively a materialized view over your raw transactional data. If your embedding store does not reflect real-time policy updates, inventory changes, or record deletions, the retrieval quality degrades silently. The Architectural Antagonist: Write Amplification I initially tried to solve this synchronization issue using legacy data pipelines. In older versions of my architecture, handling high-frequency updates and deletes to keep the knowledge base synced created massive friction. Apache Iceberg v2 used positional delete files tracking deleted rows by explicit file paths and row positions, encoding this into verbose Parquet files. This caused severe write amplification and read performance degradation because the engine had to merge too many disjointed delete files during query execution. Nights passed troubleshooting fragile workflows, custom retry systems, and overflowing memory in the JVM. Overhauling the Compute Layer: Apache Spark 4.1 The breakthrough for my team occurred when I migrated the core architecture to the newly released Apache Spark 4.1 and Apache Iceberg v3. First, I overhauled the compute layer. Spark 4.1 introduced Spark Declarative Pipelines (SDP) under SPIP SPARK-51727. This moves away from manual pipeline construction to intent-driven design. I defined the exact dataset outcomes I wanted using Python, and Spark autonomously handled the execution graph, dependency ordering, and checkpoints. Iceberg v3. SPARK-51727 Python from pyspark import pipelines as dp
import pyspark.sql.functions as F

@dp.table(name="raw_orders")
def raw_orders():
    # Establishes a continuous connection to the Kafka event stream
    return spark.readStream.format("kafka").option("subscribe", "orders").load()

@dp.materialized_view(name="transformed_context")
def transformed_context():
    # Declaratively defines the target state for our vector synchronization
    return spark.table("raw_orders").filter(F.col("status") == "COMPLETED") from pyspark import pipelines as dp
import pyspark.sql.functions as F

@dp.table(name="raw_orders")
def raw_orders():
    # Establishes a continuous connection to the Kafka event stream
    return spark.readStream.format("kafka").option("subscribe", "orders").load()

@dp.materialized_view(name="transformed_context")
def transformed_context():
    # Declaratively defines the target state for our vector synchronization
    return spark.table("raw_orders").filter(F.col("status") == "COMPLETED") More importantly for the AI use case, I implemented the Structured Streaming Real-Time Mode introduced in SPARK-53736. This bypasses traditional micro-batching mechanics to achieve sub-second latency for continuous processing and single-digit millisecond latency for stateless workloads. By configuring spark.databricks.streaming.realTimeMode.enabled to true, the engine launched long-lived streaming jobs that scheduled stages concurrently. Data is passed directly between active tasks in memory using a streaming shuffle, entirely avoiding the latency bottlenecks of traditional disk-based shuffles. spark.databricks.streaming.realTimeMode.enabled true Here is how the modernized execution flow prevents temporal lag from rotting the context: Modernizing the Storage Layer: Apache Iceberg v3 Second, I modernized the storage layer to eliminate metadata bloat. Iceberg v3 replaces positional delete files with a highly efficient binary format stored as Puffin sidecar files, known as deletion vectors. The format uses memory-optimized Roaring Bitmaps. Each deletion vector represents row positions as bits; if a bit is set, that row is considered logically deleted. The following architectural comparison illustrates exactly why legacy workflows choked on metadata, and why Iceberg v3's approach solves the merge-on-read bottleneck: The engine now maintains only a single deletion vector per data file per snapshot at write time. When a continuous Spark stream executes a CDC MERGE operation, the engine logically merges incoming deletes with the existing Deletion Vector in memory, producing a single updated Puffin file. This completely avoids the translation overhead between verbose Parquet files and in-memory representations. I ran empirical benchmarks comparing format v2 and v3 for Change Data Capture MERGE operations in my pipeline: Delete Operation Runtime: Decreased from 3.126 seconds in v2 to 1.407 seconds in v3 (a 55.0% performance improvement).
Delete File Size: Reduced from 1,801 bytes (Parquet) to 475 bytes (Puffin), a 73.6% reduction in metadata bloat.
Read Acceleration: Full table reads were 28.5% faster, and filtered reads were 23.0% faster in v3. Delete Operation Runtime: Decreased from 3.126 seconds in v2 to 1.407 seconds in v3 (a 55.0% performance improvement). Delete Operation Runtime: Delete File Size: Reduced from 1,801 bytes (Parquet) to 475 bytes (Puffin), a 73.6% reduction in metadata bloat. Delete File Size: Read Acceleration: Full table reads were 28.5% faster, and filtered reads were 23.0% faster in v3. Read Acceleration: Storage costs and S3 access costs plummeted because new deletion vectors seamlessly replace old ones without accumulating bloat. Native Row Lineage for Immutable AI Auditability To further manage AI state and ensure strict auditability, I leveraged Iceberg v3's native row lineage capabilities. The format tracks incremental changes at the row level using mandatory metadata fields like _row_id (a stable identifier) and _last_updated_sequence_number (the explicit commit snapshot). _row_id _last_updated_sequence_number This built-in capability eliminated the need for fragile custom tracking implementations in my RAG architecture. A vector database synchronization pipeline can now execute a highly efficient metadata scan to capture exact Change Data Capture deltas for incremental processing: SQL SELECT id, document_chunk, _row_id, _last_updated_sequence_number
FROM myns.transformed_context
WHERE _last_updated_sequence_number > :last_processed_sequence SELECT id, document_chunk, _row_id, _last_updated_sequence_number
FROM myns.transformed_context
WHERE _last_updated_sequence_number > :last_processed_sequence By filtering strictly by _last_updated_sequence_number, compute costs are radically reduced, and the agent's knowledge base perfectly mirrors reality. _last_updated_sequence_number Conclusion By feeding live event streams through Spark 4.1 real-time mode directly into Iceberg v3 tables utilizing deletion vectors, I engineered a pipeline that updates the vector database in milliseconds. Context rot is effectively eradicated. Building enterprise-grade generative AI requires more than simple API calls. It requires building a capability graph, connecting modules with clean interfaces, and shipping outcomes with tight feedback loops. In the AI era, the winners aren't just prompting specialists; they are the architects who build robust foundational data systems. By combining these advanced Apache frameworks, I built an infrastructure capable of supporting true autonomous intelligence. About the Author
Viquar Khan is a Senior Architect at AWS Professional Services with over 20 years of expertise in finance and data analytics. A recognized expert in large-scale distributed systems, he empowers global financial institutions to leverage AWS technologies through cutting-edge, customized data solutions. A polyglot developer and active open source contributor to Apache Spark, Kafka, and Terraform, Viquar has shaped industry standards as a member of the JSR 368 (JMS 2.1) expert group and author of KIP-1267. His technical insights have benefited over 6.7 million users on Stack Overflow. About the Author About the Author Viquar Khan is a Senior Architect at AWS Professional Services with over 20 years of expertise in finance and data analytics. A recognized expert in large-scale distributed systems, he empowers global financial institutions to leverage AWS technologies through cutting-edge, customized data solutions. A polyglot developer and active open source contributor to Apache Spark, Kafka, and Terraform, Viquar has shaped industry standards as a member of the JSR 368 (JMS 2.1) expert group and author of KIP-1267. His technical insights have benefited over 6.7 million users on Stack Overflow.

This story contains AI-generated text. The author has used AI either for research, to generate outlines, or write the text itself. 

This story contains new, firsthand information uncovered by the writer.

Immutable

Real-Time Agentic RAG: Eradicating Context Rot With Spark & Iceberg

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Dash to the New Writer Dashboard

Branches & Tags: Comparing Iceberg, Hudi, and Delta Lake Tables

Creating a Data Lakehouse using Apache Iceberg and MinIO

Data Lakehouse Table Formats: Considerations, Reflection, and Perspective

What the Interoperability Trend in Open Table Formats Signify for Enterprise Data Architectures

Navigating Apache Iceberg: A Deep Dive into Catalogs & Their Role in Data Lakehouse Architectures

Dash to the New Writer Dashboard

Branches & Tags: Comparing Iceberg, Hudi, and Delta Lake Tables

Creating a Data Lakehouse using Apache Iceberg and MinIO

Data Lakehouse Table Formats: Considerations, Reflection, and Perspective

What the Interoperability Trend in Open Table Formats Signify for Enterprise Data Architectures

Navigating Apache Iceberg: A Deep Dive into Catalogs & Their Role in Data Lakehouse Architectures

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps