The Data Architecture was working, until the GenAI arrived In today’s world, most of the enterprises are building LLM based GenAI solutions with document and database based knowledge and multi-dimensional vectors. I believe you have either explored or have already done something similar.  Then you must have watched LLM query dragging itself across multiple networks just to fetch an embedding stored in some remote vector database, or look up a database table or a document load from an object store, and then have felt the latency issues firsthand. The models waits
Our users waits
But neither our cloud cost waits nor the user’s frustration. The models waits Our users waits But neither our cloud cost waits nor the user’s frustration. This is the moment almost every enterprise reaches: the GenAI works, but the data does not. Don't forget the semantics, they are inconsistent across domains and data stores. RAG systems timeout because documents, tables, and embeddings live in separate universes. And the flashy dashboards built in BI tools have no idea how to answer semantics questions, for example “Which customers are showing signs of churn based on recent sentiment shifts?”.  Eventually, there is friction and frustration between customers and business which can't be ignored. This is not a tooling problem, it’s a data architecture problem, and in this article I present a story of how enterprises are being forced to rethink their data architecture strategy from the ground up. When Old Data Architecture met new GenAI When Old Data Architecture met new GenAI For nearly two decades, data platforms evolved linearly. This made sense when the world was about BI and traditional ML/AI workloads, but GenAI broke this linearity overnight. Now, enterprises need Relational facts and dimensions
Massive corpora of documents for grounding
Most importantly Vector and graph representations for semantic reasoning Relational facts and dimensions Massive corpora of documents for grounding Most importantly Vector and graph representations for semantic reasoning These needs are not occasional, but they are simultaneously needed and also needed at low-latency because natural language interfaces replaced traditional dashboards as the primary interface. Every natural language query became a mini-workload explosion: SQL + vector search + graph traversal + policy enforcement. Traditional architectures simply weren’t built for this world. The pain behind the pivot The pain behind the pivot Let me share a story which I’ve seen repeatedly across enterprises. Two teams ask a simple question to a natural-language LLM powered chat interface: “What is active revenue?” Both get 2 different answers. The LLM is trying its best to pick a definition based on whichever table it finds first. Nobody knows if the answer is 100% correct, and then we start blaming “GenAI hallucination”. But think about it, is it hallucination or is it simply the semantics which drifted years ago. GenAI only amplified the inconsistency at scale. Let’s take another example. A support document updates, but the embedding doesn’t. Now, your RAG pipeline uses the context which is stale for hours to days. So, you have a chatbot which is confidently answering questions with outdated advice. Is it the model’s fault or is the data architecture which was not built for freshness. But the pressure on enterprises keep increasing to provide the best in class natural-language LLM powered chat interface with freshest data and semantically correct answers. At this point, enterprises realise that their BI and traditional ML/AI optimized data architecture is cracking, and the more you scale the GenAI, the more cracks appear. Latency from pulling data across hops
Mounting cloud costs from duplicated data, documents, graph and vector stores.
Losing trust because data lineage is not tied to the interface.
Knowledge gaps because domain expertise lives in people’s heads and not in machine readable form. Latency from pulling data across hops Mounting cloud costs from duplicated data, documents, graph and vector stores. Losing trust because data lineage is not tied to the interface. Knowledge gaps because domain expertise lives in people’s heads and not in machine readable form. Enterprises don’t need new or better tools but they need a new and better architecture such that data and AI can coexist. The Realisation: GenAI Must move to where the data lives. The Realisation: GenAI Must move to where the data lives. We realise the breakthrough moment when we ask ourselves “why we are dragging the data across the network to feed GenAI, why not bring GenAI to the data instead?”. At this point, everything flips around your data architecture. We start generating embeddings inside the Lakehouse instead of externalising the embedding services
We start using in-platform vector indexes instead of remote vector databases.
Instead of building separate pipelines for semantics, metrics and entities, we start building a semantic layer with a knowledge graph as a shared meaning engine. We start generating embeddings inside the Lakehouse instead of externalising the embedding services We start using in-platform vector indexes instead of remote vector databases. Instead of building separate pipelines for semantics, metrics and entities, we start building a semantic layer with a knowledge graph as a shared meaning engine. At this point the data architecture stops fighting the GenAI and it becomes an enabler for GenAI. A semantic-First, GenAI-native Architecture This evolving data architecture is not a nice to have architecture, instead it’s the inevitable consequence of natural-language LLM powered chat interfaces becoming the dominant consumer of the enterprise data. Core philosophy of this architecture is following Bring GenAI compute to the data: All computation like embeddings, retrievals, inferences everything must run in your lakehouse where your source of truth is stored.
Your semantics should be made as the source of coherence: This philosophy is about unifying the semantic layer and knowledge graph in such a way that all systems, users and LLM interpret the data in the same way.
Hybrid retrieval must be treated as table stakes: Architecture must support SQL+ vector + graph queries as a single retrieval workflow not as a stitched pipeline.
Data architecture must have Trust embedded in itself: Data Lineage, provenance and access policies must enforce correctness at query time itself, it should not be after the fact.
There is no option but to optimise for natural-language scale: As all enterprises are realising that Natural Language Query interfaces increase the demand of data, knowledge, relationship, orchestration by 10x-100x, the architecture must adapt to cache, accelerate and cost-optimise automatically. This should not be about adding components, it’s about reshaping the enterprise data architecture foundation around GenAI as a first class workload. Bring GenAI compute to the data: All computation like embeddings, retrievals, inferences everything must run in your lakehouse where your source of truth is stored. Bring GenAI compute to the data Your semantics should be made as the source of coherence: This philosophy is about unifying the semantic layer and knowledge graph in such a way that all systems, users and LLM interpret the data in the same way. Your semantics should be made as the source of coherence Hybrid retrieval must be treated as table stakes: Architecture must support SQL+ vector + graph queries as a single retrieval workflow not as a stitched pipeline. Hybrid retrieval must be treated as table stakes Data architecture must have Trust embedded in itself: Data Lineage, provenance and access policies must enforce correctness at query time itself, it should not be after the fact. Data architecture must have Trust embedded in itself There is no option but to optimise for natural-language scale: As all enterprises are realising that Natural Language Query interfaces increase the demand of data, knowledge, relationship, orchestration by 10x-100x, the architecture must adapt to cache, accelerate and cost-optimise automatically. This should not be about adding components, it’s about reshaping the enterprise data architecture foundation around GenAI as a first class workload. There is no option but to optimise for natural-language scale How the Architecture Actually works( The Practical BluePrint) How the Architecture Actually works( The Practical BluePrint) Based on the above philosophy, the data architecture feels natural like following. This is like walking through a city where each district services a purpose. Control Plan: The City’s Rules, memory and meaning Control Plan: The City’s Rules, memory and meaning Control Plan: The City’s Rules, memory and meaning Control Plan: The City’s Rules, memory and meaning This is where trust is formed. Every dataset, feature, policy gets their identity, and  standardisation of meaning is done. This plane governs how everything is expected to behave as intended, continuously, seamlessly and in a predictable manner. Catalog & Lineage: This is like our city’s registry office Catalog & Lineage: This is like our city’s registry office Every datasets, tables, and feature is catalogued here with a unique identity and with full provenance. Anyone or everyone can ask “where did this number come from?”, the data lineage  explains the story instantly that it originates at these sources,  and transforms in this way and this is their current state.  The catalog also manages schema evolution through contracts and schema registry, such that your downstream consumers or you are not surprised when upstreams silently change their structure. If this does not exist, the GenAI systems may lose trust and can start accumulating semantic debt. Semantic Layer Semantic Layer Raw schemas don’t explain how the business thinks, and entities explain how business actually thinks. This is the layer that gives the right SQL or SPARQL or Cypher to the user’s natural language question. This is a meaning made for machines. Policy Manager Policy Manager Every query must be checked against policies before it’s executed, neither afterwards nor should it be part of an audit. It must be checked right at the point of use. RBAS, ABAC, regional residency, sensitivity rules, masking, row/column-level security, everything applies automatically. Thus purpose based access can be enforced, it should not be just aspirational. GenAI can’t be trusted unless the data architecture is trustworthy by design. Governance & Quality Governance & Quality PII/PCI/PHI detection, data quality monitoring, freshness checks, completeness checks, SLA monitoring, all must run continuously. Quality signals must flow into retrieval workflow, embossings and AI pipelines. If the data is not healthy, the system must know it and hence the GenAI also must know it. We can see that the control plane is the force which guarantees how the rest of the architecture works as expected and intended. We can see that the control plane is the force which guarantees how the rest of the architecture works as expected and intended. Data Plane: The City’s foundation and memory Data Plane: The City’s foundation and memory Data Plane: The City’s foundation and memory Data Plane: The City’s foundation and memory If the control plane governs the city, the data plane is the land where everything is built on. This is the place where enterprise’s data lives- whether it’s structured, semi-structured, or unstructured. Data is stored in open formats and interoperable formats such that any compute engine can operate without copying or duplicating the data. Object Storage(OneLake,S3, ADLS) Object Storage(OneLake,S3, ADLS) The Lakehouse sits on top of the object storage using open formats like Apache Iceberg, Delta or Parquet. This provides enterprises ACID reliability and the flexibility for any compute engine to operate on the same data without making copies. Warehouse Tables Warehouse Tables This is optimised for BI and metric-driven workloads, and these tables get materialised on top of the Lakehouse data which enables BI dashboard with governance and consistency. Document & Media Stores Document & Media Stores PDFs, HTML pages, design docs, images and audio all data live here. These are the sources GenAI with the RAG pipeline depend on to ground their answers to reflect what’s actually true inside the enterprise. When our data architecture treats text, metrics, and media as first class citizens, then only GenAI has the full context. Index Plane: The intelligence Layer Index Plane: The intelligence Layer Index Plane: The intelligence Layer Index Plane: The intelligence Layer If the data plane holds memories, the Index plane is what makes those memories semantically connected and searchable. We can say that this is the part of the city where information becomes intelligent. Vector Indexes: The City’s semantic map Vector Indexes: The City’s semantic map Every document, row and even graph node is embedded and stored in the approximate nearest neighbor. This enables semantic similarity search for RAG and Natural language driven retrieval. Architecturally, when a support document changes, the updated embedding appears here within seconds. Thus fresh context becomes the default nature of the architecture rather than a luxury. Knowledge Graph: The City’s relationship network Knowledge Graph: The City’s relationship network This is the most interesting part of the city, where entities, relationships, lineage, policies and domain rules come to life. Vector search provides “things that look similar”, however the Knowledge graph explains “how things are connected” like Which customer owns which asset
Which policies apply to which regions
Which transaction feeds which metric
Which dataset produced which report Which customer owns which asset Which policies apply to which regions Which transaction feeds which metric Which dataset produced which report It supports provenance, entity resolution and symbolic reasoning. The index plane transforms the Lakehouse from storage into cognition. Compute Plane: The City’s workforce Compute Plane: The City’s workforce Compute Plane: The City’s workforce Compute Plane: The City’s workforce The index plane enables thinking, but the compute plan is the one which acts. This is the plane where all workloads-analytics, streaming, AI run seamlessly against shared data. SQL Warehouse / Lakehouse Engine: The Analysts SQL Warehouse / Lakehouse Engine: The Analysts They are the ones which power the classic workloads, whether it’s structured queries, metrics, dashboards, or any other operational analytics running the ACID tables with highest level of reliability. Spark, Trino, Presto: The Builders Spark, Trino, Presto: The Builders These are our distributed engines that share the data whether you do ETL or ELT, batch jobs, or any kinds of transformation or ad-hoc analytics. They are the ones who convert the raw material to curated forms. Flink / Kafka Streams: The Traffic Controllers Flink / Kafka Streams: The Traffic Controllers They are the ones who handle real-time data ingestion, CDC feeds, and all types of low-latency stream processing. They ensure that fresh data flows freely across our city. AI Services: The Specialists AI Services: The Specialists We reach the busy part of the city where GenAI fits directly into the architecture. Embedding services generate embeddings in-platform, reducing latency.
RAG orchestrators combine vector + graph + lexical retrieval into unified context.
LLM inference runtimes perform prompting, fine-tuning, or adapter training.
Guardrails enforce policy, safety, and factual consistency. Embedding services generate embeddings in-platform, reducing latency. RAG orchestrators combine vector + graph + lexical retrieval into unified context. LLM inference runtimes perform prompting, fine-tuning, or adapter training. Guardrails enforce policy, safety, and factual consistency. It’s the compute plane which keeps the entire AI-native environment alive and evolving. The Experience Plane The Experience Plane The Experience Plane The Experience Plane Finally, the experience plane is where people meet intelligence, where everything this architecture does becomes visible, usable and valuable. Natural Language UX and Co-pilots Natural Language UX and Co-pilots When Users ask questions: “What was the reason churn increased last quarter?” The system routes it through the semantic layer, runs SQL, SPARQL and Vector/Graph retrieval, enforces policies and returns a sourced and explainable answer. This is the interface which makes enterprise GenAI feel like a conversation instead of just a query. BI Tools and Dashboards BI Tools and Dashboards All kinds of BI tools like PowerBI, Tableau, Fabric, Direct Lake and their visual analytics live here, which is powered by the shared semantic and Lakehouse layers beneath them. Applications and APIs Applications and APIs Finally the traditional experience plane like application and api also live in this plane. Decision intelligence, recommendations, RPA, analytics apps and everything else consumes data and AI via consistent APIs and governed access. What this unlocks What this unlocks Once enterprises adopt this AI-native, semantics-first data architecture, several things become possible. Latency drops dramatically
Costs stabilises
Answers become trustworthy
Domain expertise becomes digital
AI becomes model agnostics infrastructure Latency drops dramatically Costs stabilises Answers become trustworthy Domain expertise becomes digital AI becomes model agnostics infrastructure This is what the future of enterprise GenAI looks like: meaning driven, trust-embedded, and built for hybrid retrieval at scale. Conclusion Conclusion As we have seen data architectures were built to serve BI dashboards and traditional AI workloads for decades, now they must serve natural languages as the interface and GenAI as the primary consumer. With this evolution, the center of gravity has moved. Enterprises need to Bring AI to the data
Elevate semantics to the level of infrastructure
Unify structured, unstructured and vector/graph data
Treat trust as design principle than a compliance checkbox
Build architecture where meaning, not formats becomes the connective tissue Bring AI to the data Elevate semantics to the level of infrastructure Unify structured, unstructured and vector/graph data Treat trust as design principle than a compliance checkbox Build architecture where meaning, not formats becomes the connective tissue This should not be treated as an incremental shift, instead this should be the way foundational reshaping of data systems should be done to support intelligence, and the enterprise that will embrace it early will define the next decade of enterprise GenAI.

How GenAI is Reshaping the Modern Data Architecture

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Optimise LLM usage costs with Semantic Cache

Optimise LLM usage costs with Semantic Cache

Codex vs Factory vs Cursor: Does the Harness Really Matter for MVP Prototyping?

An Introduction to Backend Architecture

Data Product Managers and the Data Mesh

Change Data Capture to Accelerate Real-time Analytics

Optimise LLM usage costs with Semantic Cache

Optimise LLM usage costs with Semantic Cache

Codex vs Factory vs Cursor: Does the Harness Really Matter for MVP Prototyping?

An Introduction to Backend Architecture

Data Product Managers and the Data Mesh

Change Data Capture to Accelerate Real-time Analytics

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps