The Problem We All Know Too Well The Problem We All Know Too Well The Problem We All Know Too Well Let me tell you about the moment I realized we were doing data all wrong. I was sitting in yet another conference room, watching a company spend their Monday morning meeting arguing about customer numbers instead of talking about actual customers. Operations said 10,000 active customers, source systems showed 8,500 and finance had completely different numbers. Meanwhile I am trying to pull compliance reports and can't figure out which number to trust for regulatory filings. Down the hall, Mike from IT was having his own crisis. He was supposed to integrate customer data across three systems for a project going live that week. But the customer data live in Salesforce with one ID structure, Zendesk with completely different IDs and their ERO system with yet another approach. Every time he asked someone how the systems connected, they told him to ask Janet. Janet had quit two years ago. I'd seen this scene play out at company. Smart people, good technology, lots of data but somehow it all fell apart when anyone tried to use it. That's when it hit me, we were treating data like a waste product instead of something valuable. We spent millions collecting it, storing it, and moving it around but we put zero effort into making it useful for the people who needed it. My Journey to Understanding My Journey to Understanding My Journey to Understanding Over the next few years, I worked with dozens of financial institutions facing the same problems. The details were different, but the story was always the same. Trading data existed in one system, customer positions in another, regulatory reporting pulled from a third and compliance data lived in spreadsheets maintained by people who'd moved on to other jobs. The breakthrough came when I started asking a simple question: "What if we treated data like we treat our actual products?" Think about it. When companies build products, they don't just build them and walk away. They research what customers need. They design something that solves real problems. They test it. They document it. They provide support. They keep improving it based on feedback. But with data? We were building the equivalent of cards with no steering wheels, no manuals and no customer service. Then we wondered why nobody wanted to drive them. Writing the Playbook Writing the Playbook Writing the Playbook These experience eventually led me to co-author "Advanced Data Engineering Architectures for Unified Intelligence". But this wasn't an academic exercise. Every framework in that book came from real projects, real failures and real successes. Advanced Data Engineering Architectures for Unified Intelligence When we wrote Chapter 9 on Data as a Product, we weren't theorizing. We were documenting what worked in practice. As we put it in the book, treating data as a product means "packaging data so that it can be reliably discovered, understood, accessed and used by defined customers with clear expectations for quality and support." Chapter 9 on Data as a Product, "packaging data so that it can be reliably discovered, understood, accessed and used by defined customers with clear expectations for quality and support." "packaging data so that it can be reliably discovered, understood, accessed and used by defined customers with clear expectations for quality and support." That definition took months to get right, because every word mattered. "Reliably discovered" not just dumped in a folder somewhere. "Clear expectations" not vague promises about quality. "Defined customers" real people with real jobs to do. Watching It Work in Practice Watching It Work in Practice Watching It Work in Practice Let me tell you about one of my favorite success stories. I was working with a manufacturing company that had all the classic problems. Business teams couldn't agree on basic metrics. IT was drowning in integration requests. Everyone was frustrated. We started by picking one domain, HR data. Instead of thinking about HR data as "employee records that HR maintains," we started thinking about it as a product called "Employee Master Data" that served two kinds of customers. The business customers needed things like org charts that updated quickly when people changed roles. The technical customers, other systems needed clean APIs with guaranteed uptime for payroll, badge access and performance management integrations. So the HR team created what we called a data contract. This wasn't just documentation it was a real commitment about what the data would look like, how fresh it would be, and what level of quality they'd guarantee. They set up automated tests to check their promises every day. The transformation was remarkable. Sarah from marketing could suddenly get customer data in minutes instead of filing tickets and waiting weeks. Mike from IT could integrate systems through clean APIs instead of writing customer ETL jobs that broke every time someone changed a field name. The Eight Things I Learned Matter The Eight Things I Learned Matter The Eight Things I Learned Matter Through all these projects, I discovered that every successful data product has eight characteristics. These became core to our framework: First, discoverability. People must be able to find your data easily. This means real catalogs with business context, not just technical metadata. First, discoverability. Second, trust. As we wrote in the book, trust is "reinforced by tests, profiling continuous validation presented alongside the product description." People need to see that someone's checking the quality. Second, trust. Third, stable contracts. Your data needs "a stable contract describing schemas, semantics and guarantees with a predictable evolution policy." Don't break people's reports by randomly changing field names. Third, stable contracts. Fourth, explicit service levels. Make promises you can measure. "99% of daily loads complete by 6AM local time.""No more than 0.1% of records fail referential integrity checks." Real commitments, not vague hopes. Fourth, explicit service levels. Fifth, documentation that helps. Not just technical specs, but examples, gotchas and answers to questions people really ask. Fifth, documentation that helps. Sixth, access that makes sense. Security that protects sensitive data without making it impossible to use anything. Sixth, access that makes sense. Seventh, real support. When things break, people need to know exactly who to call and how to get help. Seventh, real support. Eight, observable usage. Track how people use your data products so you can make them better and understand the impact of changes. Eight, observable usage. The Organizational Revolution The Organizational Revolution The Organizational Revolution The technical stuff was only half the battle. The bigger challenge was organizational. I learned that trying to centralize everything with one massive data doesn't work. The team becomes a bottleneck, and they never understand the business context well enough to make good decisions. Instead, I started organizing around business domains. HR owns employee data. Finance owns financial data. Customer service owns interaction data. Each team has a mix of skills: product thinking, technical implementation, analytics expertise and governance knowledge. This sounds chaotic, but it works because the people closet to the business make the decisions. The HR team understands both what managers need from org charts AND what payroll systems need from employee APIs. When Technology Actually Helps When Technology Actually Helps When Technology Actually Helps The technology foundation was crucial, but not in the way most people think. The breakthrough wasn't better databases or faster processing. It was platforms that made sharing easy. Modern cloud platforms like Snowflake, Databricks and BigQuery built sharing right into their systems. Instead of copying data everywhere (and dealing with the mess that creates), teams can share data directly. The HR team maintains employee data in one place, but payroll systems, badge systems and org chart applications can all access it through clean interfaces. This eliminates the integration nightmare that Mike from IT was dealing with. No more custom ETL jobs that break when source systems change. Learning from Failures Learning from Failures Learning from Failures Not every attempt at this succeeded, and the failures taught me as much as the successes. The most common mistake I saw was creating data catalogs without real ownership. Teams would dump table definitions into some tool, call it "self-service" and wonder why nobody used it. Another common failure was writing data contracts but not enforcing them. If you promise your data will be updated every morning by 8AM, you better have alerts that tell you when that doesn't happen. Otherwise, it's just empty promises. The worst failures happened when companies focused only on technology and ignored the people changes. You can't just install software and expect teams to magically start collaborating better. My Advice for Getting Started My Advice for Getting Started My Advice for Getting Started If you want to try this approach, here's what I learned works: Start small. Pick one domain where the pain is obvious and you have a team willing to try something different. Don't try to transform your entire company overnight. Build the basic infrastructure early. You need a proper catalog, quality monitoring and documentation tools. This doesn't have to be perfect from day one, but you need the foundation. Get serious about ownership. Someone needs to own each data product not just the technical maintenance but the whole user experience. This usually means creating new roles or changing existing ones significantly. Measure everything. Track how long it take people to find and use data. Monitor quality metrics. Survey users about their experience. If you're not making data more useful, you're just rearranging deck chairs. What Success Really Looks Like What Success Really Looks Like What Success Really Looks Like You'll know this is working when people stop complaining about data and start asking for more of it. When Sarah from marketing can launch campaigns faster because she trusts the customer data. When Mike from IT can build integrations ind ays instead of months because the APIs are reliable and well-documented. That's what we achieved at that manufacturing company. They went from taking weeks to generate cross functional reports to having real-time dashboards. More importantly, they stopped having those painful Monday morning meetings where everyone argued about whose numbers were right. Why This Matter Now Why This Matter Now Why This Matter Now The companies that figure this out will have a huge advantage. Data is becoming too important to manage the old way, where it's locked in technical systems and only specialists can use it. In our book, we called this "a measurable, repeatable operating model for producing, evolving and distributing trustworthy data at scale." That's not marketing speak it's what I've seen work in practice. book "a measurable, repeatable operating model for producing, evolving and distributing trustworthy data at scale." "a measurable, repeatable operating model for producing, evolving and distributing trustworthy data at scale." The key is starting with user needs and working backward. Not starting with whatever data, you happen to have and hoping someone finds it useful. My Personal Transformation My Personal Transformation My Personal Transformation Writing this book and implementing these ideas changed how I think about data work entirely. I used to focus on technical problems, faster queries, better storage, more sophisticated processing. Now I think about user problems first. Who needs this data? What are they trying to accomplish? How can I make their job easier? book Sarah and Mike from that first meeting? They're both advocates for this approach now. Sarah can get customer insights in minutes instead of weeks. Mike can build reliable integrations instead of maintaining a house of cards. They turned data from an obstacle into an enabler. That's the real measure of success. When people start seeing data as something that helps them do their jobs better instead of something that gets in their way. Once you see the transformation happen and I've seen it dozens of times now it's impossible to go back to the old way of doing things. The Data as Product approach isn't just a better way to manage data. It's better way to run a business.

Making Our Data Actually Work for Us

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Making AI Agents Actually Do Stuff: Prompt Engineering That Works

The Noonification: How Often Do NFTs Pass The Howey Test? (1/13/2023)

Darwin's Hybrid Intelligence to Align AI & Human Goals for Startups & VCs

The Noonification: White Man (11/26/2022)

The Noonification: The Metaverse is a Sh*tshow (11/2/2022)

100 Days of AI Day 1: From Newsletter to Podcast, Leveraging AI for Audio Transformation

Making AI Agents Actually Do Stuff: Prompt Engineering That Works

The Noonification: How Often Do NFTs Pass The Howey Test? (1/13/2023)

Darwin's Hybrid Intelligence to Align AI & Human Goals for Startups & VCs

The Noonification: White Man (11/26/2022)

The Noonification: The Metaverse is a Sh*tshow (11/2/2022)

100 Days of AI Day 1: From Newsletter to Podcast, Leveraging AI for Audio Transformation

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps