Prologue It’s 8:37 a.m. on a Thursday. I’m settling into my zone in my home office at Downtown Toronto — hot chocolate in hand, quiet workspace, phone about to go silent. Then, an unexpected phone call. It’s my internet provider — let’s call them “SomeRandomNetwork” — but one of the big 3 in Canada :) unexpected phone call SomeRandomNetwork” — big 3 Caller — “Hey, we’ve noticed your Wi-Fi tends to go off around 10 p.m. every night. Just checking — any issue on your end?” Caller — “Hey, we’ve noticed your Wi-Fi tends to go off around 10 p.m. every night. Just checking — any issue on your end?” Caller — “Hey, we’ve noticed your Wi-Fi tends to go off around 10 p.m. every night. Just checking — any issue on your end?” I paused, taken aback. Then I replied politely, “No, that’s intentional. I physically disconnect the router (due to 5G signal) before bed — for my sleep.” Then I replied politely, “No, that’s intentional. I physically disconnect the router (due to 5G signal) before bed — for my sleep.” Then I replied politely, “No, that’s intentional. I physically disconnect the router (due to 5G signal) before bed — for my sleep.” The rep laughed, and so did I. But as I hung up, something about that call lingered. They had been watching my Wi-Fi habits within the intimate walls of my home. Tracking me. Even in my little corner of the world, data flowed silently — collected, analyzed, questioned. within the intimate walls of my home I couldn’t shake the thought: My house, my devices… They’re all part of a massive data-generating system. I’d spent years designing analytics platforms — harvesting, modelling, transforming data for clients — and here I was, realizing that my home life mirrored those very systems. The irony wasn’t lost on me. My house, my devices… They’re all part of a massive data-generating system. The irony wasn’t lost on me. Even choices meant to disconnect, like unplugging my router, left traces — what I now understood as data exhaust. Every action, intentional or not, painted a digital portrait. My Wi-Fi habits were just one of many invisible footprints. My offline self might embrace silence, but my digital counterpart? It never slept. Even choices meant to disconnect, like unplugging my router, left traces — what I now understood as data exhaust. data exhaust That phone call marked a turning point. It made me think about the pervasive influence of Big Data — how it observes, shapes, and sometimes manipulates the world we live in. And I realized: understanding its biases and limitations isn’t just optional anymore; it’s essential. Introduction Introduction It’s noon, and I’m cruising down the 401 toward a client meeting in Ontario. At the back of my mind, I can’t help but think about how many apps have already harvested my data, and the day’s barely halfway through. Back home, my thermostat adjusts itself based on my usual wake-up time. My ride-share app knows my weekday routines better than I do. My smart assistant logs subtle patterns — volume changes, time triggers, even tone. 401 toward a client meeting in Ontario 401 401 how many apps have already harvested my data the day’s barely halfway through. “timestamping my route with precision” In the car, my GPS records every turn, precisely timestamping my route. Music App? It’s learning my tempo preference, mood shifts, and even time-of-day listening habits to serve a perfectly curated playlist. Even if I went completely off-grid, there would still be traces: GPS pings, medical records, banking activity, streaming history. Every micro-interaction is silently recorded, analyzed, and monetized. I went completely off-grid In today’s world, modern living is deeply entangled with data, from how we shop, commute, and scroll, to how we sleep. In today’s world, modern living is deeply entangled with data, from how we shop, commute, and scroll, to how we sleep. modern living is deeply entangled with data Chapter 1 — What Is Big Data? It’s easy to think of data as rows and columns in a spreadsheet. But in reality, data today is sprawling, chaotic, and persistent. Big Data isn’t just big — it’s beyond what your everyday laptop can comfortably handle. It’s too fast, too vast, and too varied for traditional tools. And what’s considered “big” is always shifting, depending on how powerful our machines become. rows and columns in a spreadsheet. beyond your everyday laptop can comfortably handle But that’s just the surface. Life now unfolds as an interconnected web of data points: structured like a warehouse, flexible like a lakehouse, but intimate like a journal. Every swipe, every search, every GPS ping… creates a digital exhaust. Some of it clean and structured. Some of it messy and unstructured. But all of it — valuable. Life now unfolds as an interconnected web of data points: structured like a warehouse, flexible like a lakehouse, but intimate like a journal. Every swipe, every search, every GPS ping… creates a digital exhaust. Some of it clean and structured. Some of it messy and unstructured. But all of it — valuable. valuable And this is where Big Data Analytics enters. And this is where Big Data Analytics enters. And this is where Big Data Analytics enters. The Why Behind Big Data Analytics We don’t collect data for the sake of collection. We analyze it to find meaning. To spot patterns. To anticipate behaviours. From credit fraud alerts to app recommendations, the engine behind the scenes is always the same: take massive data, process it fast, and make sense of the noise. find meaning app recommendations As a developer, I live in this pipeline — building systems that integrate raw input from everywhere: sensors, mobile apps, online transactions, ride-share pings, fridge alerts, insurance forms, you name it. I live in this pipeline — building systems that integrate raw input from everywhere: Sometimes I wait and batch the data — great for dashboards and weekly reporting. Other times, the business can’t wait. So Stream processing lets us analyze data seconds after it’s created — critical in finance, healthcare, ride-sharing, even customer service chatbots. But before any analysis can happen, the data needs to be cleaned. Trust me, dirty data lies. Like I said in my Data Quality article, garbage in, garbage out. Cleaning isn’t glamorous, but it’s where good insights are born. dirty data lies Data Quality article, garbage in, garbage out. Data Quality article, garbage in, garbage Data Quality article, garbage in, garbage From Big Data to Big Insights Once cleaned, that data becomes gold. You can mine it, model it, and train it. You can predict weather, optimize delivery routes, detect disease patterns, or detect churn before it happens. Tools like deep learning, predictive models, and classification algorithms allow us to surface insights we never saw coming. Big Data brings big value — and big risks. Big Data brings big value — and big risks Security gaps. Privacy breaches. Ethical gray zones. It’s one thing to analyze data. It’s another to be trusted with it. Because in the end, Big Data isn’t just about machines. It’s about people. And the digital lives we’ve chosen — or unknowingly created. It’s about people. And the digital lives we’ve chosen — or unknowingly created. Chapter 2 — The 4 Vs and the Digital Crumbs Finally, pull into the office parking lot for my client meeting. Swipe my card. Take a ticket. Another crumb. 12 pm (Noon)— From the moment I left home, data’s been following me like a shadow. 12 pm (Noon) 8:37 am — My smart thermostat logged the hour I got out of bed. 8:37 am — 12:19 pm — My GPS tracked my route down the 401. 12:19 pm — 12:31 pm — My Apple Watch kept an eye on my heart rate as I changed lanes. 12:31 pm — 12:44 pm — A highway cam likely caught my license plate, maybe even my speed. 12:44 pm — 1:03 pm — By the time I parked, my digital footprint was miles ahead of me. 1:03 pm — And it doesn’t stop there. And it doesn’t stop there. 1:17 pm — At my downtown office, I badge in. The system logs my check-in time. 1:17 pm — My productivity apps track every click, scroll, and idle second. 2 pm to 5 pm — My messages, meetings, lunch order, even the time I spend reviewing dashboards — it’s all recorded somewhere. 2 pm to 5 pm — Even my intentions are data. I mention I need a new couch, and suddenly, my phone, TV, and browser serve up matching furniture ads like mind readers. That’s not magic. That’s tracking. Even my intentions are data. I mention I need a new couch intentions I wont get too technical how all these work, but majority of apps / devices collect personal data in some way — there’s a good and bad element to it. I wont get too technical how all these work, but majority of apps / devices collect personal data in some way — there’s a good and bad element to it. I wont get too technical how all these work, but majority of apps / devices collect personal data in some way — there’s a good and bad element to it. The Everyday Sources of Data Smartphones. Browsers. Social media. Fitness trackers. Surveillance cameras. Iot sensors. Streaming services. Credit card swipes. Loyalty programs. Your entire routine is a goldmine for data collectors. entire routine And to make sense of this flood, we turn to a foundational framework in Big Data Analytics: The 4 Vs. The 4 Vs Volume Data is big. It’s not megabytes or even gigabytes anymore — we’re talking zettabytes of data created every single year. That’s 1 billion terabytes. From personal messages to surveillance footage, the sheer size of what’s stored is almost unfathomable. big zettabytes every single year Velocity Data doesn’t just exist. It moves fast. My location updates in real-time. Spotify adjusts my recommendations before the next song. Mobile Apps know what I’ll like next before I do. Data is created, collected, and processed instantly across millions of apps and sensors. exist moves instantly Variety Forget spreadsheets. Today’s data comes in every shape and form: Structured: Spreadsheets, database tables
Semi-structured: JSON files, XML from APIS
Unstructured: Voice notes, Instagram videos, screenshots, even GIFS of me reacting to those couch ads Structured: Spreadsheets, database tables Structured Semi-structured: JSON files, XML from APIS Semi-structured Unstructured: Voice notes, Instagram videos, screenshots, even GIFS of me reacting to those couch ads Unstructured But note that each format has its quirks, storage needs, and processing challenges. Veracity Not all data is clean or truthful. Noise, duplicates, and missing values are constant threats. If my Apple Watch thinks I’m jogging when I’m driving, bad data. If my GPS pin drops in the wrong plaza, bad data. The reliability of data — its veracity — is crucial, or we risk making bad decisions from bad inputs. veracity The bottom line? The bottom line? The bottom line? Whether I’m asleep in bed or stuck in traffic, my life has become a streaming service of data points — playing on loop, 24/7. And in the world of analytics, everything counts. And in the world of analytics, everything counts Press enter or click to view image in full size Chapter 3 — GPS Told Me to Turn Here: Bias in Big Data I once read online about a guy who drove straight into a lake because his GPS told him to. At first, I laughed. Then I thought about how often I trust my apps without question, no matter how off they feel. That’s not just convenience… that’s automation bias in action. a guy who drove straight into a lake Bias is everywhere. We talk about it in hiring, media, politics — but few stop to realize: bias lives in data too. Big data may sound objective, but it’s created, collected, cleaned, and coded by humans. And we? We are the most biased variables in the system. Bias is everywhere. Bias in Data Collection — The Convenience Trap Let’s say you want to analyze workout patterns in a city. You pull data from smartwatches and fitness apps. But guess what? You’re missing people who can’t afford wearables, or simply don’t use them. That’s selection bias. selection bias Worse, your data might reflect historical bias — inequities baked into past decisions. Policing, credit scores, and healthcare — all these domains carry the weight of societal bias. And when we collect that data, we carry that weight forward. historical bias Suggestion? Widen the net. Use diverse sources. Don’t chase only clean or easy datasets. And always ask: who’s missing from this picture? Suggestion? Widen the net. Use diverse sources. Don’t chase only clean or easy datasets. And always ask: who’s missing from this picture? Suggestion? Suggestion? Widen the net. Use diverse sources. Don’t chase only clean or easy datasets. And always ask: who’s missing from this picture? Bias in Algorithms — Code Doesn’t Equal Neutral Facial recognition once misidentified dark-skinned women at a much higher rate than white men. Not by accident — but by design. The algorithms were trained on biased data, built by biased teams. once misidentified dark-skinned women This is algorithmic bias — when machine logic learns human prejudice and scales it like wildfire. algorithmic bias Sugesstion? Transparency. Use representative training data like the PPB dataset. Audit models regularly. Be loud when things go wrong. Sugesstion? Transparency. Use representative training data like the PPB dataset. Audit models regularly. Be loud when things go wrong. Sugesstion? Sugesstion? Transparency. Use representative training data like the PPB dataset. Audit models regularly. Be loud when things go wrong. Bias in Interpreting Results — We See What We Want Even when the math is sound, our minds bend the results. Ever cherry-picked a stat to back up your hunch? That’s confirmation bias. Ever stretched insights from one group to everyone else? That’s overgeneralization. Ignored results that didn’t fit your story? Reporting bias. confirmation bias overgeneralization Reporting bias Sugesstion ? Define goals early. Document every assumption. Share failures, not just wins. Sugesstion ? Define goals early. Document every assumption. Share failures, not just wins. Sugesstion ? Sugesstion ? Define goals early. Document every assumption. Share failures, not just wins. Big Data, Big Responsibility The GPS is helpful, but it doesn’t replace your eyes. Data is powerful, but it doesn’t replace your judgment. Bias isn’t just a glitch in the matrix; it is the matrix. And if we want to build better models, we first have to debug ourselves. is Chapter 4— The Entire History of You Before I wrap up, let’s showcase what a “FICTIONAL” analytical model will look like. You don’t need a giant enterprise to run a data warehouse (This section was inspired by a Netflix TV Series). a giant enterprise You are the warehouse. In my head; “A married couple is just a FULL OUTER JOIN operation between two datasets (bodies).” You are the warehouse. In my head; “A married couple is just a FULL OUTER JOIN operation between two datasets (bodies).” You are the warehouse. In my head; “A married couple is just a FULL OUTER JOIN operation between two datasets (bodies).” Your life, whether complex or quiet, generates operational logs, which flow into fact tables, enriched by dimension tables, all eventually visualized as a personal KPI dashboard. operational logs fact tables dimension tables personal KPI dashboard Every moment is recorded. Every movement, a metric. Every decision, a datapoint. Every moment is recorded. Every movement, a metric. Every decision, a datapoint. Analytical Model: Life, Logged Your Operational Event Log (Streaming in Real Time) These are the raw signals your devices generate before they’re transformed into structured data. Think of it as a Kafka stream of you. you {
  "event_time": "2025-04-25T07:31:45Z",
  "event_type": "transit_card_tap",
  "location": "Union Station",
  "device": "PrestoReader_002",
  "user_id": "me_001",
  "lat": 43.6453,
  "lon": -79.3806
} {
  "event_time": "2025-04-25T07:31:45Z",
  "event_type": "transit_card_tap",
  "location": "Union Station",
  "device": "PrestoReader_002",
  "user_id": "me_001",
  "lat": 43.6453,
  "lon": -79.3806
} Above, this is your operational data store (ODS) — where raw, messy life gets staged before it’s cleaned and modelled. Before you know it, it’s part of a fact table. operational data store (ODS) fact table Fact Tables — The Highlights of Living These are time-based, measurable events in your life. Dimension Tables — Okay, let’s give our life a bit of context These give your fact tables meaning — the who, what, where, when, and why. Arranging Your Life In Data Marts below: Arranging Your Life In Data Marts below: Your KPIS — The Dashboard of You You don’t need Tableau or Power BI to know your story, but if you did…well but if you did…well Aggregated Data Life Modelled. Life Measured. Life Lived. Life Modelled. Life Measured. Life Lived. Life Modelled. Life Measured. Life Lived. You don’t need a smartwatch to know you’ve been moving. But when that motion becomes a metric, it gains memory. Your life leaves a data trail — one that’s human, habitual, and analytical all at once. And whether you analyze it or not, that trail speaks. This final chapter looks like something lifted from an Episode of Black Mirror. I know. human habitual analytical Episode of Black Mirror. Episode of Black Mirror. Conclusion In trying to unplug, I discovered I was still connected. Even silence had a signal. Big Data wasn’t just watching — it was learning, predicting, shaping. And in that realization came clarity: “To live freely in a digital world, we must first learn to read the footprints we leave behind.” “To live freely in a digital world, we must first learn to read the footprints we leave behind.” “To live freely in a digital world, we must first learn to read the footprints we leave behind.” “To live freely in a digital world, we must first learn to read the footprints we leave behind.” Thanks for reading!

This story contains new, firsthand information uncovered by the writer.

Dash

Netflix

What You Already Know About Big Data

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Lessons From The Night I Met Dbt on Databricks

Alessio Vinassa on Building Ethical Blockchain Businesses: Technology with a Conscience

The Most Disappointing Movie Sequels Of All Time

10 Best Datasets for Geospatial Analytics (Open and Public Access)

3 Industries Harnessing the Power of Big Data: Healthcare, Law, and Retail

4 Tips To Become A Successful Entry-Level Data Analyst

Lessons From The Night I Met Dbt on Databricks

Alessio Vinassa on Building Ethical Blockchain Businesses: Technology with a Conscience

The Most Disappointing Movie Sequels Of All Time

10 Best Datasets for Geospatial Analytics (Open and Public Access)

3 Industries Harnessing the Power of Big Data: Healthcare, Law, and Retail

4 Tips To Become A Successful Entry-Level Data Analyst

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps