I've been a data engineer for years, and if there's one thing I've learned, it's this: pipelines don't explode overnight. They rot. Slowly. One shortcut at a time, one "we'll fix it later" at a time, until you're staring at a 3 AM PagerDuty alert, wondering how everything got this bad. This article is the field guide I wish I'd had when I started. These are the five anti-patterns I've seen destroy pipeline reliability across startups and enterprises alike—and the concrete fixes that brought them back from the brink. Anti-Pattern #1: The Mega-Pipeline (a.k.a. "The Monolith") What It Looks Like One giant DAG. Fifty tasks. Extract from six sources, transform everything in sequence, and load into a data warehouse—all in a single pipeline. If step 3 fails, steps 4 through 50 sit and wait. Retrying means re-running the whole thing. I inherited a pipeline like this at a previous company. It was a single Airflow DAG with 70+ tasks, and a failure anywhere meant a full retry that took four hours. The team had just accepted that "the morning pipeline" was unreliable. Why It Happens It starts innocently. You build a pipeline for one data source. Then someone asks you to "just add" another source. Then another. Before you know it, you've got a tightly coupled monster where unrelated data flows share failure domains. The Fix: Decompose by Domain Break it apart. Each data source gets its own pipeline. Each pipeline is independently retriable, independently monitorable, and independently deployable. Here's my rule of thumb: if two parts of a pipeline can fail for unrelated reasons, they should be separate pipelines. if two parts of a pipeline can fail for unrelated reasons, they should be separate pipelines. After decomposition, the same workload ran as 8 independent DAGs. Average recovery time dropped from 4 hours to 15 minutes because we could retry just the part that broke. Practical steps: Practical steps: Identify natural domain boundaries (one per source system, or one per business domain)
Use an orchestrator that supports cross-DAG dependencies (Airflow's ExternalTaskSensor, Dagster's asset dependencies, Prefect's flow-of-flows)
Introduce a shared metadata layer so downstream consumers know when upstream data is fresh Identify natural domain boundaries (one per source system, or one per business domain) Use an orchestrator that supports cross-DAG dependencies (Airflow's ExternalTaskSensor, Dagster's asset dependencies, Prefect's flow-of-flows) ExternalTaskSensor Introduce a shared metadata layer so downstream consumers know when upstream data is fresh Anti-Pattern #2: Schema-on-Pray (No Schema Contracts) What It Looks Like Your pipeline ingests data from an API or upstream service. One day, a field gets renamed. Or a column that was always an integer suddenly contains strings. Your pipeline breaks, your dashboards go blank, and nobody knows why until someone digs through logs for an hour. I once spent an entire weekend debugging a broken pipeline because an upstream team silently changed a date field from YYYY-MM-DD epoch milliseconds. No notification. No versioning. Nothing. YYYY-MM-DD Why It Happens Teams treat the boundary between systems as "someone else's problem." There's no explicit contract about what the data looks like, so any change upstream is a surprise downstream. The Fix: Schema Contracts and Validation at the Boundary Never trust upstream data. Validate it the moment it enters your domain. What this looks like in practice: What this looks like in practice: Define explicit schemas using tools like Great Expectations, Pydantic, JSON Schema, or dbt contracts. Specify column names, types, nullability, and acceptable value ranges.
Validate on ingestion. Before your pipeline does any transformation, run schema checks. If validation fails, quarantine the data and alert—don't silently propagate garbage downstream.
Version your schemas. When a breaking change is needed, version it explicitly (e.g., v1/events, v2/events). This gives downstream consumers time to adapt. Define explicit schemas using tools like Great Expectations, Pydantic, JSON Schema, or dbt contracts. Specify column names, types, nullability, and acceptable value ranges. Define explicit schemas Validate on ingestion. Before your pipeline does any transformation, run schema checks. If validation fails, quarantine the data and alert—don't silently propagate garbage downstream. Validate on ingestion. Version your schemas. When a breaking change is needed, version it explicitly (e.g., v1/events, v2/events). This gives downstream consumers time to adapt. Version your schemas. v1/events v2/events # Example: Simple schema validation with Pydantic
from pydantic import BaseModel, validator
from datetime import date

class EventRecord(BaseModel):
    event_id: str
    event_date: date
    user_id: int
    amount: float

    @validator('amount')
    def amount_must_be_positive(cls, v):
        if v < 0:
            raise ValueError('amount must be non-negative')
        return v # Example: Simple schema validation with Pydantic
from pydantic import BaseModel, validator
from datetime import date

class EventRecord(BaseModel):
    event_id: str
    event_date: date
    user_id: int
    amount: float

    @validator('amount')
    def amount_must_be_positive(cls, v):
        if v < 0:
            raise ValueError('amount must be non-negative')
        return v After implementing schema validation at our ingestion layer, silent data corruption incidents dropped to near zero. When upstream schemas changed, we caught it immediately instead of finding out from a confused analyst two weeks later. Anti-Pattern #3: The "Just Retry" Strategy (No Idempotency) What It Looks Like A pipeline fails halfway through a write operation. You retry it. Now you have duplicate records. Or worse—partial writes that leave your data in an inconsistent state. The "fix" is usually someone running a manual deduplication query, and everyone pretends it's fine. Why It Happens Writing idempotent pipelines takes extra thought. It's much easier to write INSERT INTO than to think about what happens when that insert runs twice. Under deadline pressure, idempotency is the first thing that gets punted. INSERT INTO The Fix: Design Every Write to Be Safely Repeatable Idempotency means running a pipeline twice produces the same result as running it once. This is non-negotiable for reliable data systems. Three patterns that work: Three patterns that work: Upsert/MERGE instead of INSERT. If a record already exists, update it instead of creating a duplicate. Most modern data warehouses support MERGE or INSERT ... ON CONFLICT.
Partition-based overwrites. Instead of appending, write to a date-partitioned table and overwrite the entire partition on each run. If the pipeline reruns, it replaces the partition cleanly. Upsert/MERGE instead of INSERT. If a record already exists, update it instead of creating a duplicate. Most modern data warehouses support MERGE or INSERT ... ON CONFLICT. Upsert/MERGE instead of INSERT. MERGE INSERT ... ON CONFLICT Partition-based overwrites. Instead of appending, write to a date-partitioned table and overwrite the entire partition on each run. If the pipeline reruns, it replaces the partition cleanly. Partition-based overwrites. -- Partition overwrite: safe to re-run
INSERT OVERWRITE TABLE events
PARTITION (event_date = '2025-02-06')
SELECT * FROM staging_events
WHERE event_date = '2025-02-06'; -- Partition overwrite: safe to re-run
INSERT OVERWRITE TABLE events
PARTITION (event_date = '2025-02-06')
SELECT * FROM staging_events
WHERE event_date = '2025-02-06'; Write-audit-publish pattern. Write to a staging area first. Validate the data. Then atomically swap it into the production table. If anything fails, the staging area is discarded, and production is untouched. Write-audit-publish pattern. Write to a staging area first. Validate the data. Then atomically swap it into the production table. If anything fails, the staging area is discarded, and production is untouched. Write-audit-publish pattern. I moved our team to partition-based overwrites for all batch pipelines, and the "duplicate records" Slack channel (yes, it existed) went silent within a month. Anti-Pattern #4: Logging by Vibes (No Observability) What It Looks Like The pipeline ran. Did it succeed? Well, there's no error in the logs. But also, no one checked if it actually produced the right number of rows. Or if the data arrived on time. Or if the values make sense. The pipeline is "green" in the orchestrator, but the data is quietly wrong. I call this "green but broken"—the most dangerous state a pipeline can be in, because no one is even looking for the problem. Why It Happens Engineers focus on making the pipeline run. Observability—making the pipeline observable—feels like extra work that doesn't ship features. run observable The Fix: Instrument Like You'd Instrument a Production API Treat your data pipeline like a production service. That means: Row count assertions. After every major step, assert that the output has a reasonable number of rows. Zero rows is almost always wrong. A sudden 10x spike is almost always wrong. Row count assertions. Freshness checks. Set up alerts for when data hasn't arrived by its expected time. A pipeline that "succeeds" but runs 6 hours late is still a failure from the business perspective. Freshness checks. Data quality metrics. Track null rates, value distributions, and schema drift over time. Tools like Great Expectations, dbt tests, Monte Carlo, or Elementary can automate this. Data quality metrics. Lineage tracking. Know which downstream dashboards and models depend on which upstream sources. When something breaks, you should know the blast radius in seconds, not hours. Lineage tracking. # Example: dbt test for freshness and row count
models:
  - name: orders
    tests:
      - not_null:
          column_name: order_id
      - accepted_values:
          column_name: status
          values: ['pending', 'completed', 'cancelled']
    freshness:
      warn_after: {count: 12, period: hour}
      error_after: {count: 24, period: hour} # Example: dbt test for freshness and row count
models:
  - name: orders
    tests:
      - not_null:
          column_name: order_id
      - accepted_values:
          column_name: status
          values: ['pending', 'completed', 'cancelled']
    freshness:
      warn_after: {count: 12, period: hour}
      error_after: {count: 24, period: hour} After building out a proper observability layer, our mean time to detection (MTTD) for data issues dropped from days to minutes. That alone justified the investment. Anti-Pattern #5: Hardcoded Everything (No Configuration Layer) What It Looks Like Database connection strings in the code. Table names in the SQL. Environment-specific logic is scattered across files with if env == 'prod' branches. Deploying to a new environment means a search-and-replace marathon, and one missed replacement means the staging pipeline accidentally writes to production tables. if env == 'prod' Yes, that happened. Yes, it was painful. Why It Happens Hardcoding is the fastest way to get something working right now. Configuration management feels like overengineering when you only have one environment. But you never have just one environment for long. right now The Fix: Externalize Configuration from Day One Separate what the pipeline does from where it runs. Separate what the pipeline does from where it runs. does runs Use environment variables or a config file for anything environment-specific: connection strings, bucket paths, table names, and API endpoints.
Template your SQL. Use Jinja (dbt does this natively) or your orchestrator's templating to parameterize table references and environment names.
Use a secrets manager (AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager) for credentials. Never commit secrets to version control. Not even "temporarily." Use environment variables or a config file for anything environment-specific: connection strings, bucket paths, table names, and API endpoints. Use environment variables or a config file Template your SQL. Use Jinja (dbt does this natively) or your orchestrator's templating to parameterize table references and environment names. Template your SQL. Use a secrets manager (AWS Secrets Manager, HashiCorp Vault, GCP Secret Manager) for credentials. Never commit secrets to version control. Not even "temporarily." Use a secrets manager # Bad: hardcoded everything
conn = psycopg2.connect(host="prod-db.company.com", password="hunter2")
cursor.execute("INSERT INTO prod_schema.events ...")

# Good: externalized config
import os

conn = psycopg2.connect(
    host=os.environ["DB_HOST"],
    password=os.environ["DB_PASSWORD"]
)
schema = os.environ.get("SCHEMA", "public")
cursor.execute(f"INSERT INTO {schema}.events ...") # Bad: hardcoded everything
conn = psycopg2.connect(host="prod-db.company.com", password="hunter2")
cursor.execute("INSERT INTO prod_schema.events ...")

# Good: externalized config
import os

conn = psycopg2.connect(
    host=os.environ["DB_HOST"],
    password=os.environ["DB_PASSWORD"]
)
schema = os.environ.get("SCHEMA", "public")
cursor.execute(f"INSERT INTO {schema}.events ...") Once we externalized configuration, spinning up a new environment went from a two-day effort to a 30-minute Terraform run. Conclusion These five anti-patterns share a root cause: optimizing for time-to-first-success instead of time-to-recovery. It's faster to build a monolithic, unvalidated, non-idempotent pipeline with hardcoded configs and no observability. It works on the first run. It even works on the tenth run. But when it breaks—and it will—you pay back all that time debt with interest. optimizing for time-to-first-success instead of time-to-recovery. The best data engineers I've worked with think about failure from the start. They ask, "What happens when this breaks?" before they ask, "Does this work?" That mindset shift is worth more than any tool or framework. If you're inheriting a pipeline that has some of these anti-patterns, don't try to fix everything at once. Start with observability (anti-pattern #4), because you can't fix what you can't see. Then work on idempotency, then schema contracts, then decomposition. Configuration cleanup can happen in parallel. Your future self—the one who isn't getting paged at 3 AM—will thank you.

This story contains new, firsthand information uncovered by the writer.

This story contains AI-generated text. The author has used AI either for research, to generate outlines, or write the text itself. 

Terra

PagerDuty

5 Data Pipeline Anti-Patterns That Silently Wreck Your Stack (And How I Fixed Them)

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

“Bring Your Own Agent” Meets “Bring Your Own Data”: ADBC-First Notebooks as a Governed Data UX

3 Best Hadoop Alternatives to Consider for Migration

5 Big Data Problems and How to Solve Them

6 Database Migration Tools For Complete Data Integrity & More

7 Data Analytics Tools to Boost Your Business

8 Tools You Can Use to Analyze Big Data

“Bring Your Own Agent” Meets “Bring Your Own Data”: ADBC-First Notebooks as a Governed Data UX

3 Best Hadoop Alternatives to Consider for Migration

5 Big Data Problems and How to Solve Them

6 Database Migration Tools For Complete Data Integrity & More

7 Data Analytics Tools to Boost Your Business

8 Tools You Can Use to Analyze Big Data

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps