After years of searching, there is still no cure for Digital Disposophobia After years of searching, there is still no cure for Digital Disposophobia This is first in a series that try’s to dig deeper into the Data Migration Tax Everyone “has” a 3-2-1 strategy—until they try to restore something ugly at 2 a.m. and discover they actually had 1-0-0 with vibes. The sticker on the whiteboard says three copies, two media, one offsite. The real world says: data lives in three simultaneous states (ingest, in-flight, verify), storage systems are not saints, and “offsite” that can’t be validated on demand is just an expensive daydream. three simultaneous states Why it matters Why it matters Migrations and long-term preservation don’t fail because we forgot a slogan. They fail because the pipeline and the policy are out of sync. If your second and third copies don’t move at the speed of verification, you’re left with Schrödinger’s archive—probably fine, unprovably so. At multi-petabyte scale, that gap becomes budget-eating: you pay for hardware, people, and power twice while your “redundant” copies slowly converge (or don’t). The fix is a 3-2-1 that is operational, not ornamental. operational What people miss What people miss Copies aren’t independent if the process isn’t. If Copy-2 and Copy-3 both derive from the same unverified staging area or rely on the same fragile job runner, that’s correlated risk masquerading as redundancy.
“Cloud offsite” isn’t a backup if you treat it like a black hole. If you don’t routinely re-hydrate, list, and verify sample sets (or manifests) from cloud tiers, you’ve outsourced your peace of mind to marketing.
Fixity has to be first-class metadata. Checksums (and, for objects, multipart ETag awareness) belong with the file from the first touch—not bolted on later with a heroic summer intern.
People are the third medium nobody budgets. Dual-stack years, on-call rotas, and exception triage are part of 3-2-1. If it takes a wizard to restore, you didn’t buy redundancy—you bought hero worship.
Lifecycle ≠ longevity. Auto-tier policies are not preservation policies. Your hot/cold rules should follow validation windows and risk, not just last-access dates. Copies aren’t independent if the process isn’t. If Copy-2 and Copy-3 both derive from the same unverified staging area or rely on the same fragile job runner, that’s correlated risk masquerading as redundancy. Copies aren’t independent if the process isn’t “Cloud offsite” isn’t a backup if you treat it like a black hole. If you don’t routinely re-hydrate, list, and verify sample sets (or manifests) from cloud tiers, you’ve outsourced your peace of mind to marketing. “Cloud offsite” isn’t a backup if you treat it like a black hole Fixity has to be first-class metadata. Checksums (and, for objects, multipart ETag awareness) belong with the file from the first touch—not bolted on later with a heroic summer intern. Fixity has to be first-class metadata People are the third medium nobody budgets. Dual-stack years, on-call rotas, and exception triage are part of 3-2-1. If it takes a wizard to restore, you didn’t buy redundancy—you bought hero worship. People are the third medium nobody budgets Lifecycle ≠ longevity. Auto-tier policies are not preservation policies. Your hot/cold rules should follow validation windows and risk, not just last-access dates. Lifecycle ≠ longevity validation windows and risk Make 3-2-1 operational: the playbook Make 3-2-1 operational: the playbook Define independence properly.
Fixity-first ingestion.
Verification windows as SLOs.
Staging that doesn’t lie.
Cloud offsite with teeth.
Tape as a stabilizer, not a scapegoat.
Runbook the hard parts. Define independence properly. Fixity-first ingestion. Verification windows as SLOs. Staging that doesn’t lie. Cloud offsite with teeth. Tape as a stabilizer, not a scapegoat. Runbook the hard parts. A simple architecture that works A simple architecture that works Copy-1 (Primary Service): fast object or POSIX store with fixity-on-write, immediately hash-validated.
Copy-2 (Stabilizer Medium): tape (or other long-lived medium) written from validated data with a manifest; periodic scrub reads.
Copy-3 (Offsite): separate provider/region/account with routine restore-and-verify drills and cost-capped egress rehearsals.
Glue: a small database (or log-structured store) that tracks asset → checksums → copy locations → verification events. If you can’t answer “when was the last time we proved Copy-3 existed for this asset?” in one query, build that table today. Copy-1 (Primary Service): fast object or POSIX store with fixity-on-write, immediately hash-validated. Copy-1 (Primary Service) Copy-2 (Stabilizer Medium): tape (or other long-lived medium) written from validated data with a manifest; periodic scrub reads. Copy-2 (Stabilizer Medium) Copy-3 (Offsite): separate provider/region/account with routine restore-and-verify drills and cost-capped egress rehearsals. Copy-3 (Offsite) Glue: a small database (or log-structured store) that tracks asset → checksums → copy locations → verification events. If you can’t answer “when was the last time we proved Copy-3 existed for this asset?” in one query, build that table today. Glue asset → checksums → copy locations → verification events Numbers that keep you honest (order-of-magnitude is fine) Numbers that keep you honest (order-of-magnitude is fine) Verification debt: (# assets written minus # assets independently verified). If it grows for more than a week, you’re borrowing against luck.
Mismatch threshold: agreed-upon rate that trips incident mode (e.g., >0.01% per 100k assets in a 24-hour window).
Restore SLOs: class-based targets (e.g., “tier-A: 4-hour partial, 24-hour full,” “tier-B: 24-hour partial, 5-day full”).
People math: shifts × pipelines × interventions/hour. If the model assumes zero human touches, it’s fiction. Verification debt: (# assets written minus # assets independently verified). If it grows for more than a week, you’re borrowing against luck. Verification debt Mismatch threshold: agreed-upon rate that trips incident mode (e.g., >0.01% per 100k assets in a 24-hour window). Mismatch threshold Restore SLOs: class-based targets (e.g., “tier-A: 4-hour partial, 24-hour full,” “tier-B: 24-hour partial, 5-day full”). Restore SLOs People math: shifts × pipelines × interventions/hour. If the model assumes zero human touches, it’s fiction. People math Common anti-patterns (and what to do instead) Common anti-patterns (and what to do instead) “We’ll hash later.” → You won’t. Hash now, once, at first contact; propagate the value.
“The cloud copy is our DR.” → Prove it with restores. DR you never rehearse is PR.
“Two buckets, one account.” → That’s not offsite; that’s folders with delusions of grandeur.
“The wizard knows the steps.” → Put the wizard’s brain in the runbook, then take a vacation. If anxiety spikes, the runbook is lying. “We’ll hash later.” → You won’t. Hash now, once, at first contact; propagate the value. “The cloud copy is our DR.” → Prove it with restores. DR you never rehearse is PR. “Two buckets, one account.” → That’s not offsite; that’s folders with delusions of grandeur. “The wizard knows the steps.” → Put the wizard’s brain in the runbook, then take a vacation. If anxiety spikes, the runbook is lying. If your 3-2-1 fits in a one-slide diagram and not in a runbook, you don’t have a strategy—you have clip art. If your 3-2-1 fits in a one-slide diagram and not in a runbook, you don’t have a strategy—you have clip art. So what / CTA So what / CTA Where does your 3-2-1 actually break—in fixity, independence, or rehearse-to-recover? If you had to restore 50 TB by Friday with auditors watching, what fails first: people, pipeline, or provider? Drop your failure story (names optional); I’ll trade you a checklist. Question:  How do you balance agility with regulatory demands effectively? Question How do you balance agility with regulatory demands effectively? Short Answer: Treat compliance as a design input, not an after-the-fact speed bump. The balance is: policy → pipeline → proof. Policy (what): Define verification windows and offsite rules as SLOs that map to the regs you care about (immutability/retention, independence, and evidence). Example: “Copy-2 verified ≤24h; Copy-3 (geo/off-account) verified ≤7d; 1% rolling sample/month with full re-hash.”
Pipeline (how): Make independence real. Our first two copies land minutes–hours apart via HSM, but the geo copy is pushed as raw files (not a container) to keep failure modes independent. We follow with a structure diff to catch the inevitable deltas.
Proof (show me): Every asset carries fixity at first touch; verification is a state machine, not a hope: Policy (what): Define verification windows and offsite rules as SLOs that map to the regs you care about (immutability/retention, independence, and evidence). Example: “Copy-2 verified ≤24h; Copy-3 (geo/off-account) verified ≤7d; 1% rolling sample/month with full re-hash.” Pipeline (how): Make independence real. Our first two copies land minutes–hours apart via HSM, but the geo copy is pushed as raw files (not a container) to keep failure modes independent. We follow with a structure diff to catch the inevitable deltas. Proof (show me): Every asset carries fixity at first touch; verification is a state machine, not a hope: Controls that keep you agile Button, not binder: one-click report that answers, “When was Copy-3 last proven for Asset X?”
SLOs ≠ best-effort: publish debt (assets written – assets independently verified). If it grows >7 days, you’re borrowing against luck.
Offsite independence: separate account/tenant + restore drills (cost-capped) so “offsite” isn’t just a billing line. Button, not binder: one-click report that answers, “When was Copy-3 last proven for Asset X?” SLOs ≠ best-effort: publish debt (assets written – assets independently verified). If it grows >7 days, you’re borrowing against luck. Offsite independence: separate account/tenant + restore drills (cost-capped) so “offsite” isn’t just a billing line. Snarky aside: if your compliance story is a binder, not a button, you don’t have compliance—you have décor.

The Data Migration Tax: Paying Interest on the Myth of 3-2-1

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

Inside a 34-Petabyte Migration: The True Cost of Moving a Digital Mountain

Could Blockchain and Big Data Come Together To Open Up A New Chapter in Data Integrity?

Data Integrity In Cryptography

Data Integrity Is Vital for The COVID-19 Vaccine Rollout

Data Quality: Its Definitions And How to Improve It

Inside a 34-Petabyte Migration: The True Cost of Moving a Digital Mountain

Could Blockchain and Big Data Come Together To Open Up A New Chapter in Data Integrity?

Data Integrity In Cryptography

Data Integrity Is Vital for The COVID-19 Vaccine Rollout

Data Quality: Its Definitions And How to Improve It

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps