Picture yourself transferring $1,000 from New York to London. Your bank removes the funds from your account. A currency exchange service locks in the rate. The UK bank should receive the money. The exchange service crashes after your money leaves but before it arrives. Your funds have left your account but haven't been credited elsewhere, so they are effectively stuck. Modern payment systems settle in seconds, so this failure can make transactions irretrievable and force reconciliation teams to hunt for missing transfers. Manual reconciliation fails at a scale of one million transactions per hour. And lost money drains customer confidence What we've used — and why some of those are not great Building distributed payment systems has always meant picking between a few key design choices. Two-Phase Commit (2PC) Two-phase commit locks all databases until every participant completes, making it a blocking protocol that doesn't scale. If the UK bank's API times out, your US bank's database remains blocked until the timeout resolves. I've seen entire production systems come to a standstill because one service started taking 500ms slower than normal. The maintenance burden increases as the number of interdependent steps grows. Manual Sagas The most reliable approach requires writing code for every failure case. If step B fails, execute the compensation (rollback) for step A—safe but painfully slow, which means you must maintain compensation logic for every operation: change an API call and you must update its rollback; miss an edge case and you risk data inconsistencies and overwhelmed support. If the UK bank's API times out, your US bank's database remains blocked until the timeout resolves. I have seen systems halt because a service ran 500 milliseconds slower than expected. Maintenance work increases with each added dependent step. Durable execution Durable execution maintains progress so tasks can continue without repeating work, and the platform handles state, retries, and recovery—reducing custom retry and error handling. This fixes the main problems with the other two approaches without sacrificing safety. How durable execution actually works Durable execution records every function call, API response, and decision in an append-only event log before proceeding. Event log structure Event log structure It records an event scheduling deduct_usd(account, 1000), runs the activity, and then records a completion event {id: "evt_002", type: "ActivityCompleted", result: "success"}. deduct_usd(account, 1000) {id: "evt_002", type: "ActivityCompleted", result: "success"} If the server crashes, the system checks the logs when it restarts. Because deduct_usd already ran, the system skips re-running it, uses the stored result, and then continues at get_fx_rate. deduct_usd get_fx_rate Activities vs. workflows Activities vs. workflows Workflows are deterministic and replayable; activities are the side-effecting operations they call (sending emails, calling APIs, writing to a database). Since activities can fail or return different results, the framework records their outcomes and activities should be idempotent. When you call an activity from a workflow, the framework executes the side effect and records the result. Activities must be idempotent. If deduct_usd is invoked twice, it must not deduct the same amount twice. The framework guarantees at least once execution, but your code has to handle duplicates. deduct_usd Idempotency Idempotency Create a distinct identifier based on the workflow execution ID and activity sequence number when scheduling an activity. If the ID already exists, deliver the stored outcome; otherwise execute the activity and save the outcome under that ID. State persistence and replay State persistence and replay When the workflow reaches await execute_activity(deduct_usd), the framework checks the log, sees this completed, and returns the stored result without a network call. await execute_activity(deduct_usd) The workflow repeats but side effects occur only once. The workflow advances in chunks: it executes until the next activity, then saves its state, waits for the result, and repeats. The determinism requirement Do not call DateTime.now(), random(), or other non-deterministic functions within workflows. DateTime.now() random() The system cannot reconcile differences during replay because values differ and the workflow history will not match. Your workflow history says "at timestamp X, we did Y," but during replay at timestamp Z, the code takes a different path. workflow.current_time() returns the timestamp recorded in the log, so replaying a workflow yields that logged time rather than the actual current time. workflow.current_time() Versioning Versioning What happens if you deploy new workflow code while existing workflows are still running? Active workflows remain on the code version they started with (for example, v2), while new workflows run the updated version. You can also use a feature flag in the workflow logic: if workflow.version >= 2 then use_new_logic(). The workflow stores the version number to ensure consistent replay. if workflow.version >= 2 then use_new_logic() There are two main approaches, each with different trade-offs Orchestrator model: Temporal/Cadence The orchestrator keeps workflow history and state in Cassandra or PostgreSQL; each execution is recorded as a database row, while workers run business logic and send back results. Workers poll: "Any work for me?" Orchestrator responds: "Execute this activity with these parameters." Worker executes, reports back. Orchestrator logs the result and determines the next step. Scale to millions of concurrent workflows and the orchestrator becomes the bottleneck. You write multiple events per workflow execution. A three-step workflow means 6+ database writes. Temporal uses Cassandra and sharded PostgreSQL because both are easier to scale for writes. Database-native model—DBOS/Postgres backend More and more developers favor the database-native model because it writes directly to PostgreSQL and reduces orchestration. A library wrapper saves each async function's state—including local variables and execution position—to a Postgres table after every await. await Your function gets transformed under the hood. Each await becomes a checkpoint. The library serializes your function's stack frame and stores it. On restart, it deserializes and resumes. await Simpler architecture, but you're loading your database with workflow state. For high-throughput systems, you need careful capacity planning. Your database handles both transactional data and workflow checkpoints. Regardless of which approach you choose, there's one constraint that applies to all durable execution systems: determinism. Code Example Here's a cross-border transfer with Temporal's Python SDK: from temporalio import workflow

@workflow.defn
class TransferWorkflow:
    @workflow.run
    async def run(self, transfer: TransferDetails) -> str:
        # Step 1: Deduct funds from US account
        await workflow.execute_activity(
            deduct_usd,
            transfer,
            start_to_close_timeout=timedelta(seconds=30),
            retry_policy=RetryPolicy(maximum_attempts=3)
        )
        
        # Step 2: Get current exchange rate
        rate = await workflow.execute_activity(
            get_fx_rate,
            "USD_GBP",
            start_to_close_timeout=timedelta(seconds=10)
        )
        
        # Step 3: Credit UK account with converted amount
        await workflow.execute_activity(
            credit_gbp,
            CreditRequest(
                account=transfer.recipient,
                amount=transfer.amount * rate
            ),
            start_to_close_timeout=timedelta(minutes=5),
            retry_policy=RetryPolicy(
                maximum_attempts=100,
                initial_interval=timedelta(seconds=1),
                maximum_interval=timedelta(hours=1)
            )
        )
        
        return "Transfer complete" from temporalio import workflow

@workflow.defn
class TransferWorkflow:
    @workflow.run
    async def run(self, transfer: TransferDetails) -> str:
        # Step 1: Deduct funds from US account
        await workflow.execute_activity(
            deduct_usd,
            transfer,
            start_to_close_timeout=timedelta(seconds=30),
            retry_policy=RetryPolicy(maximum_attempts=3)
        )
        
        # Step 2: Get current exchange rate
        rate = await workflow.execute_activity(
            get_fx_rate,
            "USD_GBP",
            start_to_close_timeout=timedelta(seconds=10)
        )
        
        # Step 3: Credit UK account with converted amount
        await workflow.execute_activity(
            credit_gbp,
            CreditRequest(
                account=transfer.recipient,
                amount=transfer.amount * rate
            ),
            start_to_close_timeout=timedelta(minutes=5),
            retry_policy=RetryPolicy(
                maximum_attempts=100,
                initial_interval=timedelta(seconds=1),
                maximum_interval=timedelta(hours=1)
            )
        )
        
        return "Transfer complete" Look at the retry policies. If credit_gbp fails, it retries with exponential backoff for up to 100 attempts over hours. Your workflow remains persisted while waiting; it does not hold threads or lock connections. credit_gbp Picking your tools Temporal (Go/Java) fits enterprises running millions of workflows—it requires dedicated platform teams and infrastructure investment to scale reliably. DBOS and Inngest are better for small teams that prioritize speed and low operational cost. DBOS and Inngest are more practical for smaller teams and rapidly moving companies. These tools are compatible with Postgres, Node, and Python, so you can get started in days instead of weeks. You are on the hook for less operational overhead, which matters when you are still finding product-market fit. Durable Functions is tightly integrated with Azure and stands out for Azure users, though it does lock you into Microsoft's ecosystem. Conclusion Choosing the right infrastructure matters more than complex error handling when constructing dependable distributed systems. Durable execution frameworks remove complete classes of problems—such as retry logic, state management, and partial failure recovery—by handling them at the infrastructure layer. In many cases, the code you do not write can outweigh the value of the code you write. A platform that guarantees effectively exactly-once execution and automates recovery lets you focus on business logic instead of failure handling.

This story contains AI-generated text. The author has used AI either for research, to generate outlines, or write the text itself. 

Microsoft

Durable Execution: How Fintech Solved the “Money in Limbo” Problem

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Structured Parsing Is the Key to Making LLMs Work on Large Codebases

Your Chatbot Might Be Leaking Secrets—Here’s How to Lock It Down

Improving React App Performance: Leveraging Browser APIs to help React Compiler

Choreography: A Path to Effective Application Integration

Off to the Races: 3 Effective Strategies to Mitigate Race Conditions

The Idempotence Principle in Software Architecture

Structured Parsing Is the Key to Making LLMs Work on Large Codebases

Your Chatbot Might Be Leaking Secrets—Here’s How to Lock It Down

Improving React App Performance: Leveraging Browser APIs to help React Compiler

Choreography: A Path to Effective Application Integration

Off to the Races: 3 Effective Strategies to Mitigate Race Conditions

The Idempotence Principle in Software Architecture

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps