The case for distributed execution without orchestration You have an async function. You want to run it on another machine. How hard could that be? If you’ve gone looking for the answer in the Python ecosystem, you already know: unreasonably so. Not because the problem is complex, but because every framework that offers to solve it insists on solving a dozen other problems you didn’t ask about. Want distributed execution? Great — but first, define your workflow as a DAG. Configure a state backend. Pick a serialization format from a list of four. Set up a message broker. Write a retry policy. Decide on a dead letter queue strategy. Oh, and here’s a decorator, but it only works on top-level functions with JSON-serializable arguments that are registered in a task registry that the worker must import at startup. You wanted to run a function somewhere else. You got an operating system. You wanted to run a function somewhere else. You got an operating system. The quiet assumption The quiet assumption There’s a pattern here worth naming. Somewhere along the way, the Python ecosystem decided that distributed execution is inseparable from orchestration. That you can’t have one without the other. That handing a function to a remote process is meaningless unless the framework also knows what to do when that process catches fire. It’s a reasonable instinct. Distributed systems are unreliable. Failures do happen. But there’s a difference between acknowledging that reality and baking a specific response to it into the execution layer. are do When you write a regular async function in Python, the language doesn’t force you to register an error handler before you’re allowed to `await` it. It doesn’t demand that you declare the function in a central registry. It doesn’t serialize your arguments through a format that strips away half their type information. It gives you a simple, transparent mechanism — `async`/`await` — and trusts you to build whatever policies you need on top. Why should distributed execution change any of this? Why should distributed execution change any of this? The cost of opinions The cost of opinions The frameworks that dominate this space — and I don’t need to name them — are genuinely impressive pieces of engineering. They solve real problems for real teams running real workloads. This isn’t about them being wrong. It’s about them being opinionated at the wrong layer. It’s about them being opinionated at the wrong layer. When orchestration, checkpointing, and retry logic are fused into the execution primitive, you don’t just get those features — you get their constraints. Your functions must conform to a specific shape. Your arguments must survive a round trip through JSON or a proprietary serializer. Your control flow must be expressible as a graph. Your error handling must fit the framework’s model, not yours. These constraints compound. They push you toward writing “framework code” — code that exists to satisfy the tool rather than express your intent. The function you wanted to distribute starts to look nothing like the function you would have written if distribution weren’t a concern. And if you don’t need orchestration? If you just want to fan out some computation, or stream results from workers, or run the same async pipeline across multiple machines? You’re still paying the full tax. don’t A different starting point A different starting point What if we started from the other end? Instead of building a distributed application framework and embedding an execution engine inside it, what if we just… made execution distributable? The idea is simple: Python already has the right primitives. Async functions give you concurrency. Async generators give you streaming. `await` gives you composition. Exceptions give you error propagation. Context variables give you scoping. What’s missing is the ability to say “run this over there” without giving any of that up. Not “run this over there according to my DAG definition.” Not “run this over there and checkpoint the result.” Just: this function, that machine, same semantics. That’s what I set out to build with Wool ↓ That’s what I set out to build with Wool ↓ https://github.com/wool-labs/wool?embedable=true https://github.com/wool-labs/wool?embedable=true Wool in thirty seconds Wool in thirty seconds Distributed coroutines import asyncio, wool

@wool.routine
async def add(x: int, y: int) -> int:
    return x + y


async def main():
    async with wool.WorkerPool(size=4):
        result = await add(1, 2)
        print(result)


asyncio.run(main()) import asyncio, wool

@wool.routine
async def add(x: int, y: int) -> int:
    return x + y


async def main():
    async with wool.WorkerPool(size=4):
        result = await add(1, 2)
        print(result)


asyncio.run(main()) One decorator and a context manager. The function is still an async function. You still `await` it. You still catch its exceptions. If it’s an async generator, you still iterate it. The only difference is that it executes on a remote worker process instead of the local event loop. Workers discover each other through pluggable backends — shared memory for a single machine, Zeroconf for a local network, or roll your own to suit your existing stack. There’s no broker, scheduler, or central coordinator. Workers cooperate as part of a peer-to-peer network. The topology is flat. What Wool doesn’t do is just as important. No built-in retry logic — that’s your call. No persistent state — if you need it, you know your storage better than I do. No dead letter queues, no workflow engine, no DAGs. Wool provides best-effort, at-most-once execution, and gets out of the way. doesn’t No built-in retry logic No persistent state No dead letter queues, no workflow engine, no DAGs. The bet is that the language already gives you enough to build whatever policies you need. Wool just makes the execution itself transparent. Distributed async generators import asyncio, wool

@wool.routine
async def search(query: str):
    chars = sorted(charset)
    lo, hi = 0, len(chars) - 1
    while lo <= hi:
        mid = (lo + hi) // 2
        hint = yield chars[mid]
        if hint == "higher":
            lo = mid + 1
        elif hint == "lower":
            hi = mid - 1
        else:
            return

async def main():
    async with wool.WorkerPool(size=4):
        stream = search("z")
        guess = await anext(stream)
        while True:
            if guess == "z":
                await stream.aclose()
                break
            hint = "higher" if guess < "z" else "lower"
            guess = await stream.asend(hint)

asyncio.run(main()) import asyncio, wool

@wool.routine
async def search(query: str):
    chars = sorted(charset)
    lo, hi = 0, len(chars) - 1
    while lo <= hi:
        mid = (lo + hi) // 2
        hint = yield chars[mid]
        if hint == "higher":
            lo = mid + 1
        elif hint == "lower":
            hi = mid - 1
        else:
            return

async def main():
    async with wool.WorkerPool(size=4):
        stream = search("z")
        guess = await anext(stream)
        while True:
            if guess == "z":
                await stream.aclose()
                break
            hint = "higher" if guess < "z" else "lower"
            guess = await stream.asend(hint)

asyncio.run(main()) Async generators get the same treatment. The caller drives the generator with `asend()`, `athrow()`, and `aclose()`, exactly as if it were local. Under the hood, each yield crosses a network boundary — the client sends a command, the worker advances the generator one step and streams the value back. But from the caller’s perspective, it’s just an async generator. Streaming is how you build pipelines, progressive results, long-running conversational protocols — the fact that you can distribute them without changing how they work is the whole point. What’s next? What’s next? Wool is still (very) early, but it works. Future posts will share progress and lessons learned as I continue to build this thing. If you’ve ever felt like distributed Python code shouldn’t require adopting a new programming model, Wool might be for you. Try it out — I welcome your feedback, use cases, and honest skepticism. Thanks for reading. Thanks for reading.

This story contains new, firsthand information uncovered by the writer.

This story contains AI-generated text. The author has used AI either for research, to generate outlines, or write the text itself. 

What if Python Was Natively Distributable?

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

The Noonification: Panda Power (11/28/2023)

What is Python Good for? Why Beginner Should Learn Python?

10 Python Projects with 10 Lines of Code

10 helpful Python Tips and Tricks for Beginners

The Noonification: How to Dockerize And Deploy Django Applications (10/14/2023)

11 Tips And Tricks To Write Better Python Code

The Noonification: Panda Power (11/28/2023)

What is Python Good for? Why Beginner Should Learn Python?

10 Python Projects with 10 Lines of Code

10 helpful Python Tips and Tricks for Beginners

The Noonification: How to Dockerize And Deploy Django Applications (10/14/2023)

11 Tips And Tricks To Write Better Python Code

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps