32,306 reads

Adding a pipe operator to Python

by Bogdan PopaAugust 12th, 2018

Too Long; Didn't Read

The other day, I was looking at some Python code that was made up of a sequence of deeply-nested function calls and a thought immediately got pushed up brain’s stack:

People Mentioned

Company Mentioned

featured image - Adding a pipe operator to Python

Or, “things you should never ever do in production”

The other day, I was looking at some Python code that was made up of a sequence of deeply-nested function calls and a thought immediately got pushed up brain’s stack:

Wouldn’t it be cool if Python had an operator akin to Elixir’s pipe operator?

Pipe operator?

For those of you not familiar with Elixir, the pipe operator |> can be used to turn code that looks like this:

Nested function calls.

Into code that looks like this:

Refreshing!

In essence, the pipe operator takes the expression on its left-hand side and moves it into the first argument position of the expression on its right-hand side, assuming the expression on the right is a function call.

Doing it in Python

Although adding an actual pipe operator to the language isn’t possible without changing the Python interpreter, there’s nothing preventing us from re-purposing an existing operator! That’s exactly what I set out to do.

Borrowing from shells, I figured the “bitwise or” (aka “pipe”) operator would be a good fit for this sort of functionality:

The simplest way I could come up with to repurpose the pipe operator was to rewrite functions using a decorator. Python’s built-in ast module makes this particularly easy.

All I had to do was

get a function’s source code (using inspect.getsource),
turn it into an abstract syntax tree by passing it toast.parse,
walk the tree and transform any occurrences of the pipe operator according to the rules I outlined at the end of the last section and,
finally, compile and return the rewritten function object.

That sounds a lot more complicated than it turns out to be in practice.

The ast module provides a class called NodeTransformer that implements the visitor pattern: its visit method does a depth-first search through an AST and calls any declared methods of the form visit_NODETYPE on itself for each node in the tree. As the name implies, you can use a node transformer to visit and manipulate the nodes of an AST:

Whenever the transformer encounters a binary operator, it recurses so that any transformations that need to be made on its left-hand and right-hand nodes are made first, then it checks if the current operator is “bitwise or”.

If the current node’s operator is “bitwise or” and if the right-hand side is a function call node, then it inserts the left-hand side into the first argument position of the function call and then returns the node on the right-hand side, replacing the binary operator node with the call node in the tree.

The transformer also kicks in when it sees a function definition so that it may remove the enable_threadop decorator. This’ll make sense once you take a look at the decorator itself:

The decorator takes a function as an argument, grabs its source code and removes any indentation (this is important! otherwise, decorating class methods results in a SyntaxError), parses the code and transforms the AST and, finally, it compiles and executes the function definition, returning the resulting function object.

If the transformer hadn’t removed the decorator from the final tree, then we’d have an infinite loop on our hands because enable_threadop would be called over and over again when the function definition is executed (line 13).

With all that in place, the enable_threadop decorator may be used to selectively change the behaviour of the pipe operator: