What You Probably Don't Know About Python Decorators

if you have ever been interviewed for a developer position, you heard this question:

What is a python decorator?

If you google "python developer interview questions", you can find various answers to this question and a lot of them are incorrect. For example from here

A decorator is a special kind of function that either takes a function and returns a function, or takes a class and returns a class.

Well, nope.

A decorator is a special kind of function

It can be any callable

that either takes a function and returns a function

it can return anything
it can also take a method
and in some cases it can take anything

or takes a class and returns a class.

same, it can return anything

Let's dive into python decorators and find out what they can do. I will not cover the basics, like decorator with parameters, I will give some strange examples to illustrate useful cases.

Official definition

If you look into PEP 318 -- Decorators for Functions and Methods and PEP 3129 -- Class Decorators you will see, that decorators are just syntax sugar for this -


def foo(self):
    #perform method operation
foo = decorator(foo)


# Can be replaced with this 
@decorator
def foo(self):
    #perform method operation

and this -

class Foo:
  pass
Foo = decorator(Foo)

# Can be replaced with this 

@decorator
class Foo:
  pass

Looks pretty simple and a lot of people underestimate the power of decorators. The trick is there are no restrictions for the

decorator

function.

Most of the examples in the internet will show you something like this -

def print_decorator(func):
    def wrapper(*args, **kwargs):
        print('function', func.__name__, 'called with args - ', args, 'and kwargs - ', kwargs)
        result = func(*args, **kwargs)
        print('function', func.__name__, 'returns', result)
        return result
    return wrapper


@print_decorator
def mul(a, b):
    return a * b

mul(2, 2)
### function mul called with args -  (2, 2) and kwargs -  {}
### function mul returns 4
### 4
mul(3, b=5)
### function mul called with args -  (3,) and kwargs -  {'b': 5}
### function mul returns 15
### 15
###

However you can easily do this -

@type
def func(): # func will be func = type(func) -> <class 'function'>
    return 42

print(func)
### <class 'function'>


@print
def func2(): # print doesn't return anything, so func == None
    return 42

### <function func2 at 0x7fd9c4788950>
print(func2)
### None
###

Yes, i just used build-ins

type

and

print

as decorators. It is useless, and you will never do such nonsense, but it is possible.

Essentially anything that accept class, method or function can be a decorator.

Decorator as design pattern

From GoF book - Decorator Intent

Attach additional responsibilities to an object dynamically. Decorators provide a flexible alternative to subclassing for extending functionality.

If you want to know more online from free sources, you can check this out

In short, a decorator pattern wraps an object and dynamically adds some functionality to it. This is one of the roots of the confusion surrounding decorators in python, because they have the same name, can be used to implement the same idea, but can do a lot more.

Decorator features in python

Let's look at some characteristics of decorators in Python

Decorator is the syntax feature, that can be applied to

classes
functions
methods

The decorator can do anything with a decorated object, for example

add new behaviour (classical decorator pattern from GoF book)
replace decorated object with something else
change interface of decorated class or function (can be used as implementation of Adapter pattern)
cache function calls (for example functools.lru_cache)
create relation with other objects in system (e.g. route('/path/') decorator in flask)
do nothing

Any callable can be used as decorator, including functions, methods, functors

SHOW ME THE CODE!

Let's go over few examples

Replace decorated object with something else

class FunctionTracker:
    def __init__(self, func):
        self.func = func
        self.stats = []

    def __call__(self, *args, **kwargs):
        try:
            result = self.func(*args, **kwargs)
        except Exception as e:
            self.stats.append((args, kwargs, e))
            raise e
        else:
            self.stats.append((args, kwargs, result))
            return result

    @classmethod
    def track_function(cls, func):
        return cls(func)


@FunctionTracker.track_function
def func(x, y):
    return x/y

func(4, 2)
### 2.0
func(x=5, y=2)
### 2.5
func(3, 0)
### Traceback (most recent call last):
###   File "<input>", line 1, in <module>
###     func(3, 0)
###   File "<input>", line 11, in __call__
###     raise e
###   File "<input>", line 8, in __call__
###     result = self.func(*args, **kwargs)
###   File "<input>", line 3, in func
###     return x/y
### ZeroDivisionError: division by zero

func.stats
### [((4, 2), {}, 2.0), ((), {'x': 5, 'y': 2}, 2.5), ((3, 0), {}, ZeroDivisionError('division by zero',))]
func
### <__main__.FunctionTracker object at 0x7fd35978a668>

Be careful with this example, as accumulating data in this way can lead to memory leaks.

Notice how the original

func

was replaced by an instance of

FunctionTracker

, which can be used in the same way as the original function, thanks to the

__call__

method.

Create relation with other objects in system

import json
import base64
import urllib.parse


BASE64_ENCODED_JSON = 'BASE64_ENCODED_JSON'
URLENCODED = 'URLENCODED'


class DataEncoder:
    def __init__(self):
        self._registry = {} # Mapping for algorithm_name and implementation

    def register_format(self, data_format):
        def _register_encoder(encoder_function):
            self._registry[data_format] = encoder_function
            return encoder_function
        return _register_encoder

    def encode(self, data_format, data):
        return self._registry[data_format](data)


encoder = DataEncoder()


@encoder.register_format(BASE64_ENCODED_JSON)
def base64_encoded_json(data):
    return base64.b64encode(
        json.dumps(data).encode('utf-8')
    )

# Notice that `encoder.register_format` can be used as regular function
encoder.register_format(URLENCODED)(urllib.parse.urlencode)
### <function urlencode at 0x7fe9422a86a8>


encoder._registry
### {'URLENCODED': <function urlencode at 0x7fe9422a86a8>, 'BASE64_ENCODED_JSON': <function base64_encoded_json at 0x7fe9406e0620>}
data = {'greetings': 'Hello there', 'answer': 'General Kenobi'}
encoder.encode(BASE64_ENCODED_JSON, data)
### b'eyJncmVldGluZ3MiOiAiSGVsbG8gdGhlcmUiLCAiYW5zd2VyIjogIkdlbmVyYWwgS2Vub2JpIn0='
encoder.encode(URLENCODED, data)
### 'greetings=Hello+there&answer=General+Kenobi'
###

In this example, the function

base64_encoded_json

and

urllib.parse.urlencode

are connected using decorator syntax as implementations for the

DataEncoder

class.

Flask uses a similar approach for registering functions as views via

route('/path/')

decorator; Django has

@admin.register

for registering admin classes.

It can also be used as an alternative to inheritance, in the case of

DataEncoder

I can create a subclass of

DataEncoder

with new encoding implementation instead, which isn't a best solution.

Silly example

Several decorators can be applied to one function. That means, that this

@dec2
@dec1
def func():
    pass

equals this

def func():
    pass
func = dec2(dec1(func))

Which means, that

dec2

can accept ANYTHING, that

dec1

can return.

Disclaimer: There will be weird looking code here. It is here only because I can. I would never recommend anyone use this.

def dict_from_func(func):
    return {func.__name__: func}


operations = {}

@operations.update
@dict_from_func
def mul(a, b):
    return a * b


@operations.update
@dict_from_func
def add(a, b):
    return a / b


print(mul)
### None
print(operations)
### {'mul': <function mul at 0x7fdaf17bbae8>, 'add': <function add at 0x7fdaf16a2510>}
operations['mul'](2, 5)
### 10

Here I use the

dict.update

method as a decorator, even if it is not intended for this. This is possible because

dict_from_func

returns a dict, and

dict.update

takes a dict as an argument.

As a result, I have all the decorated functions compiled in the operations dictionary.

As a side effect - all functions are replaced with

None

, because

dict.update

does not return any value.

Essentially this -

@operations.update
@dict_from_func
def add(a, b):
    return a / b

equals this

def add(a, b):
    return a / b

add = operations.update(dict_from_func(add))

You can try to win a bet with your colleagues using this example.

In conclusion

Decorators is amazing Python feature . You can use them for a variety of purposes. It's not just a “function or class that takes a function or class and returns a function or class”.