YAPDT: Yet Another Python Decorator Tutorial (intermediate)

As explained below, this is a beginner-to-intermediate tutorial, it assumes decorators are not brand new concepts to you, but you’re having trouble and implementing writing them.

Purpose of this tutorial

As I think is a common motivation to write tutorials, I wanted to create the tutorial I wish I’d read when I was at the stage to need one to explain certain concepts to me, once I was past the many, many good beginners’ tutorials out there (I recommend reading a few different ones as a beginner to really grok the concepts and see the toy examples). So I’ll assume you understand well enough that

  • Python functions are first-class objects, which means they can return functions
  • Python decorators are functions that return modified functions allowing you to apply the same modifications to different functions and preserve the DRY (Don’t Repeat Yourself) principle instead of writing the same exact code in a bunch of different function definitions (a no-no)
  • You know how to unpack *args and *kwargs, at least vaguely in concept
  • The @decoratorname syntax is just syntactic sugar for myfunction = decoratorname(myfunction). This means that, confusing to some including me at first, even though the @decoratorname comes BEFORE the function definition def myfunc(myargs), it’s as much part of the function definition as the stuff that comes after the def keyword. It’s just part of the definition you can apply to a bunch of functions without repeating yourself.
  • Additionally, I’ll be importing functools from the standard library and using the@functools.wraps()decorator factory (don’t worry if you don’t know the different between a decorator and a decorator factory, we’ll get to that) within the decorator defintion, which sounds weird and confusing but essentially is just one line of boilerplate, you can just set it and forget it. It preserves the docstring and certain properties of the original function so they don’t get replaced by that of part of the decorator function. You can leave it out if you want, but it’s easy enough to just use. If you want to get how it works, how it’s useful and what happens if you don’t use it, there are always the Python docs (which I never link to because if you’re advanced enough to understand how to do something from the Python docs you’re advanced enough to find the darn things yourself), Stackoverflow or tutorials (here’s a decent example from a very good blog,

Not the Purpose of this Tutorial

  • As I said, I won’t be covering beginners’ concepts. There are a TON of beginners’ tutorials out there. As I used to say to interns I would train (mostly coming from R and, poor things, SAS), “You learn decorators for the first time about 10 times before you get them”. As in everything in Python, it really, really, really helps to write your own and this tutorial hopes to facilitate that,
  • I won’t be covering decorators built into python like @staticmethod or @properry or those in third-party libraries like or Flask’s @app.route. This is a tutorial for writing your own decorators. But by all means, use and learn then them, especially the builtins, they’re essential at a certain point.
  • I won’t be giving examples of practical decorators you can copy and paste. StackOverflow and lots of Python blogs and websites are FULL of them! I fully believe you should be able to write your own… and then copy and paste all you want.

 

1. Difference between decorators and decorator factories

They’re written differently, which I’ll show in Part 2 right below, but here’s the essential concept:

  • If your @decorator_name is bare and doesn’t take any arguments, it’s a decorator
  • If it takes arguments, like says @retry(max_retries), it’s a decorator factory, essentially a function that creates decorator functions.

The difference is most germane in the fact that decorator factories have an extra, outer, function to write (the factory function)

The nitty gritty of this is coming up in the next section.

 

2. Boilerplate code and explanations for decorators and decorator factories

This boilerplate code will be useful for the rest of the tutorial. I have these in my snippets (I change snippet programs every few months, but that’s my weird behavior burden to bear, we’ll speak no more of it), although I’ve finally pretty much memorized them by now/

Decorators

They consist of three nested functions:

  1. The function to be decorated, referred to but by its assigned name, by a generic name, in the decorator function
  2. The wrapping function whose name will never be seen by the user, so you can call it anything (I often used “wrapped” or “wrapper” depending if I’m in a noun or a verb mood that day, or something more descriptive of the wrapped function it returns)
  3. The decorator function whose name appears after the @ symbol
import functools

def decorator_function(function_to_be_decorated):  # NB no args/kwargs
    """Decorator's docstring that described what it does"""
    @functools.wraps(function_to_be_decorated)  # NB no args/kwargs
    def wrapped(*args, **kwargs): # NB first appearance of args and kwargs
        # stuff before?
        return_value_if_any = function_to_be_decorated(*args, **kwargs)
        # stuff after?
        return return_value_if_any
    return wrapped

which is then used like:

@decorator_function
def myfunc():
    """Note that myfunc does not share a name with function_to_be_decorated above.
    That's the point, you can pass any function to the decorator."""
    # rest of function

A few notes:

  • pay attention to the #NB comments in the definition above. The first two sets of parentheses (in the decorator function and the funcrools.wraps decorator factory) contain the function to be decoratored (which I’ve called “function_to_be_decorated” to be clear and/or verbose, but must people write as func or f; the next twothe args/kwargs appear only in the wrapped function and the second appearance of the function to be decorated
  • if you don’t actually base any of the logic of the wrapped function on return_value_if_any, you can condense the two lines into th eline:
    • return function_to_be_decorated(*args, **kwargs)
    • Note that because of the parentheses, you’re returning the return value, not the function! (Always a source of confusion among beginner decorator writers, which part returns what type of thing)
  • You can specify the arguments instead of using generic *args, **kwargs if you don’t want the decorator to be general (like a timing decorator is), and restrict it to functions with those arguments.
  • Where it says # stuff before? and # stuff after? is where the decorator’s magic (er, I mean logic) happens. A decorator can do any combination of the following:
    • base its action on an arg or kwarg or any combination of them, specified or nonspecified
    • CHANGE an arg or kwarg, specified or nonspecified
    • base its action on the return value of the function to be decorated
    • CHANGE the return value of the function to be decorated
    • Do none of the above, and have logic totally independent of the function to be decorated and its properties; the classic example is a timer decorator (BTW I never write timer decorators, only context managers, but that’s a subject for another blog post, there’s nothing wrong with timer decorators)
  • As I mentioned, the only argument for a decorator function is the function to be decorated. If you want to pass any other argument, you need a DECORATOR FACTORY, which is a function that returns a decorator DEFINED WITH THAT ARGUMENT/

Without further ado…

Decorator factories

They consist of four nested functions. The extra one is an outer one; the inner three remain the same

def decorator_factory(decorator_arg_or_args):
    """Docstring, explaining the arg_or_args"""
    def decorator_function(function_to_be_decorated):
    @functools.wraps(function_to_be_decorated)
    def wrapped(*args, **kwargs):
        # somewhere in here some logic is based on decorator_args_or_args
        # stuff before?
        return_value_if_any = function_to_be_decorated(*args, **kwargs)
        # stuff after?
        return return_value_if_any
    return wrapped

One of many possible and possibly common uses of a decorator factory is a retrying decorator, when you’re not sure of the stability of a connection to a source (like a website) out of your control, e.g.

@retrying_decorator(max_tries=10, secs_between_tries=10)
def myfunctoretry():
    """Etc."""

As I said, this really returns a different decorator every time you specify different arguments, but that’s really an implementation detail you don’t have to keep track of in most cases, you just need to write it correctly.

2. Decorators are part of the function definition

This is something it took me an embarrassingly long time to grok, and some interns I’ve trained have struggled with it too. (One intern got it in about 3 seconds, but that person will be running a fortune 500 company one day, he’s way smarter than me.)

I though of giving a bunch of examples here, but it’s really a beginner concept that I’m repeating in this intermediate tutorial because I’ve seen it in some cases misunderstood by people who’ve gotten the beginner tutorials but didn’t fully appreciate this concept.

Okay, so you write your decorated function:

@my_decorator
def my_function(my args):
    # do stuff
    return something_something

It’s potentially confusing, because the syntactic sugar of the @my_decorator syntax comes before the def keyword, but all that code you wrote in the function is executed every time you call the function defined by the def keyword.

I’ll just give a simple example and we’ll move on.

If you define:

@timer
def fibonacci(n):
    # rest of function

Then every time you call fibonacci() in your code it will be timed. If you want to start timing it, you need to remove the decorator, and none of your calls to fibonacci() will be timed.

If you want a timed and a non-timed fibonacci calculator both, you could do something like:

def fibonacci(n):
    # rest of function, not timed

@timer  # defined elsewhere
def timed_fibonacci(n):
    return fibonacci(n)    

3. Decorator classes

If a class has a __call__ dunder, it can be used like a function (with parentheses syntax), and so if written in an amenable format, can be used as a decorator, after it’s instantiated. Since you can __init__ it with arguments (or just set attributes dynamically), it can work like a decorator factory, except you can change the arg after it’s instantiated. Here’s some boilerplate and an extremely simple example.

class MyDecorator:
    def __init__(self, some_arg):
        self.some_arg = some_arg

    def __call__(self, function_to_be_decorated):
        def wrapped(*args, **kwargs)
            # do stuff with self.some_arg
            some_logic_with(self.some_value)
                return func(*args, kwargs)
        return wrapped

mydecorator = MyDecorator(an_arg)

Example:

class MyMultiplier:
    """Just multiplies by a number"""
    def __init__(self, multiplier):
        self.multiplier = multiplier

    def __call__(self, f):
        def wrapped(arg):
            return self.multiplier * f(arg)
        return wrapped

multby = MyMultiplier(2)

@multby
def change_to_float(n):
    return float(n)

@multby
def wrap_in_list(a_value):
    return [a_value]

# Usage:
>>> change_to_float(2)
4.0
>>> wrap_in_list('a')
['a', 'a']
>>> multby.multiplier = 3
>>> change_to_float(2)
6.0
>>> wrap_in_list('a')
['a', 'a', 'a']

4. You can decorate class methods, and you can stack decorators

Class methods are functions, and can be decorated like any function.

def MyClass:
    "Class docstring"
    def __init__(self, some_args):
       # stuff

    @decorator_name
    def mymethod1(self):
        # etc

    @decorator_name  # maybe the same, maybe a different one
    def mymethod2(self):
        # etc.

You can also decorate entire classes, which is an advanced subject I’m not touching with a ten-foot pole here, but here’s an excellent tutorial (scroll down quite a bit): Advanced Uses of Python Decorators

Also you can stack decorators, like:

@outer_decorator
@inner_decorator
def myfunc()
   """Etc."""

Keep in mind that order matters. decorator2 decorates the myfunc decorated by decorator1, and it’s probable that you’ll get a different result (depending on the decorators) if you write:

@inner_decorator
@outer_decorator
def etc.

5. Common uses of decorators

As I mentioned, there are tons of examples of decorators you can copy and paste to get common jobs done. But I’ve worked with a few interns who because of time pressures do that, and don’t really ever get the chance to understand how they work, which is a shame, the whole idea of an internship is to walk out the door with more skills (copy-pasting doesn’t count as a skill) .The best way to learn is by doing, and it’s a great idea to try to write your own decorators that duplicate the common uses of decorators, so that if something goes wrong you can then look up a functioning version and see where you went wrong. So here are some examples.

  • Timing functions. Almost always the first example on StackOverflow. (You can always use a Context Manager for these too, writing them is a good way to learn those and their __enter__ and __exit__ dunders.)
  • Retrying functions. Bonus points for having longer times between tries per try.
  • Setting a maximum time for a function to finish executing, if the function is amenable to it (e.g. it runs in a loop)
  • Logging functions. There are tons of different types of these (and some PyPI packages that are based on this), because there are tons of different use cases for logging. They generally require a decorator factory or class, because they require parameters (such as the logger instance). P.S.: if you aren’t logging, you should be. Once you get over the cognitive overhead, it will change your life.
  • Simple debugging, sending a function’s inputs and outputs to a log or just stdout
  • Validating function inputs. Python is EAFP rather than LBYL, but it’s still useful to validate inputs sometimes depending on use case. I’ve written classes that use the same decorator with a bunch of different methods. It’s also handy to CHANGE inputs if they’re in the wrong format, e.g. if the user forgets to start a url with “http://”, you can add it.
  • Validating function outputs, for example setting a max/min.
  • Waiting/rate-limiting. Don’t want that web service to ban you for pinging them too frequently!
  • Caching/memoization. If you’ve got an pure/idempotent (i.e. a functional programming pagradigm functions) that’s also expensive (big O/long running time) that you run several times, sometimes with the same inputs, why not cache the results so you don’t have to recalculate from scratch? (Using decorators to memoize the recursive fibonacci calcualator is a classic example, and the linked tutorial also shows (and perfers, which I approve of) the built-in functools.lru_cache memorizing decorator
  • Gracefully handling database transactions, e.g. rolling back if there’s an Exception (although many DB-API packages come with their own ways to set this up)
  • Synchronization, i.e. acquiring and releasing locks in multithreading/multiprocessing applications
  • Authentication (this is what some web framework decorators do, i.e. make sure a user is logged in before they are allowed to do some task/access some resource)

Thanks so much for reading!

I welcome feedback, I JUST moved to WordPress from Blogger (yes, I joined this decade), but I’m still getting used to the framework (there are only so many hours in the day and science hasn’t figured out how to clone me to be more productive yet… it’s probably not a healthy personality trait that I find that idea attractive. I really need to figure out how to do python syntax highlighting in code blocks, for one) so I’m not 100% sure the commenting function is working properly or I have the spam filter properly tuned, so as a workaround you can send me a tweet or an e-mail and I can post it, or not, if you prefer.

I welcome all types of feedback, ESPECIALLY NEGATIVE FEEDBACK! The way I see it, there are two types of negative feedback:

  1. People pointing out mistakes or ways I could have done thing better, for which, THANK YOU! I want to make fewer mistakes and I want to do things better!
  2. People just being toxic, and I’ve been on both Twitter and Reddit (under like 10 different accounts) for many years (before the Digg migration for the latter!), so I have thick skin and can take it/ignore you if warranted.

I can’t 100% guarantee I can without exception tell the difference between the two in edge cases, so if I ignore you when you think I shouldn’t have, my apologies in advance, and if you reach out I’ll be happy to make amends.

Special shoutout and thanks to Nicolas Kruchten for reading a first draft of this article and making some very, very good suggestions for how I could be a LOT clearer in Part 2.

Have a pythonic day! (God that sounds stupid, but I had to end the article gracefully. Come to think of it, calling something I wrote stupid probably isn’t graceful. But it’s so me.)

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.