Sometimes, we encounter problems that require us to extend the behavior of a function, but we don’t want to change its implementation.
Some of those problems could be: logging spent time, caching, validating parameters, etc.
All these solutions are often needed in more than one function: you often need to log the spent time of every http connection; you often need to cache more than one data base entity; you often need validation in more than one function.
Today we will solve these 3 mentioned problems with Python decorators:
- Spent Time Logging: We’ll decorate a couple functions to tell us how much time do they take to execute.
- Caching: We’ll add cache to prevent a function from executing when its called several times with the same parameters.
- Validating: We’ll validate a function’s input to prevent run time errors.
1. Understanding Functions
Before we can jump into decorators, we need to understand how functions actually work.
In essence, functions are procedures that return a value based on some given arguments.
def my_function(my_arg):
return my_arg + 1
In Python, functions are first-class objects.
This means that functions can be assigned to a variable:
def my_function(foo):
return foo + 1
my_var = my_function
print(str(my_var(1))) # prints "2"
They can be defined inside another functions:
def my_function(foo):
def my_inner_function():
return 1
return foo + my_inner_function()
print(str(my_function(1))) # still prints "2"
They can be passed as parameters (higher-order functions):
def my_function(foo, my_parameter_function):
return foo + my_parameter_function()
def parameter_function(): return 1
print(str(my_function(1, parameter_function))) # still prints "2"
And they can return other functions (also, higher-order functions):
def my_function(constant):
def inner(foo):
return foo + constant
return inner
plus_one = my_function(1)
print(str(plus_one(1))) # still prints "2"
Another thing to notice, is that inner functions have access to the outer scope, that’s why we can use the parameter constant in the inner function of the last example. Also, this access is read-only, we can not modify variables from the outer scope within an inner function.
2. Jumping into decorators
Python decorators provide a nice and simple syntax to call higher-order functions. By definition, a decorator takes a function as a parameter, and returns a wrapper of that given function to extend its behavior without actually modifying it.
Given this definition we can write somthing like:
def decorator(function_to_decorate):
def wrapper(value):
print("you are calling {} with '{}' as parameter".format(function_to_decorate.__name__, value))
return function_to_decorate(value)
return wrapper
def replace_commas_with_spaces(value):
return value.replace(",", " ")
function_to_use = decorator(replace_commas_with_spaces)
print(function_to_use("my,commas,will,be,replaces,with,spaces"))
And after execution, the output will look like:
you are calling replace_commas_with_spaces with 'my,commas,will,be,replaces,with,spaces' as parameter my commas will be replaces with spaces
So, what is actually happening here?
We are defining a higher-order function called decorator that receives a function as a parameter and returns a wrapper of that function.
The wrapper just prints to the console the name of the called function and the given parameters before executing the wrapped function. And the wrapped functions just replaces the commas with spaces.
Now we have a decorator written here. But it’s kind of annoying to define the decorator, the function and then assigning the wrapper to another variable to finally be able to use it.
Python provides some sugar syntax to make it easier to write and read, and if we re-write this decorator using it:
def decorator(function_to_decorate):
def wrapper(value):
print("you are calling {} with '{}' as parameter".format(function_to_decorate.__name__, value))
return function_to_decorate(value)
return wrapper
@decorator
def replace_commas_with_spaces(value):
return value.replace(",", " ")
print(replace_commas_with_spaces.__name__)
print(replace_commas_with_spaces.__module__)
print(replace_commas_with_spaces.__doc__)
print(replace_commas_with_spaces("my,commas,will,be,replaces,with,spaces"))
We just annotate the function we want to wrap with the decorator function and that’s it. That function will be decorated, and the output will look the same.
wrapper __main__ None you are calling replace_commas_with_spaces with 'my,commas,will,be,replaces,with,spaces' as parameter my commas will be replaces with spaces
Now, debugging this can be a real pain, as the replace_commas_with_spaces function is overridden with the wrapper, so its __name__, __doc__ and __module__ will also be overridden (as seen in the output).
To avoid this behavior we will use functools.wraps, that prevents a wrapper from overriding its inner function properties.
from functools import wraps
def decorator(function_to_decorate):
@wraps(function_to_decorate)
def wrapper(value):
print("you are calling {} with '{}' as parameter".format(function_to_decorate.__name__, value))
return function_to_decorate(value)
return wrapper
@decorator
def replace_commas_with_spaces(value):
return value.replace(",", " ")
print(replace_commas_with_spaces.__name__)
print(replace_commas_with_spaces.__module__)
print(replace_commas_with_spaces.__doc__)
print(replace_commas_with_spaces("my,commas,will,be,replaces,with,spaces"))
And now the output will be:
replace_commas_with_spaces __main__ None you are calling replace_commas_with_spaces with 'my,commas,will,be,replaces,with,spaces' as parameter my commas will be replaces with spaces
So, now we know how decorators work in python. Let’s solve our mentioned problems.
3. The Practice
So, we need to implement cache, spent time logging and validations.
Let’s combine them all by solving a bigger problem: palindromes.
Let’s make an algorithm that, given a word, will check if it’s a palindrome. If it isn’t, it will convert it to palindrome.
palindrome.py
def is_palindrome(string_value):
char_array = list(string_value)
size = len(char_array)
half_size = int(size / 2)
for i in range(0, half_size):
if char_array[i] != char_array[size - i - 1]:
return False
return True
def convert_to_palindrome(v):
def action(string_value, chars):
chars_to_append = list(string_value)[0:chars]
chars_to_append.reverse()
new_value = string_value + "".join(chars_to_append)
if not is_palindrome(new_value):
new_value = action(string_value, chars + 1)
return new_value
return action(v, 0)
user_input = input("string to convert to palindrome (exit to terminate program): ")
while user_input != "exit":
print(str(convert_to_palindrome(user_input)))
print("------------------------------------------------------")
user_input = input("string to check (exit to terminate program): ")
Here, we have a function called is_palindrome, which given an input, returns True if its palindrome, or False otherwise.
Then, there is a function called convert_to_palindrome which, given an input, will add just as many characters (reversed, from the beginning) as necessary to make it palindrome.
Also, there is a while that reads the user input until he inputs “exit”.
The output looks like:
string to convert to palindrome (exit to terminate program): anita lava la tina anita lava la tinanit al aval atina ------------------------------------------------------ string to check (exit to terminate program): anitalavalatina anitalavalatina ------------------------------------------------------ string to check (exit to terminate program): menem menem ------------------------------------------------------ string to check (exit to terminate program): mene menem ------------------------------------------------------ string to check (exit to terminate program): casa casac ------------------------------------------------------ string to check (exit to terminate program): casaca casacasac ------------------------------------------------------ string to check (exit to terminate program): exit
As you can see, it works just fine. But we have a couple problems:
- I don’t know how long does it take it to process the input, or if its related to the length of it. (spent time logging)
- I don’t want it to process twice the same input, it’s not necessary. (cache)
- It’s designed to work with words or numbers, so I don’t want spaces around. (validation)
Let’s get dirty here, and start with a spent time logging decorator.
palindrome.py
import datetime
from functools import wraps
def spent_time_logging_decorator(function):
@wraps(function)
def wrapper(*args):
start = datetime.datetime.now()
result = function(*args)
end = datetime.datetime.now()
spent_time = end - start
print("spent {} microseconds in {} with arguments {}. Result was: {}".format(spent_time.microseconds,
function.__name__, str(args),
result))
return result
return wrapper
def is_palindrome(string_value):
char_array = list(string_value)
size = len(char_array)
half_size = int(size / 2)
for i in range(0, half_size):
if char_array[i] != char_array[size - i - 1]:
return False
return True
@spent_time_logging_decorator
def convert_to_palindrome(v):
def action(string_value, chars):
chars_to_append = list(string_value)[0:chars]
chars_to_append.reverse()
new_value = string_value + "".join(chars_to_append)
if not is_palindrome(new_value):
new_value = action(string_value, chars + 1)
return new_value
return action(v, 0)
user_input = input("string to convert to palindrome (exit to terminate program): ")
while user_input != "exit":
print(str(convert_to_palindrome(user_input)))
print("------------------------------------------------------")
user_input = input("string to check (exit to terminate program): ")
It’s pretty simple, we wrote a decorator which returns a wrapper that gets the time before and after executor, and then calculates the spent time and logs it, with the called function, parameters and result. The output looks like:
string to check (exit to terminate program): anitalavalatina
spent 99 microseconds in convert_to_palindrome with arguments ('anitalavalatina',). Result was: anitalavalatina
anitalavalatina
------------------------------------------------------
string to check (exit to terminate program): exit
As you see, there is plenty information in that log line, and our implementation was not changed at all.
Now, let’s add cache:
palindrome.py
import datetime
from functools import wraps
def cache_decorator(function):
cache = {}
@wraps(function)
def wrapper(*args):
hashed_arguments = hash(str(args))
if hashed_arguments not in cache:
print("result for args {} was not found in cache...".format(str(args)))
cache[hashed_arguments] = function(*args)
return cache[hashed_arguments]
return wrapper
def spent_time_logging_decorator(function):
@wraps(function)
def wrapper(*args):
start = datetime.datetime.now()
result = function(*args)
end = datetime.datetime.now()
spent_time = end - start
print("spent {} microseconds in {} with arguments {}. Result was: {}".format(spent_time.microseconds,
function.__name__, str(args),
result))
return result
return wrapper
def is_palindrome(string_value):
char_array = list(string_value)
size = len(char_array)
half_size = int(size / 2)
for i in range(0, half_size):
if char_array[i] != char_array[size - i - 1]:
return False
return True
@spent_time_logging_decorator
@cache_decorator
def convert_to_palindrome(v):
def action(string_value, chars):
chars_to_append = list(string_value)[0:chars]
chars_to_append.reverse()
new_value = string_value + "".join(chars_to_append)
if not is_palindrome(new_value):
new_value = action(string_value, chars + 1)
return new_value
return action(v, 0)
user_input = input("string to convert to palindrome (exit to terminate program): ")
while user_input != "exit":
print(str(convert_to_palindrome(user_input)))
print("------------------------------------------------------")
user_input = input("string to check (exit to terminate program): ")
This is a very simple implementation of cache. No TTL, no thread safety. It’s just a dictionary which keys are the hash of the arguments. If no value with the given key was found, it creates it, then retrieves it. Output:
string to convert to palindrome (exit to terminate program): anitalavalatina
result for args ('anitalavalatina',) was not found in cache...
spent 313 microseconds in convert_to_palindrome with arguments ('anitalavalatina',). Result was: anitalavalatina
anitalavalatina
------------------------------------------------------
string to check (exit to terminate program): anitalavalatina
spent 99 microseconds in convert_to_palindrome with arguments ('anitalavalatina',). Result was: anitalavalatina
anitalavalatina
------------------------------------------------------
string to check (exit to terminate program): exit
There it is. The first execution with “anitalavalatina” outputs a line informing us that the result for that input was not found. But when we input it again, that line is gone. Awesome! But we still receive spaces, let’s validate that:
palindrome.py
import datetime
from functools import wraps
def validation_decorator(validator, if_invalid=None):
def decorator(function):
@wraps(function)
def wrapper(*args):
if validator(*args):
return function(*args)
else:
return if_invalid
return wrapper
return decorator
def cache_decorator(function):
cache = {}
@wraps(function)
def wrapper(*args):
hashed_arguments = hash(str(args))
if hashed_arguments not in cache:
print("result for args {} was not found in cache...".format(str(args)))
cache[hashed_arguments] = function(*args)
return cache[hashed_arguments]
return wrapper
def spent_time_logging_decorator(function):
@wraps(function)
def wrapper(*args):
start = datetime.datetime.now()
result = function(*args)
end = datetime.datetime.now()
spent_time = end - start
print("spent {} microseconds in {} with arguments {}. Result was: {}".format(spent_time.microseconds,
function.__name__, str(args),
result))
return result
return wrapper
def is_palindrome(string_value):
char_array = list(string_value)
size = len(char_array)
half_size = int(size / 2)
for i in range(0, half_size):
if char_array[i] != char_array[size - i - 1]:
return False
return True
def should_not_contain_spaces(*args):
return False not in map(lambda x: " " not in str(x), args)
@spent_time_logging_decorator
@validation_decorator(should_not_contain_spaces, "input shouldn't contain spaces.")
@cache_decorator
def convert_to_palindrome(v):
def action(string_value, chars):
chars_to_append = list(string_value)[0:chars]
chars_to_append.reverse()
new_value = string_value + "".join(chars_to_append)
if not is_palindrome(new_value):
new_value = action(string_value, chars + 1)
return new_value
return action(v, 0)
user_input = input("string to convert to palindrome (exit to terminate program): ")
while user_input != "exit":
print(str(convert_to_palindrome(user_input)))
print("------------------------------------------------------")
user_input = input("string to check (exit to terminate program): ")
Now, this one is a little tricky. To pass arguments to the decorator we need to wrap it. Yeah, we need a wrapper of the wrapper. Thanks to that, we can pass the validation function and a message if input is invalid. The output looks like:
string to convert to palindrome (exit to terminate program): anita lava la tina
spent 87 microseconds in convert_to_palindrome with arguments ('anita lava la tina',). Result was: input shouldn't contain spaces.
input shouldn't contain spaces.
------------------------------------------------------
string to check (exit to terminate program): anitalavalatina
result for args ('anitalavalatina',) was not found in cache...
spent 265 microseconds in convert_to_palindrome with arguments ('anitalavalatina',). Result was: anitalavalatina
anitalavalatina
------------------------------------------------------
string to check (exit to terminate program): exit
3. Download the Code Project
This was an example of how to write decorators in Python.
You can download the full source code of this example here: python-decorator
Hi,
When I run your code when you added : spent_time_logging_decorator
I got :
C:\Python27\python.exe C:/Users/alotfi/vagrantMQ/projects/decorators/palindrome.py
string to convert to palindrome (exit to terminate program): oho
Traceback (most recent call last):
File “C:/Users/alotfi/vagrantMQ/projects/decorators/palindrome.py”, line 43, in
user_input = input(“string to convert to palindrome (exit to terminate program): “)
File “”, line 1, in
NameError: name ‘oho’ is not defined
Process finished with exit code 1
Thanks.
Hi! Thanks for your comment. I’ve managed to reproduce your mistake. My bad, the problem is that I work with Python 3, and you are probably running it with Python 2, I should have specified it at the beginning of the example, but all this code was tested with Python 3.4.
Thanks Sebastian, I changed input to raw_input, it’s working now.