One of the hallmarks of dynamic languages is that they try to put as much as possible off to the last minute. If we do the following
>>> 5 + 4
9
>>> "this" + "that"
'thisthat'The meaning of "+" is not decided until the arguments are evaluated. This is true even if it is used in a function that has been compiled. Only at runtime is the meaning of "+" decided.
>>> def addem(a,b): return a+b
>>> addem(5,6)
11
>>> addem("Hello","World")
'HelloWorld'Actually, even the compilation of functions is, in fact, a "runtime" activity that mostly happens as we import a module. So it is not too surprising that we can nest function definitions Like the following. Source for sample1.py
1 # sample1.py
2 def outer () :
3 print("from outer 1")
4 def inner () :
5 print("from inner 1")
6 print("from outer 2 inner at %s" % inner)
7 return inner # return the function
8
9 print("Starting")
10 myfunc = outer()
11 myfunc()
12
$ python sample1.py
Starting
from outer 1
from outer 2 inner at <function inner at 0x7f5a9e0aedd0>
from inner 1Look closely at the timing and order of the print(statements. The first print we see is from line 9. Function outer has been compiled. At line 10 outer is run and function inner is compiled. The reference to inner is returned from outer and in line 10 the variable myfunc is set to it. In line 11 myfunc is invoked and we see the print from line 5.)
Notice too, that when we print a function (line 6), we get us the name originally assigned when it was defined along with its memory address in hexidecimal.
Let's consider a slightly more complex example. Source for sample2.py
10 # sample2.py
11 def outer () :
12 print("Outer 1")
13 x = 42
14 def inner () :
15 print("Inner 1", x)
16 return inner
And now let's play with it
>>> import sample2
>>> sample2.outer()
Outer 1
<function inner at 0x9d1e224>
>>> sample2.outer()() # create inner function and then call it
Outer 1
Inner 1 42Notice that inner could access the local variable x in outer (line 13, 15). This scoping rule is usually necessary for inner functions to be useful as closures.
This time, outer returned the function it created (line 16) and we can then use it later. Most of the time we'll assign the function to a local variable so we keep it around.
>>> f1 = sample2.outer()
Outer 1
>>> print(f1)
<function inner at 0x9d1e87c>
>>> f1()
Inner 1 42Now, even if the working memory for the call to outer is discarded, that for the function inner is not since it can still be reached through "f1". And because of that, the variable x inside inner is also retained.
And, (this is an "Ah-ha" moment) x can ONLY be reached through the function referenced by "f1". It is enclosed and made completely private. This is called a closure.
But since "x" is a non-mutable variable, it's pretty boring. Let's try a variation. Source for sample3.py
17 # sample3.py
18 def outer () :
19 print("Outer 1")
20 x = 42
21 def inner () :
22 print("Inner 1", x)
23 x += 1
24 return inner
>>> import sample3
>>> f2 = sample3.outer()
Outer 1
>>> f2()
Inner 1
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "sample3.py", line 6, in inner
print("Inner 1", x)
UnboundLocalError: local variable 'x' referenced before assignment
>>>Oops! Since inner is changing the value of "x", the Python compiler assumes "x" is a local variable to inner. And it doesn't work to make "x" global because globals must be defined at the module level. So much for enclosing it.
But there is a cute way to do what we want. If "x" instead of referencing a number (or string), references a mutable like a list, a dictionary or an object, the mutable can be mutated (changed). Let's have it be an object that could provide us with a whole private namespace. Source for sample4.py
25 # sample4.py
26 def outer () :
27 print("Outer 1")
28 class Dummy : pass
29 x = Dummy()
30 x.value = 42
31 def inner () :
32 print("Inner 1", x.value)
33 x.value += 1
34 return inner
>>> import sample4
>>> f3 = sample4.outer()
Outer 1
>>> f3
<function inner at 0x9a12d14>
>>> f3()
Inner 1 42
>>> f3()
Inner 1 43
>>> f3()
Inner 1 44The class Dummy was created to simply provide an empty instance that we can add attributes to and manipulate them at will. These attributes could include functions as well as variables. The instance and everything it contains is completely tucked inside the closure.
Here's a variation on this code. You are certainly familiar with the ticket dispensers one finds in waiting rooms. You pull a number and then you can sit in line instead of standing. Let's make a function that produces dispensers that simply can't be reset. No cheating allowed. Source for makeDispenser.py
01 # makeDispenser.py
02 #
03 def makeDispenser (startNum) :
04 class NameSpace : pass
05 myStuff = NameSpace()
06 myStuff.nextNum = startNum
07 def dispenser () :
08 thisNum = myStuff.nextNum
09 myStuff.nextNum += 1
10 return thisNum
11 return dispenser
>>> import makeDispenser
>>> aDisp = makeDispenser.makeDispenser (101)
>>> bDisp = makeDispenser.makeDispenser (201)
>>> print("Get ticket from A", aDisp())
Get ticket from A 101
>>> print("Get ticket from B", bDisp())
Get ticket from B 201
>>> print("Get ticket from A", aDisp())
Get ticket from A 102
>>> print("Get ticket from B", bDisp())
Get ticket from B 202Python decorators are a syntactic mechanism to make some change in the original definition of a function. Suppose we have a function that computes the tip for a restaurant bill.
>>> def tip(amount) : return amount*.15 >>> tip(10) 1.5Now, let's assume we want to double tips. We might write a function called "bigTip" like the following.
>>> def double(x) : return x+x >>> def bigTip(amt) : return double(tip(amt)) >>> bigTip(10) 3.0
Now, if we want to force every call to the tip to generate a big tip we might think we could simply do the following.
>>> tip = bigTip >>> print(tip(10)) RuntimeError: maximum recursion depth exceeded
Well, that won't work. If you look carefully you will see the recursion loop that keeps tip calling itself. But we can reach back a bit and make double operate on the function definition level. We'll have it create a new function that calls the original function passing through the parameter and then returns the result*2.
>>> def double (func) :
... def wrapper(n) :
... return func(n) * 2
... return wrapper
...
>>> def tip(amount) : return amount*.15
>>> bigTip = double(tip)
>>> bigTip(10)
3.0
>>> tip = bigTip
>>> tip(10)
3.0
>>> print(tip)
<function wrapper at 0x9301cdc>
( or if python3 <function double.<locals>.wrapper at 0x7f857123e5e0>And this works. Notice that the variable "tip" now points to the generated function wrapper created inside double. We can no longer even access the orginal tip function. The name tip has been reassigned to the closure.
But so far, double is limited to operating on functions that take a single parameter. If we want to generalize the function parameters we need to use the special Python syntax to roll up the arguments of a function into a tuple. The rolled up tuple is generally given the name "args" and special syntax is to precede a variable name with a "*". Here's how it works.
>>> def xx(*args) : print(args)
>>> xx(1,2,3)
(1, 2, 3)
>>> xx("this", "is", [5,4,3,2,1])
('this', 'is', [5, 4, 3, 2, 1])The leading "*" forces the roll-up and is valid only in function parameters. There is also a "**" which rolls up keyword arguments into a dictionary. It is generally given the name "kwargs".
>>> def xx(*args, **kwargs) : print(args, kwargs)
...
>>> xx(1,2,3)
(1, 2, 3) {}
>>> xx(1,2, debug=True)
(1, 2) {'debug': True}Used together, we can pass arguments intact through an intermediate function. Let's rewrite double above which could be used for any function.
>>> def double (func) :
... def wrapper(*args, **kwargs) :
... return func(*args, **kwargs) * 2
... return wrapper
...
>>> def tip (amt) : return amt * .15
...
>>> tip(15)
2.25
>>> tip = double(tip)
>>> tip(15)
4.5So here our single parameter is passed through as it should be.
Now, all of this is made much more intuitive using the decorator syntax.
>>> @double
... def tax(amt) : return amt*.09
>>> tax(100)
18.0The order of events is this. First the function tax is defined, and then it is "decorated" by double. The variable "tax" is then set to whatever the decorator function (double) returns. In this case hiding the original in a closure and manipulating its computation. As we will see other things are possible too.
Notice what happens when we do the following. The secret peeks through.
>>> print(tax)
<function wrapper at 0x8528534>
( or if python3 <function double.<locals>.wrapper at 0x7f857123e5e0>
There are two more items of interest here. One is that decorators may be stacked.
>>> @double
... @double
... def tip (amt) : return amt * .15
...
>>> tip(40)
24.0A 60% tipper!! .15*2*2.
So, actually, a decorator call like "@double" expects the next line will either start a function definition or supply another (inner) decorator.
Actually, the "@" starting a line expects an expression that evaluates to a function. Now so far, that has just been a variable referencing a function that's already defined. But here is where we can take a little trip down the rabbit hole.
Consider the following decorator. Notice that it has an argument. Source for nTimes.py
01 # nTimes.py
02 #
03 def nTimes(n) :
04 "Create a decorator function that turns another functions value to n times"
05 print("In function ntimes (arg=%s) creating decorator" % n)
06 def decorator(target) :
07 print("In decorator creating the wrapper")
08 def wrapper(*args, **kwargs) :
09 print("Calling decorated function")
10 return target(*args, **kwargs) * n
11 return wrapper
12 return decoratorLook carefully and see that calling nTimes creates and returns the function decorator. Calling decorator creates and returns wrapper, the decorated version of the function definition that follows.
>>> from nTimes import nTimes
>>> nTimes(3)
In function ntimes (arg=3) creating decorator
<function decorator at 0x7f2e069066e0>
>>> @nTimes(3)
... def tip(amt) : return amt * .15
...
In function ntimes (arg=3) creating decorator
In decorator creating the wrapper
>>> tip(10)
Calling decorated function
4.5Notice carefully the order of the print(statements.)
So far, the decorators we've written have modified a function. But that is not their only use nor even the major one. Often the decorator is there just for the side effects.
Here's an example.
Suppose we have a little language processer, like some of the Python for Fun projects and we want to tie or register a token, say "+", to a function that processes the token. A common thing to do might be the following.
Define the function. Let's call it "add":
>>> def add(a,b) : return a+band then have a dictionary that will map the token "+" to our function:
>>> dispatch = {'+': add, ... }The dictionary might hold more information, like the number of arguments to pass to add.
This is where decorators can be quite elegant. This is how the equivalent might look.
@register('+')
def add(a,b) : return a+bNow, the function register, like nTimes above, must create a decorator to apply to the function add. Here's how that might look.
1 # register.py
2 dispatch = {}
3 def register(token) :
4 def decorator(func) :
5 dispatch[token] = func
6 return func
7 return decorator
8
9 def test() :
10 print(dispatch)
11
12 @register('+')
13 def add(a,b) :
14 return a+b
15
16 print(dispatch)
17
18 if __name__ == "__main__" : test()
19
20
Running it shows a function added to a dispatch table
$ python3 register.py
{}
{'+': <function add at 0x7f4ff5b2e350>}
The bottle.py minimalist web application framework uses decorators in a interesting way that is quite close to the previous example above. For example, the following
@route("/greet")
def hello () :
return "<html>Hello, whoever you are</html>"
@route("/gruess/<name>")
@route("/greet/<name>")
def greet(name) :
return "<html>Hello, nice to meet you, %s</html>" % namewill essentially "register" the function hello to a url ending with "/greet". A URL ending in either "gruess/lukas" or "greet/lukas" will call the function greet with "lukas". Notice that all 3 "@route" invokations take an argument which means the function route actually creates a different decorator function each time.
When the application is running, the string "<name>" works like a variable and is replaced by the actual word in the url. When the url is routed an args list is built from the pattern matching and passed to the function greet. Here is what you get when you run the above as part of a mini-server (the program will have all of another 4 lines or so of Python).
http://localhost:8080/greet # Call from your browser
Hello, whoever you are # returned to your window
-
http://localhost:8080/gruess/mickey
Hello, nice to meet you, mickeyIt is, of course, important that the variables in the url align with the parameters in the decorated function.
So, decorators are cute. Are they necessary? There was some resistance to adding this syntax to the Python language.
Without decorators, the bottle code might look more like this.
def hello () :
return "<html>Hello, whoever you are</html>"
def greet(name) :
return "<html>Hello, nice to meet you, %s</html>" % name
route("/greet", hello)
route("/gruess/<name>", greet)
route("/greet/<name>", greet)Certainly not wildly different. And the code within bottle.py would be much less obscure.
Still, the idea is that you will not be modifying bottle.py, just using it. And the folks at bottle have already done the debugging of that piece.
But, if I were writing an non-commercial application I would probably find extending it with custom decorators overkill. Because anyone maintaining the code, including myself, would probably have to deal with the decorators as well as the other code. This would increase the complexity of the code overall. And the major objective of programming should be simplicity.
You can download the zip file for the project here
If you have comments or suggestions You can email me at mail me
Copyright © 2012-2021 Chris Meyers and Fred Obermann
…