Some of you who follow me may have noticed a tendency of mine to “hack” programming languages more than really use them (other than reflection via annotations in languages such as Java; I hate that stuff), and today is no different. Today, we look as using what would be normal higher-order functions as Python decorators to create new functions that encapsulate the idea of both the higher-order function as well as the passed-in function under one name.
Why?
This is the second time I’ve attempted to create a way for multiline lambdas to exist in Python. The first time used the with block, as seen in the articles around creating Kotlin-like builders in Java and Python, and fully explained in its own article.
Yes, there’s the obvious thing of defining the function just before passing it into the higher-order function, but the real problem with that is that it kind of separates the logic from its use. The function takes up at least two lines above its real context. Then its name is used in the higher-order function. This does have the benefit of being able to use a helpful name, but sometimes the code is more helpful than the name.
Then there’s the issue with nesting. Probably the most commonly used higher-order functions out there are map() and filter(), but when they get nested, the code can get difficult to read:
map(transformer, filter(predicate, list))
This is just one level of nesting, and it’s already weird because it’s difficult to tell that everything after transformer is just one parameter for map(). One thing that can help is new lines and indentation, I guess.
map(transformer,
filter(predicate, list))But that opens up a whole can of worms about how to do the indenting. I’m sure some of you looked at that could and thought “I would do that differently”. I went through a few options of my own before settling on that.
So, yeah, I think about this idea a lot, and here’s another hacky idea I came up with.
Decorators
The thing is, we already have a special syntax for passing multi-line functions into a higher-order function, and that’s the decorator syntax. The part of decorators that makes them a less than ideal solution is the fact that they don’t return anything; they’re a statement that assigns the result of the decoration to the name of the decorated function.
Now, this can be used to your advantage, but we’ll start without that because I didn’t think of how that could be useful until I wrote that out, and we’ll go over how my ideas evolved along the way.
The Base Code
First, we’ll look at the code we’ll be working with for the examples. We will be using map() and filter() over a list of strings. First it will filter out the strings that aren’t “valid usernames”, then it will map them to a gmail address (simply adds “@gmail.com” to the end).
So the basic code will be:
def is_legal_username(username):
# not important
return bool_result
def to_gmail_addr(username):
# not important
return gmail_addrAnd using it would look like this:
map(to_gmail_addr, filter(is_legal_username, usernames))
Attempt 1 – Partial Definitions
In my first attempt, I effectively create decorators designed to do partial application of map() and filter(). I originally wrote them like normal decorators, but when I realized what they did, I simplified them (don’t forget to import functools to have access to partial():
def filterer(func):
return partial(filter, func)
def mapper(func):
return partial(map, func)I then applied these to the is_legal_username() and to_gmail_addr() functions.
@filterer
def is_legal_username(username):
...
@mapper
def to_gmail_addr(username):
...Now we can filter and map like this:
to_gmail_addr(is_legal_username(usernames))
This is much easier to read, but the names imply working on singular items (which is what the originals do). So, change the name of the functions to legal_usernames() and as_gmail_addresses():
@filterer
def legal_usernames(username):
# not important
return bool_result
@mapper
def as_gmail_addresses(username):
# not important
return gmail_addrWhich makes the usage look like this:
as_gmail_addresses(legal_usernames(usernames))
One could argue that you could just use the old functions and reassign partials, like this:
legal_usernames = partial(filter, is_legal_username) as_gmail_addresses = partial(map, to_gmail_addr)
But the idea is based around replacing lambdas, so the base functions are only used in this context anyway. Using the decorators (assuming you already have them defined somewhere) allows us to use locality better to show that legal_usernames() is a function used for filtering while also immediately showing the predicate used for the filtering.
Attempt 2 – Off the Rails
Disclaimer: This one is excessive and pretty crazy. You can skip to attempt 3 if you’d like. A part of me still didn’t like that old idea because you define a function that operates on a per-element basis, but it gets transformed into a function that operates on an iterable. What if we liked the look of as_gmail_addresses(legal_usernames(usernames)) over map(to_gmail_addr, filter(is_legal_username, usernames)) but already had the functions defined and also didn’t want to clutter our code with partial calls; I feel that calls to partial() should only exist within code that is already meant to be doing something “magical” (i.e. in decorators, and other metaprogramming features) or else I feel like I failed to design the code properly. Well, instead of the decorator simply redefining the function, it creates a partial filter or map, but registers it somewhere and leaves the original function along to be used as-is. Well, I thought of a surprisingly easy way to get closer to the look we want (I can get way closer or even exactly what we want, but that gets progressively more complicated/”magical”, so we’ll stop at this level). First, we’ll create a class that allows us to extend our normal higher-order functions. I’ll explain everything in the class as we use them.
class Extended:
def __init__(self, func):
self.func = func
self.extensions = {}
def __call__(self, *args, **kwargs):
return self.func(*args, **kwargs)
def __getattr__(self, item):
return self.extensions[item]
def add_extension(self, name, func):
self.extensions[name] = funcThen we can prepare our own versions of map() and filter() by extending them.
filter = Extended(filter) map = Extended(map)
And the last thing to prepare is the decorators:
def mapper(func):
ext = partial(map, func)
map.add_extension(func.__name__, func)
return func
def filterer(func):
ext = partial(filter, func)
filter.add_extension(func.__name__, func)
return funcNow we decorate the single-element functions just like we did at the beginning:
@filterer
def is_legal_username(username):
...
@mapper
def to_gmail_addr(username):
...Which allows us to call them like so:
map.to_gmail_addr(filter.is_legal_username(usernames))
Getting the names of the actual filtering and mapping functions outside of the parens is what really helps to clean it up, I feel.
Additional disclaimer: When going through the ideas on my own before writing this post, this is not what I came up with. Instead, I designed a class that would work similarly to Java’s Stream or Kotlin’s Sequence, but where it could register new methods on it similarly to how we did here, but the biggest advantage is that now we no longer need to nest calls at all:
Sequence(usernames).legal_usernames().as_gmail_addresses()
If you like this idea, I leave it to you to figure out (or you can bug me, and I’ll send you the code).
Attempt 3 – Use the Result
In this last idea, we actually use the decorated functions as lambdas because the name we use for the defined function is actually the name of the result of the call to the higher-order function.
This requires filter() and map() to be defined differently, so I’ll make my own named _map() and _filter().
def _map(iterable):
return partial(map, iterable=iterable)
def _filter(iterable):
return partial(filter, iterable=iterable)Those certainly don’t look like normal decorators, and it’s because they’re not normal decorators. They’re not designed to return a new function; they return the result of calling filter()/map() with the functions and iterables they want. Which makes their usage look like this:
@filter(usernames)
def legal_usernames(username):
...
@map(legal_usernames)
def gmail_addresses(username):
...Now, you have a variable called gmail_addresses that is an iterable created by filtering and mapping using the defined functions. The super awesome parts of this are:
- It doesn’t do nesting, and is therefore written in the order that it happens.
- All three parts are integrated together. You already know that a decorator applies and is attached to the function defined below it, plus the actual result of the call is given its name here and now. Locality, people! Locality!
If you’re trying to actually write one-off “lambdas”, I recommend this tactic over the others. To make the idea even more feasible, consider writing your higher-order functions to be called normally or as a decorator. As an example, the earlier _filter() function could be rewritten like this:
def _filter(iterable, predicate=None):
if predicate is None:
return partial(filter, iterable=iterable)
else:
return filter(predicate, iterable)Or, if you’d like to preserve the original order of parameters when called normally, you could do it like this:
def _filter(predicate=None, *, over):
if predicate is None:
return partial(filter, iterable=over)
else:
return filter(predicate, over)I chose to name the iterable as “over” so that calling it would read like “filter over this with this“.
@filter(over=usernames) def …
You can use whatever name you want. And since you can’t place non-default args in front of default ones, you have to make the others be named-only arguments.
Outro
I have too many thoughts on all of this to share it will all of you, and writing this long of an article has really zapped my mental energy for the day. Thanks for reading, and I hope you’ll come back.
| Published on Web Code Geeks with permission by Jacob Zimmerman, partner at our WCG program. See the original article here: A Weird (and Mostly Impractical) Way to Do Multi-Line “Lambdas” in Python Opinions expressed by Web Code Geeks contributors are their own. |
