Core Idioms
Consider the following task: given a list or tuple of words from a social media post, extract a list of hashtags (items that begin with #) and return them as “normal” words without the “hash.” For example, the input [’#Mary’, ’had’, ’a’, ’little’, ’#lamb’] should produce the output [’Mary’, ’lamb’].
You can build the result incrementally in a loop. Note: as tempting as it is to compare the first character of an item with ’#’ (in other words, item[0] == ’#’), that’s unsafe: some items on the list might be empty strings, and indexing would raise an error. The function startswith handles empty strings correctly.
| | data = ['#Mary', 'had', 'a', 'little', '#lamb'] |
| | |
| | result = [] # Initialize an empty list |
| | for item in data: |
| | if item.startswith('#'): # Check the condition |
| | result.append(item[1:]) # Strip the leading '#' and append |
| | |
| | print(result) # ['Mary', 'lamb'] |
This style is known as imperative programming: it works at a lower level of abstraction () and emphasizes how a result is produced, often focusing on implementation details as much as on the outcome itself. By contrast, comprehensions are a higher-level, declarative programming construct: they describe the transformation from inputs to outputs and typically abstract away the underlying implementation.
Comprehensions
The hashtag extraction task is a special case of a common Filter--Map--Reduce pattern in data processing (you can read more about the pattern in ). The pattern typically involves three steps: filtering (selecting items that meet a condition), mapping (transforming each selected item), and reducing (aggregating results).
Design Patterns | |
|---|---|
 | A design pattern is a general, reusable solution to a recurring problem in software design. It is not a complete solution but a blueprint that must be adapted and implemented in code. Read more about design patterns in . |
Python supports the Filter--Map--Reduce pattern with three types of comprehensions: list, set, and dictionary. The two general list comprehension forms are:
| | [expr for var in data] |
| | [expr for var in data if cond] |
In the first form, expr is evaluated for every var in data, and the results are collected into a list. If data is ordered (a tuple, a list, or a string), the result preserves that order. As an example, consider computing the lengths of each item in data:
| | data = ['#Mary', 'had', 'a', 'little', '#lamb'] |
| | |
| | result = [len(item) for item in data] |
| | print(result) # [5, 3, 1, 6, 5] |
In the second form, expr is evaluated only for the items that satisfy the condition cond. All other items are filtered out. In the following example, the parts of the comprehension that correspond to the filter, map, and reduce actions are placed on separate lines for clarity.
| | result = [ |
| | item[1:] # map |
| | for item in data |
| | if item.startswith('#') # filter |
| | ] # reduce (implicit list construction) |
| | print(result) # ['Mary', 'lamb'] |
How to “Comprehend” Comprehensions | |
|---|---|
 | Read a comprehension aloud by starting at the keyword for: “for every item in data, if item.startswith(’#’)...” (“filter”). Go back to the beginning and prepend “calculate” before the expression: “...calculate item[1:]” (“map”). Add “...and assemble the results” (“reduce”). This mirrors the execution order and often makes complex comprehensions easier to understand. |
If you omit the if clause in a comprehension, there is no filtering; every item is included. The expression at the front can also be trivial (just the loop variable) or even a constant. You can use the latter form to count values that satisfy the condition:
| | result1 = [item for item in data if item.startswith('#')] |
| | print(result1) # ['#Mary', '#lamb'] |
| | result2 = [1 for item in data if item.startswith('#')] |
| | print(result2) # [1, 1] |
| | print(len(result2)) # 2 |
Last but not least, if the expression is just a variable and there is no condition, the comprehension makes a copy of the original sequence, serving as an expensive equivalent of data[:].
| | result = [item for item in data] |
| | print(result == data[:]) # True |
A set comprehension uses curly braces {} instead of square brackets [] and produces a set. Think of it as essentially an application of set to a list comprehension:
| | result = {item[1:] for item in data if item.startswith('#')} |
| | print(result) # {'Mary', 'lamb'} |
Storing hashtags in a set dramatically improves the performance of membership tests because Python sets use hash tables internally (see ).
| | present = 'Mary' in result |
| | print(present) # True |
Quite expectedly, a dictionary comprehension produces a dictionary. As such, for every dictionary item, it needs a key and a value, separated by a colon:
| | {key_expr: value_expr for var in data} |
| | {key_expr: value_expr for var in data if cond} |
Now, you can precompute a mapping from hashtags to their lengths. To facilitate lookups, convert the keys to lowercase (or any other standard form):
| | result = {item[1:].lower() : len(item[1:]) |
| | for item in data if item.startswith('#')} |
| | print(result) # {'mary': 4, 'lamb': 4} |
If a hashtag happens in data more than once, the duplicates will be merged during the dictionary construction.
One limitation of comprehensions is that each produces only one output collection. If you want to split items into two groups (those that meet a condition and those that don’t), you must have two comprehensions with complementary conditions, iterating over the same data twice.
| | hashtags = {item[1:].lower() : len(item[1:]) |
| | for item in data if item.startswith('#')} |
| | justwords = {item.lower() : len(item) |
| | for item in data if not item.startswith('#')} |
This approach is inefficient, particularly for large datasets. A single-pass for loop is a better alternative:
| | hashtags = {} |
| | justwords = {} |
| | for item in data: |
| | if item.startswith('#'): |
| | hashtags[item[1:].lower()] = len(item[1:]) |
| | else: |
| | justwords[item.lower()] = len(item) |
Generators
Comprehensions create the entire output collection in memory, even if you need it piecewise, item by item.
Suppose you want the average length of the words in data. A straightforward approach involves building a list of lengths and then calculating the average:
| | lengths = [len(item) for item in data] |
| | average = sum(lengths) / len(data) |
| | print(average) # 4.0 |
Note that the list lengths exists only to be immediately reduced to a single number by summation. If data is large, so is lengths, even though the built-in function sum needs items one at a time, not all at once. Having a comprehension-like expression that produces items as needed would make this code more memory-efficient.
Enter generators. A generator expression yields (“generates”) values on demand. It looks like a list comprehension, but it is enclosed in parentheses ():
| | lengths = (len(item) for item in data) |
| | print(type(lengths)) # <class 'generator'> |
Generator expressions yield items lazily (on demand), which saves memory for large datasets
The value returned by a generator expression is an object of class generator. A generator uses lazy evaluation: it doesn’t return the results themselves but a “promise” to produce them later. That “promise” can be fulfilled explicitly by calling the built-in function next (beware that each call to next consumes the next generated value and may exhaust the generator before it is otherwise used):
| | >>> data = ['#Mary', 'had', 'a', 'little', '#lamb'] |
| | >>> lengths = (len(item) for item in data) |
| | >>> next(lengths) |
| | 5 |
| | >>> next(lengths) |
| | 3 |
| | >>> next(lengths) |
| | 1 |
| | >>> next(lengths) |
| | 6 |
| | >>> next(lengths) |
| | 5 |
| | >>> next(lengths) |
| | Traceback (most recent call last): |
| | File "<python-input-7>", line 1, in <module> |
| | next(lengths) |
| | ~~~~^^^^^^^^^ |
| | StopIteration |
At the end of the sequence, the generator raises a StopIteration exception.
Alternatively, pass the generator to an aggregating function, such as sum:
| | average = sum(lengths) / len(data) |
| | print(average) # 4.0 |
Generator expressions may be slightly slower than plain list comprehensions, and they can’t be reused: they must be re-created to iterate again. However, for large datasets, they are often essential, making the difference between feasible and infeasible list processing.
Enumerating and Zipping
Python offers iterator-centric helpers beyond basic loops.
In addition to generator expressions and custom-made generators (see below), Python provides several other built-in objects that can be used as iterators. An iterator represents a stream of data and can be used wherever you would iterate over a sequence—for example, in a for loop or a list comprehension, just like a generator. In fact, all generators are iterators, but not all iterators are generators. For example, the built-in functions enumerate and zip return iterator (but not generator) objects of their respective classes.
The built-in function enumerate returns an object that “promises” to yield (index, value) tuples from the sequence data. The first tuple is (0, data[0]), then (1, data[1]), and so on, as illustrated in the following:
| | >>> enumerated = enumerate(data) |
| | >>> next(enumerated) |
| | (0, '#Mary') |
| | >>> next(enumerated) |
| | (1, 'had') |
| | >>> next(enumerated) |
| | (2, 'a') |
| | >>> next(enumerated) |
| | (3, 'little') |
| | >>> next(enumerated) |
| | (4, '#lamb') |
| | >>> next(enumerated) |
| | Traceback (most recent call last): |
| | File "<python-input-7>", line 1, in <module> |
| | next(enumerated) |
| | ~~~~^^^^^^^ |
| | StopIteration |
You can apply the list constructor, list, to the iterator to extract all tuples at once and build a list, but this transformation defies the purpose of on-demand generation.
| | result = list(enumerate(data)) |
| | print(result) # [(0, '#Mary'), (1, 'had'), (2, 'a'), (3, 'little'), |
| | # (4, '#lamb')] |
Instead, use enumerate objects directly in for loops, comprehensions, or generator expressions—for example, to print the words with human-adjusted (1-base) indices:
| | for i, word in enumerate(data): |
| | print(f'{i + 1}: {word}') |
| | # 1: #Mary |
| | # 2: had |
| | # 3: a |
| | # 4: little |
| | # 5: #lamb |
Built-ins enumerate and zip pair indexes with values or walk multiple sequences in lockstep
The built-in function zip is a powerful generalization of the enumerate function. While enumerate pairs a single sequence with consecutive integer numbers, zip returns an object that combines N sequences into a sequence of N-tuples: the first tuple is (data1[0], data2[0], ...), the second is (data1[1], data2[1], ...), and so on, stopping at the shortest input sequence. In particular, zipping a sequence with an appropriate range of numbers produced by the range function (see ) is equivalent to using enumerate:
| | indexes = range(len(data)) |
| | zipped = zip(indexes, data) |
| | print(list(zipped)) # [(0, '#Mary'), (1, 'had'), (2, 'a'), (3, 'little'), |
| | # (4, '#lamb')] |
Possibly the most efficient application of zip is managing parallel lists (see ). Parallel lists store different attributes of the same items. For example, the item at index 0 of metals corresponds to the value at index 0 of weights. Such lists are easy to desynchronize if you insert or delete an item in one list but not in the others. Using zip allows you to iterate over the lists in lockstep, reducing the risk of mismatched data (compare this example with the ):
| | >>> metals = ['Li', 'Na', 'K'] |
| | >>> weights = [6.941, 22.98976928, 39.0983] |
| | >>> for name, weight in zip(metals, weights): |
| | ... print(name, weight) |
| | ... |
| | Li 6.941 |
| | Na 22.98976928 |
| | K 39.0983 |
It is still your responsibility to ensure that the original parallel constituents are correctly aligned.
Packing and Unpacking
Multiple assignment, introduced in , is a special case of packing/unpacking. Python packs multiple RHS expressions into a tuple and then unpacks the tuple into the LHS targets, provided that the number of items on both sides of the assignment match.
But what if it doesn’t? For general unpacking, you can use a starred target, a variable prefixed with an asterisk *, which captures all items that don’t fit into explicitly specified targets. A starred target can appear anywhere on the LHS (at the beginning, middle, or end). Here, the first character of ’Mary’ is assigned to head, and the list of the remaining characters goes into tail:
| | >>> head, *tail = '#Mary' |
| | >>> head |
| | '#' |
| | >>> tail |
| | ['M', 'a', 'r', 'y'] |
Conversely, in the following assignment, the last character is assigned to the explicit tail, the middle characters are assigned to rest, and the leading hashtag marker (#) is assigned to the conventional throwaway variable (_).
| | >>> _, *rest, tail = '#Mary' |
| | >>> rest |
| | ['M', 'a', 'r'] |
| | >>> tail |
| | 'y' |
Throw It Away! | |||||||||
|---|---|---|---|---|---|---|---|---|---|
| | Use the throwaway variable _ (a single underscore) when Python requires a variable, but you don’t intend to use its value. The name has no special meaning; it’s simply a widely used convention. Incidentally, Python’s interactive interpreter also assigns the most recent result to _:
| ||||||||
Unpacking can happen on the RHS as well. If a function requires several parameters, you can pass them as a starred sequence of the matching size. In this example, the function distance calculates the Euclidean distance between two points in the plane. The function requires two pairs of coordinates as four arguments. However, if you represent points as (x, y) tuples, you can pass them directly and let Python unpack each tuple into two arguments:
| | >>> from math import sqrt, pow |
| | >>> def distance(x1: float, y1: float, x2: float, y2: float) -> float: |
| | ... """Return the euclidean distance between the points `(x1,y1)` |
| | ... and `(x2,y2)`. |
| | ... """ |
| | ... return sqrt(pow(x1 - x2, 2) + pow(y1 - y2, 2)) |
| | ... |
| | >>> origin = (0, 0) # x1, y1 |
| | >>> other = (3, 4) # x2, y2 |
| | >>> distance(origin[0], origin[1], other[0], other[1]) # Good |
| | 5.0 |
| | >>> distance(origin[0], origin[1], *other) # Better |
| | 5.0 |
| | >>> distance(*origin, *other) # Best |
| | 5.0 |
Adding an asterisk to a parameter name in a function definition makes that parameter a “catch-all” (starred) parameter. It packs all remaining positional arguments (but not keyword arguments) into a tuple. Using a starred parameter allows the function to accept a variable number of arguments. For example, the following function returns the count of its positional arguments:
| | >>> def count_args(*args: Any) -> int: |
| | ... """ Return the number of positional arguments passed to the function. |
| | ... |
| | ... >>> count_args(1, 2, 'hello', True, None) |
| | ... 5 |
| | ... """ |
| | ... return len(args) |
| | ... |
| | >>> count_args(1, 2, 'hello', True, None) |
| | 5 |
Some arguments may be fixed (and required). They are bound to their respective parameters first; any leftover positional arguments are collected in the starred parameter:
| | >>> def count_opt(required: Any, *optional: list[Any]) -> int: |
| | ... return len(optional) |
| | ... |
| | >>> count_opt(1) |
| | 0 |
| | >>> count_opt(1, 2, 'hello', True, None) |
| | 4 |
A slightly more useful example computes the average of all its parameters:
| | >>> def average(*numbers: list[float]) -> float: |
| | ... """Return the average of all positional arguments or 0 if |
| | ... called without parameters. |
| | ... """ |
| | ... if len(numbers) > 0: |
| | ... return sum(numbers) / len(numbers) |
| | ... return 0 |
| | ... |
| | >>> average(1,2,3,4) |
| | 2.5 |
| | >>> average() |
| | 0 |
Returning 0 (or None) as the “average” of an empty input is generally not recommended. When called without arguments, the function should raise an exception (for example, ValueError). See Chapter 17, for how to do this.
