Built-In and Standard Library Helpers

Книга: Practical Programming, Fourth Edition

Python’s built-in sequence tools don’t cover every need. In this section, you’ll meet additional iteration utilities: the itertools module and the built-in functions sorted (first seen in ) and reversed (first seen in ). The function reversed and most of the functions in itertools return iterators that you can use in for loops, list comprehensions, and generator expressions, or pass to other functions that consume sequences. The function sorted returns a list, not an iterator.

Iterating with itertools

The itertools module provides more than a dozen functions for iterating over one or more input sequences. We will apply several of them: combinations, chain, and takewhile—to analyze text and perform simple text summarization. You will also encounter a few functions from the operator and collections modules.

You can prototype simple text “summaries” by chaining itertools tools

While text summarization is generally a hard problem, it can be approximated by identifying the most frequently co-occurring pairs of words. The first step toward counting word pairs is tokenization: split the text into sentences (sentence tokenization) and sentences into words, also known as tokens (word tokenization). Both tasks are challenging for natural languages such as English, Russian, Chinese, and Arabic. For more accurate results, use specialized libraries such as nltk (Natural Language Toolkit).

As a simple first step toward sentence tokenization, convert the text to lowercase to make counting case-insensitive, replace common sentence terminators (for example, ? and !) with periods, and split on periods.

	text = '''This book is about Python programming. Python is a popular and
	efficient programming language. A Python programmer is thoughtful and
	pragmatic. Be like a Python programmer.'''
	sentences = text.lower().replace('?', '.').replace('!', '.').split('.')
	print(sentences)
	# ['this book is about python programming',
	# ' python is a popular and\nefficient programming language',
	# ' a python programmer is thoughtful and\npragmatic',
	# ' be like a python programmer']

Split each sentence on whitespace and remove stopwords: frequently occurring determiners, pronouns, prepositions, modal verbs, and other parts of speech that typically don’t convey objects, actions, or properties. Now, you have a list of streams of tokens (filter iterators), one stream per sentence. Do not try to inspect the streams (for example, by converting them to lists): doing so will consume them and cause all further variables to be empty.

	stops = {'and', 'the', 'in', 'this', "don't", 'was', 'that', 'to', 'a',
	'its', 'as', 'had', 'about', 'is', 'be'}
	tokens = [filter(lambda word: word not in stops, s.split())
	for s in sentences]
	print(tokens)
	# [<filter object at 0x7f34eb4862c0>, <filter object at 0x7f34eb485a50>,
	# <filter object at 0x7f34eb4860b0>, <filter object at 0x7f34eb486170>,
	# <filter object at 0x7f34eb4861a0>]

Next, generate pairs from each token stream. The function itertools.combination yields combinations: n-element tuples of items from data without replacement. For pairs, pass 2 as the second argument. Then the itertools.chain.from_iterable function flattens the sequence of pair sequences into a single sequence, all_pairs, ready for counting.

	from itertools import combinations, chain, takewhile

	pairs = map(lambda words: combinations(words, 2), tokens)
	all_pairs = chain.from_iterable(pairs)
	print(all_pairs)
	# <itertools.chain object at 0x7f9fb6b119f0>

all_pairs is yet another stream (iterator). Inspecting it will consume it!

One way to count pairs is to build a dictionary mapping each pair to its count (as explained in ). A better way is to use the Counter class from the collections module. The latter solution based on reuse is better: it already exists, it is correct, and it is more efficient than the dictionary-based one. A Counter is a subclass of dict that stores items as keys and their counts as values. The Counter.most_common method returns all items sorted by decreasing count.

	from collections import Counter

	counts = Counter(all_pairs)
	print(counts)
	# Counter({('python', 'programming'): 2, ('python', 'programmer'): 2,
	# ('book', 'python'): 1, ('book', 'programming'): 1, ('python', 'popular'): 1,
	# ('python', 'efficient'): 1, ('python', 'language'): 1,
	# ('popular', 'efficient'): 1, ('popular', 'programming'): 1,
	# ('popular', 'language'): 1, ('efficient', 'programming'): 1,
	# ('efficient', 'language'): 1, ('programming', 'language'): 1,
	# ('python', 'thoughtful'): 1, ('python', 'pragmatic'): 1,
	# ('programmer', 'thoughtful'): 1, ('programmer', 'pragmatic'): 1,
	# ('thoughtful', 'pragmatic'): 1, ('like', 'python'): 1,
	# ('like', 'programmer'): 1})
	items = counts.most_common()
	print(items)
	# [(('python', 'programming'), 2), (('python', 'programmer'), 2),
	# (('book', 'python'), 1), (('book', 'programming'), 1),
	# (('python', 'popular'), 1), (('python', 'efficient'), 1),
	# (('python', 'language'), 1), (('popular', 'efficient'), 1),
	# (('popular', 'programming'), 1), (('popular', 'language'), 1),
	# (('efficient', 'programming'), 1), (('efficient', 'language'), 1),
	# (('programming', 'language'), 1), (('python', 'thoughtful'), 1),
	# (('python', 'pragmatic'), 1), (('programmer', 'thoughtful'), 1),
	# (('programmer', 'pragmatic'), 1), (('thoughtful', 'pragmatic'), 1),
	# (('like', 'python'), 1), (('like', 'programmer'), 1)]

The itertools.takewhile function yields items from data until the Boolean predicate function becomes false. In the example below, the predicate keeps items whose count (the second element of each (words, count) pair) is greater than 1—in other words, pairs that occurred at least twice in the original text. The result is yet another stream.

	popular = takewhile(lambda item: item[1] > 1, items)
	print(popular)
	# <itertools.takewhile object at 0x7f3f941a4800>

Finally, let’s display the popular tuples more humanely by dropping the counts. The itemgetter function from the module operator is a higher-order function: it returns another nameless function that, in turn, yields the nth element from its input:

	>>> zerogetter = itemgetter(0)
	>>> print(zerogetter)
	operator.itemgetter(0)
	>>> zerogetter(['zero', 'one', 'two'])
	'one'

Applying itemgetter(0) to each “popular” tuple yields just the word pairs: the “summary” of the text:

	from operator import itemgetter

	print(list(map(itemgetter(0), popular)))
	# [('python', 'programming'), ('python', 'programmer')]

Trying to write a solution in a single statement, a one-liner, is a fun challenge—and a great way to grow your Python skills. One-liners start from the input data (usually a sequence) and progressively apply functions, generator expressions, and comprehensions to build new sequences until the desired result is produced. One-liners are often “write-only”: powerful and fun to write, but harder to read and modify. One-liners often lead to software maintenance problems in the future. Use them responsibly: when in doubt, don’t. Here’s the text-summarization one-liner corresponding to the code above:

	print(list(map(itemgetter(0),
	takewhile(lambda item: item[1] > 1,
	Counter(chain.from_iterable(
	map(lambda words: combinations(words, 2),
	(filter(lambda word: word not in stops,
	s.split()) for s in
	text.lower().split('.')))))
	.most_common()))))
	# [('python', 'programming'), ('python', 'programmer')]

Other itertools functions worth exploring include permutations, product, groupby, and cycle. We leave them for you to discover.

Advanced Sorting

The reversed function flips the existing order, while sorted orders by value

Sorting and reversing sequences are frequently occurring operations. They are supported by the built-in functions sorted and reversed.

The function reversed yields the items in data in reversed order, from last to first. It does not sort the items:

	>>> items = [1, 'a', None, True]
	>>> reversed_items = reversed(items)
	>>> type(reversed_items)
	<class 'list_reverseiterator'>
	>>> list(reversed_items)
	[True, None, 'a', 1]

If you want to sort a sequence by value (from smallest to largest, or in reverse), you need the sorted function:

	>>> items_sortable = [1, -4, float('-inf'), 3.14159, float('inf'), 5]
	>>> sorted(items_sortable)
	[-inf, -4, 1, 3.14159, 5, inf]
	>>> sorted(items_sortable, reverse=True)
	[inf, 5, 3.14159, 1, -4, -inf]

Notice how the function correctly handles float('inf') (positive infinity) and float('-inf') (negative infinity). These values are defined by the IEEE 754 standard for floating-point numbers. Interestingly, division by infinity is allowed (for example, 1.0 / float(’inf’) yields 0.0), but division by 0 raises a ZeroDivisionError exception, making Python’s native number system mathematically inconsistent.

The IEEE 754 floating-point number standard defines a special value that is not even a number and is appropriately called nan (float('nan'), “not-a-number”). The result of any arithmetical operation that involves nan is nan. The result of any relational expression that involves nan is False, except operator !=: float(’nan’) != x is True for any x. This makes nan the only object that is and is not equal to itself!

You can use nan as a placeholder for a number that must be provided but is not. Such “ghost” numbers are referred to as missing values and are common in real-world data processing. If you want to know whether a data item x is a “ghost,” use the predicate function math.isnan(x) which returns True if and only if x is a nan.

What about sorting heterogeneous sequences? In Python 3, order comparisons between unrelated types are not supported:

	>>> items = [1, 'a', None, True]
	>>> sorted(items)
	Traceback (most recent call last):
	File "<python-input-1>", line 1, in <module>
	sorted(items)
	~~~~~~^^^^^^^
	TypeError: '<' not supported between instances of 'str' and 'int'
	>>> sorted(items, reverse=True)
	Traceback (most recent call last):
	File "<python-input-2>", line 1, in <module>
	sorted(items, reverse=True)
	~~~~~~^^^^^^^^^^^^^^^^^^^^^
	TypeError: '<' not supported between instances of 'NoneType' and 'bool'

The function sorted relies on the < operator to compare elements. The operator can compare “apples to apples” (for example, strings to strings) and “oranges to oranges” (for example, numbers to numbers), but not “apples to oranges” (for example, strings to numbers). If you must sort fruit salad, provide your own key function. The key function maps each element to a comparable value (for example, a number or a string). In the following examples, items are compared by their identities (see ) or by their printable string representations:

	>>> sorted(items, key=lambda a: id(a))
	[True, None, 1, 'a']
	>>> sorted(items, key=lambda a: str(a))
	[1, None, True, 'a']

The first result appears arbitrary; the second is more interpretable. If you need a more meaningful ordering, provide an appropriate key function!

Назад: Functional Programming

Дальше: Let’s Reiterate