Writing Algorithms That Use t he File-Reading Techniques - Practical Programming, Fourth Edition - Dmitry Zinoviev - RutLib.com

Книга: Practical Programming, Fourth Edition

Назад: Writing Example Calls Using Strin gIO

Writing Algorithms That Use the File-Reading Techniques

There are several common methods for organizing information in files. The remainder of this chapter will demonstrate how to apply various file-reading techniques to these situations and how to develop algorithms to assist with this process.

Skipping the Header

Many data files begin with a header. As described in , TSDL files begin with a one-line description, followed by comments in lines starting with a #. The Readline technique can be used to skip this header. The method ends when you read the first real piece of data, which will be the first line after the description that doesn’t start with a #.

In English, you might try this algorithm to process this kind of a file:

	Skip the first line in the file
	Skip over the comment lines in the file
	For each of the remaining lines in the file:
	Process the data on that line

Data processing
programs often consist
of input, processing,
and output stages

The problem with this approach is that you can’t tell whether a line is a comment line until you’ve read it, but you can read a line from a file only once—there’s no simple way to “back up” in the file. An alternative approach is to read the line, skip it if it’s a comment, and process it if it’s not. Once you’ve processed the first line of data, you process the rest. Written in English, this looks something like:

	Skip the first line in the file
	Find and process the first line of data in the file
	For each of the remaining lines:
	Process the data on that line

The key aspect to note about this algorithm is that it processes lines in two places: once when it encounters the first “interesting” line in the file, and again when it handles all subsequent lines:

	from typing import TextIO
	from io import StringIO

	def skip_header(reader: TextIO) -> str:
	"""Skip the header in reader and return the first real piece of data.

	>>> infile = StringIO('Example\\n# Comment\\n# Comment\\nData line\\n')
	>>> skip_header(infile)
	'Data line\\n'
	"""

	# Read the description line
	line = reader.readline()

	# Find the first non-comment line
	line = reader.readline()
	while line.startswith('#'):
	line = reader.readline()

	# Now line contains the first real piece of data
	return line

	def process_file(reader: TextIO) -> None:
	"""Read and print the data from reader, which must start with a single
	description line, then a sequence of lines beginning with '#', then a
	sequence of data.

	>>> infile = StringIO('Example\\n# Comment\\nLine 1\\nLine 2\\n')
	>>> process_file(infile)
	Line 1
	Line 2
	"""

	# Find and print the first piece of data
	line = skip_header(reader).strip()
	print(line)

	# Read the rest of the data
	for line in reader:
	line = line.strip()
	print(line)

	if __name__ == '__main__':
	with open('hopedale.txt', 'r') as input_file:
	process_file(input_file)

In skip_header, you return the first line of read data because once you’ve found it, you can’t reread it (you can go forward but not backward). You’ll want to use skip_header in all of the file-processing functions in this section. Rather than copying the code each time you want to use it, you can put the function in a file called time_series.py (because of the Time Series Data Library) and use it in other programs using import time_series, as shown in the next example. This arrangement allows you to reuse the skip_header code, and if it needs to be modified, then there is only one copy of the function to edit.

This program processes the Hopedale data set to find the smallest number of fox pelts produced in any year. As you progress through the file, you keep the smallest value seen so far in a variable called smallest. That variable is initially set to the value on the first line, since it’s the smallest (and only) value seen so far:

	from typing import TextIO
	import time_series

	def smallest_value(reader: TextIO) -> int:
	"""Read and process reader and return the smallest value after the
	time_series header.

	>>> infile = StringIO('Example\\n1\\n2\\n3\\n')
	>>> smallest_value(infile)
	1
	>>> infile = StringIO('Example\\n3\\n1\\n2\\n')
	>>> smallest_value(infile)
	1
	"""

	line = time_series.skip_header(reader)

	# Now line contains the first data value; this is also the smallest value
	# found so far, because it is the only one we have seen.
	smallest = int(line)

	for line in reader:
	value = int(line)

	# If we find a smaller value, remember it.
	if value < smallest:
	smallest = value

	return smallest

	if __name__ == '__main__':
	with open('hopedale.txt', 'r') as input_file:
	print(smallest_value(input_file))

As with any algorithm, there are alternative ways to write this code; for example, you can replace the if statement with a call to the built-in min function:

smallest = min(smallest, value)

Dealing with Missing Values in Data

We also have data for colored fox fur production in Hebron, Labrador:

	Coloured fox fur production, Hebron, Labrador, 1834-1839
	#Source: C. Elton (1942) "Voles, Mice and Lemmings", Oxford Univ. Press
	#Table 17, p.265--266
	#remark: missing value for 1836
	55
	262
	-
	102
	178

The hyphen indicates that data for the year 1836 is missing. Unfortunately, calling read_smallest on the Hebron data produces this error:

	>>> import read_smallest
	>>> read_smallest.smallest_value(open('hebron.txt', 'r'))
	Traceback (most recent call last):
	File "<python-input-1>", line 1, in <module>
	read_smallest.smallest_value(open('hebron.txt', 'r'))
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
	File "./read_smallest.py", line 23, in smallest_value
	value = int(line)
	ValueError: invalid literal for int() with base 10: ' - \n'

The problem is that ’-’ isn’t an integer, so calling int('-') fails. Missing values aren’t an isolated problem. In general, you will often need to skip blank lines, comments, or lines containing other “nonvalues” in your data. Real data sets often contain omissions or contradictions; dealing with them is just a fact of scientific life.

For the development of this algorithm, assume that the first value is an integer, because otherwise the time series would start at the second value.

To fix your code, you must add a check inside the loop that processes a line only if it contains a real value. Let’s assume that the first value is never a hyphen because in the TSDL data sets, missing entries are always marked with hyphens. So, you need to check for that before trying to convert the string you have read to an integer:

	from typing import TextIO
	from io import StringIO
	import time_series

	def smallest_value_skip(reader: TextIO) -> int:
	"""Read and process reader, which must start with a time_series header.
	Return the smallest value after the header. Skip missing values, which
	are indicated with a hyphen.

	>>> infile = StringIO('Example\\n1\\n-\\n3\\n')
	>>> smallest_value_skip(infile)
	1
	"""

	line = time_series.skip_header(reader)
	# Now line contains the first data value; this is also the smallest value
	# found so far, because it is the only one we have seen.
	smallest = int(line)

	for line in reader:
	if line.strip().isdigit():
	value = int(line)
	smallest = min(smallest, value)

	return smallest

	if __name__ == '__main__':
	with open('hebron.txt', 'r') as input_file:
	print(smallest_value_skip(input_file))

Notice that the update to smallest is nested inside the check for non-numeric values.

Processing Whitespace-Delimited Data

Another TSDL file, lynx.dat, (), contains information about lynx pelts from the years 1821 to 1934. All data values are integers. Each line contains many values separated by whitespace. For reasons best known to the file’s author, each value ends with a period. (Note that author M. J. Campbell’s name below is misspelled in the original file.)

	Annual Number of Lynx Trapped, MacKenzie River, 1821-1934
	#Original Source: Elton, C. and Nicholson, M. (1942)
	#"The ten year cycle in numbers of Canadian lynx",
	#J. Animal Ecology, Vol. 11, 215--244.
	#This is the famous data set which has been listed before in
	#various publications:
	#Cambell, M.J. and Walker, A.M. (1977) "A survey of statistical work on
	#the MacKenzie River series of annual Canadian lynx trappings for the years
	#1821-1934 with a new analysis", J.Roy.Statistical Soc. A 140, 432--436.
	269. 321. 585. 871. 1475. 2821. 3928. 5943. 4950. 2577. 523. 98.
	184. 279. 409. 2285. 2685. 3409. 1824. 409. 151. 45. 68. 213.
	546. 1033. 2129. 2536. 957. 361. 377. 225. 360. 731. 1638. 2725.
	. . .
	6313. 3794. 1836. 345. 382. 808. 1388. 2713. 3800. 3091. 2985. 3790.
	674. 81. 80. 108. 229. 399. 1132. 2432. 3574. 2935. 1537. 529.
	485. 662. 1000. 1590. 2657. 3396.

Now let’s develop a program to find the largest value. To process the file, let’s break each line into pieces and strip off the periods. Your algorithm remains the same as it was for the fox pelt data: find and process the first line of data in the file, and then process each subsequent line. However, the notion of “processing a line” needs to be examined further because each line contains many values. Your refined algorithm, shown below, uses nested loops to handle the concept of “for each line and for each value on that line.”

	Find the first line containing real data after the header
	For each piece of data in the current line:
	Process that piece

	For each of the remaining lines of data:
	For each piece of data in the current line:
	Process that piece

Once again, you are processing lines in two different places. That is a strong hint that you should write a helper function to avoid duplicate code. Rewriting your algorithm to make it specific to the problem of finding the largest value clarifies this further:

	Find the first line of real data after the header
	Find the largest value in that line

	For each of the remaining lines of data:
	Find the largest value in that line
	If that value is larger than the previous largest, remember it

The helper function find_largest finds the largest value in a line and must split the line accordingly. The string method str.split will split around the whitespace, but you still have to remove the periods at the ends of the values.

You can also simplify your code by initializing largest to -1, since that value is guaranteed to be smaller than any of the (positive) values in the file. That way, no matter what the first value is, it’ll be larger than the “previous” value (your -1) and replace it.

	from typing import TextIO
	from io import StringIO
	import time_series

	def find_largest(line: str) -> int:
	"""Return the largest value in line, which is a whitespace-delimited
	string of integers that each end with a '.'.

	>>> find_largest('1. 3. 2. 5. 2.')
	5
	"""
	# The largest value seen so far.
	largest = -1
	for value in line.split():
	# Remove the trailing period and spaces, if any
	v = int(value.rstrip(". "))
	# If we find a larger value, remember it.
	if v > largest:
	largest = v

	return largest

You now face the same choice as with skip_header: you can put find_largest in a module (possibly time_series), or you can include it in the same file as the rest of the code. Let’s choose the latter this time because the code is specific to this particular dataset and problem:

1:	from typing import TextIO
-	from io import StringIO
-	import time_series
-
5:	def find_largest(line: str) -> int:
-	"""Return the largest value in line, which is a whitespace-delimited
-	string of integers that each end with a '.'.
-
-	>>> find_largest('1. 3. 2. 5. 2.')
10:	5
-	"""
-	# The largest value seen so far.
-	largest = -1
-	for value in line.split():
15:	# Remove the trailing period and spaces, if any
-	v = int(value.rstrip(". "))
-	# If we find a larger value, remember it.
-	if v > largest:
-	largest = v
20:
-	return largest
-
-	def process_file(reader: TextIO) -> int:
-	"""Read and process reader, which must start with a time_series header.
25:	Return the largest value after the header. There may be multiple pieces
-	of data on each line.
-
-	>>> infile = StringIO('Example\\n 20. 3.\\n 100. 17. 15.\\n')
-	>>> process_file(infile)
30:	100
-	"""
-
-	line = time_series.skip_header(reader).strip()
-	# The largest value so far is the largest on this first line of data.
35:	largest = find_largest(line)
-
-	# Check the rest of the lines for larger values.
-	for line in reader:
-	large = find_largest(line)
40:	if large > largest:
-	largest = large
-	return largest
-
-	if __name__ == '__main__':
45:	with open('lynx.txt', 'r') as input_file:
-	print(process_file(input_file))

Notice how simple the code in process_file looks, because we decided to write helper functions! To illustrate the clarity, here is the same code without using time_series.skip_header and find_largest as helper methods:

1:	from typing import TextIO
-	from io import StringIO
-
-	def process_file(reader: TextIO) -> int:
5:	"""Read and process reader, which must start with a time_series header.
-	Return the largest value after the header. There may be multiple pieces
-	of data on each line.
-
-	>>> infile = StringIO('Example\\n 20. 3.\\n')
10:	>>> process_file(infile)
-	20
-	>>> infile = StringIO('Example\\n 20. 3.\\n 100. 17. 15.\\n')
-	>>> process_file(infile)
-	100
15:	"""
-
-	# Read the description line
-	line = reader.readline()
-
20:	# Find the first non-comment line
-	line = reader.readline()
-	while line.startswith('#'):
-	line = reader.readline()
-
25:	# Now line contains the first real piece of data
-
-	# The largest value seen so far in the current line
-	largest = -1
-
30:	for value in line.split():
-
-	# Remove the trailing period
-	v = int(value.rstrip(". "))
-	# If we find a larger value, remember it
35:	if v > largest:
-	largest = v
-
-	# Check the rest of the lines for larger values
-	for line in reader:
40:
-	# The largest value seen so far in the current line
-	largest_in_line = -1
-
-	for value in line.split():
45:
-	# Remove the trailing period
-	v = int(value.rstrip(". "))
-	# If we find a larger value, remember it
-	if v > largest_in_line:
50:	largest_in_line = v
-
-	if largest_in_line > largest:
-	largest = largest_in_line
-	return largest
55:
-	if __name__ == '__main__':
-	with open('lynx.txt', 'r') as input_file:
-	print(process_file(input_file))

Назад: Writing Example Calls Using Strin gIO

Дальше: Multiline Records