Part IV Machine Learning

9 The Basics of Machine Learning

Contents

In Webster’s Dictionary, artificial intelligence (AI) is defined as:

“A branch of computer science dealing with the simulation of intelligent behavior in computers.”

A subset of AI is known as machine learning (ML), which Webster’s defines as:

“The process by which a computer is able to improve its own performance (as in analyzing image files) by continuously incorporating new data into an existing statistical model.”

Machine learning was originally inspired by how the brain works. This was a branch that became known as neural networks. Analysis is structured around a simple framework of input and output signals within a large network of interconnected neurons, which resembles the human brain. This chapter presents a basic overview of machine learning algorithms that begin by replicating the input-output mechanism of a neuron.

9.1 Neural Networks

In a network of neurons, each neuron receives inputs from other neurons. These inputs have varying degrees of strength (called weights). A network is defined by the number of neurons and their weights. For example, the human brain has approximately $10^{11}$ neurons, each with $10^{4}$ weights.

When an input leads to an output, its weight leaves an in-print. Therefore, the strength of the signal can adapt from input to output. If a neuron fires along the same path, the signal can be reinforced, increasing the weight, which fixates the path in a process called learning. Weights adapt during the learning process. There are two main aspects of this type of network: reinforcement and modularity.

Reinforcement: “Neurons that fire together wire together” (known as Hebb reinforcement).

Modularity: Different areas perform different functions even if using the same input-output structure.

A neural network is composed of inputs and outputs connected through a “machine” like the brain. The machine facilitates learning though reinforcement and modularity. Examples of inputs are features, attributes, predictions, and predictive variables. Examples of outputs include classes, targets, dependent variables, and responses. Examples of machines include sets of algorithms, techniques, models. See .

Words input and output connected through a picture of the human brain. — Figure 9.1 A biological neural picture.

A human brain—or a machine learning algorithm—is a vast complex network of inputs and outputs too difficult to fully understand. One way to model it, even absent full understanding, is to reduce the network to a primordial unit as a first principle. First-principle: an irreducible unit of understanding that begins as an assumption but cannot be decomposed into smaller units. This is a starting point of analysis. A first-principle concept of machine learning is the perceptron, a concept invented by the psychologist Frank Rosenblatt in 1957.

A perceptron is a collection of inputs and weights that are fed into a machine, which combines (as a simple dot product) the inputs and the weights to yield an output. See .

See caption. — Figure 9.2 A picture of a perceptron.

The perceptron has a collection of inputs, $x1,x2,…,xn$ , which is chosen by the modeler. Typically, an extra term (called the bias) is added to the inputs, which could shift the x’s directionally (up and down). This is the analog concept to an intercept in linear regression.

The perceptron has a vector of weights $w0j,w1j,…,wnj$ , which assign the importance to the inputs. The dot product of the inputs and the weights $W^{T} X$ produce the outputs $Z_{j}$ .

The outputs of the perceptron are determined by a simple method: First, choose/obtain the inputs and, second, multiply the inputs by the respective weights. See .

This can be done in a few lines of code in Python:

The inputs are often chosen by the analyst/modeler. But how are the weights obtained? Through a process called training. We assume the output is a binary outcome giving a value of zero or one.

Binary outcome. It can only take two mutually exclusive values: True/False, On/Off, which can be represented with a value of one or zero.

The training procedure is as follows: (1) If the output is correct, do nothing; (2) if perceptron incorrectly outputs zero, add the input to the weight vector; (3) if perceptron incorrectly outputs one, subtract the input from the weight vector. If a correct set of weights exists, the process converges.

Perceptrons are simple yet powerful machines. Given enough inputs, perceptrons can learn many things. Learning is only constrained by the inputs/features included in the system.

9.2 Machine Learning Tasks

One common task of ML algorithms is classification, which is the process of separating inputs into classes of inputs. For example, we can use a perceptron to classify between zero class and one class in two dimensions.

Take points laying in a two-dimensional x-y axis. See .

Four nodes in a two dimensional space. Three nodes in blue land on the x-y axis, one purple node lands away from the axes. — Figure 9.4 A classification example.

Each point has a corresponding value in the $(x,y)$ axis where the darker color represents the value of one, or “true”, and the lighter color represents the value of zero, or “false” in .

Now we can classify between two conditions: True = 1 and False = 0. Perceptrons work well in finding linear boundaries between classes by relying on a single hyperline in two dimensions (2D) (or a single hyperplane in 3D) to separate the data points.

For example: It is possible to separate points that have a ( $x=1$ ) AND ( $y=1$ ) placement from other points. See Figure .

It is also possible to separate points that have a ( $x=1$ ) OR ( $y=1$ ) placement from other points. See Figure .

Cardinal values are assigned to four nodes in a two dimensional space. Three nodes in blue are in the (0,1), (0,0), and (1,0) positions, and one purple node is on the (1,1) position. — Figure 9.5 A classification example with ordinal binary values.

A single red line separates the three blue nodes from the purple one of the previous figure according to logical ``AND.'' — Figure 9.6 Separating between true and false for “AND.”

A single red line separates the blue node from three purple ones according to logical ``OR.'' — Figure 9.7 Separating between true and false for “OR.”

Finally, it is also possible to separate points that have a ( $x=1$ ) NOR ( $y=1$ ) placement from other points. See .

A single red line separates the three blue nodes from the purple one according to logical ``NOR.'' — Figure 9.8 Separating between true and false for “NOR.”

But classification is not always possible when no single line can separate the inputs. For example, it is not possible to separate points that have a ( $x=1$ ) XOR ( $y=1$ ) placement from other points (XOR is short for “excluding-or,” which means an “OR” condition excluding the “AND” condition). This means finding points where ( $x=1$ ) XOR ( $y=1$ ), but not ( $x=1$ ) AND ( $y=1$ ). This cannot be accomplished with a single line in our example. See .

Two red lines indicating the impossibility to clearly separate two blue nodes and two purple nodes according to logical ``XOR'' — Figure 9.9 Separating between true and false for “XOR.”

9.3 Linear Regression

Another common task of ML algorithms is linear regression. Recall the perceptron was just a graphical representation of the dot product of the inputs and the weights $W^{T} X$ , which gives the outputs $Z_{j}$ . This can also be thought of simply as a linear regression, where the output of the perceptron is simply the line that best fits a collection of inputs.

Let’s say that we have data organized in the $x−y$ plane. The y data is what we want to understand, and we assume it depends on the x data. Therefore, the output $(y)$ is a function of inputs $\vec{x_{i}}$ , where each point is represented by a vector. So in the simplest case $y \approx f (\vec{x_{i}})$ , let’s say we have a single input $x1$ , we add a bias term $x0$ to account for the intercept. A perceptron algorithm can estimate the weights for each input that best fits the line $y=w0x0+w1x1$ .

Regression analysis in machine learning requires finding the weights ( $w0$ , and $w1$ ) that provide a line that best fits our data. So, whether we are doing classification analysis or regression analysis, everything hinges on finding the weights. Machine learning treats the issue of finding weights as an optimization problem.

Optimization problem.

A mathematical analysis that is conducted to find an optimal condition or an optimal value, which often involves an objective to maximize or minimize some function, which sometimes is subject to a constraint, and it often involves iteration (repeating the process).

Optimization problems often require three pieces: 1) the objective function to optimize, 2) a set of constraints, and 3) an algorithm necessary for optimization. The goal of perceptrons in machine learning often involves prediction. Therefore, the function to optimize is some measure of prediction error. The constraints involve representing the problem. This is where the analyst or the modeler brings to bear her institutional knowledge of the particular task to accomplish. The algorithm required for optimization is often a general-purpose tool applied to the task. Often the same algorithm can be chosen for different tasks. A popular algorithm used in optimization is what is called gradient descent.

Concept box: Gradient descent: Picture a person who is blindfolded and dropped from a helicopter somewhere in a mountain range. She cannot take off her blindfold and her goal is to find the valley where it is safe. She is looking to descend from wherever she is down to the lowest point in the area. She cannot see, but she is armed with a walking stick. One way to find the valley is to use her walking stick to palpate in every direction touching the walking stick in every direction around herself and wherever she detects the biggest drop (largest gradient descent) step in that direction. Then repeat the process (iteration), using her walking stick in every direction and step in the direction where the biggest drop is. She keeps descending down the mountain through the biggest directional drops. Wherever she might be in the mountain, she always palpates in a circle and she always moves toward the direction with the biggest drop. She keeps repeating the process. When she gets to a position where upon palpating around, she does not find a drop in any direction, she has reached the valley. In other words, using her walking stick she cannot find another direction in which to move farther down, so she has reached her goal. However, she may just found a lowest point in an immediate area (a local valley). Has she found the lowest point in the whole mountain range (the global valley)? With her current walking stick, she has no way to conclude whether she has found a local or a global valley. One thing to do is to exchange her walking stick for a longer one to facilitate a larger step. Imagine she gets a much larger stick that allows her to palpate and leap over longer distances. An advantage of a larger step size is that she could presumably reach the valley faster (since she is traveling faster) and she lowers the likelihood of getting stuck in a local valley. The disadvantage of a bigger step size is that she could skip right over a given valley. She could end up traveling faster, but also spending more time traveling without finding the global valley because she is more likely to repeatedly skip over it, multiple times and, potentially, forever.

The task of machine learning is centered around formulating a hypothesis that we have some functional form that allows us to combine inputs with weights to produce some output. This hypothesis $hw(X)$ is a function of inputs X and weights w.

The learning procedure can be described as follows:

1.: We begin with a vector of inputs or features $X^{T}$ .
2.: We have a hypothesis that involves the dot product of inputs and weights $(w)$ . We apply the constraint and the learning algorithm to the hypothesis $hw(X)$ and we get a predicted output.
3.: We compare the predicted output to the observed output and we get an error function.
4.: We use the error to rescale the weight and loop into another step for the hypothesis. See .

This is an iterative process repeating i times, which means the algorithm first makes a prediction. Then, from its prediction, the algorithm registers an error function—quantifying the difference between the predicted value from the weight-contingent hypothesis $hw(X)$ and the actual value y—and feeds the error function $J_{w}$ back into the learning algorithm before repeating the process again.

It is common to assume a functional form for the error to be quadratic. Quadratic errors have two nice properties:

1.: The signs of the errors do not offset each other because the square of a number is always positive, whether the number is positive or negative. This means that if one iteration has an error of −3 and the next iteration yields an error of +3, without assuming a quadratic we run the risk that the two errors offset each other, and we could conclude incorrectly that the algorithm performs without error across the two iterations.
2.: A quadratic error penalizes larger errors more than smaller errors. A quadratic error of two is four, while a quadratic error of three is nine.

So, the error term $J_{W}$ can be written as minimizing the sum of squares across i iterations.

\begin{matrix} J_{W} = \frac{1}{2 m} Σ_{i} {[h_{W} (X^{(i)}) - y^{(i)}]}^{2} . \end{matrix}

Gradient descent involves finding the minimum error by updating the weights W while following the slope of the error function (keeping track of how the error responds to the vector of weights $\vec{W}$ ) and iterating (repeating) until we converge to the lowest possible value of the error.

The algorithm requires taking a stand on two conditions:

1.: How large should the step size α be?
2.: What values should the weights start from ${\vec{W}}^{(0)}$ (called initialization).

The following formula shows how the error function varies with respect to the weights on a given iteration i:

\begin{matrix} \frac{δ}{δ W_{i}} J_{W} = \frac{1}{m} X^{T} (h_{W} (X) - \vec{y}) . \end{matrix}

The function that updates the weights each iteration is given by:

\begin{matrix} W_{i} = W_{i} - α \frac{δ}{δ W_{i}} J_{W} . \end{matrix}

These can be easily coded into Python as follows:

The full training/learning procedure is given below:

The process will iterate (repeat) while the error is larger than some arbitrary value (1e-6), typically called “tolerance”. Once the error climbs down to a value equal or lower than this, the learning algorithm has converged to a low error given an optimal set of weights $\vec{W}$ and outputs $\vec{y}$ .

Application. Let us apply this to the following question.

Can daily movements in the 1-year U. S. Treasury rate be explained by movements in the 10-year Treasury rate?

In essence, we want to conduct a linear regression between the 1-year and the 10-year treasuries and find whether they are related. Let our output y denote the 1-year treasury and our input $x1$ denote the 10-year treasury rate. The form of the regression is given by $y=w0+w1x1$ , where $w0$ represents the intercept of the line we want to fit and $w1$ represents the slope of the line. Essentially, we are using a single feature (the 10-year treasury) to explain or predict a single output (the 1-year treasury) given two weights for intercept and slope of a best fit line.

Let’s say that we collected 11 points of data for both treasury rates and graph them. See .

The first thing to do will be to add an arbitrary vector of 11 values to initialize the bias (11 zeros, 11 ones,… the values do not matter because the algorithm will update them through the learning procedure) and append them to the vector of features; in this case, X just has the one feature (11 values of the 10-year treasury).

We then decide on a step size and initialize the weights.

The full learning/training procedure is given below:

The last few lines instruct that every third iteration the algorithm should print the value for the hypothesis, the error, and the weights in brackets [ $w0$ , $w1$ ] representing the predicted value of the intercept and slope.

This sample output shows that it took 54 iterations for the downhill technique to reach the lowest value (the value to reach the tolerance specified) with final values for the intercept and slope of 0.3125 and 1.2602. See .

Charting the error function reveals that by the 40th iteration, the downhill climb largely reached the bottom and by 54th the process has converged. See .

Plotting the line with our estimated values of the slope and intercept reveals that the two treasury rates are positively correlated. For every 1% increase in the 10-year rate, we would, on average, expect a 1.26% increase in the 1-year treasury. See .

9.4 Classification

Classification is another typical ML task. When doing classification, probability is a critical tool. Linear regressions extend lines in a straight direction from positive infinity to negative infinity, well outside a range of acceptable values for probabilities that take on values between zero and one.

Therefore, linear regressions that we built based on our hypothesis of a dot product of weights and features, may be less useful for classification tasks. The good news is that we can build on the learning algorithm we used for linear regression by incorporating new hypotheses into otherwise the same procedure.

In the case of linear regression, the hypothesis was itself the dot product of inputs and weights (w). We now rewrite it ahead of the hypothesis step so the hypothesis can take other forms. This will allow us to apply other learning algorithms. The rest—prediction, error collection and weight updating—remains the same.

More generally, most ML hypotheses will require what is called an activation function ϕ. In the case of linear regression, the activation function was the identity matrix, meaning no activation was required beyond collecting the weights and inputs and deriving the dot product, denoted by $Z≡XTw$ . Therefore, if the activation function was the identity matrix $ϕ=I$ , the hypothesis was itself the dot product $ϕ(z)≡XTw$ . For classification objectives, we will consider other activation functions ϕ.

Logistic Regression. Logistic regression predicts the probability of a value belonging to a given class. Imagine you have 11 clients with a savings account at a bank you manage who applied for a loan in the first trimester of last year. Your clients’ savings accounts ranged from $10K to almost $80K and you can find which clients qualified for the loan (1) or failed to qualify for the loan (0). See .

Simple inspection reveals a clear-cut threshold of $40K. It seems like all clients with savings exceeding that number qualified for the loan and those below the $40K did not. So there is a clear transition from not being able to take out a loan to qualifying for one at an amount of $40K of savings.

This indicates any client with less than $40K in savings will have a 0% chance of qualifying for the loan, and any client with savings greater than $40K will have a 100% chance of getting it. Because there is a distinct separation at $40K, a logistic regression is going to “jump” from 0% to 100% at that boundary.

Of course, real life rarely works out this way. Let’s say you gathered more client data since then and got a more realistic picture, where the middle of the range has a mix of clients qualifying and not qualifying for the loan. The way to interpret this is the probability of clients qualifying for a loan gradually increases with more savings, which can be used as collateral against repayment failures.

Because of this overlap of points in the middle, there is no distinct cutoff when clients qualify, but instead a gradual transition from 0% probability to 100% probability (“0” and “1”) of getting a loan. See .

More generally, this logistic regression results in a curve indicating a probability of belonging to the true (1) category, which in this case means a client gets a loan. As savings increase, the number of qualifying clients also increases and, thus, the probability of getting a loan increases.

A Logistic Regression is a classification tool that predicts a true or false value for one or more variables. Training data must have outcomes of 0 (false) or 1 (true), but the regression outputs a probability value between 0 and 1.

An S-shaped curve (a logistic or sigmoid function) is fit to the points and then used to predict probability.
If a predicted value (the y-axis) is less than 0.5 it is typically categorized as false (0), and if the predicted value is greater than/equal to 0.5, it is typically categorized as true (1).

For linear regression, the activation function was an identity. For logistic regression, the activation function will be a sigmoid function to predict the probability of a value belonging to a given class. Using the sigmoid/logistic function, we can map weighted inputs to a range between zero and one [0,1]. See .

And at a certain threshold (say 0.5), the value is classified as True = 1 above it and a False = 0 below it. See .

Since we now have a binary outcome, we are going to need a different error term. For linear regression, the error function was the square difference. For logistic regression, the error function is called cross entropy, which is given as follows:

\begin{matrix} J_{W} = - \frac{1}{m} [y^{T} log (h_{W} (X)) + {(1 - y)}^{T} log (1 - h_{W} (X))], \end{matrix}

which measures the distance between two probability distributions, where

\begin{matrix} h_{W} (X) = - \frac{1}{1 + e^{- X \vec{W}}}, \end{matrix}

which converts the labels to probabilities; for example, an instance with a label = 1 has a probability 1 of belonging to that class. The gradient descent formulas are the same as those used in linear regression.

Linear and logistic regressions share the same learning procedure, except for the activation function and the corresponding error function.

Different activation functions $ϕ(z)$ are essentially different algorithms (just change the error… the rest is the same). Activation functions should typically be: nonlinear, differentiable, nondecreasing. They should also have the ability to compute new features, where each layer builds a more complex representation of the data. See Figures and .

Rectified linear (RELU) functions have become popular because they are easier (faster) to train than sigmoids, resulting in faster learning because it is a stepwise linear regression (lower degree of nonlinearity than sigmoid). See .

9.5 Forward Propagation

Having multiple activation functions allows us to generalize the concept of the perceptron. Forward propagation uses the perceptron for any activation function.

Forward propagation is a multilayer perceptron system involving four steps: 1. Obtain the inputs. 2. Multiply the inputs by their respective weights. 3. Calculate the output using a selected activation function. 4. Use the output of this perceptron as an input for the next. See Figure .

This means we construct multiple layers of perceptrons, where the output of each perceptron can feed as an input into the next layer. However, as we saw earlier in a single perceptron setting, updating the weights required calculating an error term and feeding it back to the algorithm so the weight could be updated.

One way to propagate the errors backward and update the weights is to have a RELU activation function for each input on the first layer and then have a linear activation function on the second layer, which builds up the linear regression from the values computed by the RELUs in the previous step. See Figure .

We need to quantify our errors to determine how correct our predictions are. As discussed before, this is achieved with loss ( $J_{W}$ ) functions: Two common ones are the quadratic loss function and cross entropy.

In a process called regularization, extra penalty terms are typically added to the error functions. This minimizes the value of each weight, which allows the algorithm to converge faster by constraining the parameter space within which we look for the true values.

Two common penalty terms involve adding the absolute value of the weights, called the Least Absolute Shrinkage and Selection Operator, or LASSO:

\begin{matrix} \hat{J_{W}} = J_{W} + λ Σ_{i j} | ω_{i j} | \end{matrix}

or adding a quadratic term for the weights:

\begin{matrix} \hat{J_{W}} = J_{W} + λ Σ_{i j} ω_{i j}^{2}, \end{matrix}

which is called an “L2” Ridge regression.

LASSO drives less important weights to zero, while Ridge makes the weights roughly equal across all features. These are two different methods to speed up convergence and feature selection.

9.6 Backward Propagation

Once forward propagation is accomplished, and a measure of error is quantified in each layer, we must move backward and update the weights. The error at the output layer is a weighted average difference between the predicted output and the observed one.

Let $δ^{L}$ be the error for the final layer, written as the difference between the predicted value ( $h_{ω}$ ) and the data (y). We then move backward to the previous layer by multiplying $δ^{L}$ times the derivative of the activation function as a way to propagate backward through the layers until the first layer is reached, where the inputs have no errors $δL≡0$ by definition.

Application: The MNIST Database. In 1994, the National Institute of Standards and Technology (NIST), which is part of the U. S. Commerce Department, developed a large database of handwritten digits to facilitate image processing and optical recognition systems to speed up mail sorting for the U. S. Post Office. This modified NIST dataset, or MNIST, consists of 70,000 28×28 black-and-white images of handwritten digits extracted from two NIST databases—one written by American Census Bureau employees (60K images) and one by American high school students (10K images). The MNIST dataset became the original canonical dataset for deep learning with 70,000 grayscale images of handwritten digits between zero and nine. Since there are 10 categories of numbers (one category per digit), 7,000 images can be used for training and 1,000 images for testing (algorithm validation) for each number.

We can write a deep learning algorithm that will recognize these numbers. To do this, we build a neural network with three layers: One input layer (X), one hidden layer (σ), and one output layer (Y)—with two perceptrons (θ) that connect the layers.

The inputs are each of the 28×28 pixels for each of the digits. Each pixel takes on a value from zero to one, ranging from white to black, with any value in between corresponding to any shade of gray. See an example of these looked like in .

10 boxes with a black background and handwritten digits in white from zero to nine. — Figure 9.24 National Institute of Standards and Technology modified numbers (MNIST).

We first stack the value of 28×28 into a 784×1 column vector containing all these values. We specify a single hidden layer composed of 50 neurons, which we can parcel out as a 50 vectors, and the output layer should have 10 vectors, one for each of the digits 0–9. The forward propagation stage moves sequentially from the input to the output layer. At the end, the value of the guess with the highest probability is quantified. Then we propagate backward along the lines described previously.

We add a bias term on the input and hidden layers for forward propagation. The bias is not needed for backward propagation. shows the process.

The forward propagation algorithm can be written in a few lines of code.

Then, the predict function effectively implements the actual model. It forward propagates the inputs through all the layers with the correct activation functions at each step and returns the final output.

Then, the backward propagation algorithm returns the information from the output layer back to the input layer. This algorithm is built on top of the single-gradient descent, which is extended to incorporate multiple layers, where the weights for each layer need to be adjusted separately.

The training procedure essentially remains the same as what was discussed for linear regression. This facilitates graphing cost functions and accuracy curves for the algorithm. The ancillary files include the Python code for identifying a single digit with 92% prediction accuracy from the MNIST dataset (also provided). See Figures and .

A line that shows a fast decrease at first followed by a more gradual decrease later in the cst function as iterations increase. — Figure 9.26 Cost function.

Nonlinear increases in the accuracy of training and testing datasets as iterations increase. — Figure 9.27 Training and testing accuracy.

9.7 Data Considerations

In the typical practice of machine learning, many details must be considered. For example, many algorithms are linear—or use Euclidean distances—that are heavily influenced by the numerical values/units used (cm vs. km, for example). Therefore, using features with very different range of values in the same analysis can cause numerical problems. To avoid scaling difficulties, it is common to rescale the range of all features so that each feature follows within similar ranges of values.

One simple method is to convert the raw data to fall within a range of values falling between zero and one, $x∈[0,1]$ , in what is called unity-based normalization.

\begin{matrix} \hat{x} = \frac{x - x_{min}}{x_{max} - x_{min}} . \end{matrix}

In the context of machine learning, this pre-processing of data is also called feature scaling. For some purposes, the modeler might want to extend the scale from zero to one to arbitrary values, say $x∈[a,b]$ . This can be done with a simple adjustment to the formula:

\begin{matrix} \hat{x} = a + \frac{(x - x_{min}) (b - a)}{x_{max} - x_{min}} . \end{matrix}

Other methods to pre-process raw data before feeding it to machine learning algorithms include normalization $(\hat{x} = \frac{x}{‖ x ‖})$ and standardization $(\hat{x} = \frac{x - μ_{x}}{σ_{x}})$ . There are myriad other methods to pre-process and filter the data prior to beginning the learning algorithm.

9.8 Inference and Prediction

The objective of machine learning is statistical inference, which involves learning something from the data—not to repeat the data back, but to infer something new from the data. So we want to find a hypothesis, a story, a summary, a label, that fits the data. An important concern is what is called overfitting.

The typical outcome of a machine learning algorithms is to make a prediction from the data. Imagine that the pre-processed data we fit an algorithm is the material that it uses to “learn.” Think of it as a practice exam. When we are ready to obtain a prediction from our machine learning algorithm, we implement the algorithm. Think of this as an exam with high stakes. Overfitting relates to “memorizing” the answers to practice questions instead of generalizing to questions we have not seen before (inference). Machine learning algorithms can be very susceptible to overfitting—meaning they can have a tendency to “regurgitate back” the answers to the practice exam instead of correctly answering the new “exam questions.”

A common approach to address the major concern of overfitting is to withhold some portion of the data the algorithm is to learn from, thereby splitting the data into two subsets: Training and Testing. So a machine learning algorithm proceeds in two stages. First, the algorithm is trained using only the training dataset. Second, we evaluate results in the previously unseen testing dataset.

Broadly speaking, learning in machine learning algorithms can be supervised by a human or it can be left unsupervised. In the category of machine learning algorithms that involves supervised learning, the human analyst supervises the connection between the training and the testing. The analyst may decide how much the algorithm needs to train and how much it needs to be tested. In a simple example of unsupervised learning, all the human analyst needs to do is to provide the data and prompt the algorithm for an outcome, and the algorithm trains a random portion of the data and tests over another portion. The algorithm splits the sample between training and testing at a random point.

Generally, there may be different rules on how to split the sample. One way to do this is with a multiple split, effectively splitting the dataset in an arbitrary number of (k) parts. Then, train the algorithm separately in k parts and evaluate in one. To mitigate the randomness of that choice, training can be repeated k times in order to average the results. This is called k-fold cross validation.

Since we now have two facets of learning—training and testing—there opens an opportunity for error in both. So, now we have to track error in training and error in testing. The overall error in the model is called the bias. And the difference between the training error and the testing error is called the variance. The model’s complexity is driven by the number of features, the number of hyper-parameters to estimate, the number of layers in the forward and backward propagation, and other aspects of the model.

It is generally difficult for a machine learning algorithm to learn from an overly simplistic model. In principle, as the model increases in complexity, the error in both testing and training will typically decline. Therefore, as the model increases in complexity, the bias will decline as the errors in training and testing begin to decrease (often in tandem). Thus, the more complex the model, the less error in training. Think of this as getting the practice questions in advance. The more attention (complexity) you dedicate to the practice, the better your performance will be in the practice questions. So, the error in testing decreases monotonically with the model complexity—the more complexity, the less error in training.

On the other hand, while some complexity will decrease the error in testing, too much complexity may begin to increase the testing error again. So, there may be an inflection point in testing error where an optimal level of complexity reduces testing error, but more complexity than that optimal level will begin to increase the testing error again—even if the training error continues to decline. This means that an overly complex model will see further reductions in training error along with increases in testing error. In other words, the difference between the training and testing errors (the variance of the model) widens as the complexity of the model increases.

An overly complex model reduces the bias (as the error in training declines) but increases the variance (as the error in testing increases). This means there is a trade-off between bias and variance in machine learning algorithms. The hope is to minimize the bias and the variance, so the objective is to find a “sweet spot” level of complexity. Too little complexity and the algorithm is more prone to biased predictions. Too much complexity and the algorithm will have a tendency to yield more variance in its predictions (with differing results every time we run it).

Why is it the case that too much complexity will generally increase the error in testing? Think about adding complexity as paying more attention (studying more). If we pay little attention (i. e., the model is overly simplistic) we may be prone to large errors in practice questions (training) and in the exam (testing). As we begin to pay more attention and study more, we may learn a lot from our practice questions, which should help prepare us for the exam. So paying more attention (increasing complexity) should help reduce error in both training and testing. However, more and more complexity relies more and more on proper training. So adding more and more complexity would be like memorizing the practice questions, which would further reduce the error in training, but it may not help—indeed it may increase—the error in the exam (testing). In other words, unless the exam is an identical copy of the practice questions—in which case the objective would be recall, rather than learning—too much complexity (memorizing the practice questions) will yield excellent performance in training and comparatively poorer performance in testing. This is related to the issue of overfitting, which we discussed earlier, where in-sample predictions can be made to look artificially great but out-of-sample forecasts can be widely inaccurate.

Imagine we computer-simulate a mountain range with randomly placed peaks, slopes, plains, and valleys. We also simulate a virtual hiker represented by a machine learning algorithm. And we now task the simulated hiker (our algorithm) with finding the deepest valley—the point with the lowest altitude—in our simulated mountain range. The algorithm is nearly blind. It cannot see the whole mountain range. It can only see one (simulated) step in every direction. In other words, the hiker does not get a global view of the valley; it gets a local view. We endow the virtual hiker with the ability to look one step into every direction, learn where the lowest point is within that confined circle step, and step into that lowest local point. Once it has moved, repeats the process, looks around, learns where the lowest point is and moves there.

In this story, walking is learning. So, the size of the simulated step is the learning rate. Imagine that we, humans, supervise the algorithm by modulating (or fine tuning) the learning rate of the algorithm. If the learning rate is very high, the step size is too large, so the virtual hiker jumps across many peaks and valleys in every bound, making it highly unlikely it will ever find the lowest valley. If the learning rate is high, the step size is moderately large, so the hiker may find the general region of the valley, but it jumps across multiple spots within the valley never finding the lowest point. In both of these cases we would say the algorithm never converges. Conversely, if the learning rate is slow (think baby steps), the algorithm may find both the general region of the valley and the lowest point in that valley, but it may take a long time to find it. In this case, we would say the algorithm converges (too) slowly. Therefore, the objective is to find the optimal step size (the optimal learning rate) that does not endlessly jump over the lowest point, and it does not take too long to find it.

9.9 A General Scaffolding for Machine Learning Algorithms

We do not want to build a machine learning algorithm from scratch every time. The good news is that neural networks are extremely modular in their design. Modular code allows easy expansion to an arbitrary number of layers. The structure of a neural network can be described as a list of weight matrices and activation functions.

Activation functions are important. They are essentially similar algorithms with different specifications for the quantification of their error terms. Activation functions generally need to be nonlinear and nondecreasing. The AI engineer also needs to keep track of the gradients of the activation functions. This means that activation functions must also be differentiable. Differentiability is important for the gradient descent feature of learning optimization (remember the hiker who is blindfolded in our earlier example descending into the lowest point of the valley).

Activation functions can be used to compute new sets of features because they can be appended to each additional layer in the neural network scaffolding of a machine learning algorithm. Each layer builds up a more abstract representation of the data.

Again, a machine learning algorithm is modular in design, which means the AI engineer can use most (similar blocks) of the same code and only tweak what she needs for her particular application. For example, the next few lines of Python code instantiate an activation function base class, which provide an interface to both the activation function and its derivative.

This code snippet will help calculate an activation function and its derivative. Then, this base class can be applied to functional forms of various activation functions. Below, we can extend the base class with multiple activation functions.

Then we describe the model with a set of weights. First, we initialize the weights. Then, we specify how many layers we want in the algorithm. Then we can modularize so that we can easily add or subtract layers from the algorithm as needed. All of this is shown below:

Next, we specify a model, where sigmoid is an object that contains both the sigmoid function and its gradient as defined in the previous snippets.

Prediction involves for-looping to the forward propagation function for each layer and its corresponding activation function. The following snippet defines a forward propagation function as well as a predict function. The predict function now takes the entire model as input and it must loop over the various layers.

Finally, once the algorithm has been initialized with weights and beginning from the input layer, it has propagated forward through the various layers and activation functions and has arrived at a prediction in the output layer, a predicted value is reached. Then, the algorithm will comb backward from output back to input to update the weights and proceed propagating forward again. Backward propagation is similar to forward propagation, but starts at the end and goes backward through each layer and its corresponding activation function, taking account of the prediction errors along the way and updating the weights accordingly. At the end, it returns the list of the changes of all the deltas and the overall error is updated before beginning the forward propagation again. Machine learning is an iterative process.

As we have discussed, a machine learning algorithm includes: input layers, hidden layers, and output layers. These can be organized in different ways in what is called neural network architectures. In the figure below, the yellow nodes are the input layers, the green nodes are the hidden layers, and the red nodes are the output layers. See Figure .

In this chapter, we have discussed the perceptron (P), and the feed forward (FF) architectures. A deep neural network (DFF) is essentially a neural network with more layers (and possibly more neurons in each layer). It is not just scale that makes the FF algorithm ‘deep.’ Importantly, DFF allows for more connections between nodes across layers. Propagation in DFF takes longer, it generally requires more time to converge, and it requires more computing power.

Another type of architecture is known as an Auto-Encoder (AE), which has two distinct features. One, there are fewer neurons in the hidden layer than there are inputs. Two, the outputs to be computed are the same as the inputs. See .

So why bother? Because the AI engineer may not be interested in learning about the outputs. AE is useful if the objective is to find the value of the nodes in the hidden layer that facilitates an accurate and reliable recovery of the inputs. An AE grabs the input data and finds an internal representation of the data (finds features from the data). Then, from those features, can the model guess what the input was? In other words, the objective may not be to learn about new outcomes from the features, but to learn about the features so that the neural network can recover the inputs. AE is useful for file (e. g., zip) compression, noise reduction, and dimension reduction, among other applications.

FF neural networks proceed from input to output through hidden layers. Deep learning just adds more layers and more connections between nodes, but information still flows from the inputs to the outputs. Another type or neural network is a recurring neural network (RNN). Information is circular (it recurs) in an RNN so it goes back from the output to the input. RNNs accomplish this circularity by adding an extra input that corresponds to the previous output. So, the objective is to recall what the last output was and use it to try to guess what the next answer should be. Each output depends implicitly on all previous outputs. In RNNs, input sequences generate output sequences. Instead of stacking layers next to each other, RNNs stack layered networks next to each other. Many large language models (LLMs) are built as RNNs.

Thinking About It…

Machine learning was inspired by biological neural networks. Neural networks consist of interconnected neurons that receive weighted inputs. A perceptron is the basic unit, combining inputs and weights to produce outputs. Learning occurs through weight adjustment based on input-output patterns. Machine learning tasks include classification: separating inputs into classes (e. g., AND, OR, NOR operations), and linear regression: fitting lines to data points.

Key technical concepts include: gradient descent: an optimization algorithm for finding minimal error; activation functions: including linear, sigmoid, and RELU functions; forward propagation, which involves moving data through network layers; and backward propagation, which requires updating weights based on error calculations.

Machine learning can be used for a large number of applications. In this chapter, we discussed an application to predict binary outcomes in a loan approval example and recognizing handwritten digits using neural networks.

9.10 Glossary

Activation Function: A mathematical function that determines the output of a neural network node. Common types include linear, sigmoid, and RELU (Rectified Linear Unit) functions. It transforms the input signals into output signals.
Artificial Intelligence (AI): A branch of computer science dealing with the simulation of intelligent behavior in computers.
Backward Propagation (Backprop): The process of calculating gradients and updating weights from the output layer back to the input layer to minimize prediction errors.
Bias Term: An extra input added to neural network layers that allows the model to learn patterns that do not pass through the origin, analogous to the intercept in linear regression.
Binary Outcome: A result that can only take two mutually exclusive values (e. g., True/False, 0/1), commonly used in classification problems.
Classification: The process of categorizing inputs into distinct classes or categories. A common machine learning task where the goal is to predict which category new data belongs to.
Cross Entropy: An error function used in classification tasks that measures the difference between predicted probability distributions and actual values.
Epoch: One complete pass through the entire training dataset during the learning process.
Forward Propagation: The process of moving data through a neural network from input layer to output layer, where each layer’s outputs become inputs for the subsequent layer.
Gradient Descent: An optimization algorithm that iteratively adjusts weights to minimize error by moving in the direction of the steepest descent of the error function. It is analogous to a person who is blindfolded using a walking stick to find the lowest point in a valley.
LASSO (Least Absolute Shrinkage and Selection Operator): A regularization technique that adds the absolute value of weights to the error function, helping drive less important weights to zero.
Linear Regression: A machine learning task that finds the best-fitting straight line through a set of points, used for predicting continuous values.
Logistic Regression: A classification algorithm that predicts the probability of a binary outcome using a sigmoid function to map inputs to values between 0 and 1.
Machine Learning (ML): The process by which a computer improves its own performance by continuously incorporating new data into an existing statistical model.
Neural Network: A network of interconnected nodes (neurons) that processes information through weighted connections, inspired by the human brain’s structure. Each neuron can receive inputs from other neurons and produce outputs based on those inputs.
Overfitting: An unwanted result of overusing the available data or parameters in a model, which may lead to “too-good-to-be-true” replication of the in-sample data we have, but very poor learning about the data/question we are investigating and we want to know more about. It regurgitates back what we know but it does not provide much insight on what we do not know. Overfitting is a major concern in machine learning algorithms.
Perceptron: A fundamental unit of neural networks invented by Frank Rosenblatt in 1957, consisting of inputs and weights that are combined to produce an output. It serves as the basic building block for more complex neural network architectures.
Regularization: A technique to prevent overfitting by adding penalty terms to the error function, constraining the model’s parameters. Common types include LASSO and Ridge regression.
RELU (Rectified Linear Unit): An activation function that outputs the input directly if positive, and zero otherwise. Popular in modern neural networks due to its computational efficiency.
Ridge Regression: A regularization technique that adds squared weights to the error function, helping make weights roughly equal across all features.
Weights: Numerical values that determine the strength of connections between neurons in a neural network. These values are adjusted during the learning process to improve the network’s performance.