5

The Trading Edge Is a Number, and Here Is the Formula

On their first day on the job, every market participant learns that the key to survival is to have a trading edge. Everyone talks about it, but no one really articulates what it boils down to. In this chapter, we will unveil one of the most well-guarded secrets in the finance industry. Behind the curtain is not an old man pulling some levers, it is only a simple formula every middle school student knows. Then, we will decompose this formula into two distinct modules: signal and money management.

Within the signal module, we will explore how to time entries and exits, and the properties of the two strategy types that seem to provide a trading edge: trend following and mean reversion. We will visit a popular mean reversion strategy called pairs trading.

We will continue this discussion in before covering the money management module in more depth in .

Along the way, we will cover the following topics:

• Importing libraries
• The trading edge formula
• A trading edge is not a story
• Regardless of the asset class, there are only two strategies

Technical requirements

You can access color versions of all images in this chapter via the following link: .

You can also retrieve source code for this chapter via the book's GitHub repository. To access the repository link, follow the steps in the Download the example code files section in the Preface.

Importing libraries

For this chapter and the rest of the book, we will be working with a few more libraries than usual. If they are not installed on your machine, you know the drill:

pip install –upgrade

So, please remember to import them first:

# Import Libraries  import pandas as pd  import numpy as np  import arcticdb as adb  from arcticdb import LibraryOptions  from dateutil.relativedelta import relativedelta  import requests  from io import StringIO  import yfinance as yf  %matplotlib inline  import matplotlib.pyplot as plt  import mplfinance as mpf    import statsmodels.api as sm  from statsmodels.tsa.stattools import coint, adfuller  from statsmodels.tools.tools import add_constant  from statsmodels.regression.linear_model import OLS

We have imported the libraries we will be working with. Now, let's define this elusive concept called trading edge.

The trading edge formula

"Information is not knowledge."

– Albert Einstein

Who said that science fiction hasn't found its way into the austere world of finance? Ask any hedge fund manager about their edge and you will enter a world of crusaders against corporate cabals, financial Sherlock Holmeses patiently piecing the information puzzle together, and visionaries investing in the next new [insert the next disruptive technological buzzword here…].

Everyone in the trading business will say that you need an edge to make money. Yet, they will never tell you how to build a sustainable one, presumably for fear that dissemination could erode theirs. Understandably, the trading edge has been this mysterious secret sauce.

There are broadly three common types of edge: technological, information, and statistical, which will be considered over the next few sections.

Technological edge

Any retail trader today has access to more information and computing power than any top-tier institutional investor 10 years ago. Anyone with a little bit of Python skill can scrape data off the internet and process it via machine learning or artificial intelligence much faster than any traditional research department. A hedge fund-grade cloud infrastructure costs less than 50 dollars per month. Anyone can learn how to code, analyze data, trade, invest, and manage a portfolio for free on YouTube, Quora, or any other learning platform. Technology has brought democracy to the world of finance.

Everyone wants to be Jim Simons, but no one wants to take care of the plumbing. When asked what the number one problem facing programmatic traders is, a third of responses were related to data processing, storage, and server management. Market participants need to be expert system engineers to ensure continuous connectivity. Then, they have to polish their development skills to write code that not only machines but more importantly humans can read.

The first iteration of a strategy is like the stone imprisoning Michelangelo's David. It takes a lot of patient carving to reveal the ephebe. When written poorly, it takes a lot more time to disentangle the other Italian cultural heritage: spaghetti. After all this is taken care of, market participants can finally focus on designing alpha-generating strategies. Then comes a whole other set of difficulties: scaling in/out, risk management, all the juicy stuff this book is about. The transmutation of billion-dollar ideas into bug-free code is still a daunting technological hurdle. Bottom line, it takes enormous amounts of work to make money work for you.

A simple analogy would be the personal computer industry circa Apple 1. Everyone wanted to play Space Invaders. Only a rare die-hard bunch of people were willing to build their own computers to play games. Today, every other Bitcoin enthusiast wants to learn algorithmic trading. Very few people are willing to put in sweat, blood, and tears to build their own platform. This book will give you the building blocks to build your own strategy. AI will considerably speed up the learning process. You cannot however "vibe code" your way to a robust trading edge. You need proper foundations to ask the right questions and come up with a solid architecture. This is what this book intends to do.

Information edge

The industry has traditionally operated on the belief that information gives an edge. This has been either privileged information, like getting access to top management and better treatment by analysts, or its more sinister cousin, inside information. Any information edge gets arbitraged away fast.

The information edge never really made a difference in the first place. If all it took to beat the Street was a better information edge, then, in theory, the metaphorical animal kingdom of big trading houses with their parliaments of analysts, schools of Ph.D.s, prides of fund managers, murders of traders, drifts of sell-side researchers, herds of brokers, and kaleidoscopes of expert opinions, with corporate access on speed-dial and enough money to bail out half a continent, would consistently outperform the market. In practice, venerable institutions have limped behind low-tech, plain-vanilla index funds Exchange-Traded Funds (ETFs) for every year on record. Information gives only a temporary edge that gets arbitraged increasingly quickly over time. Information wants to be democratized.

Statistical edge

A statistical edge does not get arbitraged away easily. It exists and persists by design. It is not "what" but "how" you trade that matters. In the next sections, we will look at ways to build a robust statistical trading edge. We will not try to tell you how to pick stocks differently, but how to squeeze more juice out of those you have already picked.

A trading edge is not a story

"What gets measured gets managed."

– Peter Drucker

Trading edge is not a story. A trading edge is a number, and the formula is composed of a few functions:

1. Arithmetic gain expectancy: In execution trader English, this is how often you win multiplied by how much you make on average minus how often you lose times how much you lose on average. This function is the classic arithmetic expectancy, present in every middle school introduction to statistics and conspicuously absent in a Finance MBA curriculum. When talking about trading edges or gain expectancy, market participants default to the arithmetic gain expectancy. It is easy to grasp and calculate. The simplicity of this formula imposes itself, even to those who do not understand its sophistication.

   def expectancy(win_rate,avg_win,avg_loss):  return win_rate * avg_win + (1-win_rate) * avg_loss

2. Geometric gain expectancy: Profits and losses compound geometrically. Geometric gain expectancy is mathematically closer to the expected robustness of a strategy.

   def geometric_expectancy(win_rate,avg_win,avg_loss):      return (1+avg_win) **win_rate*(1+avg_loss) **(1-win_rate)-1

3. The Kelly criterion is a position sizing algorithm that optimizes the geometric growth rate of a portfolio. Kelly has a fascinating history, starting with the 18th-century mathematician Daniel Bernoulli, re-discovered by J.L. Kelly Jr., and popularized by the legendary Edward Thorp. Kelly uses the same ingredients as the previous expectancies but cooks them a bit differently. Ed Thorp wrote books that changed two industries: "Beat the Dealer," after which casinos had to change the rules of blackjack, and "Beat the Market," which had a profound influence on quantitative finance.

   def kelly(win_rate,avg_win,avg_loss):      return win_rate / np.abs(avg_loss) - (1-win_rate) / avg_win

   We will take off where we left in , regime definition. We will simply re-use what we built. We calculated the regime across the entire S&P 500 for the absolute and relative series. We stored the data in arcticDB. We will now retrieve this data. First, we recycle the initialization function from the previous chapter to retrieve the data.

   def initialise_adb_library_local(uri_path, library_name):      uri = f"lmdb://{uri_path}" # this will set up the storage using the local file system      ac = adb.Arctic(uri)      library = ac.get_library(library_name, create_if_missing=True)    return library	

4. Implementing helper functions to initialize local Arctic database connections and efficiently retrieve specified columns across multiple tickers for universe-wide analysis.
   • _adb_library_local(): we construct LMDB URI path from directory and library name, create Arctic connection, and get or create named library with dynamic schema support.

     Next, this function creates a df for a single field across an investment universe:

     def adb_concat_single_column(library, symbols, column_name):      symbols_list = [symbol for symbol in symbols if symbol in library.list_symbols()]      comp_list_from_adb = [library.read(symbol, columns=[column_name]).data.rename(columns={column_name: symbol}) for symbol in symbols_list]      return pd.concat(comp_list_from_adb, axis=1)	

   • adb_concat_single_column(): Filter symbol list to only include those present in the library to avoid missing-data errors; read specified column for each valid symbol and rename to ticker name for clarity; concatenate all series horizontally into wide-format DataFrame. Both functions enable efficient data pipelining: initialize once, then retrieve any column(s) across the full S&P 500 universe in a single call.

     We initialize the S&P 500 library we stored in the previous chapter. We want to retrieve the 'Close' price. We do not need to retrieve the relative series to calculate pairs.

     library_SP500 = initialise_adb_library_local('data', 'SP500')  print('S&P 500: ',len(library_SP500.list_symbols()[:]), end =', ')  px_df = px_df = adb_concat_single_column(library_SP500, library_SP500.list_symbols(), 'Close')  print(px_df.shape)

We create a mock strategy for demonstration purposes.

Rudimentary strategy simulation

This is by no means a production-grade simulation. The objective is simply to explain concepts such as trading edge. At this stage, we do not need the full Michelin star 5-course menu to explain mayonnaise.

Let's use a ticker we saw in a previous chapter and create a df. It is only fitting we use General Electric (GE) to illustrate gain expectancy (GE).

ticker = 'GE'  ticker_raw_data = library_SP500.read(ticker).data	

Next, we will recycle the relative score we calculated in Chapter 4. The score is a composite number made from several regime methods. There was no attempt at optimizing either the parameters, such as breakout or moving average duration, or even weights in the weighted average calculation. If the score is above 0.5, go long. If the score is below -0.5, go short. It doesn't get more rustic than that.

data = ticker_raw_data[['score_rel','rClose', 'Close']].copy()    data['longshort'] = np.where(data['score_rel'] > 0.5 , 1, np.where(data['score_rel'] < -0.5, -1, 0))  data['shifted_longshort'] = data['longshort'].shift(+1)  data[-400:-50][['longshort','shifted_longshort',]].plot(figsize=(16, 3), style=['k', 'orange'], title=f'{ticker}: Long/Short Signal', grid=True)

Converting regime scores into trading signals and calculating daily/cumulative returns for both long-short strategy and buy-and-hold reference positions.

1. We extract regime score (score_rel), relative close price (rClose), and absolute close from raw ticker data; create binary long/short signals where score > 0.5 → long (+1), score < -0.5 → short (-1), else flat (0).
2. We shift signals forward by one day to simulate real trading and avoid look-ahead bias (trades entered day-after-signal), visualizing the last 400 bars to observe signal timing. Let's say a signal pops up today. We will only be able to put on the trade tomorrow. We do not include transaction costs or slippage.

This is better illustrated with the following graph:

Figure 5.1: GE Long/Short signals shifted forward

Signals are in black. Trading dates are shifted by one day. This makes manipulation easier.

Next, let's calculate relative returns using log returns.

data['log_returns_rel_1D'] = np.log(data['rClose'] / data['rClose'].shift())  data['cumul_rel_returns'] = data['log_returns_rel_1D'].cumsum().apply(np.exp) - 1  data[['rClose', 'cumul_rel_returns','score_rel',]].plot(figsize=(16, 4), secondary_y=['rClose'], style=['k',  'grey','y:'], title=f'{ticker}: relative Series, Returns and Score', grid=True)	

Calculating daily log and cumulative returns.

1. We calculate daily log returns.
2. We use the cumulative sum apply exponential and subtract by 1 to calculate cumulative returns.
3. We plot the relative series, Close price and score.

   

   Figure 5.2: GE Long/Short relative series, returns and score

The simplest technique to simulate Long and Short is to assign positive and negative signs, respectively. We multiply the shifted column by the return, tally them up, and voila!

data['longshort_rel_1D_returns'] = data['log_returns_rel_1D'] * data['shifted_longshort']                       data['longshort_cumul_rel_returns'] = data['longshort_rel_1D_returns'].cumsum().apply(np.exp) - 1  data[[ 'cumul_rel_returns', 'longshort_cumul_rel_returns','score_rel', 'shifted_longshort']].plot(figsize=(16, 5), secondary_y= ['shifted_longshort','score_rel'], style=['k', 'grey', 'y:','c:'], title=f'{ticker}: Cumulative Relative Returns, L/S Returns, Score Rel, L/S Score', grid =True)	

This produces the following chart.

Figure 5.3: GE Cumulative Relative Returns, L/S Returns, Score Rel, L/S Score

This strategy tends to give back a lot of the gains during sideways markets. There are lots of false positives, which erode the equity curve. This is something we will come back to later in this chapter.

Next, let's calculate the gain expectancy. We will first calculate and visualize the daily profits and losses.

def daily_profits(daily_returns):      profits = daily_returns.copy()      profits[profits < 0] = np.nan      return profits    def daily_losses(daily_returns):      losses = daily_returns.copy()      losses[losses > 0] = np.nan      return losses  winning_days = daily_profits(data['longshort_rel_1D_returns'])  losing_days = daily_losses(data['longshort_rel_1D_returns'])    plt.figure(figsize=(15, 3))  plt.plot(winning_days, 'g', label='Winning Days')  plt.plot(losing_days, 'r', label='Winning Days')  plt.title(f'{ticker}: Winning & Losing Days', fontsize=16)  plt.ylabel('Daily Profits & Losses', fontsize=14)  plt.grid(True, linestyle='-', alpha=0.6)  plt.show()

We create two series, winning days and losing days. We assign NaN to loss-making days on the winning days series and vice versa for the losing days. This gives the following chart:

Figure 5.4: GE Winning and Losing Days

Decomposing daily strategy returns into winning and losing days, then visualizing profit/loss distribution to assess trade quality.

1. daily_profits(): we copy daily returns series and mask all losses (set < 0 → NaN) to isolate winning trades for aggregation.
2. daily_losses(): Similarly, we copy daily returns series and mask all gains (set > 0 → NaN) to isolate losing trades for aggregation.
3. We extract winning/losing days from strategy returns (longshort_rel_1D_returns); plot both series on the same chart with green for wins, red for losses.
4. Visual inspection reveals win/loss clustering, magnitude patterns, and drawdown severity to inform trade quality assessment.

This generates a series of profitable versus unprofitable days plotted over time, enabling visual identification of high-consequence loss periods and win streaks.

We will use those two series to calculate the win and loss rate, as well as the average win and loss rate. We will calculate the gain expectancy over a rolling period of 100 days. Percentages look neat. It is easier to grasp. Here is the code:

window = 100  win_rate = winning_days.rolling(window).count() / window  loss_rate = losing_days.rolling(window).count() / window  avg_win = winning_days.fillna(0).rolling(window).mean()  avg_loss = losing_days.fillna(0).rolling(window).mean()    plt.figure(figsize=(15, 3))  plt.plot(win_rate, 'g--', label='Winning Days')  plt.plot(loss_rate, 'r--', label='Winning Days')  plt.title(f'{ticker}: Win & Loss Rates', fontsize=16)  plt.ylabel('Win & Loss Rates', fontsize=14)  plt.grid(True, linestyle='-', alpha=0.6)  plt.show()

We are computing rolling-window statistics such as win and loss rate, and average win and loss over a lookback period to visualize the evolution of the gain expectancy over time.

1. We define a rolling window size (e.g., 100 days). We count winning days per window divided by window size to yield rolling win-rate. Similarly, we compute the rolling loss-rate from losing days.
2. We compute a rolling average of winning trades (ignoring zeros) and a rolling average of losing trades to track average magnitude per win/loss over time.
3. We plot win/loss rates on the same chart (green dashes for wins, red dashes for losses) to visualize shifting trade quality across time; separately plot average win/loss magnitudes.
4. We overlay rates and magnitudes. This reveals periods of consistent wins versus clustered drawdowns, enabling regime-specific performance assessment.

This produces the following chart:

Figure 5.5: GE rolling Win and Loss Rates

A simple way to read the chart is that the strategy wins more often when the green line is higher than the red line. It does not necessarily mean it will be profitable. To know if it makes money, we need to superimpose the average profits and losses over the same period.

Let's also plot the average profits and losses.

plt.figure(figsize=(15, 3))  plt.plot(avg_win, 'g:', label='Winning Days')  plt.plot(avg_loss, 'r:', label='Winning Days')  plt.title(f'{ticker}: Average Profits & Losses', fontsize=16)  plt.ylabel('Average Profits & Losses', fontsize=14)  plt.grid(True, linestyle='-', alpha=0.6)  plt.show()

This generates the following chart:

Figure 5.6: GE Rolling Average Profits and Losses

Average profits are, in general and over time, bigger than average losses. The biggest profits are larger than the largest losses in absolute terms. This indicates that the strategy is right skewed.

In practice, it is advisable to use 2 durations. The long duration establishes that the strategy actually makes money over time. It does not have to make money all the time. It just needs to make money over time. The shorter duration captures cyclicality. It is used for asset allocation purposes. For example, the above strategy gives back profits in sideways markets. It would make sense to reduce exposures accordingly.

After this practitioner interlude, let's do a deep mathematical dive:

data['trading_edge'] = expectancy(win_rate,avg_win, avg_loss).ffill()  data['geometric_expectancy'] = geometric_expectancy(win_rate ,avg_win, avg_loss).ffill()  data['kelly'] = kelly(win_rate, avg_win,avg_loss).ffill()  data[window *3:][['trading_edge', 'geometric_expectancy', 'kelly']].plot(secondary_y = 'kelly', style = ['b-o', 'y-', 'c'], figsize=(16, 3), title=f'{ticker}: Trading Edge, Geometric Expectancy, Kelly Criterion', grid=True)

We calculate the trading edge, geometric expectancy, and Kelly criterion

1. We calculate the rolling arithmetic expectancy using the expectancy function and propagate using .ffill().
2. We calculate the rolling geometric expectancy using the eponymous function and propagate using .ffill().
3. We calculate the rolling Kelly criterion.
4. We plot the three lines.

Here is what the three lines look like.

The strategy is not always active. We therefore need to stitch the parts together using the .ffill() method.

Figure 5.7: GE Trading Edge, Geometric Expectancy, Kelly Criterion

This Flintstone pre-historic strategy seems to be working fine for GE. It does not have any of the refinements that we will slap on in the next chapters. Experienced market participants often use a partial Kelly. This is a fraction of what Kelly suggests, something like one third or half Kelly. What may be the mathematically optimal position sizing is often beyond the psychological tolerance of market participants. Those three lines on the chart are a variation on the same theme. The ingredients are the same, but the cooking is a bit different. The key takeaway is: nothing happens until the gain expectancy turns positive. Sharpe, Sortino, Jensen, Treynor, and information ratios are all well and good, but they come after the gain expectancy is positive.

All of these formulae can be decomposed into two modules:

• Signal module: Win/loss rate. These are the returns generated from entry and exit signals.
• Money management module: Average profit/loss. These are contributions generated from returns multiplied by bet sizes.

This demystifies the trading edge as something that can be engineered from the ground up. Market participants pretend they have superior knowledge of the future thanks to their analytical superpowers and machine learning crystal balls. Meanwhile, casinos publicly advertise randomness. Still, the former have lumpy returns, and the latter consistently print money day-in, day-out. This is not a coincidence. The former believe knowledge is an edge, while casinos engineer their edge. The job of every market participant is therefore to optimize their trading edge. Over the next few chapters, we will consider how to engineer these modules and maximize the trading edge, starting with the signal module, before moving on to the money management module and position sizing.

So, without further ado, let's start with the signal module.

Signal module: entries and exits

"You've got to know when to hold 'em,

know when to fold 'em,

know when to walk away,

and know when to run."

– Kenny Rogers, The Gambler

Let's start gently by strangling a classical myth once and for all: "you will make money as long as you are right 51% of the time." Wrong. By their own admission, the industry's top performers with decades-long track records often claim unimpressive long-term win rates. Conversely, Long-Term Capital Management (LTCM) boasted an exceptionally high win rate until its blowup.

You will make money so long as you have a positive trading edge. The first thing we need to engineer is the signal module. As we saw in , false positives cannot be eradicated. Randomness is a feature, not a bug. Every basket comes with a few bad apples. The key is to design policies to spot those bad apples, deal with them before they spoil the basket, and move on.

Entries: stock picking is vastly overrated

The stock market is the only competitive sport where people like to hand out medals before the race starts. The industry is built on the cult of the stock picker. Everyone loves to talk about their top stock picks. Ninety percent of the energy of market participants is focused on making the right calls. Unfortunately, the same 90% of market participants will fail to beat their benchmark 5 years in a row.

In quants trader jargon, this is called correlation. When correlation endures year after year for every year on record, r-squared—causality in quants vernacular—goes to 1. In execution trader English, insanity is when the average frustrated market participant puts more and more energy and resources into stock picking, yet consistently fails year after year. Bottom line, 90% of market participants predictably fail because they consistently focus on the wrong thing: entry. Maybe it is time we entertained the possibility of a different approach.

According to Professor Jeremy Siegel, the long-term average return of the stock market is +7%. That leaves little room for pleasantries on the short side. Ideas are cheap and plentiful. Every single human being has a bag full of million-dollar ideas. The difference between ideas and profits is called execution. As we saw earlier, many market participants fail on the short side because they enter when ideas germinate, not when they are fully ripe for the picking. Underperformance is the financial equivalent of the stomach ache we get when we eat fruits that have not ripened yet.

This book will not give you a silver bullet methodology. The purpose of this book is to introduce a probabilistic view of the markets so that you can become a statistically better version of yourself. This book is merely a cookbook for the ingredients you bring to the table. Fundamental, technical, and quantitative short sellers have all made money, but they could not have succeeded without getting probabilities on their side first.

There are two times when you need to be cognizant of probabilities: stock selection and entry. In execution trader English, this means there are two times when you should listen to what the markets have to say before pulling the trigger.

• First, you need to short stocks that are either in sideways or bear regimes. Shorting stocks in a bull regime is a bet against a bull market. At some point, you will get the horns. In , we learned an objective way to reclassify stocks according to their regime. Since short selling is a relative game (stocks' prices dropping in absolute terms is a laggard indicator), regime change often takes the form of sector rotation in the equities market. Yesterday's darlings are tomorrow's dogs.
• The second time when you need to be aware of probabilities is entry. Market participants coming from the long side like to buy on breakouts. The same logic does not translate well on the short side. Remember that almost all market participants are in the markets to buy long. Some people have been waiting on the sideline for an opportunity to buy on dips. Fast-falling markets are often followed by rapid bear market rallies. There is a short window when probabilities are in favor of short sellers.

The best time to enter a short position is when a bear market rally rolls over. We saw earlier a visual representation of gain expectancy. In the following chapters, we will look at techniques designed to increase your trading edge. They extend beyond short selling. Some may not apply to your style or strategy.

Let's start with the exits.

Exits: the transmutation of paper profits into real money

Some market participants put a lot of emphasis on entry while neglecting exit. Let's take a real life analogy to illustrate the importance of exits. The only soldiers who walk into battle without an exit strategy are called "kamikaze", literally "divine wind" in reference to Japanese pilots during world war 2. They do not expect to come back alive. Since you probably expect your trading to thrive, the best time to develop an exit strategy is before you "get boxed in," by entering a position.

In the financial services industry, the only thing cheaper than toilet paper is back-tests and simulation print-outs. This is just paper money. No one gets hurt. Real money is the only thing that matters. Let's take the idea one step further. The only time when you know how much "real money" was made, or lost, is after closing a trade. Anything before that is called paper profit. Bottom line: exits matter.

We have all been conditioned to believe that our initial choices are what matter the most. If we have the right internship, graduate from the right university, or choose the right first job, then our careers will be smooth. Life, however, is what happens to us when we have other plans. We often underestimate the role of luck in our lives. Being at the right place at the right time has made a lot of lucky people rich. Those individuals, however, rarely succeed on their first attempt. They try, fail, and pivot until something finally clicks.

Since market participants like to buy and hold, marriage is arguably a sensible metaphor. Every marriage starts with an optimistic "happily ever after," yet roughly half of them end up as a divorce statistic. Getting married is easy. Getting divorced is a life-altering event. If you have not considered the eventuality before getting married, then it will be emotionally and financially devastating. Bad marriages can be saved and sometimes turned around. Bad divorces can't. Bad entries can be salvaged. Bad exits can't. This is when the P&L gets printed.

As you will see in the coming chapters, 5 of the 7 steps to increase your trading edge deal with losses. Another good analogy is personal finance. There are two ways you can increase your savings. Either you maintain your current spending and go for a better-paying job, or you restrict your expenses and save the difference. The former works well if you have a stable high salary.

If your earnings are entirely variable, then you would be wiser to maintain a low fixed cost base and pocket whatever difference you glean from the markets. You will get the same big paydays as the other players, but the difference is you will not need them to stay afloat. In financial creole, this focus on expenses is called "cutting your losses short."

Let's look at the only two types of return distributions.

Regardless of the asset class, there are only two strategies

Jack Schwager often points out that there is no universal holy grail. Market wizards come in all shapes and forms, sometimes even with contradictory strategies. They have one thing in common though. They excel at managing risk and controlling losses. They consistently focus on the downside. Winning positions take care of themselves. Market participants' job is to take care of losers.

Market participants usually define themselves by "what" they trade (asset class, markets, time horizon), rarely "how" they trade. Regardless of the asset class, there are only two types of strategies: trend following and mean reversion. The reason why there are only two strategies is not entries but exits. How you choose to close a trade determines your dominant trading style. The mean reversion camp closes early when inefficiencies are corrected. The trend-following crowd loves to ride their winners. A classic example is value versus growth. Value investors buy undervalued stocks and then pass the baton to growth investors who ride them into the sunset.

For example, a mean reversion market participant would buy a stock at a Price to Book Ratio (PBR) of 0.5, close the position when PBR reverts back to 1, and move on. A trend follower would buy the same name at the same valuation but would ride it deep into euphoric territory. They would eventually close the position after this once-obscure, under-researched name finally made it to the "Strong Buy" lists of tier 1 investment banks and predictably underperformed thereafter.

Some readers may argue that value investors are more the mean reversion kind and growth managers the trend following type. That is often true but not always the case. The best counterexample and probably the ultimate value trend follower is Warren Buffett. He invests in undervalued businesses that he intends to hold ad perpetuum. Your positions are not Philippe Patek watches. You do not hold them for future generations. At some point, you will close your positions. Only three things can happen: the price goes up, down, or nowhere. In execution trader English, You either lose, make, or waste money. You need a scenario for each case. You need a plan to realize profits, mitigate losses, and deal with freeloaders. Our objective is to engineer a better statistical trading edge. If there are only two strategy types, then it is important to spend time understanding how they behave, their payoffs, and risks, and whether they are mutually exclusive or compatible.

Trend following

"Money is made in the sitting and waiting."

– Jesse Livermore

Trend following strategies rely on the capital appreciation of a few big winners. Systematic Commodities Trading Advisors (CTAs) look for breakouts and place protective trailing stop losses. Technical analysts look for entry points and ride the upside. Even fundamental stock pickers qualify as trend followers. Instead of price action, they follow improvement in fundamentals, earnings momentum, or even news flow. Bottom line, whether they consciously admit it or not, the default mode for market participants is trend following. Since trend following is the dominant model for market participants, we use a rudimentary trend following strategy in this book.

This is what the P&L distribution of trend followers looks like:

Figure 5.8: Trend Following distribution of returns

We can observe the following properties:

• Low win rate: Somewhere between 30 to 50%. The mode, the peak number of trades, is in the loss-making section of the distribution. For every Alphabet, Amazon, or Apple, there are countless Nestcapes, Ataris, or MySpaces.
• Alpha generation: Trend-following strategies rely on the capital appreciation of a few winners to offset numerous small losses. This introduces a lag between profits unfolding over time, and losses promptly denting the equity curve. Losses are quickly dealt with while winners take time to mature.
• Right skew: A right-tailed P&L distribution gives a favorable tail ratio. Winners are bigger than losers. The above chart is a visual representation of the time-honored "cut your losers, ride your winners." Again, it takes time to achieve those outsized returns. Meanwhile, the equity curve has to deal with the frog smudge along the way.
• Cyclicality: Markets do not trend all the time. Styles come in and out of favor. When a style does not work, losses start mounting. Successful trend followers are prompt to recognize their mistakes and keep individual losses small. When their style is in favor, they ride the trends into the sunset.
• Relatively low turnover: Turnover is a function of cyclicality. When markets are trending up or down, there is no need to do anything. Money is made in the sitting and waiting, as legendary trader Jesse Livermore used to say. When a style goes out of favor, turnover will increase as more frogs will be churned.
• Volatility: This is a consequence of cyclicality. Market participants go through periods of lackluster returns.

Trend followers kiss a lot of frogs. The risk is that the cumulative smudge of all the delicious batrachians may not outweigh the fairness of a few princesses. The risk with trend-following strategies is in the aggregate weight of losses over profits.

The most pertinent risk metric for trend-following strategies is to compare cumulative profits and losses. This is the gain-to-pain ratio, also commonly known as the profit factor or profitability factor, favored by the prophet Jack Schwager. It states that when cumulative profits in the numerator exceed losses, the ratio is above 1, and vice versa for loss-making strategies. This is yet another version of the arithmetic gain expectancy, as a ratio instead of a delta.

def profit_ratio(profits, losses):      pr = profits.ffill() / abs(losses.ffill())      return pr    rolling_profit_ratio = profit_ratio(winning_days.fillna(0).rolling(window).sum(), losing_days.fillna(0).rolling(window).sum()).ffill()  cumulative_profit_ratio = profit_ratio(winning_days.fillna(0).cumsum(), losing_days.fillna(0).cumsum()).ffill()    plt.figure(figsize=(15, 3))  plt.plot(rolling_profit_ratio[window * 3:], 'b', label='Rolling Profit Ratio')  plt.plot(cumulative_profit_ratio[window * 3:], 'r', label='Cumulative Profit Ratio')  plt.title(f'{ticker}: Rolling & Cumulative Profit Ratio', fontsize=16)  plt.ylabel('Profit Ratio', fontsize=14)  plt.grid(True, linestyle='-', alpha=0.6)  plt.show()

Calculating profit ratio metrics comparing total winning amounts to total losing amounts, measured both over rolling windows and cumulatively, to assess strategy profitability consistency.

1. profit_ratio(): We define helper function dividing cumulative/rolling profits (sum of winning trades) by absolute cumulative/rolling losses (sum of losing trades) with forward-fill to handle NaN edges.
2. We compute rolling profit ratio: divide rolling sum of wins (over 100-day window) by rolling sum of losses to track period-specific win-to-loss magnitude. This reveals periods where wins outweigh losses.
3. We compute cumulative profit ratio: divide total wins from inception by total losses from inception to track long-term profitability trend; should rise if strategy has consistent edge.
4. We plot both ratios on the same chart (blue for rolling, red for cumulative) starting after the 3× window; intersection and divergence patterns reveal edge stability.

This time series of profit ratios enables identification of periods where average winning trade magnitude exceeds average losing magnitude, signaling sustainable strategy edge.

Figure 5.9: GE Cumulative and Rolling Profit Ratios

Profit ratio needs to be above 1 for a strategy to be viable long-term. It doesn't need to make money all the time. It just needs to make money over time. Trend-following strategies post impressive but volatile returns. They go through long periods of drawdowns. The rolling profit ratio can dip for extended periods of time such as 2019 through 2020. Their main challenge is to keep cumulative losses small. Profits only look big to the extent that losses are kept small.

Do not underestimate the mental toll. It will take courage to step up and take another trade, knowing it will probably be yet another loser. In a nutshell, trend followers get paid handsomely for the mental discomfort of maintaining discipline, while watching their capital erode.

After all, the turtle traders amassed hundreds of millions of dollars despite a 30% win rate. They took trades expecting them to fail. They satisfied their need to be right not by becoming better stock pickers but by being exceptional risk managers.

Next, let's look at mean reversion strategies.

Mean reversion

"The most important of these rules is the first one: the eternal law of reversion to the mean (RTM) in the financial markets."

– John C. Bogle

Mean reversion strategies compound numerous small profits. They rely on the premise that extremes eventually revert to the mean. They arbitrage market inefficiencies. Mean reversion strategies essentially capture the time it takes for inefficiencies to correct. For example, the price of a warrant may look cheap compared to its underlying stock. Over time, prices will converge, and the gap will close. Mean reversion strategies post low-volatility, consistent returns. They perform well during stable market regimes: bull, bear, or sideways. Stationarity is a key assumption in mean reversion strategies. They rely on the stability of the relationship between their own historical price, otherwise known as autocorrelation in industry jargon.

The markets of predilection for mean reversion strategies are typically sideways phases where prices oscillate in semi-predictable fashions. They may find bull or bear phases more challenging, but there is ample empirical evidence of talented managers performing in those markets. Again, risk management is what separates the pros from the "tourists".

Mean reversion strategies perform poorly during regime changes. For example, long high beta (high sensitivity to the markets), short low beta (defensive stocks) will do wonders in a bull market but will give back a lot of performance as the regime transitions to sideways and bear. The darlings of the last bull market often lead the way down in the ensuing bear market. They also perform poorly during tail events, one of those extreme market moves that lie in the "tail" of a probability distribution. Short gamma funds performed well for years until they spectacularly blew up in three weeks during the 2008 GFC.

This is what a distribution of returns looks like for a mean reversion strategy.

Figure 5.10: Distribution of returns for mean reversion strategies

The characteristics of mean reversion strategies are:

• Alpha generation: Mean reversion strategies have high turnovers. They compound small profits quickly.
• Left skew: Left-tailed P&L distribution. Unfavorable tail ratio: losers are bigger than winners. The above chart is a visual representation of the time-honored adage "you can't go broke taking profits." It turns out you probably can if losses are too big. Mean reversion strategies arbitrage inefficiencies. Trades are closed when inefficiencies are corrected. Sometimes, those inefficiencies persist. Since the original premise is that inefficiencies are bound to correct anyways, stop losses are not enforced. This leads to rare but devastating blows.
• High win rate: Often above 50%. Markets are not efficient all the time on all time frames. The mode of trades is in the profit section of the distribution.
• Moderate to high turnover: In practice, inefficiencies appear more frequently than the Efficient Market Hypothesis (EMH) and subsequent academic theories generally suggest.
• Consistent low-volatility small profits: Mean reversion strategies arbitrage multiple small inefficiencies during stable market regimes.
• Potentially devastating left-tail losses: Mean reversion strategies perform in established markets where inefficiencies correct. They fail during regime changes when past inefficiencies persist. Interestingly enough, trend following strategies start to perform well right around the time mean reversion strategies falter.
• Slow recovery: Mean reversion strategies bank small profits but post big losses. It takes a lot of small wins to offset a few big losses.

As we saw earlier, mean reversion strategies post consistent small profits but suffer rare but game-ending setbacks. The risk for mean reversion strategies is in the tail. A few devastating losses have the power to sink the ship. After all, the captain of the Titanic had a 99% win rate. The most relevant measure of risk is therefore the ratio of the biggest profits to the worst losses, or tail ratio:

def rolling_tail_ratio(cumul_returns, window, percentile,limit):      left_tail = np.abs(cumul_returns.rolling(window).quantile(percentile))      right_tail = cumul_returns.rolling(window).quantile(1-percentile)      np.seterr(all='ignore')  	tail = (right_tail / left_tail).clip(-limit, limit)      return tail    def expanding_tail_ratio(cumul_returns, percentile,limit):      left_tail = np.abs(cumul_returns.expanding().quantile(percentile))      right_tail = cumul_returns.expanding().quantile(1 - percentile)      np.seterr(all='ignore')  	tail = (right_tail / left_tail).clip(-limit, limit)      return tail

Defining functions to measure return distribution skewness by comparing right-tail (gains) to left-tail (losses) magnitudes, quantifying strategy bias toward large winners versus large losers.

Let's explain those functions:

1. rolling_tail_ratio(): We calculate rolling window quantiles at specified percentile (e.g., 5th percentile for left tail, 95th percentile for right tail). We divide the right-tail value by absolute left-tail value to create ratio metric. We apply min/max limits to cap extreme outliers and suppress numpy warnings for division operations.
2. expanding_tail_ratio(): Same calculation using expanding window instead of rolling window; tracks tail ratio evolution from inception to present, capturing long-term distribution asymmetry trends.
3. Both functions use clip to constrain tail ratio within [‑limit, +limit] bounds, preventing inf/nan from division by near-zero values.
4. Positive tail ratio (>1) indicates right-skewed distribution with larger gains than losses; negative or <1 indicates left-skewed distribution with larger losses.

Next, we will explore a classic mean reversion strategy: pairs trading.

Pairs trading across sectors of the S&P 500

Pairs trading is conceptually simple to understand. Pick two issues that move in tandem in a predictable fashion. Whack them back to the mean every time they deviate too far.

Numerous scripts on the internet suggest testing vast swaths of data to uncover hidden pairs. While the autocorrelation may work for some time, the relationship may break over time. We will take a much simpler approach. We will look for pairs within the same sector. After all, nothing resembles a utility stock more than another utility stock. Goldman Sachs (GS) may have a different culture and different revenue streams than JP Morgan (JPM), but at the end of the day, those are still bank stocks. Flocks of the same feather fly together as we saw earlier in .

1. So, first, we will download the S&P 500 and its constituents. We will group stocks by sector.
2. We will test for cointegration. This goes one step further than correlation. Stocks may move together but they may still diverge over the long term. We want stocks with a stable relationship. For example, utility stocks have more predictable relationships than Facebook and MySpace.
3. We will produce a df of pairs by sectors and a list of pairs with the index. Pairs with the index can be useful as they swing from outperformers to underperformers. The relationship tends to be notoriously unstable. It is therefore advisable to periodically retest those pairs.
4. We will then calculate the historical spread between those pairs as well as the ratio. We will calculate the Augmented Dickey-Fuller (ADF) to test for stationarity.
5. We will then look for a boring pair in a predictable sector.
6. We will calculate z-scores of the historical spread and ratio at various durations and concatenate with the main df. This gives a quick screening tool to enter pairs when the relationship is overextended
7. We will test various z-scores for entry and exit thresholds.
8. We will plot the equity curves for the top and bottom combinations of z-scores, entry and exit thresholds.

Let's start with the investment universe:

url = 'https://en.wikipedia.org/wiki/List_of_S%26P_500_companies'  headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64)}'}  response = requests.get(url, headers=headers)  df_SP500 = pd.read_html(StringIO(response.text))[0]  df_SP500 = df_SP500.rename(columns={'Symbol':'ticker', 'Security': 'name', 'GICS Sector':'sector','GICS Sub-Industry':'sub-industry'})  df_SP500['ticker'] = df_SP500['ticker'].str.replace('.', '-', regex =False)  df_SP500 = df_SP500.sort_values(by = ['sector','name']).set_index('ticker')  df_SP500 = df_SP500[df_SP500.index.isin(library_SP500.list_symbols())]    bm_ticker = '^GSPC'   ;  bm = 'SP500' ; ccy = 'local' ; dgt = 2  df_SP500.groupby('sector').count()	

Downloading S&P 500 constituents list with sector and company metadata, filtered to only include tickers available in the Arctic database.

1. FetchingS&P500constituentslistfromWikipediausingHTTPrequestswithuser-agentheader parse HTML table to extract ticker, company name, GICS sector, and sub-industry classifications.
2. We standardize ticker format by replacing dots with dashes (e.g., 'BRK.B' → 'BRK-B') to align with database naming conventions; sort by sector and company name for logical grouping.
3. We filter the DataFrame to retain only tickers present in library_SP500.list_symbols(), removing any delisted or missing-data tickers from the database. There are periodical additions/deletions in the index. We want to slice the index only for the symbols present in arcticDB to avoid errors.
4. We set 'ticker' as index and define global trading parameters: benchmark ticker (^GSPC), benchmark label, currency ('local'), and decimal precision (2).
5. We display a sector-wise breakdown showing the number of constituents per sector for universe composition verification.

Next, let's define the co-integration functions we will be using across the S&P 500 constituents.

def cointegration_test(df, cutoff):      n = df.shape[1]      score_matrix = np.zeros((n, n))      pvalue_matrix = np.ones((n, n))      colname = df.keys()      pairs = []      for i in range(n):          for j in range(i+1, n):              p1 = df[colname[i]]              p2 = df[colname[j]]              cointegration_matrices = coint(p1, p2)              score = cointegration_matrices[0]              pvalue = cointegration_matrices[1]              score_matrix[i, j] = score              pvalue_matrix[i, j] = pvalue              if pvalue < cutoff:                  pairs.append((colname[i], colname[j]))      return pairs, pvalue_matrix, score_matrix	

This function performs pairwise Engle-Granger cointegration tests across all asset combinations computes the t-statistic (ADF test on residuals) and p-value for each pair populates symmetric matrices and filters pairs below the p-value cutoff (e.g., 0.05) to identify statistically significant long-run equilibrium relationships.

This function is the engine that will look for pairs across a population of tickers. Cointegration simply means that pairs are correlated and do not diverge over time. The relationship between pairs is stable over time. Deviations predictably revert to the mean.

Overextension is measured either by subtracting or dividing one stock by the other. We then calculate the distance to the mean expressed in standard deviations, commonly referred to as z-score.

def historical_spread(df, pair):      pair_1 = df[pair[0]]      pair_2 = df[pair[1]]      pair_1_with_const = add_constant(pair_1)  # Add constant without overwriting      results = OLS(pair_2, pair_1_with_const).fit()      coeff = results.params[pair[0]]  # Extract the non-constant coefficient      spread = pair_2 - (coeff * pair_1)      return spread    def price_ratio(df,pair):      return df[pair[0]] / df[pair[1]]    def price_spread(df,pair):      return df[pair[0]] - df[pair[1]]    def rolling_Z_score(df, t):      Z_score = (df - df.rolling(t).mean()) / df.rolling(t).std(ddof=0)      return Z_score	

There is a difference between historical spread and price spread. The former tests for stationarity. It assumes the historical spread is constant and applies a fixed coefficient. A trading signal appears when it deviates from historical norms. Price spread is the actual price difference between one asset and the other.

Next, we will process the entire S&P 500 by sectors. We will test cointegration over 3 years.

cutoff = 0.05 ; lookback_years = 3  coint_window = px_df.index.max() - relativedelta(years=lookback_years)  df = px_df[coint_window:].copy()  sector_list = list(set(df_SP500['sector']))  sector_pairs_dict = {}  ; bm_pairs_list = []    for sctr in sector_list:      sector_tickers = [bm_ticker] + list(df_SP500[df_SP500['sector'] == sctr].index)      print(f'{sctr}: {len(sector_tickers)} stocks, ',end="")      pairs, pvalue_matrix, score_matrix = cointegration_test(df.loc[:,sector_tickers].fillna(0), cutoff)      sector_pairs_dict.update({sctr: pairs})      bm_pairs_list += [bm_pair for bm_pair in pairs if bm_ticker in bm_pair]      print(f'{len(pairs)} pairs,')    print('bm_pairs_list:', len(bm_pairs_list), bm_pairs_list)  sector_pairs_df = pd.DataFrame.from_dict(sector_pairs_dict, orient='index').T  sector_pairs_df

Screening for cointegrated pairs within each GICS sector and against the S&P 500 benchmark, building a dataset of statistically significant long-run equilibrium relationships for pairs trading.

1. Wedefinecointegrationwindow:extract3-yearhistoricalpricedatafromfullS&P500priceuniverse set p-value cutoff (e.g., 0.05) for statistical significance threshold on cointegration tests.
2. For each GICS sector: we extract sector constituents from the filtered S&P 500 universe. We prepend the benchmark ticker (^GSPC) to the sector list to identify pairs correlated with the overall market pass sector + benchmark to cointegration_test() function.
3. We perform pairwise cointegration testing on sector-specific prices. We store all cointegrated pairs by sector in a dictionary. We separately capture benchmark pairs (pairs containing ^GSPC) in bm_pairs_list for market-hedged strategies.
4. We reshape sector_pairs_dict from a dictionary to a DataFrame with sectors as columns and pairs as rows (pivot structure). This enables efficient pair lookup and aggregation across sectors.
5. We print progress: display sector name, total stocks (sector + benchmark), number of cointegrated pairs for each sector; display summary of benchmark pairs and final DataFrame structure.

This produces a sector-indexed pairs discovery with cointegrated relationships identified for 11+ GICS sectors benchmark pairs enabling market-neutral strategy pairs and a foundation for spread-based mean-reversion signals across sector portfolios.

Let's see how many pairs were generated:

sector_pairs_count = pd.concat([df_SP500.groupby('sector').count()['name'],                             sector_pairs_df.count()],axis=1, keys=['#Stocks', '#Pairs'])  sector_pairs_count

Here is the output:

Table 5.1: Sector indexed pairs

This gives 688 pairs, bigger than the size of the index. This is obviously an investment universe too big to be monitored. We need to perform additional tests to reduce this raw data sample.

Next, we want to test the stability of the relationship, or stationarity. Not only do we want pairs to move in tandem over time, we want to make sure they move together by the same measure. We can either calculate the spread or the ratio.

historical_spread_dict = {} ; price_spread_dict = {} ; price_ratio_dict = {} ; pairs_list = []    ;   failed_list = []  df_clean = df.ffill().fillna(0)  pairs_valid_list = []    x= 0  whilex< len(sector_pairs_df.columns):      print(x+1,sector_pairs_df.columns[x],sector_pairs_df.iloc[:,x].count(),end = ' pairs:')      y= 0      while(y<len(sector_pairs_df)) & (pd.notnull(sector_pairs_df.iloc[min(y,len(sector_pairs_df)-1),x]))  :          raw = sector_pairs_df.iloc[y,x]          pair = eval(raw) if isinstance(raw, str) else raw          try:              spread = abs(historical_spread(df_clean,pair))              adf_spread = adfuller(spread)[1]                                ratio = price_ratio(df_clean,pair)                      adf_ratio = adfuller(ratio)[1]                        adf_avg = (adf_spread + adf_ratio) / 2              ifadf_avg<= cutoff:                  pairs_valid_list.append(pair)                  pair_dict = {'pair':pair, 'sector':sector_pairs_df.columns[x], 'adf_spread': adf_spread, 'adf_ratio': adf_ratio, 'adf_avg': adf_avg}                  pairs_list += [pair_dict]                                historical_spread_dict.update({pair: spread})                              price_ratio_dict.update({pair:ratio})                      price_spread_dict.update({pair: price_spread(df_clean,pair)})              print(pair,end = ',')          except Exception as e:              failed_list.append(pair)          y +=1      x += 1      print('')    historical_spread_df = pd.DataFrame.from_dict(historical_spread_dict).round(3)  price_ratio_df = pd.DataFrame.from_dict(price_ratio_dict).round(3)  price_spread_df = pd.DataFrame.from_dict(price_spread_dict).round(3)  pairs_df = pd.DataFrame.from_dict(pairs_list).round(4)  if not pairs_df.empty and 'pair' in pairs_df.columns:      pairs_df = pairs_df.set_index('pair')  else:      print('Warning: no pairs found - pairs_df is empty')  print('price_spread_df:' ,price_spread_df.shape      ,'price_ratio_df:',price_ratio_df.shape,'pairs_df:', pairs_df.shape)  print(f'Failed list: {len(failed_list)}, {failed_list}')

For each cointegrated pair, we compute regression-based and simple spreads, verify stationarity using ADF tests on both representations, filter pairs meeting statistical criteria, and construct final spread DataFrames for trading signal generation. Let's go through the code:

1. We forward-fill and zero-fill price data (df_clean) to handle missing values.
2. We iterate over sector_pairs_df using a double while loop: 1) outer loop over sectors (columns), 2) inner loop over pairs within each sector (rows). Pairs are stored in a 2D DataFrame indexed by sector.
3. For each pair, we compute OLS spread and price ratio, and run the ADF test on both.
4. We accept pair if average ADF p-value ≤ cutoff (5%).
5. We store valid pairs' spread, ratio and price spread series in dictionaries.

This produces three aligned wide-format DataFrames (spreads/ratios as columns, dates as rows): historical_spread_df (regression residuals), price_ratio_df (ratios), price_spread_df (differences); pairs_df metadata with sector and ADF statistics; all pairs meet stationarity threshold, enabling mean-reversion trading.

We have reduced the investment universe to 160+ pairs from the original 500+ long list.

If we assume reversion to the mean, then we need to measure how pairs oscillate around it. This is measured in standard deviations around the mean. This is called a z-score. Since we a priori do not know which time frame best captures the oscillation, we will first calculate rolling z-scores over multiple durations.

duration_list = list(range(20, 101, 20))  z_df_dict = {}  for t in duration_list:      z_df_dict[f'z_spread{t}'] = round(rolling_Z_score(price_spread_df, t),2)      z_df_dict[f'z_ratio{t}'] = round(rolling_Z_score(price_ratio_df, t),2)

Computing rolling z-score on spread and ratio DataFrames across three distinct timeframes (short, medium, long) to generate scale-independent entry/exit signals.

1. We define duration_list: create list of rolling window sizes [20, 40, 60, 80, 100] days using range(20, 101, 20) to span short-term reactive signals through longer-term trend filters.
2. For each timeframe in duration_list, we apply the rolling_Z_score() function to price_spread_df, producing z-normalized spreads. We repeat for price_ratio_df, producing z-normalized ratios.
3. Z-score transformation: (spread - 20/60/100-day rolling mean) / rolling standard deviation, standardizing each spread/ratio to zero mean and unit variance over respective windows.
4. We store the results in z_df_dict with keys naming convention (e.g., 'z_spread20', 'z_ratio60', 'z_ratio100') for easy retrieval. All z-scores are rounded to 2 decimals for signal clarity.
5. Dictionary structure enables lookup of any pair's z-scores across all six representations (3 windows × 2 spread types) for threshold-based trading signals.

This produces a dictionary{XE"pairstrading:across,sectorsofS&P500"}ofmultiplez-scoreDataFramesindexedbydurationandspreadtype; all spreads/ratios are converted to a standardized normal distribution, enabling consistent threshold-based entry triggers (e.g., z >= 2.0 for overbought, z <= -2.0 for oversold) across different spread magnitudes.

We then build a df of the last values across all the durations:

  z_last_list = []  for z in z_df_dict:      z_last_list += [z_df_dict[z].tail(1).T.rename(columns={z_df_dict[z].index.max(): z})]  z_last_df = pd.concat(z_last_list,axis=1)    z_last_df.index.name = "pair"    if not pairs_df.empty and 'adf_avg' in pairs_df.columns:      pairs_df = pd.concat([pairs_df, z_last_df], axis=1).dropna(subset=['adf_avg'])      pairs_df = pairs_df.loc[:, ~pairs_df.columns.duplicated(keep='last')]  else:      print('Warning: no pairs found - skipping z-score merge')  pairs_df.head()

Extracting the most recent z-score values across all timeframes and spread types, merging with pair metadata to create an enriched screening DataFrame showing current pair positioning.

1. We initialize an empty z_last_list to accumulate transposed z-score snapshots for concatenation.
2. We iterate through z_df_dict containing 10 z-score DataFrames (5 windows × 2 spread types: z_spread20/40/60/80/100, z_ratio20/40/60/80/100).
3. For each z-score DataFrame: we extract the last row (most recent date) using .tail(1). We transpose (.T) to convert pairs from columns to rows. We rename the single column from the date timestamp to the z-score identifier (e.g., 'z_spread20', 'z_ratio100').
4. We append each transposed single-column DataFrame to z_last_list. After the loop completes, we concatenate all horizontally using pd.concat(z_last_list, axis=1) to create z_last_df with pairs as rows and 10 z-score columns.
5. We set index name to "pair" for clarity; merge z_last_df with existing pairs_df metadata (sector, adf_spread, adf_ratio, adf_avg) using horizontal concatenation on pair index.
6. We drop rows missing ADF statistics (dropna(subset=['adf_avg'])) to retain only qualified cointegrated pairs; remove duplicate columns (.loc[:, ~pairs_df.columns.duplicated(keep='last')]) from any prior merge collisions.

This produces a consolidated pairs_df DataFrame with sector classification, stationarity metrics (ADF p-values), and latest z-scores across all 10 timeframe/spread-type combinations. This enables immediate screening for extreme positioning (e.g., |z| >= 2.0 on multiple timeframes) signaling high-confidence mean-reversion entry candidates.

Now that we have a lot of pairs to choose from, let's pick one and run with it. If you trade for thrills, you are in the wrong business. Good trading is supposed to be boring. So, let's pick the most stable pair in the most boring sector: utilities.

utilities_pairs_df = pairs_df[pairs_df['sector'] == 'Utilities'].sort_values(by='adf_avg')  utilities_pairs_df

Ranking pairs within a sector:

1. We filter pairs_df to a single sector (Utilities).
2. We sort by adf_avg ascending (smaller p-values → stronger stationarity).
3. We change the sector name to inspect others; set ascending=False to surface weaker candidates.

This produces a table like this:

Table 5.2: Utilities p-values

Let's extract the pair with the lowest average adf score. ('CMS', 'PPL') has the lowest adf average at the time of writing. The code has built-in flexibility to allow whichever pair comes up to run smoothly. That should be as uneventful as it gets. We will plot the price of each constituent of the pair and then the spread to visually verify they move together.

pair =  utilities_pairs_df[utilities_pairs_df['adf_avg']>0].index[0]  px_df[[pair[0], pair[1]]].plot(figsize=(15, 3), style=['k', 'orange'], secondary_y = pair[0], title=f'{pair[0]} & {pair[1]}: Price Series', grid=True)    df_pair = px_df[[pair[0], pair[1]]].ffill().copy()  df_pair['spread'] = round(df_pair[pair[1]] - df_pair[pair[0]],5)  df_pair['spread_daily_log'] = np.log(df_pair['spread']/df_pair['spread'].shift()).round(5)  df_pair[[pair[0], pair[1], 'spread']].plot(figsize=(15, 3), style=['k', 'orange', 'y:'],secondary_y=['spread'], title=f'{pair[0]} & {pair[1]}: Price Series & Spread', grid=True)

Visualizing raw price series for both assets in a selected pair to assess co-integration quality, relative price divergence, and mean-reversion opportunity identification.

1. We extract the first valid pair from utilities_pairs_df (filter adf_avg > 0 to exclude edge cases). We extract both ticker price columns from px_df universe-wide price DataFrame.
2. We plot both leg prices on a dual-axis chart (black/orange) to observe historical co-movement and correlation strength.
3. Visual inspection reveals correlation strength, divergence episodes, and relative volatility between pair constituents enabling validation of statistical cointegration relationship.
4. We create df_pair DataFrame with forward-filled prices for both legs; compute simple difference spread (leg2 - leg1) rounded to 5 decimals.

This dual-axis time series chart displays raw prices for both pair components, enabling visual confirmation of mean-reverting behavior and identification of current spread positioning relative to historical norms. Let's have a look at the chart:

Figure 5.11: (CMS, PPL) prices

Stocks appear correlated over the years. Next, let's take a closer look at the historical spread between the prices.

Figure 5.12: (CMS, PPL) prices and spread

The historical spread seems stable over time. The whole game of mean reversion is about identifying when the spread moves too far out of whack and arbitraging it back, with the expectation that it will revert to the mean. It comes down to choosing a duration for the z-score.

Next, let's take a look at all the z-scores in the z_df_dict

z_pair_dict = {}  z_pair_dict = {f'{z}': z_df_dict[z][pair] for z in z_df_dict if pair in z_df_dict[z].columns  z_pair_df = pd.concat(z_pair_dict, axis=1)  z_pair_df.plot(figsize=(15, 4), title=f'{pair[0]} & {pair[1]}: Z-Score Spread & Ratio', grid=True)  z_pair_df.columns

Visualizing all six z-score time series (3 windows × 2 spread types) for the selected pair to assess mean-reversion signals across multiple timeframes.

1. We build z_pair_dict by filtering z_df_dict for the current pair. We extract the pair column from each of the six z-score DataFrames (z_spread20/60/100, z_ratio20/60/100) if the pair exists.
2. We concatenate all six z-score series horizontally into z_pair_df with standardized column names
3. We plot all series on a single chart with distinct styles (dotted/dashed/solid lines, varied colors).
4. Visual overlay reveals signal convergence/divergence across timeframes: short-term z-scores react quickly to spread deviations, long-term z-scores filter noise; extreme readings (|z| > 2) on multiple timeframes signal high-confidence entry opportunities.

This produces a multi-timeframe z-score chart for the selected pair, showing standardized spread positioning across 20/60/100-day windows. This enables the identification of mean-reversion entry signals when multiple timeframes align at extreme z-score thresholds.

Figure 5.13: Z-scores of spread and ratio

We have tested for stationarity. The relationship seems stable over time. Things revert to the mean for all durations tested. The shorter the duration, the higher the frequency. Let's add the cumulative returns of the pair spread to see how signals fit with actual returns.

Let's calculate the daily and cumulative returns of the spread.

df_pair = px_df[[pair[0], pair[1]]].ffill().copy()  df_pair['spread'] = round(df_pair[pair[0]] - df_pair[pair[1]],5)  df_pair['spread_daily_log'] = np.log(df_pair['spread']/df_pair['spread'].shift()).round(5)  df_pair['cumul_spread_returns'] = df_pair['spread_daily_log'].cumsum().apply(np.exp) - 1  df_pair = pd.concat([df_pair, z_pair_df], axis=1)  plot_cols = [ 'cumul_spread_returns'] + list(z_pair_df.columns)  df_pair[plot_cols][z_pair_df.index[0]:].plot(figsize=(15, 3), grid = True, secondary_y = list(z_pair_df.columns), title=f'{pair[0]} & {pair[1]}: Cumulative Spread Returns & Z-Scores', style = ['k'] )

Merging cumulative spread returns with multi-timeframe z-scores into a unified DataFrame:

1. We forward-fill leg prices from the selected pair; compute simple difference spread (leg1 − leg2) rounded to 5 decimals for precision.
2. We calculate daily log returns of spread: ln(spread_t / spread_t−1), rounded to 5 decimals; compound into cumulative returns via np.exp(cumsum) − 1 to track spread performance.
3. We concatenate cumulative returns with z_pair_df (six z-score series) horizontally into unified df_pair; align all data to z_pair_df start date.
4. Welot cumulative spread returns on the primary y-axis (black line) and all six z-scores on the secondary y-axis (varied colors/styles) for dual-perspective visualization.

The visual overlay shows: spread performance trending versus z-score extremes; identify entry candidates where |z| >= 2.0 across multiple timeframes coincides with spread inflection points.

Figure 5.14: Z-scores of spread and ratio and cumulative returns

Now that we have verified that mean reversion seems to work, the next step is to simulate pair trading. Next, we will build a function that simulates pairs trading.

def pairs_trading_simulator(z_score, entry_threshold, exit_threshold):      """      Parameters:          z_score (pd.Series): The z-score series for the pair.          entry_threshold (float): The z-score threshold to enter a trade.          exit_threshold (float): The z-score threshold to exit a trade.      """      position = 0  ; trades = []      for date, z in z_score.items():          if position == 0:              if z > entry_threshold:  # Short entry                  position = -1                  trades.append([date, position])              elif z < -entry_threshold:  # Long entry                  position = 1                  trades.append([date, position])          elif position == 1:  # Long position              if z > -exit_threshold:  # Exit long                  position = 0                  trades.append([date, position])          elif position == -1:  # Short position              if z < exit_threshold:  # Exit sho                  position = 0                  trades.append([date, position])      trades_df = pd.DataFrame(trades, columns=["Date","position"])      return trades_df

This function enters a long when the deviation has gone below the negative threshold and vice versa for a short. Positions are closed when the spread reverts to an exit threshold. Let's go through the code:

Converting a z-score signal into discrete long/short/flat positions using symmetric entry/exit thresholds:

1. We initialize position=0 and an empty trades log.
2. We iterate over the z-score series:
   1. If flat and z > entry_threshold | enter short (-1); if z < ‑entry_threshold | enter long (+1); log the trade.
   2. If long and z > ‑exit_threshold | exit to flat (0); log the exit.
   3. If short and z < exit_threshold | exit to flat (0); log the exit.
3. We collect all position changes into trades_df with Date and Position columns.

This produces a DataFrame of dated position flips (long, short, flat) driven by z-score threshold crossings, ready for PnL backtesting.

So, we have multiple durations for z-scores. We will next test when to enter and exit positions based on values of the z-score across all durations.

entry_thresholds = np.arange(1.7, 3.1, 0.1)  exit_thresholds = np.arange(0.1, 1.6, 0.1)  results_list = []  cum_returns_dict = {}  z_score_cols = [col for col in z_pair_df.columns if col.startswith('z_')]    for entry_threshold in entry_thresholds:      for exit_threshold in exit_thresholds:          z_pair_df_temp = z_pair_df[z_score_cols].copy()          pos_list_temp = []              for z in z_score_cols:              pos_df = pairs_trading_simulator(z_pair_df_temp[z], entry_threshold, exit_threshold)              z_pair_df_temp = pd.concat([z_pair_df_temp, pos_df.set_index('Date')], axis=1)              z_pair_df_temp.rename(columns={'position': f'pos_{z}'}, inplace=True)              pos_list_temp += [f'pos_{z}']                z_pair_df_temp = z_pair_df_temp.loc[:, ~z_pair_df_temp.columns.duplicated(keep='last')]          z_pair_df_temp[pos_list_temp] = z_pair_df_temp[pos_list_temp].shift(+1).ffill()                for pos_col in pos_list_temp:              df_pair_temp = df_pair[coint_window:].copy()              position_series = z_pair_df_temp[pos_col].astype(float).fillna(0)                        df_pair_temp[f'{pos_col}_daily_log_rets'] = df_pair_temp['spread_daily_log'] * position_series              df_pair_temp[f'{pos_col}_cum_log_rets'] = df_pair_temp[f'{pos_col}_daily_log_rets'].cumsum()              df_pair_temp[f'{pos_col}_cumul_returns'] = np.exp(df_pair_temp[f'{pos_col}_cum_log_rets']) - 1                        daily_rets = df_pair_temp[f'{pos_col}_daily_log_rets']              final_return = df_pair_temp[f'{pos_col}_cumul_returns'].iloc[-1]              max_drawdown = (df_pair_temp[f'{pos_col}_cumul_returns'] / df_pair_temp[f'{pos_col}_cumul_returns'].cummax() - 1).min()                        mean_ret = daily_rets.mean()              std_ret = daily_rets.std()              sharpe = (mean_ret / std_ret * np.sqrt(252)) if std_ret > 1e-10 else np.nan          			  													  	              results_list.append({'entry_threshold': round(entry_threshold, 2), 'exit_threshold': round(exit_threshold, 2),                  'z_score': pos_col, 'sharpe_ratio': sharpe, 'final_return': final_return, 'max_drawdown': max_drawdown})    results_df = pd.DataFrame(results_list)  results_df = results_df.sort_values('sharpe_ratio', ascending=False).round(4)

Systematically testing all combinations of entry and exit z-score thresholds across multiple z-score durations. It takes some time to process: we test of entry/exit combinations. We want to identify the optimal parameter sets that maximize the Sharpe ratio) while minimizing drawdowns. Here is how we do it step by step:

1. We define threshold ranges. Entry thresholds start from 1.7 to 3.1 in 0.1 increments. Exit thresholds start from 0.1 to 1.6 in 0.1 increments. This creates 14 × 15 = 210 threshold combinations for comprehensive optimization coverage.
2. We extract z-score columns from z_pair_df (all series starting with 'z_'). We filter only standardized spread/ratio signals for position generation.
3. Nested loop: for each entry/exit threshold pair, iterate through all z-score columns. We call pairs_trading_simulator() to generate position series with current thresholds. We concatenate position columns to temporary z_pair_df_temp. We shift positions forward +1 day and forward-fill to align with trading delays.
4. For each position column: we copy df_pair (prices + spreads). We multiply spread daily log returns by position series (float conversion). We compute cumulative log returns via cumsum(). We compound into cumulative percentage returns using np.exp() ‑ 1.
5. We calculate performance metrics: final cumulative return (last row), maximum drawdown via cummax() comparison, Sharpe ratio = (mean_daily_return / std_daily_return) × √252 with division-by-zero protection.
6. We append results dictionary (entry/exit thresholds, z-score identifier, Sharpe, final return, max drawdown) to results_list; iterate through all 210 × 6 = 1,260 strategy combinations.
7. We convert results_list to results_df. We sort by Sharpe ratio descending (highest risk-adjusted returns first). We create a pivot table by entry/exit thresholds with mean Sharpe as the aggregation metric.
8. We visualize the pivot table as a heatmap with seaborn: entry thresholds on the y-axis, exit thresholds on the x-axis, color intensity representing Sharpe ratio (red = negative Sharpe, green = positive Sharpe); enables quick identification of optimal threshold zones.

This produces a results_df with 1,260 rows (all threshold × z-score combinations) ranked by Sharpe ratio. We generate a heatmap visualization revealing optimal entry/exit threshold combinations with highest risk-adjusted returns. This identifies parameter regions where strategy consistently outperforms regardless of z-score window selection.

Let's publish the code before plotting the heatmap.

pivot_sharpe = results_df.pivot_table(values='sharpe_ratio', index='entry_threshold',     columns='exit_threshold', aggfunc='mean')  plt.figure(figsize=(12, 5))  import seaborn as sns  sns.heatmap(pivot_sharpe, annot=True, fmt='.2f', cmap='RdYlGn', center=0)  plt.title(f'{pair[0]} & {pair[1]}: Sharpe Ratio Heatmap by Threshold Combination', fontsize= 15, fontweight='bold')  plt.ylabel('Entry Threshold', fontsize= 15, fontweight='bold')  plt.xlabel('Exit Threshold', fontsize= 15, fontweight='bold')  plt.tight_layout()  plt.show()

Next, let's visualize the heatmap:

Figure 5.15: Heatmap of Sharpe ratio by entry and exit threshold combination

This shows the optimal region for entry and exit regardless of the duration and type (spread or ratio) of z-scores. This is really interesting. Rather than extremes, it seems like entering around 1.7 standard deviations, and closing when it dips below 1.2, produces a high Sharpe ratio.

When Z-scores reach extreme territory (>2 standard deviations), the Sharpe ratio declines. Moral of the story: arbitrage frequent small inefficiencies and close early.

Next, let's look at the top and bottom 10 Sharpe ratios. We will calculate the cumulative returns and plot the results on a chart.

n = 10  top_n = results_df.head(n)  bottom_n = results_df.tail(n)      fig, ax = plt.subplots(figsize=(15, 5))  for topbottom in ['Top','Bottom']:      if topbottom == 'Top':          colors = plt.cm.Greens(np.linspace(0.5, 0.9, n))          data = top_n      else:          colors = plt.cm.Reds(np.linspace(0.5, 0.9, n))          data = bottom_n        for idx, (_, row) in enumerate(data.iterrows()):          cum_rets = cum_returns_dict[int(row['i'])]          label = f"{topbottom.capitalize()} {idx+1}: {row['z_score'].replace('pos_','')}; E={row['entry_threshold']}, X={row['exit_threshold']}; Sharpe={row['sharpe_ratio']:.2f}, Ret={row['final_return']:.2%}, DD={row['max_drawdown']:.2%}"          ax.plot(cum_rets, linewidth=2, label=label, color=colors[idx], linestyle='-' if topbottom=='Top' else ':')    ax.set_xlim(left=coint_window, right=px_df.index.max())  ax.set_title(f'{pair[0]} & {pair[1]}: Top {n} vs Bottom {n} Equity Curves', fontsize=15, fontweight='bold')  ax.set_ylabel('Cumulative Returns', fontsize= 15, fontweight='bold')  ax.set_xlabel('Date', fontsize= 15, fontweight='bold')  ax.legend(loc='upper left', fontsize= 8, framealpha=0)  ax.grid(True, alpha=0.6)  ax.yaxis.set_major_formatter(plt.FuncFormatter(lambda y, _: f'{y:.0%}'))    plt.tight_layout()  plt.show()

Let's explain this bit of code. We compare the equity curves of the top versus bottom-performing parameter combinations.

Objective: Visualize and compare equity curves for the top and bottom 10 parameter combinations ranked by Sharpe ratio.

Now let's look at the steps:

1. We slice results_df (already sorted by Sharpe descending) into top_n (best 10) and bottom_n (worst 10).
2. For each group, we retrieve the pre-computed cumulative returns series directly from cum_returns_dict using the integer index i. There is no recomputation, guaranteeing consistency with the Sharpe/return metrics.
3. We plot top strategies in green (solid) and bottom in red (dotted), labeling each with z-score type, entry/exit thresholds, Sharpe, return and max drawdown.
4. We constrained the X-axis to the cointegration window to avoid applying the relationship outside the period it was tested on.

This produces an equity curve chart with 20 strategies (10 green top performers + 10 red bottom performers).

Figure 5.16: CMS/PPL Top versus Bottom 10 equity curves

The tight clustering of top curves versus divergent bottom curves reveals parameter robustness. The legend provides complete strategy specifications for each curve.

Now, this chart is too optimistic for production. We ignored slippage, transaction costs, and dividends received or paid. More importantly, we optimized for one pair. The same set of parameters will probably not produce the same results for other pairs. In financial creole, we are guilty of the crime of overfitting. The classic solution would be to run the same test across other pairs and compare. We would perform what is called out-of-sample data analysis. This is what every sensible financial analyst should do: optimize across a sample, test for robustness on out-of-sample data.

Mean reversion strategies produce small and consistent gains, with some rare but ulcer-provoking losses along the way. Stop losses are not effective for mean reversion strategies. Positions are entered around peak efficiency and expected to revert to the mean. Yet, sometimes they go a little closer to the sun like Icarus, before their wings melt away back to the mean. This is why the stationarity test is critical. As soon as stationarity breaks down, stop trading the same pair. No one wants to be caught long MySpace and short Facebook, hoping for a reversion to the mean.

Overfitting may however be the right approach in the specific case of pairs trading. We want to trade a behavior that we have observed on two separate issues. This behavior is pair specific. It may not translate well across other pairs. There may be commonalities among pairs, yet there may not be universality. The important part is the stability of the relationship.

Summary

Let's recap what we have learned in this chapter. The mysterious, mystical, mythical, magical trading edge is nothing but a little formula we learned back in school called gain expectancy. The trading edge is expressed through three related formulas: arithmetic expectancy measures average profit per trade. Geometric expectancy assesses compounding and long-term robustness. The Kelly criterion optimizes the geometric growth through position sizing. All three rely on the same inputs: win rate, average win, and average loss.

Regardless of asset class, timeframe, or instrument, all strategies ever traded fall into two buckets: trend following or mean reversion. Trend following strategies have low win rates (<50%). They rely on a few large winners to offset many small losses. They are right skewed: tail events are profitable. Profits come from keeping losses small. Mean reversion strategies have high win rates and frequent small gains. Prices are expected to misbehave before reverting to the mean, thereby rendering stop-losses counterproductive. They exhibit left-skewed return distributions: they suffer rare but potentially catastrophic losses. They perform poorly during regime changes. They depend critically on stationarity and the stability of the relationship between securities. Trend following and mean reversion strategies have opposite payoffs and risk profiles.

Pairs trading was introduced as a staple mean‑reversion strategy. Pairs are selected within sectors to improve stability. Cointegration ensures long‑term relationship stability. ADF tests confirm stationarity of spreads and ratios. Z‑scores standardize deviations, define systematic entry and exit rules. Parameter testing showed that moderate deviations, not extremes, often produce better risk‑adjusted returns. The core lesson: trading success comes from engineering probabilities, managing risk, and keeping losses small.

In the next chapter, we will look in detail at the money management module and one of the most underrated topics in fund management: position sizing.

Get this book's PDF copy, code bundle, and more

Scan the QR code (or go to ). Search for this book by name, confirm the edition, and then follow the steps on the page.

Note: Have your invoice handy. Purchases made directly from the Packt website don't require an invoice.