Книга: Algorithmic Short Selling With Python
Назад: Chapter 6: Position Sizing: Money Is Made in the Money Management Module
Дальше: Part 3: The Long/Short Game: Portfolio Construction

7

Refining the Investment Universe

This chapter focuses on refining the investment universe for short-selling by filtering out ideas that are theoretically attractive but practically dangerous. Short-selling is asymmetric by nature: liquidity evaporates; crowding creates explosive short squeeze risks, and corporate actions can distort prices for long periods. As a result, success on the short side depends less on finding "good stories" and more on weeding out bad setups. This chapter introduces the fundamental and risk-based filters required to build a robust, tradable short universe.

By the end of this chapter, you will be able to systematically narrow a broad equity universe into a short-ready subset, avoid structural short-selling hazards, interpret valuation and beta signals in context, and understand how these filters support more resilient long/short portfolio construction.

In this chapter, we will cover the following topics:

Liquidity is the currency of bear markets

The way to approach liquidity on the short-side is radically different. On the long-side, liquidity increases as more investors are drawn to rising prices. Early birds end up selling to a much larger pool of market participants. Market participants can afford to build positions over several days of trading volume. The bid/ask spread tends to narrow on the way up and widen on the way down.

On the short side, when investors liquidate their positions, it is a one-way street. After a beating, they don't come back for round two. Nothing captures the emotional journey of the Long market participants more faithfully than the Kübler-Ross model. Market participants grieve their losses. Each phase even has its distinctive signature on the markets. Interest wanes, and as the pool of market participants shrinks, so does liquidity. Early bear markets are a lot more liquid than later-stage ones. Short-sellers exit into thinner liquidity than they entered.

Add short squeezes to the mix, and you've got a recipe for an explosive cocktail. The last thing you want is to be caught in a short squeeze with a large position that you cannot cover without serious market impact. On the short side, the way to approach liquidity and market impact is not about how long it takes to build a position, but about how easy it is to get out.

This limits the investment universe to issues where liquidity is not a problem. Average daily trading value should be well above 1 million dollars. When risk is "on", small-mid caps are where the action is happening. Meanwhile, on the short side, boring blue chips go out of style.

When risk is "off", market participants gravitate toward larger capitalizations, where liquidity remains abundant. Reverting the Long small/Short large caps trade is not easy. Shorting small caps is a bloody sport. Liquidity evaporates, such that you can dock a supertanker in the bid/ask spread. So, gains can be wiped out at the first squeeze.

Importing libraries

We will be working with the pandas, numpy, yfinance, and matplotlib libraries. So, please remember to import them first:

import pandas as pdimport numpy as npimport requestsfrom io import StringIOimport arcticdb as adbfrom arcticdb import LibraryOptionsimport yfinance as yf%matplotlib inlineimport matplotlib.pyplot as plt

Now that we have imported the libraries, let's proceed to download the information for all constituents of the S&P 500 index.

Downloading information for all S&P 500 constituents

First, we download the list of the current constituents of the S&P 500 via Wikipedia. Note that it has survivorship bias. Constituents added and deleted are not featured. So, this is purely for educational purposes.

For better data integrity, no survivorship bias, and yet still affordable, I would personally recommend Norgate (). I have no affiliation with the company, yet they have some of the best quality data. For now, let's proceed with the free data sources Wikipedia and yfinance:

url = 'https://en.wikipedia.org/wiki/List_of_S%26P_500_companies' headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'}  response= requests.get(url, headers=headers)  df_SP500 = pd.read_html(StringIO(response.text))[0]  df_SP500 =df_SP500.rename(columns={'GICS Sector':'sector'})  df_SP500.set_index('Symbol', inplace=True)  bm_ticker = '^GSPC' tickers = [bm_ticker] + df_SP500.index.tolist()

Once we have a dataframe with the tickers for all the constituents, we can proceed to extract data for individual securities.

  1. Fetch data: Request Wikipedia page with browser headers to avoid blocking.
  2. Extract table: Parse HTML and select the second table [1] containing company data.
  3. Clean up: Rename: 'GICS Sector' is renamed 'sector' and sets ticker symbols as index.
  4. Verify: Display the first 5 rows.
  5. Result: DataFrame indexed by ticker (e.g., AAPL, MSFT) with columns for company name, sector, date added, and so on.

Now that we have published the function, let's download the information for each current constituent of the S&P 500 index.

sp500_info_dict ={}  for symbol in tickers:      try:          ticker = yf.Ticker(symbol)          info = ticker.info          sp500_info_dict[symbol] = info      except Exception as e:          print(f"Error fetching info for {symbol}: {e}")          sp500_info_dict[symbol] = None      sp500_info_df = pd.DataFrame.from_dict(sp500_info_dict, orient='index')

Let's go over the logic step by step:

  1. Loop through tickers: Iterate over each S&P 500 symbol.
  2. Fetch data: Use yf.Ticker(symbol).info to get company information (market cap, PE ratio, dividends, short interest, etc.).
  3. Error handling: If fetch fails, store None and print an error message.
  4. Build DataFrame: Convert dictionary to DataFrame with tickers as the index.

Extracting useful information for all S&P 500 constituents

Now that we have created a dataframe with all the information available, let's narrow down to the fields useful to our purpose.

# Market Cap correction  sp500_info_df['marketCap_calculated'] = sp500_info_df['sharesOutstanding'].mul(sp500_info_df['currentPrice'])  sp500_info_df['market_Cap']  = np.where(sp500_info_df['marketCap'] == 0, sp500_info_df['marketCap_calculated'], sp500_info_df['marketCap'])  sp500_info_df['market_Cap']  = round(sp500_info_df['market_Cap'] *1e-6)    # Average Liquidity calculation  value_traded  = round(sp500_info_df['averageVolume'].mul(sp500_info_df['currentPrice']) *1e-6)  value_traded_avg10  = round(sp500_info_df['averageDailyVolume10Day'].mul(sp500_info_df['currentPrice']) *1e-6)  sp500_info_df['value_traded'] = np.where(~pd.isna(value_traded_avg10), value_traded_avg10, value_traded)    # Crowded shorts  sp500_info_df['shortfloat'] = round(sp500_info_df['sharesShort'].div(sp500_info_df['floatShares']),3)  sp500_info_df['shortPctFloat'] = np.where(~pd.isna(sp500_info_df['shortfloat']),                                            sp500_info_df['shortfloat'], sp500_info_df['shortPercentOfFloat'])    sp500_info_df[['exDividendDate', 'dividendDate']] = sp500_info_df[['exDividendDate', 'dividendDate']].apply(lambda x: pd.to_datetime(x).dt.strftime('%Y-%m-%d'))    sp500_info_df[['sector','exchange','market_Cap', 'value_traded', 'currentPrice',#'sharesOutstanding', 'averageDailyVolume10Day', 'averageVolume',   'shortPctFloat', 'shortRatio', #'sharesShort', 'floatShares', 'shortPercentOfFloat',  'exDividendDate', 'dividendDate','dividendYield', 'fiveYearAvgDividendYield', 'payoutRatio',  'trailingPE', 'forwardPE', 'priceToBook', ]]

Let's go through the code:

  1. Market cap fix: Calculate market cap from shares × price when API value is zero; convert to millions.
  2. Liquidity metric: Compute daily dollar volume traded (prefer 10-day average over standard average).
  3. Short interest: Calculate short % of float from raw shares data; fill missing values from the API field.
  4. Date formatting: Convert dividend dates from Unix timestamps to YYYY-MM-DD format.
  5. Display subset: Show key columns for liquidity, shorts, valuations, and dividends.

Liquidity is the currency of bear markets. Market capitalisation is calculated as share price times number of outstanding shares. Logically speaking, the larger the number of shares available for trading, the more liquid.

There may be endless potential shorts in the small cap universe. Market efficiency is a direct function of liquidity. The more illiquid, the longer it takes to arbitrage an inefficiency. In other words, market impact on the way in and out is not worth the price of inefficiencies. During bear markets, the market capitalizations of small caps melt like butter in the sun. And yet, they are particularly difficult to short. Selling short exacerbates the selling pressure while absence of demand drags prices down.

Downstream from market capitalization is liquidity. As a general rule of thumb, only short issues with an average trading value well above $1 million. Anything below is unlikely to have decent borrow available.

Next, we will avoid a common tourist trap: crowded shorts.

Crowded shorts

"Elvis has left the building"

Back in 2007, I developed a little weapon of mass short destruction (WMSD) eponymously named "squeeze box." The mechanics might have been counterintuitive, but it had a surreal accuracy rate to predict potential short squeezes. All it took was a long-only manager to take a minuscule speculative long position. Since the buy/sell equilibrium was already out of balance, this would push the price up and flush the "tourists" out. They would frantically cover, which would rapidly morph into a short squeeze. At this point, the long-only manager would leisurely exit his long position with a comfortable profit. When I realized that this program directly hurt my friends back in the hedge fund world, this WMSD was permanently dismantled.

Moral of the story: Eliminate all crowded shorts—issues where borrow utilization exceeds 50%—from your investment universe. They should not even be on your radar screen. The way to make money on the short side is to find what institutional investors are liquidating and ride their tails. Good short stories usually make up for bad short trades. Every bit of information has a tick price tag attached to it. Do not wait for all the pieces of the puzzle to fit. By the time a story has deteriorated enough to become an obvious short candidate, institutional investors will have liquidated their long positions. The only people left will be short-sellers fighting over a dry bone. This leads us to a simple and powerful supply and demand indicator called borrow utilization.

Borrow utilization is the ratio of borrow taken out for short-selling divided by the supply of shares available for borrow. Large institutions have lending programs wherein they lend a portion of their long holdings for a fee. When they trim those positions, supply dries up.

Meanwhile, as shorts gain popularity, demand for borrow increases. Demand on the numerator and supply on the denominator is what makes borrow utilization the most effective measure of both institutional ownership and popularity. Institutions are considered stable long-term shareholders. They engage in stock lending programs to earn interest and juice up their returns. For example, borrow utilization north of 50% simply means that the shorting appetite exceeds institutional lending programs. Since institutions are the major players in the stock lending markets, the risk/reward ratio evidently deteriorates.

Borrow utilization can vary significantly based on a company's ownership structure. Some shareholders do not participate in lending programs. For example, tightly held companies have few shareholders, who, for inscrutable reasons, often fail to appreciate their babies being the targets of vicious bear raids. Unfortunately yfinance does not feature borrow utilisation. Yahoo has two fields.

Short ratio is the number of days it takes to cover the existing shares shorted. It is calculated as the number of shares shorted divided by the average volume. The longer the duration to cover, the more painful and protracted a short squeeze would be. The rationale would be that short sellers' frantic short cover would push prices up and force other participants to cover as well. In theory, it makes sense. In practice, it might fuel and exacerbate a short squeeze for one or two days and then things would revert back to normal.

short_ratio = sp500_info_df['shortRatio'].rank(pct=True).sort_values( ascending=False).index[:10].tolist()  sp500_info_df.loc[high_short_ratio, ['shortName','sector','shortPctFloat', 'shortRatio']].round(3).head(10)

Let's explain this snippet:

This gives the following table at the time of writing:

Figure 7.1: Top 10 short ratios

Figure 7.1: Top 10 short ratios

The second ratio is the shares shorted as a percentage of the free float. This is more interesting than the short ratio. Free float is the number of a company's shares that are available for public trading, excluding shares held by insiders, governments, or long-term strategic holders.

short_float = sp500_info_df['shortPercentOfFloat'].rank(pct=True).sort_values(ascending=False).index[:10].tolist()  sp500_info_df.loc[high_short_float, ['shortName','sector','shortPercentOfFloat', 'shortRatio']].round(3).head(10)

This gives us the following table:

Figure 7.2: Top 10 short as a percentage of the free float

Figure 7.2: Top 10 short as a percentage of the free float

The most heavily shorted stocks are less than 1/5 of the free float. Anything beyond the top 10 represents less than 10% of the float. This means that short sellers do not constitute a material threat to share prices. Let's illustrate this further by calculating a sector average.

sp500_info_df.groupby('sector')[['shortPctFloat','shortRatio']].mean().sort_values(by='shortPctFloat', ascending=False).round(3)

This gives us the following table:

Figure 7.3: Top 10 short as a percentage of the free float

Figure 7.3: Top 10 short as a percentage of the free float

It is high time we put an enduring myth to rest. Short sellers do not cause share prices to melt. Shorts represent an average of 3.5% of the free float of the S&P 500. It would take 3.4 days to cover the entire short selling across the most liquid equity market on the planet. That proves beyond the shadow of a doubt that even with unfriendly intentions, foul language, short sellers still could not tank share prices.

Short-selling is an expensive sport. Short-sellers are charged borrowing fees. These range from general collateral (GC) on easy-to-borrow issues to prohibitive rates on hard-to-borrow or crowded shorts. As borrow gradually gets taken out, the quality of the remaining pool deteriorates. Adding to this, hard-to-borrow issues do not only fetch usury rates, they are sometimes callable. This means the lenders have the option of withdrawing their lending (shares lent) back on short notice. This is referred to as "recall."

When recalls happen, short-sellers are left to either locate borrows elsewhere or close their positions. Recalls sometimes cause short squeezes.

The last tourist short-sellers to get on board are often too excited to think twice about borrowing callable stocks. When a recall happens, they cannot locate borrow and are forced to close. Since selling pressure has already reached a climax, the new buying pressure creates an imbalance in the supply/demand. Share price rises effortlessly. This triggers other short-sellers to stop losses, and they proceed to cover. This rapidly snowballs into a full-blown short squeeze that flushes out not only tourists but the more seasoned short-sellers as well. Bottom line: stay away from issues where other short-sellers tap into low quality borrow.

Crowded shorts tend to have lower sensitivity to the market than the more popular issues. When the market hits a periodic air pocket, crowded shorts barely move. Meanwhile, high flying stocks on the long side nosedive. In relative terms, this means that crowded shorts outperform and longs underperform. One possible explanation for this phenomenon is the lack of participation from institutional and retail investors. There is no institutional long money in crowded shorts, so when a periodic pullback happens, there is no one to sell. Bottom line, crowded shorts do not edge long positions when it really matters. Simply said, when everything else tanks, crowded shorts do not drop as much. It may sound counterintuitive, but crowded shorts relatively outperform the markets during pullbacks.

Conclusion: Just like dieters need to remove ice cream and potato chips from reach, weed out crowded shorts from your investment universe altogether.

Next, we will look for shorts hiding behind high dividend yields.

The fertile ground of high dividend yields

Ex-growth companies often increase their dividend yield to bring dividend support to their share price. They realize they underperform the benchmark. So, they compensate lackluster returns with generous dividend policies. This effectively deters short-sellers from engaging in bear campaigns. While a high dividend yield might stem immediate short-selling pressure, it does not prevent share price from falling long term. This is what makes the high-yielding stock universe fertile ground for profitable, peaceful short-selling.

Let's prove that point using the sp500_info_df. In , we defined defensive and cyclical sectors. Cyclical sectors lead the charge while defensive sectors trail the benchmark during bull markets. Let's calculate the average dividend yield by sector.

sp500_info_df.groupby('sector')[['dividendYield', 'fiveYearAvgDividendYield', 'payoutRatio']].mean().sort_values(by='fiveYearAvgDividendYield', ascending=False).round(2)

This is how we did it:

This gives us the following table:

Figure 7.4: Average dividend yield and payout ratio by sector

Figure 7.4: Average dividend yield and payout ratio by sector

Utilities and Consumer Defensive sectors have 5-year average dividend yields at 3.34% and 2.78%, respectively. Those are quintessentially defensive sectors that proudly lag the index. Meanwhile, consumer cyclicals and technology sport 5-year average yields of 2.12.33% and 1.41.58%, respectively. Growth stocks often reinvest their earnings into the business. Apple, once the largest market cap in the multiverse, has long sported a stingy dividend policy as a badge of honor.

The good news about dividends is that they are predictable corporate events. There are a few events that short-sellers need to be mindful of. The first two are record and payment dates. Short-sellers need to curb their enthusiasm in these moments. Companies sometimes decide to raise dividends around the time they make earnings announcements. Articulate your trading around those dates, and you will have a surprisingly pleasant time. Dividend record dates are also called ex-dates. Short-sellers are liable for dividends when holding a short position through a record date. Keep a dividend calendar close to your short positions.

You will need to convert the Unix date into YY-mm-dd format when using the yfinance library. Here is a quick snippet of code:

sp500_info_df[['exDividendDate', 'dividendDate']] = sp500_info_df[['exDividendDate', 'dividendDate']].apply(lambda x: pd.to_datetime(x).dt.strftime('%Y-%m-%d'))  sp500_info_df[['exDividendDate', 'dividendDate']]

Another way of reframing high-yield stocks as viable short candidates is to focus on the underlying reason for the dividend being so high in the first place. Growth stocks usually have miserable dividend yields. They need their cash to reinvest in their businesses. Stable, mature companies have excess cash but unsexy long-term prospects. A little bit of dividend goes a long way to court investors.

Next, we will look at a legal price control scheme.

Share buybacks

"A man generally has two reasons for doing one thing. One that sounds good and the real one"

– JP Morgan

Conventional wisdom has it that corporates will buy back their shares on the open market when they are undervalued. In practice, They are just as clueless as sell-side analysts when it comes to timing undervaluation. Share buy-backs peaked in late 2007 and bottomed out in March 2009.

Proponents for share buy-backs argue that there are no better use of their cash and cheap credit than buying back their shares. This creates a trickle-down effect and benefits the economy at large. This shareholder supremacy dates back to a school of thought pioneered by Milton Friedman in the 80s'. Corporates have become increasingly big players on the markets through share buy-back programs. The rally that followed President Trump tax cut was primarily driven by corporates buying back their own stock.

Adversaries argue that cash is siphoned from stakeholders: employees, customers, R&D and suppliers, then channeled toward shareholders. Simply said, the coach skimps on the players but spreads largesse onto the fans. The majority of top executives' compensation comes from stock options. They therefore have a direct incentive to prop up share prices to pad their stock option plans. The most powerful way is to buy back stock. This reduces liquidity, inflates earnings per share, and drives share price up.

The Corona pandemic of 2020 shed a crude light on this hypocrisy and settled the debate once and for all. Share buybacks siphon the life out of companies. This obsessive focus on short-term market impact is not aligned with the long-term interests of companies.

This is, however, besides the point for short-sellers. Corporates have pockets deep enough to artificially levitate share prices. Short-sellers should therefore refrain from shorting companies that engage in share buyback programs.

The good news is that share buybacks are highly correlated to market gyrations. Share buybacks evaporate when most needed, during corrections.

Next, we will finally talk about fundamental analysis.

Fundamental analysis

"Time is on my side"

– Mick Jagger

Fundamental analysis gives its letters of nobility to stock market analysis. Few endeavors are as intellectually stimulating as stock analysis. Regime definition can greatly simplify the work of fundamental analysts.

When fundamental analysts do not consider market regime, they essentially try to answer a theoretical question. They try to figure out why a stock should go down. There are multitudes of reasons. They may eventually be vindicated. Yet they take on massive timing, reputational, and ultimately business risk. They hope markets will agree with their thesis before investors lose patience.

On the other hand, subjecting fundamental analysis to market regimes tries to answer a practical question: why is this stock going down? The good news is the answer generally falls in only three buckets.

  1. It can be a temporary mispricing. Regime may have switched from bull to sideways. After a vigorous bull phase, stocks tend to pause and digest the advance. This phase is known as consolidation. Stocks may be dead money for a while. If so, the solution is simple. Dead money walking looks a lot better with a vigorous haircut. Trim your positions. Re-allocate capital to fresh ideas. If stocks start to perform again, there will be ample time to restock.
  2. If a few stocks within a sector start to underperform in unison, this suggests sector rotation. The solution is to switch horses and go on with the race. We saw sector rotation in . Sell short sectors that start to underperform and go long nascent outperformers.
  3. If one stock does not behave in line with its sector, it might indicate some stock specific problems. This is the time to shine for fundamental analysts. This is the time to spot the real profitable structural shorts out there. Value traps often go unnoticed because they have all the exterior signs of value stocks. They often have generous dividend policies. Valuations are at a discount relative to their peers, which makes them look superficially attractive. Yet, they stubbornly underperform. They are cheap and stay cheap for a reason. The job of a fundamental analyst is to find out why.

On the short side, conventional fundamental analysis is difficult to do. Asymmetry of information is something short-sellers must contend with. Companies rarely volunteer bad news. Analysts keep their "buy" ratings until the fateful day when they happen to read in the press that companies have filed for bankruptcy the night before.

Sell-side cheerleaders may however have some involuntary marginal utility. In the permafrost tundra "Buy rating", the code word for underperformance is "Buy for the long-term investors". Look for stocks where ratings are not refreshed, earnings models gather dust, where even maintenance research is not properly maintained. One phone call to confirm that "those are stocks for long-term investors" and you are in business.

Next, we will consider valuations. This is far from trivial.

Valuations

Once you have distilled your investment universe, it is time to pull the trigger. Next, let's take a look at average sector valuations:

sp500_info_df.groupby('sector')[['trailingPE', 'forwardPE', 'priceToBook']].median().sort_values(by='trailingPE', ascending=False).round(1)

How it works:

  1. Group by sector: Aggregate stocks by GICS sector classification.
  2. Calculate medians: Use median (not mean) for trailingPE, forwardPE, and priceToBook to avoid outlier distortion.
  3. Sort by trailing PE: Rank sectors from most to least expensive based on historical earnings.

This gives the following table:

Figure 7.5: Average trailing and forward-looking PER, PBR

Figure 7.5: Average trailing and forward-looking PER, PBR

Technology and internet stocks have been sporting gravity-defying valuations and yet they have spearheaded this bull market. Note that the forward-looking price to earnings (PE) appear reasonable. This means that earnings momentum is still strong. Those valuations are rich, but they deliver solid earnings growth. This gap between trailing PE, PE calculated on past earnings per share (EPS), and forward-looking PE, calculated on estimates, justifies high expectations and high valuations. Should earnings disappoint, the market will not fail to teach some table manners to those insolent stocks. Meanwhile, the valuations of defensive sectors are below market average. Those stocks have committed the ultimate crime on the markets. They are boring. There is no story to write about. No-one gets excited about a food stock baking bread. Consequently, the lack of demand for those sectors in a strong bull market pushes valuations down.

Now, let's go back to the growth premium for a bit. Growth stocks often sport insolent valuations. In "The Little Book That Beats the Market", Joel Greenblatt used Microsoft to illustrate a simple point: a great company can still produce poor investment returns if you pay too high a price for its earnings. In the early 90s, Microsoft insolently traded at insolently rich valuations, sometimes 50–60× earnings or more. Yet, the company delivered year after year. Microsoft was an exceptional business: high growth, high return on capital, dominant products. In the late 90s, growth started to slow down. Valuations compressed and the stock went nowhere for years. Bottom line, do not short stocks with rich valuations alone. Short high valuations only when earnings momentum decelerates. In financial creole, this is called "multiples compression". For example, this means a PE of 35 will come down to 20, because the growth premium disappears. Joel Greenblatt showed that even if Microsoft continued growing strongly, those already priced-in expectations made the future returns unattractive.

Price to Book Ratio (PBR) is a reflection on the general health of the balance sheet. Expensive PBR means that book per share (equity + retained earnings) is expensive relative to the current share price. In other words, growth and investments have been financed either by issuance of shares or by debt. In accounting parlance, this means investing cash flow has been financed by the financing cash flow instead of operating cash flow. In any case, this fragilizes the balance sheet. When the music stops, highly leveraged companies often have a rude encounter with Newtonian physics. Never keep highly leveraged companies on the long side of the book through a bear market.

Next, we looked at sectors. In Chapter 3, we assigned sectors labels such as defensive or cyclicals. Let's take a look at beta and see if we can build a mathematical foundation.

Beta

"CAPM and beta imply that buying riskier stocks should give you higher returns, but the data says the opposite".

Joel Greenblatt

Beta measures how much a stock moves relative to its benchmark. High beta means stocks move "faster" than the market and vice versa for low beta stocks. For example, a beta of 1.2 means that when the benchmark moves by 1%, the issue would move by 1.2%. Market participants really need to understand how beta works and more specifically how to balance the Long/Short net beta exposure.

In the following subsections, we will download historical prices since January 2015. We define and calculate monthly beta. We perform some sector analysis. We define high and low beta custom indices. We calculate daily log and cumulative returns. We then drill down to the highest and lowest betas' returns.

Calculating beta from historical prices across the S&P 500 constituents

Without further ado, let's download the historical prices. We recycle the arctic database we created in Chapter 5. We start by defining the functions.

def initialise_adb_library_local(uri_path, library_name):      uri = f"lmdb://{uri_path}" # this will set up the storage using the local file system      ac = adb.Arctic(uri)      library = ac.get_library(library_name, create_if_missing=True,)# library_options=LibraryOptions(dynamic_schema=True))      return library    def adb_concat_single_column(library, symbols, column_name):      symbols_list = [symbol for symbol in symbols if symbol in library.list_symbols()]      comp_list_from_adb = [library.read(symbol, columns=[column_name]).data.rename(columns={column_name: symbol}) for symbol in symbols_list]      return pd.concat(comp_list_from_adb, axis=1)

Then we download Close prices in a df called px_df.

library_SP500 = initialise_adb_library_local('data', 'SP500')  px_df = adb_concat_single_column(library_SP500, library_SP500.list_symbols(), 'Close').round(2)  px_df_rel = round(px_df.divide(px_df[bm_ticker], axis=0).mul(px_df.iloc[0,list(px_df.columns).index(bm_ticker)]),1)  px_df_rel = px_df_rel.round(2)

Accessing local Arctic database to retrieve closing prices for all S&P 500 stocks and compute relative performance versus the benchmark.

  1. We initialize the library_SP500: Create or connect to the SP500 library in the local LMDB database at the data folder.
  2. We retrieve Close prices: Use adb_concat_single_column() to extract Close column for all symbols, concatenate into wide-format DataFrame. We round absolute prices to 2 decimals (px_df).
  3. We calculate relative prices: Divide each stock by benchmark (^GSPC) and normalize to benchmark's opening value; round to 1 decimal (px_df_rel).
  4. We clean data: Remove any time zone information and standardize format.

This produces two DataFrames: px_df contains absolute closing prices, while px_df_rel shows each stock's relative performance indexed to the benchmark.

Next, we will define beta and calculate monthly betas.

def beta(df,bm_col):      log_returns = np.log(df/df.shift(1))      var = log_returns[bm_col].var()      cov = log_returns.cov()      beta = round(cov.loc[:,bm_col] / var,2)      return beta

In high school parlance, beta is a covariance matrix. The formula is as elegant as it is mathematically simple. We calculate a covariance matrix with the benchmark and then divide it by the variance matrix to measure the dispersion.

Here is how we coded it:

  1. Compute log returns: Calculate log(price_t / price_t-1) for all stocks and benchmark.
  2. Calculate variance: Compute the variance of benchmark returns.
  3. Compute covariance: Build covariance matrix between all stocks and the benchmark.
  4. Calculate beta: Divide each stock's covariance with the benchmark by the benchmark variance.
  5. Round results: Return betas rounded to 2 decimal places.

Next, we will calculate the monthly beta. In theory, we could use the method apply to calculate a rolling daily beta. This would, however, take time. Beta does not move much from one month to the next in regular market conditions. So, we will resample the px_df dataframe to month ends and calculate beta as a multiple of 12 months.

This is what the code looks like:

bm_col = '^GSPC'  window = 12 * 3    def monthly_beta(df, bm_col, window):      month_end_list = df.resample('ME', closed='left', label='right').last().index.to_list()      beta_df = pd.DataFrame( )      for t,dt in enumerate(month_end_list):          beta_monthly_df =  beta(df[dt:month_end_list[t + window]],bm_col)          beta_df = pd.concat([beta_df,beta_monthly_df],axis =1)          beta_df = beta_df.rename(columns={bm_col: month_end_list[t + window]})          if month_end_list[t + window] == month_end_list[-1]:              print(t,dt,month_end_list[t+window])              break      return beta_df    beta_df =  monthly_beta(px_df, bm_col, window)  beta_df.tail()

Computing monthly updated betas using a 3-year rolling window to track how systematic risk evolves over time.

  1. Create month-end list: Resample price data to month-end dates for period boundaries.
  2. Set rolling window: Use 36 months (12 × 3) for beta stability while capturing regime changes.
  3. Loop through months: For each month, calculate beta using data from that month through +36 months ahead.
  4. Build beta DataFrame: Each column represents betas as of a specific month-end date.
  5. Stop at data end: Break loop when reaching the last available month.

This generates a dataFrame with betas for each stock (rows) calculated at each month-end (columns), showing how risk profiles change across market cycles.

Once we have calculated beta for the entire market, we will update the arctic database.

monthly_beta_df = beta_df.T  for ticker in library_SP500.list_symbols():      df =  library_SP500.read(ticker).data      df = df.dropna(subset =['Close','High','Low','Open' ])      df = pd.concat([df,monthly_beta_df[ticker]],axis =1)      df = df.rename(columns={ticker:'beta'})      df['beta'] = df['beta'].ffill()      df = df.loc[:, ~df.columns.duplicated(keep='last')]      library_SP500.write(ticker, df)

Writing rolling 36-month beta values to Arctic database for all S&P 500 tickers, ensuring each stock record includes a monthly-updated systematic risk metric.

  1. Transpose beta_df: we convert from ticker-indexed to date-indexed format with tickers as columns (monthly_beta_df).
  2. Loop through all tickers: For each symbol in library_SP500.list_symbols(), read complete ticker data from the database.
  3. We drop missing OHLC data: Remove rows with NaN in price fields to ensure clean data before beta merge.
  4. We concatenate beta monthly_beta_df[ticker] column with ticker DataFrame on matching dates (axis=1). We rename the concatenated column to beta; apply forward-fill to propagate values through data gaps.
  5. We remove duplicate columns and write the updated DataFrame back to the library.

AllS&P500stockrecordsinArcticdatabasenowcontainmonthly-rollingbetavalues forward-filled values ensure continuous coverage from the first available beta date forward.

Calculating average beta by sector

Next, we will concatenate the beta_df with the sector column to the sp500_info_df. We will perform some sector analysis.

df_SP500_beta = pd.DataFrame()  df_SP500_beta = pd.concat([sp500_info_df['sector'], beta_df], axis=1)  sector_beta_pivot = pd.pivot_table(df_SP500_beta, values=list(beta_df.columns),                                     index= ['sector'], aggfunc="mean").T  sector_beta_pivot.plot(figsize=(15,4),title = 'Beta by Sector')  plt.legend(loc='lower left')

Aggregating individual stock betas to the sector level and plotting time series can help identify which sectors have high or low beta.

This is how we do it:

This produces a line chart revealing which sectors exhibit higher or lower (e.g., Technology, Utilities) systematic risk and how sector betas shift during bull and bear markets.

Figure 7.6: Average beta by sector

Figure 7.6: Average beta by sector

The above graph sets mathematical foundations on something we named in . Cyclical sectors such as technology and consumer cyclicals are also often referred to as high beta. Conversely, defensive sectors, such as utilities and consumer defensive, trail the benchmark with sub-1 betas.

This paints an interesting dichotomy of the market. Go short defensive sectors in a bull market and go long cyclicals. This echoes something we saw in . Defensive sectors such as consumer staples and utilities are impervious to the economic cycle. They do not go down during downturns on the basis that people need food and electricity. They do not participate in economic upturns either. Meanwhile, cyclical sectors, such as consumer cyclicals and technology, move with the economic cycles.

Next, let's calculate the daily log and cumulative returns across sectors. Note that we will calculate a simple lateral average, otherwise referred to as equal weight.

Calculating average returns by sector

We have calculated the average beta by sector. Next, we will calculate returns in absolute terms and relative to the index for all the constituents. We will then calculate the average by sector. Let's start with the daily log and then calculate the cumulative returns.

daily_log_returns = np.log(px_df/px_df.shift(1))  daily_rel_log_returns = daily_log_returns.sub(daily_log_returns[bm_col],axis=0)  cum_returns_abs = daily_log_returns.cumsum().apply(np.exp)-1  cum_returns_rel = daily_rel_log_returns.cumsum().apply(np.exp)-1

Computing daily log returns in both absolute and benchmark-relative terms for performance attribution.

Here is how it works:

This results in four DataFrames, enabling analysis of both absolute performance and alpha generation—excess returns over the S&P 500 index.

Next we will calculate and chart absolute and relative returns by sector.

sector_abs_returns = pd.concat([sp500_info_df['sector'],cum_returns_abs.T],axis=1)  sector_avg_abs_returns = sector_abs_returns.groupby('sector').mean().T  sector_avg_abs_returns.plot(figsize=(15,4), title='Sector Average Absolute Returns vs S&P 500', grid=True)  plt.legend(loc='upper left')  plt.show()    sector_rel_returns = pd.concat([sp500_info_df['sector'],cum_returns_rel.T],axis=1)  sector_avg_rel_returns = sector_rel_returns.groupby('sector').mean().T  sector_avg_rel_returns.plot(figsize=(15,4), title='Sector Average Relative Returns vs S&P 500', grid=True)  plt.legend(loc='upper left')  plt.show()

Comparing sector-level absolute and relative returns to identify which sectors out/underperform the S&P 500 benchmark.

Here are the steps:

  1. Merge sector labels with cumulative return time series to organize returns by sector classification.
  2. Group returns by sector and calculate mean performance across all constituent stocks within each sector.
  3. Plot both absolute returns (total gains) and relative returns (excess over benchmark) for sector comparison.

This produces two visualizations showing sector performance trends—absolute chart reveals total returns while relative chart highlights which sectors generated alpha (outperformance) versus the benchmark.

Figure 7.7: Sector cumulative average absolute returns

Figure 7.7: Sector cumulative average absolute returns

As Warren Buffet famously said: "A rising tide lifts all ships." All sectors have gone up in tandem. Apart from technology, which has dwarfed the pack, it is quite difficult to sort the winners from the losers.

Next, we will display the equal-weight cumulative relative returns by sector.

Figure 7.8: Sector average relative returns

Figure 7.8: Sector average relative returns

Relative returns paint a better picture of the performance of sectors over time. Next, let's build custom indices by beta.

Calculating average returns by sector

Now that we have calculated returns and beta by sector, let's put all the pieces together and build custom indices: high and low beta. High beta index will be the simple average (equal weight) of the two highest beta sectors: "technology" and "consumer cyclical." Conversely, low beta will be the simple average (equal weight) of the two lowest beta sectors: "utilities" and "consumer defensive."

Let's publish the code:

high_beta_sectors = ['Consumer Cyclical', 'Technology']  low_beta_sectors = ['Consumer Defensive', 'Utilities']  high_beta_rel_returns = sector_rel_returns.loc[sector_rel_returns[      'sector'].isin(high_beta_sectors)].drop(columns=['sector'])  low_beta_rel_returns = sector_rel_returns.loc[sector_rel_returns[      'sector'].isin(low_beta_sectors)].drop(columns=['sector'])  high_beta_abs_returns = sector_abs_returns.loc[sector_abs_returns[      'sector'].isin(high_beta_sectors)].drop(columns=['sector'])  low_beta_abs_returns = sector_abs_returns.loc[sector_abs_returns[      'sector'].isin(low_beta_sectors)].drop(columns=['sector'])  beta_cum_returns = pd.DataFrame()  beta_cum_returns[bm_col]= cum_returns_abs[bm_col].copy()  beta_cum_returns['high_beta_rel_avg'] = high_beta_rel_returns.mean(axis=0)  beta_cum_returns['low_beta_rel_avg'] = low_beta_rel_returns.mean(axis=0)  beta_cum_returns['high_low_beta_rel_avg'] = beta_cum_returns['high_beta_rel_avg'].sub(      beta_cum_returns['low_beta_rel_avg'] )    beta_cum_returns['high_beta_abs_avg'] = high_beta_abs_returns.mean(axis=0)  beta_cum_returns['low_beta_abs_avg'] = low_beta_abs_returns.mean(axis=0)  beta_cum_returns['high_low_beta_abs_avg'] = beta_cum_returns['high_beta_abs_avg'].sub(      beta_cum_returns['low_beta_abs_avg'] )

Analyzing performance differences between cyclical high-beta sectors (Technology, Consumer Cyclical) and defensive low-beta sectors (Utilities, Consumer Defensive).

How it works step by step:

  1. Define sector groups: Select high-beta sectors (Technology, Consumer Cyclical) and low-beta sectors (Utilities, Consumer Defensive).
  2. Filter returns: Extract returns for stocks in each sector group from sector_rel_returns and sector_abs_returns.
  3. Calculate averages: Compute mean returns across all stocks in high-beta and low-beta groups.
  4. Build comparison DataFrame: Combine benchmark, high/low beta averages, and their spread (high minus low).

Next, let's plot the relative and absolute returns:

beta_cum_returns[[bm_col,'high_beta_rel_avg','low_beta_rel_avg', 'high_low_beta_rel_avg']].plot(      grid = True, figsize= (15,4), title = 'High Low Beta Sectors Relative Returns')    beta_cum_returns[[bm_col,'high_beta_abs_avg','low_beta_abs_avg', 'high_low_beta_abs_avg']].plot(      grid = True, figsize= (15,4), title = 'High Low Beta Sectors Absolute Returns')

Comparing high-beta versus low-beta sector performance.

  1. Plotting relative returns: Chart showing relative performance versus S&P 500 for both groups and their difference.
  2. Plotting absolute returns: Chart showing total returns including benchmark for comparison.
  3. Result: Two charts demonstrating beta factor premium—high-beta sectors outperform in bull markets but underperform in bear markets, while low-beta sectors provide defensive characteristics. The spread line shows the cyclical nature of the beta factor.

The absolute chart shows the returns without adjustment for the benchmark returns. This is why it looks so much better. The relative chart shows the excess returns over the benchmark.

Let's start with the absolute returns:

Figure 7.9: High and Low Beta v. S&P 500 absolute returns

Figure 7.9: High and Low Beta v. S&P 500 absolute returns

The high beta index has really outperformed the low beta one. Drawdowns, however, are not commercially viable. Investors will not sit through routine –20~40%.

Next, let's publish the relative returns:

Figure 7.10: High and Low Beta versus S&P 500 relative returns

Figure 7.10: High and Low Beta versus S&P 500 relative returns

On the surface, it may seem like the net or high beta index underperforms the S&P 500 index. Those are the excess returns over the index. If the high and net beta underperformed the index, this would be negative, like the low beta index. Bottom line, these are promising excess returns. It is important to understand that high betas will take returns to exhilarating highs but also gut-wrenching lows.

Before we start loading the truck on high beta indiscriminately, we should probably be aware of the low beta anomaly. According to the Capital Asset Pricing Model (CAPM), risk is positively correlated with returns. This means high beta stocks carry inherently more risk and should generate high returns. In practice, low beta stocks tend to generate high returns in long-only portfolios over long durations.

In a long short context, however, things tend to move differently. When markets are in bullish territory, go long high beta and short low beta. In bearish markets, keep the same stocks, switch sides. The trick is to negotiate stock changes in regime and sideways markets. The solution may not be stock selection but risk management as we will see in .

Next, let's see what happens when we pick the top and bottom 5 betas.

Calculating the average returns for the top and bottom 5 betas

In the following block of code, we drill down to the individual stock level. We select the top and bottom 5 betas at the end of every month. We calculate the average returns for the following month. The portfolio is rebalanced every month. We do not factor in transaction costs.

pos_count = 5  top_beta_returns_list = []  bottom_beta_returns_list = []  beta_returns_df = pd.DataFrame()    for t in range(1,len(beta_df.columns)):      period_start = beta_df.columns[t-1]      period_end = beta_df.columns[t]      top_beta = beta_df[period_start].nlargest(n = pos_count).index      bottom_beta = beta_df[period_start].nsmallest(n = pos_count).index      top_beta_returns = daily_rel_log_returns.loc[period_start:period_end, top_beta].iloc[1:].mean(axis=1)      bottom_beta_returns = daily_rel_log_returns.loc[period_start:period_end, bottom_beta].iloc[1:].mean(axis=1)      top_beta_returns_list.append(top_beta_returns)      bottom_beta_returns_list.append(bottom_beta_returns)    top_beta_returns_df = pd.concat(top_beta_returns_list)  bottom_beta_returns_df = pd.concat(bottom_beta_returns_list)    beta_returns_df[f'top_{pos_count}_beta']  = top_beta_returns_df  beta_returns_df[f'bottom_{pos_count}_beta']  = bottom_beta_returns_df  beta_returns_df[f'net_{pos_count}_beta']  = beta_returns_df[f'top_{pos_count}_beta'].sub(beta_returns_df[f'bottom_{pos_count}_beta'])  beta_returns_df[bm_col] = daily_log_returns[bm_col]  beta_returns_df = beta_returns_df.cumsum()  beta_returns_df.plot(figsize=(15,4), grid = True, title=f'Top vs Bottom Beta {pos_count} Stocks Cumulative Returns'  )

Testing monthly rebalanced strategy: long top-beta stocks, compare to bottom-beta stocks to measure beta factor premium.

This is how we do it step by step:

  1. Each month, select the 5 highest-beta and 5 lowest-beta stocks based on the previous month's rankings.
  2. Calculate mean daily relative log returns across the 5 stocks in each portfolio during the holding period.
  3. Concatenate all daily returns across holding periods into a continuous time series.
  4. Compute the long-short spread (top minus bottom), add benchmark returns, and apply cumulative sum.
  5. Plot cumulative performance showing how each strategy performs over time versus the benchmark.

This code produces the following chart:

Figure 7.11: Top/Bottom 5 Betas versus S&P 500 cumulative relative returns

Figure 7.11: Top/Bottom 5 Betas versus S&P 500 cumulative relative returns

That tame low volatility line? This is just the S&P 500. To say that this strategy is rock n' roll would be the understatement of the year. Going long top betas and shorting the bottom ones will deliver spectacular results in strong bull markets. It will however be devastating in any other times. This illustrates the low beta anomaly mentioned earlier in this chapter. High beta is positively correlated with risk, but not always returns.

Top and bottom betas may be too volatile or too defensive. Perhaps lower and higher decile betas would have better return profiles. But for now, let's see what happens when we select for the biggest rises and drops in betas.

Simulating a beta momentum strategy

Going long top and short bottom betas is not a commercially viable strategy. The volatility is too high to be an investable product. Let's take a different approach. see how rising and falling beta momentum would fare. In the next block of code, we will select the top 5 largest increase and decrease in beta, calculate the average return and rebalance the portfolio on a monthly basis. The objective is to see if returns are positively correlated with rising or falling beta.

First, we publish the code then we will explain it step by step:

beta_diff = beta_df.pct_change(axis=1, fill_method = None).dropna(axis=1, how = 'all')  top_beta_diff_returns_list = []  bottom_beta_diff_returns_list = []  beta_returns_diff = pd.DataFrame()    for t in range(1,len(beta_diff.columns)):      period_start = beta_diff.columns[t-1]      period_end = beta_diff.columns[t]      top_beta = beta_diff[period_start].nlargest(n = pos_count).index      bottom_beta = beta_diff[period_start].nsmallest(n = pos_count).index      top_beta_diff_returns = daily_rel_log_returns.loc[period_start:period_end, top_beta].iloc[1:].mean(axis=1)#.sum()      bottom_beta_diff_returns = daily_rel_log_returns.loc[period_start:period_end, bottom_beta].iloc[1:].mean(axis=1)#.sum()      top_beta_diff_returns_list.append(top_beta_diff_returns)      bottom_beta_diff_returns_list.append(bottom_beta_diff_returns)  top_beta_diff_returns_df = pd.concat(top_beta_diff_returns_list)  bottom_beta_diff_returns_df = pd.concat(bottom_beta_diff_returns_list)  beta_returns_diff[f'rising_{pos_count}_beta']  = top_beta_diff_returns_df  beta_returns_diff[f'falling_{pos_count}_beta']  = bottom_beta_diff_returns_df  beta_returns_diff[f'net_{pos_count}_beta']  = beta_returns_diff[f'falling_{pos_count}_beta'].sub(beta_returns_diff[f'rising_{pos_count}_beta'])  beta_returns_diff[bm_col] = daily_log_returns[bm_col]  beta_returns_diff = beta_returns_diff.cumsum()  beta_returns_diff.plot(figsize=(15,4), grid = True, title=f'{pos_count} Fallling V. Rising Betas Cumulative Returns'  )

Backtesting beta momentum strategy (rising versus falling betas). This code tests a strategy based on beta changes rather than absolute beta levels, targeting stocks with rising or falling systematic risk.

Let's go through the code step by step:

  1. Calculate month-over-month percentage change in beta values for all stocks to identify momentum in risk profiles.
  2. Each month, select the 5 stocks with the largest beta increases (rising risk) and 5 with the largest beta decreases (falling risk).
  3. Calculate mean daily relative log returns for each portfolio during the holding period.
  4. Concatenate daily returns into a continuous time series, compute the long-short spread (falling minus rising), and add the benchmark.
  5. Plot cumulative performance to determine if betting against beta momentum (short rising, long falling) generates alpha.

Result: Chart showing high-beta stocks typically outperform in bull markets while low-beta stocks provide downside protection in bear markets.

Let's publish the chart:

Figure 7.12: Beta momentum versus S&P 500 cumulative relative returns

Figure 7.12: Beta momentum versus S&P 500 cumulative relative returns

Fast rising beta stocks look like the kind of stocks that would be featured in Jim Cramer's popular show Mad Money. They have a compelling story in theory but tend to be gravitationally challenged in practice.

Rising beta stands out as being inversely correlated with returns. The faster beta rises, the worse the returns. This is a counterintuitive finding. It flies in the face of CAPM. Classic financial education postulates that returns should be correlated with risk. Rising risk is inversely correlated with returns.

Conversely, falling betas are positively correlated with excess returns. This means that issues whose sensitivity to the market decreases tend to outperform over time. In other words, once issues fade from the limelight, volatility abates, and they perform better. This rudimentary beta momentum analysis has inherent flaws such as survivorship bias. It is by no means an exhaustive study. It is, however, worth digging deeper and extending to other markets.

In summary, beta is a double-edged sword. Market participants who fail to understand it can find themselves on the wrong side of a trade. The easiest way to manage the risk associated with beta is to size positions accordingly. Inverse beta is a popular method that equalizes risk. For example, let's take two stocks A and B with betas of 1.25 and 0.8 (0.8 is the inverse of 1.25), then assign them raw weights of 0.5% fixed risk. If we want to neutralize risk using the inverse beta, their respective size would look like this:

Now, let's assume we would want a net beta-neutral portfolio with A on the long side and B on the short one.

If we want to neutralize the beta, i.e. a net beta neutral portfolio, we need to have a bigger short book. This mechanically leads to a negative net exposure of -0.225%. This explains why seemingly market neutral portfolios, with net exposure hovering around 0, often have a residual bullish tilt. We are now touching on an important concept that we will revisit in the next chapter on the Long/Short toolbox: net exposure versus net beta. For now, let's summarize what we have learned in this chapter.

Summary

This chapter provided a structured process for boiling down a broad investment universe to a practical list of securities suitable for short selling. Because of the inherent asymmetrical risk in short selling, the chapter emphasized strict filtering based on liquidity, crowding, corporate actions, fundamentals, valuations, and beta.

The first filter is liquidity, described as the currency of the bear markets. Liquidity often evaporates during selloffs, so short positions must be chosen with the exit in mind before entering. Small-cap stocks, despite offering many theoretical opportunities, are generally excluded due to thin trading volume, limited borrow availability, and painful short squeeze risk. A minimum of roughly $1 million in daily trading value is a practical threshold.

Next, the chapter highlights "crowded shorts" as perilous. When borrow utilization rises above about 50%, the risk of recalls and forced short cover increases sharply, while downside sensitivity to market declines weakens. Short ratio and short as a percentage of float are briefly described.

The chapter then examines dividend and buyback policies. High‑dividend stocks are sometimes defensive underperformers and are potential short candidates, since elevated yields often signal ex-growth fundamentals. In contrast, companies actively conducting share buybacks will artificially support their stock price, making them unattractive for shorts.

The fundamental analysis section explains that stocks fall for three main reasons: mispricing, sector rotation, or company-specific deterioration. The focus is on identifying value traps: firms with superficially cheap valuations but declining profitability or weakening balance sheets.

Finally, the chapter details valuation and beta effects. High valuations alone are insufficient for shorting; the key warning sign is expensive multiples paired with slowing earnings momentum. Beta analysis shows that sector betas vary widely across regimes. Rising betas tend to precede underperformance, while falling betas often precede outperformance, helping guide long/short portfolio construction.

In the next chapter, we will expand on beta and revisit concepts such as gross and net exposure that we saw in on position sizing.

Get this book's PDF copy, code bundle, and more

Scan the QR code (or go to ). Search for this book by name, confirm the edition, and then follow the steps on the page.

Image

Image

Note: Have your invoice handy. Purchases made directly from the Packt website don't require an invoice.

Назад: Chapter 6: Position Sizing: Money Is Made in the Money Management Module
Дальше: Part 3: The Long/Short Game: Portfolio Construction