12 Measuring and reporting results

Reporting consists of evaluating our ML system’s performance based on its final goal and sharing the results with teammates and stakeholders. In chapter 5, we introduced two types of metrics: online and offline metrics, which generate two types of evaluation: offline testing and online testing. While offline testing is relatively straightforward, online testing implies running experiments in real-world scenarios. Typically, the most effective approach involves a series of A/B tests, a crucial procedure in developing an efficient, properly working model, which we cover in section 12.2. This helps capture metrics that either directly match or are highly correlated with our business goals.

Measuring results in offline testing (sometimes called backtesting) is an attempt to assess what effect we can expect to catch during the online testing stage, either directly or through proxy metrics. Online testing, however, is often a trickier story. For example, recommender systems rely heavily on feedback loops, so the training data depends on what we predicted at the previous timestamps. Moreover, we don’t know exactly what would have happened if we had shown item X or Y to user A instead of item Z, which appeared in historical data at timestamp T.

Finally, when the experiment is done, we are ready to report the effect of the model on the business metrics we are interested in. What do we report? How do we present the result? What conclusion should we make about the further steps of system elaboration? In this chapter, we answer questions that might arise during the process.

As we already mentioned, metrics are interconnected within a hierarchical structure. There are proxy metrics for other proxy metrics for metrics of our interest (please see chapter 5). Consequently, there are plenty of ways to conduct the offline evaluation.

Prediction evaluation is the fastest way to check the model’s quality and iterate through different versions, but it is also usually the farthest from business metrics (the relationship between prediction quality and online metrics is often far from ideal).

The offline evaluation procedure is trustworthy and valuable for assessing quality. However, there is rarely a direct connection between offline and online metrics, and learning how an increase in offline metrics A, B, and C affects online metrics X, Y, and Z is a regression problem by itself. To bridge the gap between offline and online metrics and make offline testing more robust, we could gather a history of online tests and online metrics and related offline metrics to calculate the correlation between offline and online results.

Our goal is to avoid overbooking (missed revenue due to offering too many discounted tickets) and minimize unsold seats (lost revenue). Assume the number of tickets sold reflects the passengers’ actual demand and that we adjust prices at the beginning of each day based on our forecast. We are okay with this rough estimation for the sake of illustration.

The actual number of tickets sold during the 4 days is [120, 90, 110, 80]. The predicted ticket sales are

The ticket prices for each day are [$200, $220, $230, $210].

The total revenue (we choose the minimum of sold and forecasted tickets for each day) is

Certainly, we’ve oversimplified the real-world context where the model makes forecasts and affects decision-making (e.g., how many tickets to release for sale, how to adjust prices dynamically). Here, the whole transition is done by multiplying by price and, optionally, subtracting flight costs.

Nonetheless, this example illustrates how we can transition from model performance metrics, which may not fully capture the practical effect of predictions, to business indicators like revenue. These business metrics can be reported to the team before running A/B tests.

A good example of this kind of environment would be time-series forecasting, which is relatively simple to simulate thanks to the availability of labels, auxiliary information (e.g., historical costs or turnover of goods in stock), and disambiguation in the outcome. That is usually not the case for online recommendation systems, such as news, ads, or product recommendations, where we deal with partly labeled data; we know only impressions of content items (views, clicks, and conversions) that we showed to a particular user, while those that we did not show remain without user feedback as there is always a chance that had we produced an alternative order of impressions, everything might have been very different.

Let’s review how such an environment could be created in the case of a recommender system.

Sometimes online recommendation systems are solved via the multi-armed bandits and contextual bandits approaches (usually, this subset of recommender systems is called real-time bidding, but in a nutshell, its goal remains the same—to provide the best pick from a variety of options to maximize a specific reward). There is an agent, which is our algorithm, that “interacts” with an environment. For example, it consumes information about user context (their recent history) and chooses which ad banner to show to maximize CTR, conversion rate (CVR), or effective cost per mile (eCPM). This is a typical reinforcement learning problem with a great feedback loop influence: previously shown ads change the future user behavior and give us information about their preferences.

The so-called “replay” algorithm is a simple (but not the most intuitive) way to build a simulation for real-time recommendations, derived from the paper “Unbiased Offline Evaluation of Contextual-bandit-based News Article Recommendation Algorithms” by Lihong Li et al. (). Here we summarize the paper due to its practicality:

A visual example is divided into several pieces in figures 12.1 to 12.5 that demonstrate the work of an unbiased estimator, with figure 12.1 displaying the real history of events and figures 12.2 to 12.5 showcasing simulated events based on historical data.

For instance, after building a new version of a search engine pipeline, we can either ask a group of experts to estimate the relevance of search results from 1 to 5 for some benchmark queries or choose which output is more relevant, comparing the old and new versions of search results. The second (comparative) formulation tends to produce a more robust evaluation as it excludes the subjectiveness of score estimates. In the case of generative large language models, human evaluation and hybrid approaches (auxiliary “critic” model, trained on human feedback) are the most popular options to measure the quality of generations.

While human evaluation has low throughput and high costs that may impede frequent use, its capacity to yield highly precise and trustworthy assessments (compared to fast but limited automated methods) cannot be downplayed.

After formulating the hypothesis and our expectations, we choose a splitting strategy determined mainly by the nature of the system we deploy and how it will affect the existing processes.

Next, we decide on the statistical criteria and conduct simulation tests, both A/A (where we compare the same system against itself to check type I error rates) and A/B (where we simulate a desired effect through the addition of the noise of a desired magnitude), as a sanity check on historical data. This helps us confirm whether we meet the predefined type I and type II error levels. A comprehensive description of that is beyond the scope of this book, but if you are interested in more details, a great starting point would be Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing () by Ron Kohavi et al.

Finally, we have everything we need to launch our A/B test. Remember that the design of the experiment is fixed beforehand and cannot be changed on the go. Next we briefly outline the basic stages of A/B testing.

To zoom in, let’s examine hypothetical pricing systems applied in different domains:

When splitting by a nontypical key, aiming for group (bucket) similarity is essential. For instance, if you’re compelled to divide geographically, the chosen areas should be similar and possess comparable economic indicators (they should match each other).

Finally, there is a more advanced splitting strategy called switchback testing. This technique divides data into region-time buckets and randomly and continuously switches between different models (see figure 12.7). It ensures that each region will be in both the control group and the testing group for an approximately equal amount of time. For further details, see “Design and Analysis of Switchback Experiments” by Iavor Bojinov et al. ().

Every experiment usually has three kinds of metrics:

In short, statistical criteria are used to quantify the probability of receiving specific results under specific assumptions. For example, the probability of receiving t-statistics equal to or greater than 3 generally will be less than 0.01 (if there’s no difference between the groups and assumptions for which the t-test is met).

The most common statistical test used in A/B testing is the t-test we just mentioned. It has many modifications. For example, Welch’s t-test is relatively easy to interpret and is similar to Student’s t-test, but it can be used even when the variances of two groups are not equal. The statistic for the Welch t-test is calculated as follows:

By statistical significance, we mean that the probability of capturing a specific value of a statistic or even a more extreme one under the assumption that the null hypothesis is true (we call it p-value) is less than significance level α (effectively, the same as type I error; typically, α = 0.05). P-value, along with the significance level, is a way to standardize the results of any statistical test and map them to a single reference point rather than to define critical values for each test in particular.

While there are many different statistical tests available, the choice of a particular test depends on the context of the experiment and the nature of the data. That said, providing a detailed guide on choosing the most appropriate test is beyond the scope of this book. For a more detailed discussion on selecting the right statistical test for A/B testing, we once again recommend Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing () by Ron Kohavi et al.

The t-test is widely used because it is a robust test that can handle a variety of situations, it is easy to implement, and it provides results that are easy to interpret. However, the choice of a specific test should always be made considering the type of data you are dealing with, the nature of your experiment, and the assumptions each test requires.

A simulated A/A test involves randomly sampling two groups with no incurred difference and applying a chosen test statistic. We repeat these actions many times (say 1,000 or 10,000 times) and expect the p-value to follow a uniform distribution (if everything is correct). For instance, in 5% of simulations, the test will reject the null hypothesis for α = 0.05. The exact percentage of cases when the test does catch a difference for two groups is a simulated type I error rate.

A simulated A/B test does a similar thing, but now we add an uplift of a specific size (usually MDE of interest) to the second group B (we assume that groups A and B are the same size as they will be during the real A/B test). Again, we apply our test statistic and run it 1,000 to 10,000 times. After that, we reexamine the distribution of resulting p-values and count how many times we rejected the null hypothesis (“the p-value passed the threshold”). The percentage of cases when the chosen test rejects the null hypothesis is an estimation of sensitivity (1 – β), which is the probability that the test catches the difference if there is one. If we subtract sensitivity from 1, we get a type II error rate (the probability of ignoring the difference between A and B when there is one). If calculated type I and II error rates fit predefined levels, then the statistical test is picked properly, and the sample size is estimated correctly (see figure 12.8).

There are different ways to tackle this problem, but since this is a narrow topic that is out of the scope of this book, we will not cover it in detail. For your convenience, however, we list them here so you can use them as keywords for a more in-depth look:

Also, when conducting an A/B test, we should consider fairness and biases among different segments of users, which can affect target and auxiliary metrics. This analysis is similar to what we covered in depth in chapter 9. For further reading, we recommend “Fairness through Experimentation: Inequality in A/B Testing as an Approach to Responsible Design” by Guillaume Saint-Jacques et al. ().

Figure 12.9 shows a funnel representing the range of effects that we cannot detect with statistical significance. If the uplift moves outside this funnel, the effect is significant. Please note that without a specific design, you cannot peek into the test as many times as you wish until you see the desired results. For further reading on the subject, we recommend “Peeking Problem—The Fatal Mistake in A/B Testing and Experimentation” by Oleg Ya () and “Sequential A/B Testing, Further Reading Choosing Sequential Testing Framework—Comparisons and Discussions” by Mårten Schultzberg et al. ().

This funnel can be calculated based on the MDC. Here, we need to solve the same equation we saw earlier—however, this time for different time periods—but we cannot make a decision until a period of time that we calculated in advance has passed. You could ask: what’s the point of using it then? The answer is to have an emergency stop criterion to abort the experiment if results are outside the funnel and negative—the least expected, desirable, and probable outcome. For any sequential testing framework mentioned by the link earlier (e.g., mSPRT), we can make a decision as soon as results are outside the funnel or the experiment time has ended.

In general, an experiment should run according to the predetermined design unless there’s a catastrophic drawdown in key metrics (a significant negative effect) that would result in a meaningful financial loss had the experiment continued.

On the other hand, if you see positive effects in key metrics, the best practice is to stick with the initial plan and let the experiment run its full course. It allows you to confirm that these positive effects are sustained over time and are not a result of short-term fluctuations.

What about those borderline cases where some metrics are positive, some are negative, and others hover around the MDE? It’s crucial during the experimental design phase to decide how you will interpret these mixed outcomes and set priorities among the different key metrics.

If you can’t wait to see the final results, consider using methods like sequential testing, which allows for repeated significance testing at different stages of the experiment. These methods come with their own assumptions and considerations, so make sure you understand them fully before proceeding.

Deciding whether to stop or continue an experiment is a complex task that requires careful consideration of the metrics, potential outcomes, and their effects. Document these decisions in your experimental protocol, including criteria for possible early termination.

Not every experiment is successful. According to our experience, it is quite good to have 20% of A/B tests show a statistically significant difference (recall that usually if the false positive rate is 5%, the margin is 15%). And remember: if the test was statistically significant, it doesn’t mean that the results are positive or massive.

Suppose we decide that the A/B test is successful and the effect is significant. In that case, we have two options:

If the experiment is successful, a debrief document will include suggestions for similar experiments in other products. In case of failure, it is important to discuss what should be done differently in future tests and what mistakes should be avoided, as well as to develop new control metrics to prevent similar failures in the future earlier during an experiment.