14 Monitoring and reliability

This means that the model will inevitably be prone to degradation over time, as well as experience cases of unexpected behavior, due to the difference between the data it was trained on and the data it will receive in real-life conditions.

These are risks that cannot be eliminated but that you need to be prepared for and able to mitigate so that your system remains effective and valuable to your business over the long term.

In this chapter, we’ll cover the essence of monitoring as part of ML system design and the sources of problems your ML model may encounter as it operates. We will also explore how you want to monitor typical cases in system behavior change and how you need to respond to them.

Without proper monitoring, even the most well-trained and accurate models can begin to degrade over time due to changes in the data they are working with. This is especially evident during times of significant change, such as the recent COVID-19 pandemic, where models used for tasks like predicting item availability, credit risk, and many more had to face enormous challenges. By implementing monitoring systems, we can identify and address problems with data quality and ensure that our models continue to make reliable predictions.

If somebody asks us to make a simplified high-level review of a generic ML system, it will most likely look like figure 14.1.

However, the probability that such a combination of factors can happen is extremely low, as we are always dealing with a multitude of inputs that can change with varying degrees of probability and effects on the model. Let’s break down all four components from figure 14.1 to understand what challenges we may encounter.

Another thing is that the system we have might be probabilistic/generative by nature. The dialogue with ChatGPT (as of July 2024) shown in figure 14.2 is a rather illustrative example.

The screenshot shows four different (though similar) replies from ChatGPT. While behaving within a specific range is something we would expect, not having the same output for the same input makes it more challenging to check the health of the system. Which reply is acceptable, and which is not? As of January 2023, our experience with ChatGPT showed that it is not rare to receive several answers in a row that are completely different or contradict each other, although this was partially fixed in later versions.

Another example would be a lag in time, causing shifts in understanding of the world to happen much later than the event that prompted these shifts. A good example here is the Madoff investment scandal, which was revealed only decades after the original shenanigans. For many years, signals coming from it were reviewed as signals of a successful hedge fund, but later, it turned out to be a Ponzi scheme. In other words, for many years, the incoming data was labeled as benevolent, only eventually turning out to be fraudulent.

If the model was designed to support a specific business process, it may no longer be effective if the process changes over time. For example, say the model was set up to prioritize customer support for the incoming tickets based on the future revenue of those users, and the new process is to focus on the quickest tickets to solve. Similarly, if the model is not being used in the way it was intended or is not being used at all, it may not be able to deliver the expected business value. To mitigate these risks, it is important to continuously monitor the performance of the ML model and the business value it is delivering and to make adjustments as needed to ensure that the model is able to support the desired business objectives.

In the following sections, we cover the most common cases of what can go wrong, focusing primarily on the first three stages. We discuss how to monitor and detect them, what to do if something has changed, and how to incorporate this into the design document so that you are able to plan beforehand and not after something terrible happens. Before we dive into potential malfunctions and ways to handle them, let’s talk about the main components of monitoring ML systems. In her article “Monitoring ML Systems in Production. Which Metrics Should You Track?” (), Elena Samuylova highlights four integral components of ML system monitoring, represented as a pyramid in figure 14.3.

Monitoring the health of an ML system is an important aspect of production deployment. It is essential to ensure that the system is up and running and to track its performance characteristics to meet service-level objectives. Many of the same monitoring practices used in traditional software systems—such as application and infrastructure monitoring, alerting, and incident management—can also be applied to ML systems. These practices can help ensure the health and performance of the ML system and allow for timely intervention if any problems arise. By using the tools and practices developed for traditional software systems, it is possible to effectively monitor and manage the health and performance of an ML system in production.

There are various metrics that can be monitored, depending on the deployment architecture of the ML system. These can include service usage metrics such as the total number of model calls, requests per second, and error rates, as well as system performance metrics such as uptime, latency, cold start time, error rate, and resource utilization metrics such as memory and GPU/CPU utilization. It is important to carefully select a few key metrics that quantify different aspects of service performance, often referred to as service-level indicators. These metrics can help identify problems with the system and allow for timely intervention to prevent failures or degraded performance.

It is important to log events and predictions with their timestamps to monitor and debug your ML system. In the context of an ML system, we recommend recording data on every prediction if this is feasible and fits the budget (there are cases when storing every intermediate model output may drastically affect the profit margin), including the input features and the model output. This will allow you to track the performance of the model and identify any problems that may arise. As a general rule, you should log every prediction unless there are specific circumstances, such as privacy regulations or the use of edge models on a user’s device, that prevent you from doing so. In these cases, it may be necessary to develop workarounds to collect and label some of the data.

In addition to prediction logs, it is also important to have software system logs—timestamped events that provide information on what is happening within your ML-powered application. Having access to these logs can be helpful for debugging problems that may arise and for improving the model’s performance. There are multiple tools available to help you centralize and analyze these logs, like Prometheus and Grafana among open source tools or AWS Cloudwatch or Datadog among cloud services.

Logs and metrics should be stored in systems, allowing for exploration and alerting. There are multiple software solutions for this matter. Good examples of self-hosted systems are the ELK stack (Elasticsearch, Logstash, and Kibana) for logs and Prometheus or Victoria Metrics for metrics. Similar systems are often provided by all-in-one cloud providers (e.g., AWS Cloudwatch) and more dedicated companies like Datadog. As we mentioned in chapter 13, connecting your ML system to a proper logs/metrics toolset is a must, though inexperienced engineers often underestimate the importance of this step.

One of our favorite examples is when a job processes zero rows and reports success (expected green square in the UI reporting the successful execution of the extract, transfer, and load job), so everyone is happy—until 1 week passes and we find out that there was a problem in the upstream data sources.

In addition, the data processing stage of an ML pipeline can also be prone to problems. This can include problems with the data source, data access problems, errors in the SQL or other queries used to extract the data, infrastructure updates that change the data format, and problems with the code used to calculate features. In batch inference systems, detecting and correcting these problems is possible by rerunning the model. However, in high-load streaming models, such as those used in eCommerce or banking, the effect of data processing problems can be more severe due to the large volume of data being processed and real-time decisions made based on model outputs.

Sometimes, these outages may only affect a subset of the data, making the problem harder to detect. Additionally, a corrupted source may still provide data, but it may be incorrect or misleading. In these cases, it is crucial to keep track of unusual numbers and patterns to identify potential problems. These failures can be both gradual and instant. Let’s imagine two computer vision systems used for industrial needs: one is used on sterile assembly lines where photos of half-assembled devices are taken and analyzed, and the other is used on gigantic ironworks where it monitors how metal scrap turns into new alloys. The first could have an instant failure of lighting, and thus, the datastream suddenly becomes dark. The second is affected by dust and temperature; thus, the camera lens degrades over time.

If a data source problem is detected, it is important to assess the damage and take appropriate action, such as updating, replacing, or pausing the model if necessary.

Some data source problems are, unfortunately, inevitable. Imagine building a recommendation system for a marketplace where some merchants fill in every attribute of their goods and others ignore optional fields. In this case, a smart choice is to think about it in advance during the feature engineering stage to make the model ignore such cases.

Let’s take a look at a more concrete example:

For example, in a content or product recommendation engine, one model may predict the popularity of a product or item. In contrast, another model makes recommendations to users based on the estimated popularity. If the popularity prediction model is incorrect, this can lead to incorrect recommendations provided by the second model.

A similar problem can occur in a car route navigation system, where a model predicts the expected time of arrival for various routes, and another model ranks the options and suggests the optimal route. If there is a problem with the model that predicts the expected time of arrival, this can lead to incorrect route recommendations, which in turn can affect traffic patterns.

These types of interconnected systems can be at risk for problems if something goes wrong with one of the models, leading to a chain reaction of negative actions throughout the system. It is essential to carefully design and monitor these types of systems to ensure that they are functioning correctly.

For example, in a demand forecasting or e-commerce recommendation system, changes to the product catalog can affect the model’s understanding of the data. Similarly, updates to business systems or the introduction of new data sources or APIs can also cause problems for the model if it has not been trained on new data. (See chapter 6 for more information.)

To mitigate the affect of data schema changes, it is vital to design the model with this possibility in mind and to educate business users about the potential consequences of these types of changes. Data quality monitoring can also help identify and address these problems as they arise.

One example of training-serving skew is a model trained on a limited set of crowdsourced images that performs poorly when applied to real-world images with a wide range of data formats and image quality. Similarly, a model trained on high-quality images in a lab setting may struggle to perform well on real-world images taken in poor lighting conditions (see figure 14.5).

Data quality monitoring is somewhat specific to ML, as it involves ensuring that the data we use to train and make predictions with an ML model meets specific expectations. However, it is also necessary for other analytical use cases; here, existing approaches and tools can be reused.

Traditional data monitoring is often performed at a macro level, such as monitoring all data assets and flows in a warehouse, but ML requires monitoring to be more granular, focusing on particular model inputs. In some cases, you can rely on existing upstream data quality monitoring. Still, additional checks may be required to control feature transformation steps, real-time model input, or external data sources.

Various metrics can be monitored to ensure the quality of the data that is being used in an ML model. Some common types of metrics and checks are as follows:

In most cases, it may be useful to validate inputs and outputs separately for each step in the pipeline to identify the source of the problem more easily. This can be particularly helpful if your pipeline is complex and involves multiple steps, such as merging data from different sources or applying multiple transformations. By running checks at different stages of the pipeline, you can locate the source of any problem and debug them more quickly. On the other hand, if you only validate the output of the final calculation and notice that some features are incorrect, it may be more difficult to identify the source of the problem, and you may need to retrace each step in the pipeline.

The choice of metrics to monitor will depend on various factors such as the model’s deployment architecture (e.g., batch, live service, or streaming workflows), the specifics of the data and real-world process, the importance of the use case, and the desired level of reactivity. For example, if the cost of failure is high, more elaborate data quality checks may be necessary, and online data quality validation with a pass/fail result may be added before acting on predictions. In other cases, a more reactive approach may be sufficient, with metrics like average values of specific features or the share of missing data tracked on a dashboard to monitor changes over time.

One of the challenges of data quality monitoring in ML is the execution of the monitoring process itself. Ensuring the quality of data fed into ML pipelines is critical, but setting up a monitoring system can be time-consuming. This is especially true if you need to codify expert domain knowledge outside the ML team or if you have many checks to set up, such as monitoring raw input data and postprocessed feature values.

Another challenge is managing a large number of touchpoints in the data quality monitoring process. It is important to design the monitoring framework to detect critical problems without being overwhelmed. If your monitoring system sends tens of false alerts daily, at some point you will just ignore its signals, so finding the right balance of sensitivity is crucial.

Finally, tracing the root cause of a data quality problem can be difficult, especially if you have a complex pipeline with many steps and transformations. Data quality monitoring is closely connected to data lineage and tracing, and setting this up can require additional work.

We highly recommend reading a paper from Google called “Data Validation for Machine Learning” ().

The following is a highly condensed summary of the paper that features a list of actions that we recommend to effectively monitor your system:

We recommend setting up a data governance framework to ensure that the data validation system is properly maintained and aligned with business objectives. This might involve establishing roles and responsibilities for data quality management, as well as establishing processes for data quality improvement and problem resolution.

Model decay, also known as model drift or staleness, refers to the phenomenon of a model’s performance degrading over time. This can occur for a variety of reasons, such as changes in the data or the real-world relationships that the model was trained on. The speed at which model decay occurs can vary widely; some models can last for years with no need for updates, while others may require daily retraining on fresh data. One way to monitor for model decay is to regularly track key performance metrics and compare them to historical baselines. If the metrics start to degrade significantly, it could be an indication of model decay (see figure 14.6).

We review output drift as one of the markers to monitor and detect model drift, as it can indicate a change in the model’s performance or a change in the relationship between the input and output data, which either belongs to concept drift or data drift. By identifying and addressing output drift, you can help ensure that the model continues to produce reliable and accurate results.

Model drift can lead to increased model error or incorrect predictions, and in severe cases, the model can become inadequate overnight. It is important to continuously monitor for model drift and take appropriate action to maintain the accuracy and effectiveness of the model (see chapter 8 for more context).

One example of data drift is when an ML model trained to predict the likelihood of a user making a purchase at an online marketplace is applied to a new population of users who were acquired through a different advertising campaign. If new users come from a different source, such as Facebook, and the model did not have many examples from this source during training, it may perform poorly on this new segment of users. Similarly, data drift can occur if a model is applied to a new geographical area or demographic segment or if the distribution of important features in the data changes over time.

To address data drift, it may be necessary to retrain the model on the new data or build a new model specifically for the new segment of data. Monitoring the data and the model’s performance can help identify data drift early on and take corrective action before the model’s performance deteriorates significantly. As data drift is not something inherently wrong with the data, it has to be fixed on the model level instead.

There are several types of concept drift: gradual concept drift, sudden concept drift, and recurring concept drift. Gradual or incremental drift occurs when external factors change over time, leading to a gradual decline in the model’s performance (see figure 14.8). This type of drift is usually expected and can be caused by a variety of factors, including changes in consumer behavior, changes in the economy, or wear and tear on equipment when the data is coming from physical sensors.

To estimate how quickly a model will age, it is helpful to perform a test using older data and to measure the model’s performance with different frequencies of retraining. This can give an indication of how frequently the model should be updated with new data to maintain its accuracy. It is also essential to consider the effect of external factors, such as changes in the market or the introduction of new products, which may affect the relationships between the model’s inputs and outputs and cause concept drift.

Monitoring the performance of ML models and regularly retraining them as necessary is a critical aspect of maintaining their effectiveness in a production environment.

Sudden concept drift usually happens due to external changes that are sudden or drastic and can be hard to miss. These types of changes can affect all sorts of models, even ones that are typically “stable.” For example, the COVID-19 pandemic affected mobility and shopping patterns almost overnight, causing demand forecasting models to fail to predict surges in certain products or the cancellation of most flights due to border closures. In addition to events like the pandemic or stock market crashes, sudden concept drift can also occur due to changes in interest rates by central banks, technical revamping of production lines, or major updates to app interfaces. These changes can cause models to fail to adapt to unseen patterns, become obsolete, or become irrelevant due to changes in the user journey.

In an ML system, it is common for certain events or patterns to recur over time. For example, people may display different behavior during the holiday season or on certain days of the week. These repeating changes, also known as recurring drift, can be anticipated and accounted for in the design of the system (see figure 14.9). For example, we can build a group of models to be applied in certain conditions or incorporate cyclic changes and special events into the system design to account for such recurring drift and prevent a decline in model performance.

There are hundreds of different metrics that can be calculated to evaluate the performance of an ML model (see chapter 5). We will cover several categories into which these metrics can be grouped.

Model quality metrics evaluate the actual quality of the model’s predictions. These metrics can be calculated once ground truth or feedback is available and may include

Prediction drift occurs when the model’s predictions change significantly over time. To detect prediction drift, you can use statistical tests, probability distance metrics, or changes in the descriptive statistics of the model’s output. We recommend reading a post by Olga Filippova () for more details.

Input data drift refers to changes in the input data used by the model. This can be detected by tracking changes in the descriptive statistics of individual features, running statistical tests, using distance metrics to compare distributions, or identifying changes in linear correlations between features and predictions. Monitoring for input data drift can help identify when the model is operating in an unfamiliar environment.

Outliers are unusual individual cases where the model might not perform as expected; it is important to identify these and flag them for expert review. Please note that this is different from data drift, where the goal is to detect the overall distribution shift. Outliers can be detected using statistical methods and distance metrics.

The monitoring process can’t be done without covert challenges, though. These include

If new labels are available and the training pipeline is up and running, it is very tempting to hit the retrain button (you can find more information in an article by Emeli Dral, ). But there are things we can do prior to that if we dig deeper.

Check whether the problem is related to data quality or if it is a genuine change in the patterns the model is attempting to capture. Data quality problems can arise from a variety of sources, such as data entry errors, changes to the data schema, or problems with upstream models. It is important to have separate checks in place to monitor for data quality problems and to address them as soon as they are detected (see section 14.3).

If data drift is genuine, try to investigate the cause of the shift. This can help us understand how to address the problem in an optimal way and maintain the model’s accuracy and effectiveness.

One way to start this process is to plot the distributions of the features that have experienced drift, as this can provide insights into the nature of the change. Another helpful step would be consulting with domain experts who may have insights into the real-world factors that could be causing the shift.

In cases where concept drift is also detected, there is a chance that the relationships between features and the model output may have changed, even if the individual feature distributions remain similar. Visualizing these relationships can provide further insights into the nature of the drift. For example, plotting the correlations between features and model predictions can highlight changes in these relationships. Additional insights may also come from plotting the shift in pairwise feature correlations.

It is important to determine whether the observed drift is meaningful and warrants a response. This can involve setting drift detection thresholds tailored to a given use case and set to be alerted to potentially significant changes.

However, it is often necessary to iterate on these thresholds through trial and error, as it can be difficult to predict in advance exactly how data will drift over time. When a drift alert is triggered in production, we need to carefully assess its nature and magnitude, as well as the effect it may have on the model’s performance. This may involve consulting with domain experts or conducting further analysis to gain a deeper understanding of the changes. Based on this assessment, you will be able to decide whether to address the drift. In some cases, you may be able to understand the causes of the drift and decide to (temporarily) accept the changes rather than take action to address the problem.

For example, suppose additional labels or data are expected to become available in the near future. In that case, it may be worth holding off on taking action until this information can be taken into account. If it was a false alert, we can change the drift alert conditions or the statistical tests, as well as discard the notification to avoid receiving similar alerts in the future.

In some cases, the model may still perform despite the drift. For example, if a particular class becomes more prevalent in the model’s predictions but this is consistent with the observed feature drift and the expected behavior, there may be no need for any further action. In these situations, it may be possible to continue using the model as is without the need to retrain or update it. However, given the potential consequences of this decision, monitoring the model closely to ensure it remains effective is crucial.

Adapt the preprocessing used in the pipeline. For example, say your system uses images captured by a camera in a factory. At some point, the factory manager decides to upgrade the light bulbs, so the assembly line is now well lit, and so are the images. Once it affects the model performance due to drift, one solution can be applying an artificial “darkening” function to mimic the original data distribution.

Retrain the model using updated or new data. This is often the most straightforward approach and can be effective if the necessary data is available. By retraining the model using updated or new data, you should improve its performance and adapt to changing patterns in the data (if nothing is broken in the data pipelines). If you follow the good practices from chapter 10, retraining the model should be relatively simple.

Instead of hitting the retrain button, you can consider developing an ML model that is more robust to drift. This could involve applying more robust model architectures (e.g., those designed for online learning) or using techniques such as domain adaptation to make the model more resilient to changes in the data distribution. The following are some additional details on options for addressing data drift:

To do this, you can start by analyzing the changed features of the data and identifying any potential correlations with the model’s performance. For example, if you see a shift in the distribution of a feature (e.g., location), you might try filtering the data by that feature to see if it is causing the low performance.

Once you are able to identify low-performing segments of the data, you can decide how to handle them. One option is to route your predictions differently for these segments, either by relying on heuristics or by manually curating output. Alternatively, you can single out these segments and wait until you have collected enough new labeled data to update the model.

Finally, processing outliers in a separate workflow can also help limit errors under data drift. Outliers are individual data points that are significantly different from the rest of the data, and processing them separately can help to ensure that the model can continue to operate effectively.

Another option to address data drift is to apply additional business logic on top of the model by either making an adjustment to the model prediction or changing the application logic. This approach can be effective but difficult to generalize and can have unintended consequences if not done carefully.

A good example of this approach is manually correcting the output, which is common in forecasting demand. Business rules for specific items, categories, and regions can be used to adjust the model’s forecast for promotional activities, marketing campaigns, and known events. In case of data drift, a new correction can be applied to the model output to account for the change.

Another example is setting a new decision threshold for classification problems. The model output is often a probability, and the decision threshold can be adjusted to assign labels based on a desired probability. If data drift is detected, the threshold can be changed to reflect the new data.

Alternatively, you can consider using a hybrid approach where you combine ML with non-ML methods, especially if you have a small amount of data or if you need to make predictions in a dynamic environment where relationships between variables are constantly changing. In some cases, using a non-ML solution can be more robust and easier to maintain because it does not depend on data patterns and can be based on causal relationships or expert knowledge. However, it may not be as flexible or able to adapt to changing circumstances as ML models. It’s important to carefully consider the tradeoffs and choose the appropriate solution for a given problem. See section 13.4.

Retraining the model may mean either full retraining for a smaller model or some kind of finetuning for a larger one, depending on how serious the drift is and what the computational budget is.

You may have to modify the model’s scope or the business process (this can be done by shortening the prediction horizon, postponing the predictions to have more time for data accumulation, or increasing the frequency of model runs). With this approach, it is important to maintain effective communication with business owners and other system consumers, ensuring that everyone is prepared for changes and is able to handle them.

Additional attention should be paid to recurring drift or seasonality. There are many possible reasons for it, such as increased shopping on Black Fridays or altered mobility patterns on holidays or payday at the end of the month. Consider incorporating these periodic changes or building ensemble models to take them into account. This can help prevent a decline in the model’s performance, as these predictable changes (recurring drift) are expected and can be accounted for.

To effectively handle seasonality in a ML system, it is important to train the model to recognize and respond to these periodic changes. For example, if an ML model is used to forecast loungewear sales, it should be able to recognize and anticipate an increased demand on weekends. In this case, the model correctly predicts the known pattern of increased shopping on weekends. A predictable change like that will not require an alert, as it is a regular occurrence.

To design it effectively, consider ways in which the model and the real-world process behind the data are likely to change. Depending on the needs and constraints of the model, there may be a variety of approaches to detecting and responding to drift.

For instance, if the model’s quality can be directly calculated using timely obtained true labels, you can ignore distribution drift. Instead, you can focus on identifying and addressing problems such as broken inputs using rule-based data validation checks.

On the other hand, if the model is being used in a critical application with delayed ground truth and interpretable features, you may need to switch to a more comprehensive drift detection system. This might include a detailed data drift detection dashboard and a set of statistical tests to help identify changes in data distribution and correlations between features. If you know the model’s key features and the business value they provide, you can assign different weights to those features and focus on detecting drift in those. If not, you can consider the overall drift of multiple weak features but increase the threshold for defining drift in this case, compared to a few key features to avoid false positives.

Ultimately, the best approach to drift detection will depend on the specific needs and constraints of the model and may involve a combination of different methods and metrics. So carefully keep in mind the potential for false-positive alerts and the consequences of the model’s failure and design a system that can effectively detect and respond to drift while minimizing disruption and downtime.

However, there is a reason why business KPIs are taking their place at the top of the pyramid in figure 14.5. Monitoring business metrics and key performance indicators is crucial for understanding the business value of your ML system. Tracking metrics such as revenue and conversion rates (if your model revolves around user acquisition, for example) allows you to determine whether the ML system meets its goals and exerts a positive effect on the business.

It is also important to involve both data scientists and business stakeholders in the monitoring process to ensure that the ML system is meeting the needs of the business. We also advise tracking both absolute and relative values to gain a more comprehensive understanding of the ML system’s effect.

Most importantly, whatever technically honed metrics you define for your ML system, they are nothing but the result of gradual cascading of business KPIs. And it is for this reason that ultimately the effectiveness and sustainability of any system directly or indirectly affects the success of your employer or even your own business.