11 Features and feature engineering

While the role of feature engineering is not crucial for a system designed with a deep learning core in mind, no machine learning (ML) practitioner can ignore their role. In a sense, framing some fancy multimodal data into a deep learning model or even making a prompt for a large language model is a specific way of feature engineering, and that’s why classic feature-related techniques like feature importance analysis are still very relevant.

This chapter explores the art and science of creating effective features. We will cover tools that help determine the most valuable features for the system, the engineering challenges we can face, the factors and tradeoffs we should consider while selecting the right subset of features, and how we can ensure that the selected features are reliable and robust.

To secure a fruitful and streamlined modeling process, you should always make sure you assemble an effective feature engineering strategy while designing a system. This plan will become a compass to guide the team through identifying and engineering the most impactful features while minimizing the risk of wasted efforts. By prioritizing iterations in the proper order and charting the course, we can avoid potential pitfalls and ensure our actions add value to the end goal.

Feature engineering in ML is similar to crafting prompt structures in generative models, such as large language models and text-to-image generators. Both features and prompts serve as enhanced inputs that guide the model’s “attention focus” (literally or figuratively) toward the most relevant data aspects.

The most obvious way to fetch a new feature is to add a new data source to your data pipeline or use a column that previously was not incorporated into the dataset. This data source can be either internal (e.g., an existing table in the database) or external (e.g., buying data from a third-party provider). On the one hand, these new features are low-hanging fruit with a valuable contribution to the model’s performance. On the other hand, they require most of the data engineering efforts, take a lot of time to manage, and may cause infrastructural problems, as greater complexity always requires more effort for maintenance.

If new sources are not used, there are two alternatives—to transform the existing features or to generate new features based on a combination of two or more of the existing ones.

Transforming numeric features includes scaling, normalization, and mathematical functions (e.g., using logarithms to improve distribution skewness). The type of model dictates the appropriateness of transformation. For instance, a common thing will be finding no increase in gradient boosting metrics after applying monotonic transformations to its features because the core element of the algorithm—a decision tree—is invariant under monotonic transformations of inputs.

When dealing with time-series data, it’s common to utilize transformations such as lags (shifting the feature’s values backward in time to create new features), aggregates (calculating measures like mean, max, or min over a specific time window), or generating statistical features from past data, such as the standard deviation or variance over different time periods.

Quantile bucketing (or quantization) is a specific case of transformation. It converts continuous features into categorical features by grouping them into discrete buckets based on their values. For example, Uber applies this approach in its DeepETA network (; see figure 11.1).

Categorical features often necessitate transformations, which can be accomplished through methods like one-hot encoding, mean target encoding, ordinal encoding (this method ranks categories based on some inherent order), or the hashing trick, which allows handling large-scale categorical data. It is important to note that while being powerful, mean target encoding can easily lead to data leakage if not properly implemented, as it uses information from the target variable to create new features.

For sequential data like text, we can use techniques such as bag-of-words, term frequency-inverse document frequency (TF-IDF), and BM25 to transform the data into a form that can be processed by ML algorithms. It is worth noting that these methods lose information about word order; this disadvantage can be partially addressed by using longer N-grams instead of single words (unigrams). We can also use pretrained language models such as BERT to represent input data in a low-dimensional embedding space, which we can feed to the final model.

Remember that we can represent almost any sequential data as tokens, not texts. For example, in industries like online retail and media streaming services, we can interpret a user session as a sequence of visited product pages or watched videos. Each visited page will have its learnable representation (an embedding). Afterward, we can use these embeddings in our recommendation system as a prompt in the “next page prediction task” to get an idea of what product/video the user is looking for.

If we want to use product embeddings in the tabular dataset, one of the common options is to utilize the distances between products. Examples of features here would be

What about merging signals from multiple features into one? When we have multiple features in our dataset, we can combine them to create a feature that is more informative or meaningful for our model. For example, instead of having separate features for the number of clicks and the number of purchases a user has made on an eCommerce site, we can combine them to create a new feature such as “purchase-to-click ratio,” which might be a better indicator of the user’s buying intent.

In the case of a taxi aggregator company, instead of having separate features for the “number of rides” and “total distance traveled,” we could combine them to create a new feature like “average distance per ride,” which might provide more valuable insights into drivers’ and passengers’ behavior.

We should also consider the relationship between the existing features. For example, absolute product sales for a certain period may provide little information. However, comparing them to sales of other products in the same category or sales in previous periods may reveal valuable patterns or trends. Combining signals from multiple features can create new features that capture more complex relationships in the data and improve the model’s performance.

The technique of combining multiple features is usually referred to as feature interactions or feature cross. This technique is especially important for linear models because such features may unlock the linear separability of data points.

Model predictions can be thought of as a specific case of a feature where the output of the model is used as an input to another model or system. This approach is sometimes called model stacking. While using model predictions as features can be powerful and effective, it poses some engineering challenges and risks.

The simplest example of using model predictions as a feature is target encoding (). In this approach, the categorical feature is encoded by the mean target value (with a certain degree of regularization) and is used as a feature in the model. However, there is a risk of data leakage where the encoding is based on information from the training data that is not available during inference. This can result in overfitting and poor performance on the new data if we don’t use advanced validation techniques like nested cross-validation (see chapter 7).

Another example is using third-party models (e.g., weather forecasts as a feature in a demand prediction model). While weather data can be highly informative, there is a risk that forecasts may need to have the necessary historicity. In such cases, forecasts with the necessary historicity are preferable to forecasts with higher precision. Besides, relying on external data sources can introduce additional dependencies and risks beyond the ML team’s control.

Finally, using third-party or open source models as feature extractors in deep learning systems can pose risks, too. While the generated embeddings can absorb useful patterns in the data, there is a danger of model drift or instability if the external model is updated without proper versioning and vice versa—with no updates, the model may lose its value due to a drift in your data. This can result in unexpected behavior and drastically drop the performance of your ML system.

To mitigate these risks and challenges, it is important to design the feature engineering pipeline carefully and have robust testing and monitoring procedures in place (the former is described in chapters 10 and 13; the latter can be found in chapter 14). This can include using cross-validation and other techniques to prevent leakage, validating external data sources and models, and having processes in place for monitoring and updating features over time.

ML models can often be seen as black boxes that provide no insight on how they arrive at their predictions. This lack of transparency can be problematic for engineers, stakeholders, or end users who must understand the rationale behind decisions provided by a given model.

In pursuing model transparency, we employ two key concepts: interpretability and explainability. Both are aimed at demystifying the workings of an ML model:

In addition, feature importance analysis can increase trust in our ML system. This is particularly important in high-stakes domains such as medicine and financial fields. While the General Data Protection Regulation () does not strictly enforce explainability, it does suggest a level of transparency in automated decision-making, which could be beneficial or even essential in many ML applications ().

Identifying the most important features explains the variables driving the model’s predictions and the reasons behind them. This information can help optimize those features to boost the model’s performance and remove irrelevant or redundant features to improve efficiency. Additionally, it can guide us through debugging by, for example, detecting overfitting or evaluating the usefulness of newly added features.

Navigating the terrain of feature importance analysis can be daunting, but having a map of available methods can show us the right direction. These methods can be broadly classified based on their properties, such as type of model, level of model interpretability, and type of utilized features.

On the other hand, deep learning models, which automatically learn feature representations from data, present unique challenges for importance analysis. Given the complex, nonlinear transformations and the high level of abstraction, there is more involved than simply looking at the model’s parameters to understand feature importance. Instead, we rely on advanced techniques like saliency maps, activation maximization (read more in “Visualizing Deep Convolutional Neural Networks Using Natural Pre-Images” by Aravindh Mahendran et al., ) or layer-wise relevance propagation (read more in “Layer-wise Relevance Propagation for Neural Networks with Local Renormalization Layers” by Alexander Binder et al., ) and its successors to make sense of what is happening inside neural networks. Please bear in mind that the list of examples is not exhaustive, and none of them is truly universal, as the problem of explainable deep learning is not solved in general and remains in an active research stage.

In their turn, model-agnostic methods treat a model as a black box. They often involve perturbing the input data and observing the effect on the model’s output. Examples of model-agnostic methods include

Some examples of methods that focus on individual predictions include local interpretable model-agnostic explanations (LIME); see the paper "Why Should I Trust You?" by Marco Tulio Ribeiro et al. (), which approximates the decision boundary around a particular input using a simpler, more interpretable model (e.g., linear regression), and anchor explanations (learn more in the paper “Anchors: High-Precision Model-Agnostic Explanations” by Marco Tulio Ribeiro et al. (), which identify a rule that sufficiently “anchors” a decision, making it interpretable by humans.

However, in the context of deep learning, especially with data types such as images, audio, or text, feature importance can become less clear and more challenging. Deep learning models, by nature, automatically learn hierarchical representations from the data, often in a highly abstract and nonlinear manner. In these cases, a “feature” can refer to anything from a single pixel in an image to a single word or character in a text or a specific frequency in an audio signal to complex attributes, like the location of an object in an image, the sentiment of a sentence in a text, or a specific sound pattern in a voice record.

Despite that, it is still possible to gain insights into what the model considers important in raw input data and which patterns it pays attention to. Let’s explore a few techniques for feature importance analysis in deep learning:

This is where feature selection comes in. By carefully selecting the most informative features, we can improve our system’s performance and interpretability while reducing its complexity and training time. We will explore the techniques, best practices, and potential pitfalls of feature selection and learn how to choose the right features for our specific ML problem.

However, just as not all plants in the garden will thrive, not all features will benefit the model, and at some point, we will have to prune away dead or unproductive plants (in our case, discard irrelevant or redundant features) to sustain healthy growth. This cycle of nurturing and pruning, of adding and reducing, is a constant in the life of ML systems as we continually refine and improve our feature sets.

The ancient Greek philosopher Heraclitus once said, “Opposition brings concord. Out of discord comes the fairest harmony.” This also holds true in ML, where we achieve optimal performance by keeping the balance between generating new features and carefully selecting the most informative ones.

Perfect personalization becomes worthless if it slows prediction by 300 ms, causing negative perception. Therefore, lightweight personalization with moderate quality is more appropriate than a model that accumulates all possible inputs from a user but makes them wait.

Filter methods work by filtering features independently from the model, using simple ranking rules based on the statistical properties of a single feature (univariate methods) or the correlation with other features (multivariate methods). These methods are easily scalable (even for high-dimensional data) and perform quick feature selection before the primary task.

The order in which characteristics are ranked in univariate filter methods is determined by the intrinsic properties, such as feature variance, granularity, consistency, correlation with the target, etc. Afterward, we leave top-N features as our subset and either fit the model or apply more advanced, computationally intensive feature selection methods as the second feature selection layer.

In multivariate methods, we analyze features in comparison with each other (e.g., by estimating their rank correlation or mutual information). If a pair of features represent similar information, one of them can be omitted without affecting the model’s performance. For example, the feature interaction score (regardless of the way it is measured) can be incorporated into an automatic report. When the score is high, it triggers a warning for potential reduction in the model’s performance before the training begins.

Wrapper methods focus on feature subsets that will help improve the quality of the model’s results used for selecting based on a chosen metric. We call them this because the learning algorithm is literally “wrapped” by these methods. They also require designing the right validation schema nested into outer validation (to choose the right validation schema, please see chapter 7).

Wrapper algorithms include sequential algorithms and evolutionary algorithms. Examples of sequential algorithms are the sequential forward selection, where features are included one by one starting from the empty set; SBE (sequential backward elimination), where features are excluded one by one; and their hybrids—floating versions when we allow inclusion of an excluded feature, and vice versa. In evolutionary algorithms, we stochastically sample subsets of features for consideration, effectively “jumping” through the feature space. A common example of an evolutionary algorithm is to run a variation of differential evolution in a binary feature mask space where “1” indicates an included feature and “0” denotes an excluded one.

The main disadvantage of these methods is that all of them are computationally intensive and often tend to converge to local optima. Despite that, they provide the most accurate evaluation of how the subset affects the target metric. Use them carefully, especially if your hardware specs are limited.

With embedded methods, we use an additional “embedded” model (which may or may not be of the same class as our primary model) and make decisions based on its feature importance. A good example is the Lasso regression, due to the ability of the L1-regularization to turn the coefficients to zero if they are not relevant to the target variable, as shown in figure 11.9.

Embedded methods lie somewhere between filter methods and wrapper methods as far as the required computation and selection quality are concerned.

When it comes down to choosing the methods to start with, we prefer rough cutoffs for the initial reduction of the number of features rather than using Lasso selector or RFE, depending on which one outputs more meaningful subsets. Computationally expensive methods may have better performance, but faster methods tend to be good enough for initial feature pruning, especially if you suspect that some features are total garbage.

There are plenty of dummy and still-useful methods that also serve reasonable feature selection baselines. For example, we can take a feature, shuffle its values, concatenate it to the initial dataset, and train a new model. If the importance of a given feature is less than the importance of this random feature (often called a shadow feature), it is likely to be irrelevant to the problem. We can label this algorithm as a trivial instance of the wrapper methods.

Imagine investing weeks of effort into engineering sophisticated features only to stumble upon a conversation with a colleague during a coffee break where you discover that another team has already implemented and tested exactly the same features. Alternatively, a colleague approaches you, seeking an implementation of a specific feature. While you confirm its existence, you realize that your code lacks proper documentation and reusability due to heavy dependencies on other code in your repository. Consequently, your colleague decides that an easier way is to reinvent the wheel and develop their own implementation. This inconsistency may lead to each engineer implementing and calculating each feature on their own, causing the company to waste an unacceptably large amount of resources (see figure 11.10).

The best way to start is to analyze the existing extract, transform, and load pipelines of all teams. The following are questions you should be ready to ask:

Here we would like to stress once again that building your own custom feature store is a huge and expensive project. That’s one of those cases when you should consider reusing an open source solution or a product from a third-party vendor. Some popular options are Feast, Tecton, Databricks Feature Store, and AWS Sagemaker Feature Store.

Writing is commonly done in batches. We don’t prefer to compute a whole dataset when we can do it simultaneously, although this is a widely used antipattern. Moreover, updating the features for the last few days helps us overwrite some temporary corruptions or unavailability on the data warehouse side that may occur just before our daily feature update. It is worth noting that “commonly” is not a common case—features can be appended or updated on different schedules. For example, some are computed by a long job on a daily basis, and some are lightweight enough to be streamed in near real-time.

The critical aspect of reading features is usually latency. We must ensure that the infrastructure we are building for our feature store meets our nonfunctional requirements. Sometimes we can combine precomputed features with real-time features (those that require the most recent events) during read operations, as shown in figure 11.13.

In particular, one of the most straightforward patterns is calculating features, which should be done in advance but not when we ask the feature store to gather a dataset. For example, a good time to update features is when our database finishes processing orders for the previous day.

A closely related optimization technique is to split the calculation into multiple steps:

Suppose some engineer implements a new version of a feature, and your system treats it as if it’s the same feature as before, with no differentiaton. You’ll be lucky if the calculation is exactly the same but done faster, for example. But if the calculation principle is changed (even a bit)—or worse, if the engineer changes the data source for the same feature—it leads to inconsistency in the precalculated feature. The values of the feature before and after the update can significantly differ from each other, and you don’t want to mix values from old and new calculation methods. A well-designed feature store will automatically overwrite the updated feature or, even better, write it to a new table while backfilling all previous values with available history.

Each dataset should capture in its meta info not only which features it contains for a given range of timestamps but also versions of these features at the time of calculation. It allows us to easily roll back to an old version of a dataset and completely reproduce the results of an old model that was developed, for example, 2 months ago. This pattern is similar to the libraries’ version freezing for our application.

A pattern we discussed earlier, preaggregation, can be considered a parent feature for the final features. We highlight them (level 1 features) along with their child features (level 2 features, etc.) in figure 11.15.

Lineage tracking also helps engineers and analysts rapidly explore the source of each feature, thus simplifying debugging and improving their understanding of the origin of outliers or other surprising behavior.

It means there is no need to work with features as isolated columns and write a separate data pipeline for each. We naturally prefer to implement our feature store in such a way that it consolidates computations for features derived from the same data sources. Thus, a single entity of computations would be a batch of features, not a single one:

Other things that can be shown to users are feature importance, value distribution, category, the owner, update schedule (daily, hourly), key (user, item, or user-item, or item-day, category-day), feature lineage, ML services that consume this feature, and other meta info.