6 Gathering datasets

There is an old popular quote about real estate: the three most important things about it are location, location, and location. Similarly, if we were to choose only three things to focus on while building an ML system, those would be data, data, and data. Another classic quote from the computer science world says “garbage in, garbage out,” and we can’t doubt its correctness.

Here we’ll break down the essence of working with datasets, from finding and processing data sources to properly cooking your dataset and building data pipelines. As a culmination of the whole chapter, we will look at datasets as a part of design documents, using the examples of Supermegaretail and PhotoStock Inc.

Some datasets become valuable when enriched or annotated/labeled. Annotation means combining a raw dataset with proper labels or, in other words, creating a closely tied dataset and connecting it with a new one. It is a popular pattern of mixing a unique proprietary dataset with a public data source and getting a much more valuable dataset as a result.

Ad tech companies, mentioned earlier, may benefit from joining datasets as well. Let’s take a look at a classic example. A company operates a free-to-play game, which means it has a lot of players, and only some of them pay money. The number and list of paying customers is kept secret and available only to the game’s publisher. At the same time, its partnering ad network has millions of detailed user profiles derived from behavior. When these datasets are combined (see figure 6.1), it opens a great marketing opportunity: the company can target its new ad to potential customers who are similar to its paying players. Data exchanges like this increase the efficiency of online marketing and thus are one of the powers driving the modern web.

Speaking about combining data sources, Arseny once helped kickstart a startup as an advisor. The company worked in the agri-tech area, helping farmers and related companies to increase their operating efficiency, and its secret sauce was based on datasets. The way it worked was as follows. One data source was public, using satellite images of the planet available thanks to several space initiatives like Landsat by NASA and Copernicus by ESA (where you can fetch countless images of agricultural lands). But having those images alone could not bolster the efficiency of the startup, as it lacked information describing these agricultural lands. The main problem is most agricultural companies are far from innovative, and there is no single solid source of data on what crops were grown, the yield results, and more. Such data is poorly digitized but is really valuable for multiple business needs: it can be used to reduce the amount of fertilizer used, to estimate future prices for food commodities, and so on. Eventually, the team implemented smart ways of gathering such data and merging it with the huge photo database. An ML system built on these joint datasets helped the company to grow rapidly.

Defining data sources and the way they interconnect is the first cornerstone of solving the data problem for the system. But raw data is often almost useless until we make it available for the system and ML model, filter it, and preprocess it in other ways.

We’ve gathered a list of techniques you can use to properly cook your dataset, with each technique presented in a separate subsection. This list is not strictly ordered, and the order of actions may vary depending on your domain, so there is no single universal answer. In some cases, you can’t filter data before labeling, while sometimes filtering happens multiple times throughout the cooking process. Let’s briefly touch on these techniques.

Data filtering is a very domain-specific operation. In some cases, it can be done fully automatically based on a set of rules or statistics; in others, it requires human attention at scale. Experience shows that the end approach normally lies somewhere between those two extremes. A combination of the human eye and automated heuristics is what works best, with the following algorithm being a popular approach: look through a subset of data (either randomly or based on some initial insight or feedback), find patterns, reflect them in the code to extend coverage, and then look through a narrower subset.

While the absence of data filtering leads to extensive noise in the dataset and thus worse performance of the whole system, overly aggressive filtering may have a negative effect as well: in some cases, it may distort data distribution and lead to a worse performance on real data.

Sometimes features are not engineered manually but created by a more complicated model; unlike “regular” features, they can be non-human-readable vectors. In these cases, it’s more precise to use the term representations, although at some level of abstraction, they’re the same thing: inputs to the ML model itself.

For example, in a big ML-heavy organization, there may be a core team building a model that generates the best possible representations of users, goods, or other items at scale. Their model doesn’t solve business problems directly, but applied teams can use it to generate representations for their specific needs. That is a popular pattern when we’re talking about such data as images, videos, texts, or audio.

Let’s imagine you’re building a medical assistance product, a system that helps radiologists analyze patients’ images. It is one of the most popular ML applications in medicine that is still very complicated to design properly. A use case may seem simple: a doctor looks at an image and judges if there is a malignancy, so you want doctors to label the dataset.

There are numerous ways to label it (see figure 6.2), including

In some situations, labeling data with the proper level of detail is nearly impossible. For example, it could take too much time, there may not be enough experts to cover enough data, or the toolset is not ready. In these cases, it is worth considering using weakly supervised approaches that allow the use of inaccurate or partial labels. If you’re not familiar with this branch of ML tricks, we recommend “A Brief Introduction to Weakly Supervised Learning” () for an overview; more links for further reading are at Awesome-Weak- Supervision ().

Efficient data labeling requires a choice of a labeling platform, proper decomposition of tasks, task instructions that are easy to read and follow, easy-to-use task interfaces, quality control techniques, an overview of aggregation methods, and pricing. Additionally, best practices should be followed when designing instructions and interfaces, setting up different types of templates, training and examining performers, and constructing pipelines for evaluating the labeling process.

When datasets need to be labeled manually, there are two main ways to go: an in-house labeling team or third-party crowdsourcing services. In a nutshell, the former provides a more controlled and consistent labeling process, while the latter can scale up the annotation process quickly, possibly at a lower cost. The simplest heuristic of choosing one over another is based on the labeling complexity: once it requires specific knowledge or skills, it makes sense to develop an in-house team of curated experts; otherwise, using an external service is a good option.

There are dozens of crowdsourcing platforms available; the most popular is probably Amazon Mechanical Turk. As we opt for width over depth in this book, we will not focus on features of different platforms. Instead, we focus on generic properties of labeling with crowdsourcing:

Let’s imagine we have three labelers annotating the same object for a binary classification. The problem is tricky, so each one is only right in 70% of cases. So we can expect a majority vote of three labelers to be correct in 0.7 * 0.7 * 0.7 + 0.3 * 0.7 * 0.7 + 0.7 * 0.3 * 0.7 + 0.7 * 0.7 * 0.3 = 78.4%. That’s a nice boost!

These numbers should be interpreted with a grain of salt. Condorcet’s jury theorem implies assumptions that are not exactly met here: labelers are not completely independent, and their origin of error is correlated (this may be the case when data samples are noisy and hard to read). But still, an ensemble of labelers is more accurate compared to a single labeler, and this technique can be used to improve labeling results.

This idea can be modified to decrease costs. One algorithm modification we have faced is the following:

The heuristic described here is a simple though powerful baseline. However, if data labeling is a key ingredient for your problem and you see the need to invest some effort, there is a lot of research dedicated to better design. Some of the materials we recommend are “A Survey on Task Assignment in Crowdsourcing” () and a tutorial by crowdsourcing vendor Toloka.ai ().

Labelers’ consistency and mutual agreement may and should be measured. While ML practitioners often tend to use regular ML metrics to estimate it, those who have a statistical background may recall a concept called interrater reliability and its separate set of metrics (Cohen’s kappa, Scott’s pi, and Krippendorff’s alpha, to name a few). These metrics help keep the labeling team’s work reliable, filter out unscrupulous labels (thus greatly improving the overall system performance), and, in some scenarios, give a solid upper bound for the performance of your model (recall the human-level performance we mentioned earlier in chapter 3).

Labeling teams require proper tooling, which may boost efficiency and improve the quality of created labels. If you go with third-party tooling, it’s very likely it will have a toolset for the most popular problems; for your own team, you need to set up or create your own. Aspects of choosing an existing solution or building a brand-new one are very domain specific, although the general approach is: the more popular the ML formulation of your problem, the more chances you have to find a modern quality toolset for the labeler team. As an example, there are multiple software solutions to create labels for image detection at scale, but once you need to annotate videos with 3D polygon meshes, you’re very likely to require something custom made.

Recent advances in large foundation models have become a game changer in the labeling process as well: they often can be used to gather initial labels in a few-shot setup with the accuracy higher than nonexpert human labelers. One of many examples of a tool used for large language model labeling is Refuel Docs (), notable for being focused on text data; however, using foundation models is also possible in other domains (e.g., addressing universal segmentation models like Segment Anything [] or multimodal solutions like LLaVA []).

There are various algorithmic tricks that may help simplify/reduce the efforts required for the data labeling aspect of building a system, but we will not go into an in-depth review of those in this book, as they are too subject-specific. What we would like to additionally highlight is that very often efforts invested in the data labeling process and tools can be a driver for the success of your ML system.

As we discussed earlier, while recipes for datasets may vary, most likely, your dataset will be cooked properly. It’s okay to reveal new aspects of the problem, thus adjusting the dataset structure over time, and reflect them accordingly in how the dataset was processed. All these differences should be reflected in metadata.

Another source of changes is testing and bug fixing. Imagine that you work for a ridesharing company and are to solve a problem of estimating how long it will take to reach point A from point B. You start with historical data your company generated, and there is a preprocessing job that extracts the information from logs and stores it in your favorite database, so you end up with a table like latitude_from, longitude_ from, datetime_start, datetime_finish, distance, data_source.

At some point, the company makes a big acquisition of a competitor, and you decide to add its data to your ETL process. A new engineer on the team starts coding, new sources are attached, new data points are added to your dataset with the same schema, and suddenly your model performance drops. What happened?

Fast forward: your company stored distance in kilometers, while the acquired team stored distance in miles. An engineer who handled the integration knew it and implemented the support of miles for the new source but accidentally enabled it for the old one as well. So new data samples are correct for the new data source but not for the old one. If you had a metadata field like preprocessing_function_version, you could easily find affected samples. Without it, you need to gather the dataset from scratch after the code defect is discovered.

Defects like this are not the only scenario where you may need to reveal the date of data sample origin. Some ML systems may affect data sources, creating a phenomenon called a feedback loop. A very common example of a feedback loop is recommender systems: let’s say a marketplace sells many items, and there is a block on the website titled “You may also be interested in these goods.” As a naive baseline, the company could put in the items that are most popular overall. But when this baseline is replaced with an ML system, new items appear, and old leaders lose their popularity in favor of new recommended items. With the poor design of such a system, there is a chance for an item that may not be that good to gain its place among popular ones and dominate for a long time. Storing the information when the sample was generated and what versions of related systems were live at the moment is crucial (not enough, though!) to avoid feedback loops.

Another important scenario related to metadata is stratification, a process of data sampling that shapes the distribution of various subgroups. For example, say a company started its operation in country X, gathering a lot of data related to customers from X. Later the company reached a new market, Y, and has a goal of providing the same level of service, including accuracy of ML models, for customers in both countries. It will require representing customers from X and Y in training and test datasets with proper balance.

Stratification is crucial for validation design and resisting the algorithmic bias, and both topics will be covered in future chapters.

First, not all data samples are equally useful. Growing the dataset size is only useful if new objects help the model learn something new and related to the problem. Adding samples very similar to existing ones is pointless; the same goes for extremely noisy data. That’s why there is a popular research avenue dedicated to finding the most samples for dataset extension called active learning. The simplest intuition is to enrich the dataset with samples where the model was incorrect (this signal is often fetched using humans in the loop processed) or demonstrated the lowest confidence. The academic community has developed tens of methods related to active learning; see a recent survey (e.g., “A Comparative Survey of Deep Active Learning” by Xueying Zhan et al., ) for more information.

One more aspect related to dataset size is sampling. While in most cases the question we ask ourselves is “How do we get more data?” sometimes it is “What part of data should I use to increase efficiency?” It usually happens when a dataset needs no manual labeling and is generated by a defined process—for example, the clickstream in popular B2C web services (search engines, marketplaces).

Sampling is effective when a dataset is not only huge but also tends to be imbalanced and/or may contain a lot of duplicates. Different strategies can be applicable here, and the most common of them are based on stratification, where you split data into groups (based on key features or algorithmically defined clusters) and limit the amount of data used for each group. The amounts don’t have to be equal—for example, if the temporal component is relevant to your problem, it’s very likely you will want to prioritize fresh data over old groups.

Another point is about data noisiness. A while ago there was a strong consensus in the ML community that a few clean samples are better than multiple noisy samples. Things have changed recently, however; the progress with large models like GPT-3 and CLIP demonstrated that at a certain scale, manual filtering is almost impossible (processing a dataset of 500 billion text tokens or tens of millions of images would cost a fortune), but using a huge amount of weakly or self-supervised (automatically, using heuristics) data works, so for some tasks massive imperfect datasets are more suitable than smaller hand-picked ones.

Not every model can be improved by adding new data because of the uncertainty of origin. Uncertainty primarily comes from two central origins:

It is often a problem for startups or companies trying to launch products in new markets or verticals. And, as often happens in the ML world, the go-to solution is approximation. Since we don’t have data perfectly matching our problem, we need to find something as close as possible. What data will be close depends on the problem, so let’s look at some examples.

For this example, let’s imagine a company focused on employee safety that builds products monitoring how workers follow safety rules in various environments. The new product should check if hard hats were worn on factory grounds. When the product is available, there will be a lot of data coming from customers who agree to share it, but before that, customers’ cameras are not available for the company.

So how do we approach the problem?

In every scenario, we need to keep in mind that data is not a representative sample of the real distribution we’re going to work with when the system goes live. This means validation results should be taken with a grain of salt, and replacing such proxy datasets with more realistic ones is a top priority. Let’s talk about it when discussing the properties of a healthy data pipeline.

Reproducibility means you should be able to create a dataset from scratch if needed. There should be no golden data file in your storage that is crafted by a wise engineer with a bit of dark magic. But there must be a software solution (a simple script, more advanced pipeline, or big system) that allows you to create such files again using the same origin as used before. This solution needs to be documented and tested well enough so every project collaborator is able to run it when needed.

The reasoning behind reproducibility is similar to the infrastructure-as-code paradigm popular in the DevOps community. Yes, it may seem easier to create an initial dataset manually in the first place, but it’s not scalable in the long run and is very error-prone. Next time—when a new batch of data is going to be added—you may forget to run a preprocessing step, thus causing implicit data inconsistency, a hard-to-detect problem that will affect the whole system’s performance.

Consistency itself is the key. ML problems often have situations when labels are partially ill-defined, and defining a strict separating plane is impossible. In such cases, experts who do labeling tend to disagree with each other.

A very demonstrative example from our background is related to the credit card transactions classification problem. Customers are expected to give each transaction a label describing the purpose of the expense. The initial label taxonomy contains “food and drink,” “bars and wineries,” and “coffee houses.” A rhetorical question arises: What label is more suitable for payment for a coffee and a sandwich ordered in a wine bar? A partial solution for this example would be to have some kind of protocol on how ties are to be broken and how blurry boundaries are resolved (e.g., “when a merchant type and purchased item type are conflicting, the first to be opened takes precedence”)—it greatly simplifies model training and especially validation.

Consistency is not only about labels. All aspects of data should be consistent: what the data origin is, how data is preprocessed, what filtering is applied, etc. It is relatively easy when the system is being built within a small company and way more challenging for an international corporation, and some formal definitions can be helpful. Once you feel there is a chance of misreading a core term used in the data pipeline, it may be useful to add it to the design doc to make sure the whole team is on the same page.

One more aspect of consistency is how data is gathered at the system-building stage and what the system inputs are during system usage. It is a common problem, and in ML terms, it leads to distribution mismatch, a problem affecting model performance and fairness that is not easy to detect before the system goes live.

We recently mentioned the feedback loop phenomena—the model affects data origin and thus violates the consistency assumption. We also described situations when data is not available until the system launch, and this also may lead to a mismatch. Problems like this are still among the most challenging aspects of applied ML, and you can never ignore them while designing a system.

Data consistency is always an open question, so you should care about it at every stage of ML system development, from initial drafts to long-term maintenance. We will get back to this aspect in chapter 14 when discussing monitoring and drift detection (there will even will be a short but rather cool campfire story from one of our friends).

Last but not least is availability. This property is actually an umbrella for two ideas: availability for a system and availability for engineers. The first can also be named reliability; a system designer should be very critical of nonreliable data sources. Here is a good situation for a negative example—one where things can ultimately go wrong. Imagine a system that depends on a third-party API enriching your data stream, and everything was good until it wasn’t—a small company powering the API lost its key site reliability engineer and thus the ability to handle the infrastructure. If your system’s dependency on this data source is critical, their problems become your problems.

Of course, it doesn’t mean using third-party APIs is not an option. As we mentioned earlier, using external solutions is often a good practice—and not only provided by giants like Amazon, Google, and Microsoft. But data availability is important, and these risks should be taken into account seriously. There are some services provided by vendors that can be less reliable (we can’t imagine a top-priority problem caused by the outage of an experimental visualization tool), but data sources are not one of them.

The same philosophy goes for internal systems (e.g., those controlled by your peers from other teams). There are different control gears for external and internal systems: for the former, service-level agreements are used to estimate risks, while internal systems exist only in mature organizations. At the same time, in smaller companies, it is easier to be aligned with a team maintaining the system and effectively reducing related risks.

It is also worth mentioning that problems related to data availability are not strictly software related. Even if all related systems are built and maintained properly, there may be problems of more sophisticated origins—for example, caused by information security and privacy rights (such as new legal regulations that affect using personal data or your key customer CEO’s decision that their data should not leave their infrastructure anymore).

Availability for engineers can be overestimated: you can hear something like “Come on, our engineers can fetch the data even if it’s not a simple operation—they are professionals and will handle the tech difficulties, if any.” Most likely it is true—we assume our readers and their colleagues are outstanding professionals, but you can’t ignore time constraints. Imagine that an engineer went for lunch with their buddy from another team; they chatted about work-related stuff, and after a cup of coffee, a new hypothesis related to the system of current interest popped up.

If data is attainable easily, you can make an informative decision relatively quickly by pulling the dataset, aggregating some statistics, and maybe even running basic experiments. If the idea gets confirmed in the first naive approach, it can be prioritized higher, meaning more resources should be allocated. Who knows, maybe it is going to be a significant improvement in the future.

Otherwise, if pulling data and making data-driven decisions is time-consuming, the hypothesis is likely to be ignored (“Well, it might be interesting, but I have so many things to do and finding the data will take so long!”). To avoid these cases, we recommend dedicating a share of engineering efforts to building tools that will make datasets more available to people who should work with data. While it is somewhat applicable to any engineering productivity tools, return on investment is especially high when the tools improve data availability. As we said in the very beginning, it’s hard to overestimate the importance of quality data for ML systems, and thus smoothing interactions with the datasets is a good investment in the long run.

This is sufficient depth in the information on consistency and availability at this point in the book. We will return to these two properties of the pipeline in greater detail later in chapter 10. Meanwhile, it’s time to move on to the practical part of the chapter: the design document.