1 Essentials of machine learning system design

While all of the positions are valid, we believe that to become a seasoned expert in ML system design, you have to encapsulate expertise from each of the backgrounds. Note that while some of the things we discuss in this book are specific to ML systems, others will be familiar to those readers who have already built non-ML software systems (you will find this information in chapters 2, 13, and 16). This is because ML system design, although a new phenomenon, is still based on the classic fundamentals of software development.

But first we need to discover what ML system design is as a whole. In this opening chapter, we will suggest our take on the definition of ML system design and support it with examples from personal experience, both our own and those of our colleagues; we will describe the perfect persona for the position and share cases from our personal experience of why a coherent and consistent approach to designing an ML system will save you a lot of time in the long run and will help in delivering short-term business wins, which is crucial to gain the trust of colleagues in this new working method from the early stages.

However, after reaching out to a variety of people for their opinions on the outline, we saw that the term itself caused discord in opinions and interpretations. Perhaps this is due to the fact that the industry has long had a definite list of job titles, which gives candidates a relatively clear understanding of what set of functions and responsibilities they are applying for. The positions of software engineer, research engineer, ML engineer, and so on each entail a certain classic set of functions enshrined in textbooks and eloquently stated in job descriptions.

So, is there a job title associated directly with the term ML system design? Currently, there’s no position completely tied to the scope we’ll be describing in this book, but if you meet a person fulfilling this scope, their position will most certainly be data scientist.

In our attempts to understand the nature of this connection, we reached various people working in data scientist positions and eventually realized that the role implies a rather vast and vague list of responsibilities. Indeed, you can find 10 different people working in 10 different companies as a data scientist and ask them what they do—and you’ll end up hearing about 10 completely different things:

While contemplating this, we found that in the case of ML system design (or rather what later received such a name), the situation is quite the opposite: there is a phenomenon without a common name but with a clear set of functions and responsibilities, and what needs to be done is to organize them and bring them to a coherent structure of interrelated functions.

In the following chapters, we will be giving our own perspective on ML system design and even suggesting unconventional ideas and solutions, but before we dive in, we’d like to suggest our own definition:

But it is not only the term machine learning system design itself that has been questioned by those who have seen this book’s synopsis or walked through the table of contents. We received a host of questions on various aspects of the book; the following are those we found the most notable:

Coming from deep tech companies, we got used to calling people who do ML machine learning engineers, but the difference between ML engineer and software engineer is getting slimmer, especially since some prominent people call machine learning software 2.0 (). At the same time, data scientist is a job title mostly tied to people who do product analytics and work with metrics, insights, etc. (Please note we are speaking about our experience in deep tech companies, but since these companies employ thousands of highly qualified pros and gradually set standards for the whole industry, we tend to take this approach as a benchmark.)

When people interview for an ML engineer position, they mostly walk through the software engineer hiring loop topped with additional sections, with ML system design being one of the most important. It is used to draw out signals about a candidate’s expertise, maturity, and ability to overview complex systems and decompose them into blocks of interdependent tasks. This is not an easy exercise, as a candidate has only 40–45 minutes to present their design of a system randomly picked by the interviewer.

Eliezer Yudkowsky, a modern AI writer and philosopher, wrote, “The most dangerous habit of thought taught in schools is that even if you don’t really understand something, you should parrot it back anyway” (). It is very applicable to the tech interview flow in some companies: the interviewer provides a puzzle and expects a particular answer to be parroted back. After the interviewee is hired and becomes an interviewer of their own, the bad practice gets reinforced, and the company continues hiring people with perfectly memorized knowledge fragmentarily drawn from various fields. There is no guarantee these people understand the whole picture, and this is what we came across while conducting interviews ourselves.

We interviewed and hired ML engineers for various companies. Some were at the start of their career, some were seasoned experts, and some were solid software engineers switching to ML. However, there was a specific commonality among those who didn’t get through the interview: while working on the ML system design section, they were concentrating too much on details, never getting to the bigger picture.

For us, these failures indicated an expectations mismatch; as young hiring managers, we were convinced that a person who knew all the algorithms, tools, and patterns would be a good fit for the role by default. But later we saw that sometimes people just couldn’t combine their pieces of knowledge into an integrated vision.

In addition, building systems in a real environment is overwhelmingly different from discussing them during interviews. One can learn dozens of popular ML system design questions (“How would you design a job recommendation system for a LinkedIn-like website”?) and be puzzled when a similar problem occurs in real work.

But let’s ignore the interview part for a while. ML experts get hired for a reason: companies need them to build, maintain, operate, and improve systems—and not just for the sake of writing some code or closing Jira tickets. Businesses need reliable ML systems to reach objectives and solve problems.

Building ML systems requires a wide scope of skills. To put it briefly, a person in charge must be able to answer three questions:

There are few people who can combine all these skills at the proper level. However, a lot of ML systems are being built these days, and someone has to design them. From our experience, it is common for an ML system to be designed by either a bright ML expert (because it’s ML) or an experienced software engineer (because it’s a system). They do the job but often struggle in the areas where they don’t shine.

To sum it up, the confusion around ML system design is more typical for candidates with less expertise on the one hand and hiring managers or recruiters who are looking for that jack of all trades on the other. However, if we look at it from the point of view of an executive officer or an expert, a much broader picture appears. They know that you hire these specialists to build, maintain, and improve ML systems, and their end performance working on ML systems becomes the ultimate benchmark of their career growth.

We believe that it’s the fusion of data scientist and software engineer with experience in academic ML that constitutes an expert in ML system design. People who end up designing ML systems may come from various backgrounds—software, practical ML, academic ML, data research—and we hope our hands-on experience aided by small bits of theory will help them close the gaps, systemize their knowledge in the areas they’re familiar with, and feel more confident in the areas where they’re lacking precious experience.

Some projects are simple enough that they don’t require that thorough of an approach. Let’s take construction as an example. You could probably build a shed without an initial blueprint. But when your ambitions spread further to the level of a house or a skyscraper, you can’t get away with not using a prearranged, detailed plan. ML system design is an architectural approach to engineering ML systems that incorporates the experiences of hundreds of experts who have worked in dozens of companies on a multitude of projects.

This process worked, but it had a weak spot: someone had to decide the structure of all these blocks (what are the highest-level components, what’s their structure inside, and so on to the lowest level of implementation). The most responsible decisions were made by software architects—experienced engineers who worked with many systems.

This kind of approach is usually associated with the Waterfall methodology and Big Design Upfront paradigm. In other words, it assumes software projects start slowly and are deeply analyzed and documented before the first line of a real system code is written. This approach was and continues to be reliable but inertial and bureaucratic. In a world of rapid changes, the project could lose its initial sense before finishing.

Opponents of such slow but steady approaches are often enthusiasts of something called the agile software development paradigm. The authors of Manifesto for Agile Software Development () stated four main values:

Let’s summarize this little historical overview: at some point, industry faced a spectrum of software engineering processes, from a heavily regulated one led by software architects to the chaotic, anarchist “screw the hierarchy” hacker-style way of building things. And, as it often happens, things got mixed. More traditional companies tend to become more agile, and most anarchist startups mature, introducing processes and separate roles.

This mixture leads to a consensus that dominates tech companies these days: instead of delegating all the decisions to dedicated people like software architects, we will keep it the responsibility of regular software engineers; let them both design systems and write code for these systems. But this level of freedom didn’t wipe out the initial need for decisions: someone still has to have a final word on how things are structured. Someone has to be responsible for the system design. Every engineer may be involved here and there, but seeing the whole picture is critical.

Skills in implementing low-level pieces of a designed system are not the same as skills in designing a proper system. That’s why deep tech companies tend to have separate interview sections to check a candidate’s skills in writing effective code (aka algorithms section) and designing systems: it’s expected that engineers will wear both hats. The split between those two can be different: usually, junior engineers are silent readers of design documents, and senior engineers are authors or active contributors.

In a nutshell, there is a consensus: a solid software engineer should be able to operate on different levels of abstraction, from low-level implementation to high-level architecture decisions.

Everything we have said so far about the definition of system design is fair for any software—we didn’t mention something ML related. However, not everyone who can successfully design a software system will succeed in designing an ML system—it’s a very specific subset of systems. While designing an ML system, the person in charge should keep in mind many aspects that are not relevant to regular software. In this book, we’ll focus on these aspects; readers interested in more general system design questions can look into other literature.

This book is structured as a comprehensive practical guideline on how to build complex, properly functioning ML systems in various domains, regardless of the size of the company you work for. This guideline includes

Each chapter is a high-level checklist mandatory for every ML system. Note that while not all the items must be fulfilled, each of them must be remembered and considered.

In addition, each chapter answers the question regarding why and when the given item is important. It also includes a description of the landscape (what techniques and tools are suitable for the item). The description is systematized (not just a list of 100 buzzwords), although not necessarily exhaustive, as we believe that an experienced reader will be able to compare the example case with something from their background and draw their own conclusions. At the same time, we try not to slip into a typical textbook or course on classic ML or deep learning.

We come from quite different (and therefore, very intercomplementary) backgrounds: both of us have been involved in ML projects with over 20 years of combined “mileage” in a variety of roles, companies, and environments—from pre-seed startups to multibillion-dollar international corporations. Sometimes we worked long hours as individual contributors. Other times, our work primarily implied rapid team growth and coaching talented and aspiring young engineers. We have witnessed and have been part of successes and failures, big acquisitions, and massive job cuts. And, of course, we’ve discussed a lot of the successes and failures of ML projects with our friends.

But no matter how different our backgrounds are, there’s one thing we strongly agree on: ML projects almost never fail because their participants can’t use algorithms properly. There may be multiple reasons for a failure: a misdirected or completely unnecessary task, sloppy data handling, an unscalable solution with no growth potential—this list could go on and on.

There is a pattern so popular that we’ll have to repeat some stories in different parts of the book: a deep expert in a narrow area focuses a lot on their area of expertise—maybe picks some similar areas but still doesn’t get the big picture. As a result, some important nuances are missed, and it leads to project failure, missed deadlines, and violated budgets.

While books on ML usually provide the “right” answers, our main objective is quite the opposite. What we’d like to do is to teach you how to ask the right questions. These might be the questions you ask yourself, your teammates, users, stakeholders—you name it. Each one of us, as tech industry professionals, accumulates tons of valuable information but can’t always connect the dots. This is where timely questions help structure all the knowledge around us.

We’ve split the book into four main parts so that its structure is in line with the life cycle of any system—research, creation, improvement, and maintenance.

The first two parts are based on the early stages of machine learning system design. Throughout part 1, we’ll focus on the overall awareness and understanding of the problem your system needs to solve and define the steps needed before system development has started. This phase rarely involves writing code and mostly focuses on small prototypes or proofs of concepts. Part 2 delves into the technical details of the early-stage work. This stage requires a lot of reading and communicating, which is crucial for understanding a problem, defining a landscape for possible solutions, and aligning expectations with other project participants. If we compare an ML system to a human body, it’s about forming a skeleton.

The third part is focused on intermediate steps. In this stage of a system life cycle, the schedule of responsible engineers is usually flipped. There is much less research and communication and more hands-on work implementing and improving the system. Here we focus on questions such as how to make the system powerful in multiple dimensions: solid, accurate, and reliable. Continuing our human body metaphor, the system grows its muscles.

The final part is all about integration and growth. For an inexperienced observer, the system may seem ready to go, but this impression is tricky. There are multiple (engineering, mostly) aspects that need to be taken into account before the system goes live successfully. In the software world, a system failure is rarely a disaster like in civil engineering, but it’s still an unwanted scenario. So, at this stage, you will learn how to make your system reliable, maintainable, and future-proof. If you’re not tired of human body metaphors, this is where the system gets a mind and gains wisdom because untamed strength can lead to nothing but trouble.

Overall, the opening chapters will contain more general information, which is nonetheless crucial for framing the problem and sets the core fundamentals for building a well-functioning ML system. The further you go into the book, though, the more complex and in-depth the material becomes, providing you with practical examples and exercises. Starting from the very next chapter, we will introduce two fictional cases, radically different from one another, that we will carry through the whole book. Both will require an ML system to solve their problems, and both will evolve as you continue exploring.

In every part of the book, we always prefer intuition over comprehensiveness. There are many aspects to building ML systems, and each one deserves a book of its own. However, we don’t plan on writing a separate book on data gathering and preparation, another one on feature engineering, and another one on metrics. Instead, we describe the top of the iceberg and review the landscape structure while supporting our thoughts and points with links to noteworthy papers so readers can both familiarize themselves with top-level examples and add their own specific knowledge to the provided framework. We also do not aim to explain details related to particular libraries or engines. We will mention notable examples in certain chapters, but they are only for illustrative purposes in favor of higher-level abstractions.

Real systems are always more complicated than examples we see in blog posts, conference talks, and, of course, interviews. For all of these scenarios, people talk about high-level abstractions, but in reality, the devil is in the details. That’s why we believe getting some intuition on problem-solving is so important: a successful ML system designer should be able not only to recognize some recipe from the cookbook and reproduce it but also to adapt themselves for company-specific details that can flip the table sometimes.

We hope this book will be useful for

Improving the system is even more important. As we’ll describe in more detail in the upcoming chapters, building systems from scratch is a relatively rare event. People outside the industry may think software engineers spend most of their time writing code, while in reality, as we all know, way more time is dedicated to reading code. The same goes for systems: much more effort is usually spent improving and maintaining existing systems (which requires an in-depth understanding of system internals), not building them from scratch.

The difference between improving and maintaining is somewhat blurry. For the sake of clarity here, we define improvements as adding new functionality or changing existing functionality significantly and maintenance as keeping existing functionality working in a constantly changing environment (new customers, new datasets, infrastructure evolution, etc.).

Some principles included in the book are mostly focused on ML system improvement. They help identify weak spots and growth points of a system and even new applications sometimes.

Finally, some principles are more oriented toward ML system maintenance. The sad truth is that very often systems are maintained by teams who didn’t participate in building them. So it’s a double-edged sword: the building team should keep some principles in mind to simplify the lives of their followers, and the maintenance team should understand the principles to be able to understand the whole system logic in a timely manner and find proper workarounds to keep the system alive over a long period of time.

It is safe to say that close to 100% of ML projects that didn’t have a well-written design document have failed, whereas a sweeping majority of those systems that were thoroughly planned found success. Although it should not necessarily be a complex, multipage piece of documentation, and it is often enough to have several pages of condensed information, the design document, in this case, plays two major roles. Not only does it set proper priorities within a project, but it also helps explain whether you actually need this project in the first place and drags your gaze away from the core idea (you might be too focused on the project itself) to see the whole picture. Please see chapter 4 for details.

After working for multiple businesses, we can firmly say that once there’s structured documentation describing all aspects of your system functionality, any activity, from onboarding newly hired employees to applying core changes, is implemented many times faster. Instead of searching for the one and only loremaster who keeps all the knowledge to themselves (but still won’t guarantee precision), you can address a certain document in the library.

To summarize, we strongly believe that arranging a design document, preceded by asking your business the right guiding questions and setting up proper goals, is the key to success for your ML system—or a reason to cancel the project at the earliest stage, which is also a positive outcome, considering how much time, effort, and money you can save by dropping an unwanted activity. We’ll dedicate at least three chapters to this stage of the project, as this is the most crucial part you’ll have to deal with.