Learnings from AMLD — Getting ML into production by Mikio Braun

Mikio Braun held a great talk at AMLD 2020 on the topic of getting ML into production. Mikio is staff scientist at GetYourGuide and was previously working as senior datascientist for Zalando. He might have an idea or two about applied ML, so I thought it’s worth sharing my takeaways from his talk.

His main theme: How do we “do ML”? Is algorithms + tools + data all there is to it? Maybe, but surely they all have further facets to explore. Mikio focuses on three topics:

1. Where and when to apply ML (and when not?)
2. In which architecture or pattern?
3. Using which teams & organizational setups?

Where to apply ML?

It usually helps to visualize value streams for identifying where ML could be applied. This provides a great tool to assess opportunities and discuss them with business partners.

From value stream…

…to potential ML applications.

Second, you can only apply ML where there is good, curated data. Not only must data be available, but it needs to be maintained and catered in an adequate way. This goes back to Takeaway #1 from my AMLD Keynotes post. Mikio refers to the famous AI hierarchy of needs. Often engineering teams (e.g. front-end devs) are not aware of the importance and implications of the data they are producing. But since it’s the fuel to analytics machines, it’s crucial to get this right.

Third, connecting the Business Problem to the ML Problem is non trivial and stuff gets overlooked. Very seldomly is ML the direct solution to the client’s problem. ML often only indirectly supports tasks the company does to solve the client’s problem.

Let’s take Recommendation for example. The business problem might be to find products “similar” to what the client was interested in earlier. But, since “similar” is hard to program and hard to explicitely automate this notion needs to be connected to an explicit, analytical notion, so it can be leveraged by ML. This is where the disconnect happens and where we have to be very careful. The business and the ML metric are often different.

If we fail to come up with a clean connection between the business problem and the ML problem, it might be better to stop, rather than to pretend.

Takeaway 1: Apply ML only where it makes sense along your value chain, where you have the data and where you are able link it smoothly to the business problem.

What architectures or patterns?

ML is usually just a (very) small part of an overall process pipeline. Datascientists tend to focus on this small part and lose focus on the rest (e.g. data extraction, monitoring, post-processing). However, this rest becomes crucially important in the overall architectural picture. In the last few year’s we’ve seen architectural patterns (think “design-patterns”) emerge in many of them.

For integration of ML into the larger application, Mikio lists four main ways:

1. Scoring: A candidate set, loaded e.g. from the backend database, is scored by the model and the model returns the top candidates as result (e.g. recommendation).
2. Preprocessing: The model receives inputs from the application and processes these in some way (e.g. object detection).
3. Prediction: The model is asked to estimate some probability (e.g. p(a|s)).
4. End2End: The model itself is directly integrated in the application process

For serving he lists three patterns:

1. Embed model in service that runs naturally
2. Precompile and put the “rules” into a databse (e.g. compute once a day and put in database for system to fetch)
3. Serve the model directly. This needs preprocessed features (e.g. how often client bough last 10 days) in a database or other service that is available.

For preprocessing Mikio listed two patterns:

1. Preprocess Batch-wise and load the batch features into a feature store using a batch-update at a certain interval.
2. Preprocess Online. Directly calculate features once new information arrives and save them into feature store.

Takeaway 2: Be aware of ML design patterns and decide consciously which patterns you chose in your own projects.

Which team & organizational setups?

Two main questions dominate the organizational setup: How should Datascientists work with software engineers? And: How should Datascientists work accross the company?

Working with software engineers

Datascience is iterative and hypothesis driven by nature. There is a lot of uncertainty. Classical developement on the other hand starts often more focused with clearer outcomes in mind, projects tend to work on the same code-base for years. These are very different modes. If there is a lack of willingness to understand the other side bad things happen.

There are two potential modes of working with software engineers:

• ML people experiment and design a new model & process, then handover the docs, then engineers build.
• IT builds a system with a very clear interface and datascienists have to learn to satisfy this interface. Often the preferred way (if the organizational setup allows for it)

Working accross the organization

Organizational Patterns… almost a classic at this point. Mikio lists three patterns, each of them has different strengths.

1. Center of excellence is especially good for starting out, to collect experience quickly and effectively and get enough “fire power” fast.
2. Over the fence-Style, in specialized Silos. This works well to optimize each team’s local goals, but overall-process gaps ensue sooner or later.
3. Cross-functional teams are best in terms of optimizing the overall-product. To keep up expert exchange and leverage synergies e.g. among datascientists it’s helpful to set up “guilds” or “tech-exchange groups”.

Big Companies have decided to separate ML from engineering teams currently. Air BnB is an example, Facebook supposedly as well. But in general people don’t really know yet, companies are still figuring out when to apply which model. And for sure factors like company size and product types makes a difference.

Takeaway 3: Datascience requires collaboration accross the organization, design your working-mode with other teams accordingly.

This is part of a series of summaries from AMLD 2020. Follow and check my other work for more articles on applied machine learning and advanced analytics.