This blog post is an updated version of part of a talk I gave at GOTO Amsterdam last year. The talk is also available for watch online.
Delivering value and positive impact through machine learning product initiatives is no easy task. One of the main reasons for this complexity is the fact that, in ML initiatives developed for digital products, two sources of uncertainty intersect. On the one hand, there is the uncertainty related to the ML solution itself (Will we be able to predict what we need to predict with sufficient quality?). On the other hand, there is uncertainty related to the impact that the entire system will be able to provide (Will users like this new functionality? Will it really help solve the problem we are trying to solve?).
All this uncertainty makes failure in ML product initiatives relatively common. Still, there are strategies to manage and improve the chances of success (Or at least survive through them with dignity!). Getting machine learning initiatives off on the right foot is key. I talked about my main learnings in that area in a previous post: start with the problem (and define how the predictions will be used from the beginning), start small (and keep it small if you can), and prioritize the right data (quality, volume ). , history).
However, starting a project is just the beginning. The challenge of successfully managing an ML initiative and generating positive impact continues throughout the project lifecycle. In this post, I'll share my top three learnings on how to survive and thrive during machine learning initiatives:
- Embrace uncertainty: innovation, stop, turn and failure.
- Surround yourself with the right people: roles, skills, diversity and network.
- Learn from data: right direction, being able to improve, detect failures and have a plan.
It is really difficult (even impossible!) to plan ML initiatives in advance and develop them according to that initial plan.
The most popular project plan for ML initiatives is the Machine Learning Lifecycle, which divides the phases of an ML project into business understanding, data understanding, data preparation, modeling, evaluation, and implementation. Although these phases are drawn as consecutive steps, in many representations of this life cycle you will find arrows pointing backwards: at any point in the project, you can learn something that forces you to return to a previous phase.
This results in projects where it is very difficult to know when they will be finished. For example, during the evaluation step, you may realize using model explainability techniques that a specific feature was not coded well, forcing you to return to the data preparation phase. It could also happen that the model is not able to predict with the quality you need and could force you to go back to the beginning in the business understanding phase to redefine the project and the business logic.
Whatever your role in an ML initiative or project, it is key to recognize things won't go according to plan, to accept all this uncertainty from the beginning and use it to your advantage. This is important for both managing stakeholders (expectations, trust) and for you and the rest of the team (motivation, frustration). As?
- Avoid overly ambitious time or delivery constraints, ensuring that machine learning initiatives are perceived for what they really are: innovation that needs to explore the unknown and has high risk, but also high reward and potential.
- Know when to stop, Balancing the value of each incremental improvement (ML models can always be improved!) with their price in terms of time, effort and opportunity cost.
- Be prepared to swing and fail, continually taking advantage of the learnings and insights that the project gives you, and decide to modify the scope of the project, or even eliminate it if that is what those new learnings tell you.
Any project begins with people. The right combination of people, skills, perspectives and a network that empowers you.
Gone are the days when machine learning (ML) models were confined to the data scientist's laptop. Today, the true potential of ML is realized when models are implemented and integrated into business processes. This means that more people and skills need to collaborate to make it possible (data scientists, machine learning engineers, backend developers, data engineers…).
The first step is to identify the skills and roles that are required to successfully build the end-to-end machine learning solution. However, it requires more than a set of roles that encompass a list of skills. Having a diverse team that can bring different perspectives and empathize with different user segments has been shown to help teams improve their ways of working and create better solutions (“why having a diverse team will improve your products“).
People don't talk about this enough, but the key people to executing a project go beyond the team itself. I refer to these other people as “network”. The network is made up of people who you know are really good at specific things, who you trust to ask for help and advice when needed, and who can unblock, accelerate or empower you and the team. The network can be made up of your business stakeholders, the manager, staff engineers, user researchers, data scientists from other teams, customer support team… Make sure you build your own network and Identify who is that ally that you can turn to depending on each specific situation or need. .
A project is an opportunity for continuous learning and many times the learning and knowledge comes from verifying the correct data and monitors.
In ML initiatives there are three large groups of metrics and measures that can provide a lot of value in terms of learning and ideas: model performance monitoring, service performance, and final impact monitoring. In a previous post I delved into this topic.
Verifying the right data and monitors while developing or deploying machine learning solutions is key to:
- Make sure you are moving in the right direction: This includes many things, from designing the solution correctly or choosing the right features, to understanding whether the project needs to be pivoted or even stopped.
- Know what or how to improve: understand whether outcome goals were met (e.g. through experimentation or a/b testing) and drill down into what went well, what didn't, and how to continue delivering value.
- Detect failures early and have a plan: to enable quick responses to issues, ideally before they impact the business. Even if they impact the business, having the right metrics should allow you to understand the why behind the failure, keep things under control, and prepare a plan to move forward (while maintaining the trust of your stakeholders).
Effectively managing machine learning initiatives from start to finish is a complex task with multiple dimensions. In this blog post I shared, based on my experience first as a data scientist and then as an ML product manager, the factors that I consider key when approaching an ML project: accepting uncertainty, surrounding yourself with the right people and learning from others. data.
I hope these insights help you successfully manage your machine learning initiatives and drive positive impact through them. Stay tuned for more posts on the intersection of machine learning and product management 🙂
