Product Management in ML /AI companies
It's complicated
I’m assuming that we’re not questioning whether ML / AI is the right solution that your organization is trying to address. But I do assume that an ML / AI company is building it themselves. Licensing an ML-based solution doesn’t make you an ML / AI product manager.
No domain and ML experience as a PM
Do you need to be a domain expert to work in product?
Let’s have a look at Jua.ai the product which I’m leading right now. We have an ML-based weather prediction model which claims to produce weather forecasts better than what the industry has otherwise.
We also deliver our product through an API to our typical B2B / Enterprise customers.
Let’s take away the Machine learning part for a second and let’s say we just produce weather forecasts. I’m not a weather expert at all and I didn’t feel at all that this held me back. I can still obsess over the outcomes of bad weather as a personal experience and how it’s all connected on a global scale. Will I ever be a meteorologist? Probably not. But I already learned in 4 months a ton about weather and climate.
If this was an ordinary product we would have our answer. No, you don’t. You might not even have to be technical as long as you understand the consumers extremely well.
Machine learning and being ‘technical’
Here’s where it gets tricky.
Machine learning is not just a different language in which programs run. It’s a completely different beast in how things are done from start to end.
It’s very powerful and complex. And chances are you have absolutely 0 clues on what’s going on inside of it all.
Let’s go back to the example. If we add ML to the mix we have a problem. I would be swimming if I wasn’t a technical person at Jua. I’m not a former ML engineer but I know how to develop software. I know it well enough to know intrinsically what problems come with it.
Nothing is ever simple and when something smells complex it’s probably going to break down. Improvements to the product are not happening like in other products. We’re not surfacing a user need and then in a week we have a “solution” or feature for it.
A PM not having any technical background in an environment like this would be like hiring someone that doesn’t know how to operate a laptop. You’re making it unnecessarily hard on yourself. You won’t understand the interconnected mess we’re in. The amount of data we deal with is massive, and the technology itself changes almost every week based on what’s written in research papers.
And you have a lot of different people and functions in the mix. It’s the classical problem a lot of hardware companies have:
There’s a different type of engineer for almost everything. The time when you could separate your engineers into frontend/backend is over.
The effect of it is you have many more breaking points due to more people involved in the entire process. If you ever thought that you have to be defensive in estimating efforts then you’re in for a surprise with an ML product.
There is no process or RICE framework that is going to save you from this, if you don’t have a sense of software architecture in some way you will be stumped by the additional complexity.
Connecting the ‘business’ side with a complicated offering
Machine Learning / AI-driven products follow the same rough rules depending on whom you’re selling to:
B2C: The driving force for your distribution will be Word of Mouth and how easy it is for fans of your product to onboard other users. Simple value proposition, fast time to value etc.
B2B: Word of mouth is still relevant but loses relevance as the main distribution source, it’s more about whether you can generate insights that reach people in B2B companies. This can happen through generated content, insights, and brand distribution over Linkedin. It’s a tough and long game.
Enterprise: Insights alone start to lose relevance unless things are already burning and the companies are actively looking for new solutions. The one thing an enterprise cannot and will not compromise on though is stability. And that comes in 3 dimensions:
The operational stability of your product
How stable your company and offers are to be around in a couple of years.
How reliable and fast you can fix problems in your product
The integration cost for enterprise businesses is staggering no matter how “simple” your tool is. They have to adapt internal documentation, processes, and so forth and it always costs them more than you think.
Enterprises are typically in cost optimization mode for everything and have therefore good visibility over such problems. At least in theory.
How does this matter for Machine Learning products and PMs?
As is typical for new technology that swells up from the ground, they are messy and unstable. While machine learning is nothing ‘new’ it’s a radical change in the mass market.
Depending on whom you’re selling to you might want to prioritize different parts of what you’re offering and develop.
Prioritizing stability and reliability, release schedules, etc. can become actual differentiators in the market early on if you choose to sell to enterprises for instance.
Product Management in more modern environments is very focused on repeating the mantra: Acquisition! Engagement! Retention! but it’s very B2C-centric. B2B typically deals with low-data environments due to having fewer potential users. Combine that with a product that is not easy to optimize and dissect and you have the perfect problem:
You can’t measure what your users do easily
You can’t measure how your product reacts easily
You have to completely relearn what kind of metrics matter. Inside an ML model different metrics matter. And they differ depending on whom you are building for. You need to understand those at least to some degree otherwise you can’t possibly design an experiment to determine whether what you’re planning to change is driving meaningful success for your customers.
What’s the implication for product leaders?
More than ever I am convinced that directive non-collaborative environments in Machine Learning are doomed to fail. We don’t know of any substitute so far for delivering customer insights to engineering and vice versa. You cannot develop a model in a shed somewhere outsourced and then hope that it somehow performs in the market.
Products need adjustment constantly and you need to have a plan on how to deliver customer feedback back to engineering with more accuracy than ever. A simple statement from a customer like,
“Well the output just wasn’t that good, it was confusing”
can turn into a massive project to figure out what was meant and it likely won’t end up in a simple UI fix.
Have a strategy for data sources
The more good data your model has the better. You need to have a product strategy to make sure you’re not suddenly cut off by it. Even if you don’t license anything, if you rely on scraping specific sources what will you do if those are suddenly not accessible anymore due to regulation or other events?
Observability is not a nice to have
It is an absolute must as a PM to have an airtight view of what your model is doing. It goes beyond service uptime and whether it’s delivering “something”.
You need to make sure that you can evaluate the performance of the model continuously. Is the data/output by the model delivered accurately? Is the product doing what it’s supposed to?
Sometimes the only saving grace for that is to have a tight grip on user success outcomes. It might be that you deliver something correctly but the user doesn’t understand it. That still means failure.
You cannot sit in your ivory tower and control everything from your chair. You need to be closer than ever with your customers and that means talking to them.
The gist
If you are a PM managing a machine-learning product and you are not great at gathering qualitative feedback you need to fix that immediately. If you don’t have at least some technical understanding of what’s happening with AI you’ll land in rough waters as well.
Data literacy is also necessary to some point due to it being the lifeline of any Machine Learning model.
Or you just accept that don’t want to work in AI/ML ever. Depends on whether that’s an option in the long term of course :)
Generally, applicable advice on what to “do” now is difficult as there are different degrees in any company to how “close” you are to the model itself.
At the very least familiarize yourself with popular ML tech by using ChatGPT or other services… learn their limitations and how it works under the hood. Whether you want to go all out and learn about computer science is in the end all up to you.
But we might be facing a future where not being technical at all for product managers simply doesn’t exist.
Maybe.