3 Leadership in AI / ML learnings
Why is everything so complicated
This is written from my perspective as the Head of Product at Jua.AI. We build an end-to-end Machine Learning model for weather predictions
Trust and your users
No matter how watertight you build your QA and testing coverage. Things go wrong with AI products all the time. You change something on one end and somewhere else things break without anyone noticing.
While it should be self-evident you can combat this by having superior support. Easy to contact, fast to handle, and relevant. The correct way here is not to try to apply tests everywhere.
Specifically, ML products are constantly changing due to them being driven by research papers. That means your code architecture changes which makes test coverage even harder.
Move fast and break sh** but also correct it fast. Specifically, incentivize teams on reaction time. (I have a team that has an OKR to discover and react to failures <1h)
The curse MVP building
There’s no such thing as an MVP if your product is out there and is fueled by ML/AI. The number one mistake I have to correct is usually around tracking.
I don’t let any product iteration go out without watertight tracking. There are no two ways about it. If you cannot possibly understand what’s going on in a Machine Learning model (the typical ML black box) then you have to have a tight grip on your tracking.
You can for instance notice that something is wrong with your release if you see people start to visit your support page at an increased rate. It doesn’t tell you what’s wrong but at least you know something is off.
If you’re not looking on the regular or setting up custom alerts on these outcome metrics (for instance how many documents your users process) it’s all for nothing as well.
People commonly understand Minimum Viable Product under MVP. I advocate to understand it as a Minimum Verifiable Product though. If you don’t think through how it verifies anything you failed. Just running and hoping to see something interesting in the data is not how things should be.
If the customers are not unhappy with it, you don’t need to wrestle control with testing.
If there is no tracking you won’t ship.
Garbage In - Garbage out
I need every model team, every AI team to think about Data that comes into their model. While Machine Learning is amazing it’s terribly easy to break. If your product relies on being retrained all the time, how do you ensure access to this data source, whatever it is?
Are you really willing to risk your entire product if you don’t have an alternative plan on how to secure data access to it?
Having a secure failure scenario especially if you rely on public data (so no guarantee that you will have access forever) is something I never had to consider.
But it’s common sense when you license business-critical technology. We use SLAs there to ensure uptime because we pay money. If you create public content or images for free who says that you have access to them in a good enough quantity in the future?
You don’t. At least be prepared for it.