AI introduces Tech Debt into your product even if you don't build AI products

Beep bop. The AI wave is coming and your lifeboat is not good enough.

(This is a crosspost from my other main blog, with time I will publish all AI topics exclusively on this substack)

Product / Growth Managers have to expand their knowledge profiles a lot. One of those is how to manage tech debt. It’s a really sticky situation and far from any fun but if you ignore it you risk your product and team’s influence.

The one thing you can get buried by easily in tech is debt. Tech debt.

The typical cycle of “I’ve been here for 2 years and my backlog is an unmanageable avalanche, I need to quit”

Understanding it as a form of investment can unlock your velocity hard for your product team in the long term. Some key components are changing now due to AI and you should be prepared to deal with them or you get swooped up.

Product Managers are getting buried by tech debt more often than not.

I wrote another article (paid) on the original topic here on my other substack

Tech Debt is an investment, not a burden

Let’s set the scene. What hasn’t changed in the past months is who your ally is in any organization. It’s still your engineering manager, the VP of Product/Engineering, or the CTO.

What has changed and will accelerate is how your engineers are operating. They become more efficient. And some of our antiquated processes don’t work anymore. More efficiency doesn’t mean necessarily more impact though.

I had a particularly interesting discussion with a PM who told me that their teams seem to use ChatGPT without disclosing it at times in places where they shouldn’t. Responding to tickets and PRDs and while it seems this way it’s getting harder and harder to spot. The problem is not the usage of ChatGPT but those responses just eating up time without producing too much tangible outcomes.

Point estimations are also completely useless now for some types of stories and tickets.

What’s the connection to tech debt?

If we’re thinking about the classical cycle of having a hypothesis, tying this to an OKR or Strategy, and then building features to address those we usually also need to decide on the scope of them.

“How solidly do we build them, and how well is the feature maintainable? How much of an MVP should it be? Do we fix it afterward?”

The number of MVPs

The more MVPs you produce the more tech debt you introduce by definition. And you can do this intently by understanding that some of these MVPs will yield a lot of impacts, compensating for the others.

If we assume that your teams will produce more with less this number is starting to steadily increase.

Another compounding factor is that every company now wants to potentially add “AI” to their pricing page so there are a ton of feature requests coming in left and right. The need for cadence has increased dramatically.

The quality of MVPs

But as we said, if tech debt is an investment then it comes at a cost. Every MVP introduces a little bit of interest that you have to pay back. Unfortunately, early anecdotal reports are pointing in a not-so-good direction. While it’s easier to produce something, the quality is not necessarily better.

What we still haven’t solved with AI is the integration of whatever code it finds into an existing codebase. You still need to make it work on a bigger scale and integrate it well. Code doesn’t just work by itself, it’s a part of a bigger codebase no matter how clean it looks from the outside.

This creates an unfortunate effect: the hit rate of your MVPs to generate meaningful impact with your customers is going down. That means you should more often than not not even ship anything simply because the underlying code is not good enough anymore. The other problem is that QA is becoming a nightmare:

QA for MVPs

Ok, we produce more at a worse quality by definition. While QA for code snippets may have arguably become easier - actual QA for interfaces and overall flows has become harder.

I also have never met anyone that particularly enjoys QA duty and oftentimes product teams are structured nowadays in a way that whoever builds something is running it. Meaning that engineers are starting to do more QA work than engineering work.

How that shift turns out we’ll see but if the majority of your day is assigned to architectural firefighting and QA this might not be the job you signed up for originally. A decrease in morale never helped code quality overall.

The big killer for PMs. Validation

Bringing it all together this kills most PMs day to day. On one hand, the market is flooded with competition to no end and is projected to accelerate due to further advancements in AI. So it gets harder to find the right things to build to develop any impact.

An antidote to that is to get a tight grip on validating your outcomes. That takes time, a lot of time. Solid hypothesis, post-experiment analysis. Maybe you freed up some time yourself due to AI by not having to write that many PRDs (I doubt it though) but this is an additional burden.

At the same time, you have much more output to validate.
So what are you supposed to do?

Do less, but better? Or do more and just close your eyes on quality?

The solution is in metrics and the pulse from your engineers. Let’s look at how to leverage this in an OKR and what potential Health Metrics could look like, they will save you.

Have a plan, leverage that debt

Look beyond your short-term planning. If you’re fortunate to work in an A/B testing-driven culture you probably have already a framework to measure success.

What’s often forgotten is a dedicated resource that deals with introduced tech debt.

But just dedicating 20% of your time never works in an industry that struggles with tieing anything to hourly attendance rate.

So what are you left with? Your old companion, outcomes. You definitely can do a rough analysis of how flipping annoying it is for your engineers to deal with certain problems. From the obvious benefits like happier people, you can actually put numbers to this.

If we focus on our outcomes and customer success in the number of tasks they do with our products with can do the same for our engineers.

Don’t start to measure the number of lines they produce, a surprising metric here is the subjective opinion of engineers working with specific systems. Engineers are exceptional at knowing what they actively hate working with.

Let’s say your CI/CD pipeline is just in an absolute disarray and you know it’s being used x amount of times per day by x amount of teams then there is a case to be made to actually “fix” it. How? By targeting what annoys engineers. Tie together the collective annoyance of people by the number of times that something is used and what impact it has on the business.

As a product or engineering leader, this is far more impactful than reserving a certain amount of time to tech debt that never gets used.

I use this frame of mind for product teams for their OKRs

1. 1 Optimization Bet: What is the biggest lever we can have by optimizing our current product team’s influence (optimizing onboarding flows, or existing features, etc.) and what is the expected uplift?

2. 1 Innovation Bet: What is a moonshot that is totally blowing the lid off but has a low chance of succeeding by providing something new? What’s the potential upside?

3. 1 Techdebt Bet: If we could magically fix one thing that we introduced in the past, what would it be? What would make us internally much much happier and affects the majority of either our users or us internally? You are allowed to go beyond your team’s influence area. (Tech Debt often times lives across teams)

That doesn’t mean that for every growth or product team you will have an evenly distributed OKR goal setting but if you look at it this way you will start to see opportunities in a different way.

We still have to balance our goal setting overall in product portfolios but I need teams to think in this way.

Nail that tech debt bet

What metrics should we look out for when we think about tech debt?

As usual, 90% of you that are reading this are in product teams that don’t know how to make any of the following visible. Focus your job on this, be creative and keep. it. simple.

The metrics for that haven’t changed from my original article with one little exception:

1. Stability metrics like uptime and intelligent metrics for heavier users on how successful they are. If this drops too much it needs to be prioritized → Heavier users are more affected by stability issues on your platform, and intelligent tracking of their journeys is a powerful tool to make sure this important cohort of users is not forgotten.
   Stability metrics can be more flexible if you are in a highly volatile market like AI features (Users do expect some rough times)
   → Done by: Product & Engineering

2. Risk analysis metrics Security risks are always fun. We cannot wait for them to happen but they could have a big impact. Expecting your teams to sit down and evaluate all risks on 2 dimensions once every year makes a lot of sense:

   1. How likely is it that this risk is going to occur?

   2. How bad is the damage if it occurs?

   A good example is a downtime of 5 hours. Maybe a year ago this cost you 10k in revenue. But now since you have 10x your user base the damage is dramatically scaling up. Risks change with scale and making this visible is important. The bigger you get the more likely they also start to appear. It’s a vicious cycle.
   I.e. It’s common for services to be interrupted if a team deploys new code. With 1 team that means maybe a 5-minute interruption for the customers per day. With 100 teams that become an unmanageable mess and catching up with infrastructure and pipeline projects is an absolute nightmare at this stage.
   Listing those risks with the team is a team exercise, putting a number on the bad ones is the job of a good product manager. This makes them comparable
   → Done by: Product & Engineering

3. Custom metrics: Whatever you need to measure tech debt, as long as you have buy-in from your leader and establish the correlation they can make the case for you afterward. Help me help you defend whatever you’re coming up with. Buy-in is not generated by convincing me as a product leader it’s generated by giving me tools to defend this against others.
   I’m not going to the CEO and say “Well they focus on tech debt for 3 months but I don’t really understand why, so f*** your sales requests.”

Don’t stop here. If you succeed and get that impact from handling tech debt that you hoped for SHOW IT. Visualize it, celebrate it. If you don’t make it visual on the regular what you did for the company you will be put on the bench and go right back where you started: producing features because Gary from Sales said so.

Keep that backlog clean

It’s cool that your backlog is collecting more and more ideas but it’s no use if you never get to finishing any of it. I’m a big fan of a simple process:

• We meet regularly and clean out epics (I’m working at that level, stories work too) on a bi-weekly basis

  • The team can move interesting stories onto the save bench which protects them from elimination for 3 months

  • After 3 months the thing gets deleted unless it’s completely rewritten. This might sound stupid but there needs to be some “punishment” for insights that we don’t leverage. If they are not good enough to be dealt with right now they will never be dealt with.

• Be realistic, don’t use your backlog as an excuse to deal with requests from others just because you’re afraid of saying no.

Tech Debt has this annoying habit of creeping up on teams.

Don’t ignore it, it won’t ignore you either.