Using Metrics To Verify AI-Driven Productivity Increases (Or Decreases?)

This is the fifth in a series of posts, entitled “AI, The Panda Strike Way,” exploring the strategic approach we take to help our clients navigate the risks and realize the massive quality payoffs of the AI era.

In the last post in this series, we discussed the importance of measuring outcomes in our software processes, particularly when adopting new technologies like AI. We asked: How can we know whether AI is making us more productive—or perhaps even less so? Data-driven decision-making is particularly important given the hype surrounding the technology and the heavy reliance on claims from vendors with a vested interest—or on the subjective experiences of developers. Making matters worse, even seemingly reliable metrics (like raw PR counts), when used in isolation, can be gamed—Goodhart’s Law—or mask underlying problems (like rising defect rates).

However, before we throw up our hands and declare software productivity entirely unmeasurable—which is a conclusion some researchers have drawn, so it’s far from the worst answer we could give—let’s see if we can find a path forward.

Choose Metrics In Tension Permalink

Fortunately, we also have evidence that metrics can reveal useful insights when used in combination, as in the SPACE framework. At the same time, we want metrics we’re likely to already have across a wide range of projects and over a long enough period to establish a pre-AI baseline. This tends to rule out many interesting approaches grounded in the research. So the question is: Can we distill from the scholarship an approach that reasonably approximates what we’re trying to measure, using data that’s practical to collect?

Let’s choose two metrics that (a) we’re likely to have available and (b) are in tension: median PR turnaround time and defect rate. Being in tension means it’s difficult to game both at once. For example, if we simply skim PRs to cut our turnaround time, our defect rate is likely to increase. Similarly, all else being equal, reducing our defect rate probably means spending more time on PRs. Thus, if we’re improving both metrics at the same time, it’s a strong signal that our productivity is genuinely improving.¹

Normalize Across Projects Permalink

Since we want to normalize these across projects, removing variables like team size or cadence (the duration of sprints or iterations) from the equation, we’ll use the relative change on a given project rather than absolute metrics. We do this for each metric by:

1. Picking a starting period for our baseline, large enough to give us an adequate sample size.
2. Adding one to avoid division-by-zero errors (or near-zero inflation) in step 3.
3. Taking the inverse of the measure (so that larger values are always better).
4. Scaling that baseline value to 100, and normalizing future measurements relative to it.

Let’s call the resulting scores our effective PR rate (EPR) and effective defect rate (EDR). To get an overall score for a project, we take the harmonic mean of the EPR and EDR, which (among other things) makes the result even harder to game. And, as a final flourish, we can use a weighted harmonic mean—weighted by team size—to roll project scores up into an organizational metric.²

Our pre-AI baseline will be 100, so if we see a number like 110, that’s a 10-point gain, indicating that our productivity is increasing. Conversely, a number like 90 suggests productivity is decreasing.

There are a few nuances to how we compute this. For example, we’ll want to group smaller projects together—or use larger sampling intervals—to avoid skewing results with projects that lack enough data to be reliable on their own. Similarly, we’ll want to filter out inactive projects. The good news is that, so long as we’re careful, we can produce a meaningful estimate of changes in our productivity over time.

Calculating The EPR Permalink

A few additional notes on the EPR:

• For smoothing (the “add one” in step 2), pick a sensible unit relative to the median PR turnaround time—typically whatever unit you already measure it in, such as one day.
• Don’t forget to count PRs carried over from a prior period.
• To avoid rewarding poor PRs, multiply the turnaround rate by the acceptance rate before scaling to 100 in step 4. For example, if your smoothed turnaround time is 2 days and your acceptance rate is 75%, the pre-normalized EPR is ½ per day × 75% = ⅜ per day.
• If possible, use code churn (how much code is modified or deleted shortly after being committed) as a check on the EPR and an early warning sign that defects are likely to increase.

Know Your Limits Permalink

Even so, our overall metric is still only an estimate and doesn’t truly measure productivity. It’s simply a signal. We can confidently say that an improvement of 20 points means our productivity is probably increasing, but we can’t really say that it’s improved by 20%, per se. All we can say is that the metric has improved by 20% and that we believe this is a strong signal. And it still rests on key assumptions—for example, that, organizationally, we’re building things that create business value.

And, again, this presumes that PR turnaround and defect rates are measurable across projects. We discussed the importance of software process in making effective use of AI in a previous post in this series. But we can at least keep those requirements to a minimum: Are you (a) reviewing updates to your code and (b) capturing defects in a bug tracker? Regardless of how you construct your productivity metric, you’ll need at least some basic software process that’s consistently followed over a large enough sample of projects. Without that, you literally cannot measure your productivity—so you can’t know for sure whether AI, or any other tool or technology, is helping or hurting it.

1. In the previous post in this series, we referred to defect density, or defects per line of code. However, a better measure for our purposes is the defect rate—the defects produced in a given time interval. After all, as we’ve already discussed, code volume is a poor proxy for productivity, particularly when AI agents can generate large amounts of code. With a defect metric, we’re trying to capture quality, and defect rate yields a cleaner signal than defect density. ↩

2. To see why using a harmonic mean makes the metric harder to game, consider a scenario where we double our EPR (a score of 200) but also double our defect rate (halving our EDR to roughly 50). A simple average gives us 125, which looks like an overall improvement. But the harmonic mean gives us 80, suggesting a decline, which more accurately reflects the distressing spike in the defect rate. ↩