Is AI Actually Making Your Software Developers Less Productive?

In our previous posts, we’ve discussed how best to adopt AI tooling into your software development workflows. But how can you tell if it’s working? How do you know if your teams are being more productive? Or if the AI tooling is introducing bottlenecks? And if they are being more productive, how can you be sure it’s due to the AI and not due to something else?

If your organization already actively collects productivity metrics, you’re in a good position to see whether productivity is improving after adopting AI tooling. Even then, however, this is a great time to review your metrics, since the use of AI itself can affect the metrics you should be looking at. And if you aren’t collecting such metrics yet, this is an opportune time to start.

Which Metrics? Permalink

But which metrics? You might think that, by now (in the year 2026 as I write), this question would be long settled. However, what we’ve learned over the years is that conventional metrics—while useful for measuring things like the likelihood of project success—aren’t great at measuring productivity.

What’s more, there’s also just the question of how we define productivity in the first place. Does productivity incorporate quality? Does it incorporate the likelihood of success? And how can we integrate all these factors into a single measure that tells us whether we’re getting more done?

What Are We Measuring? Permalink

We also have to consider the possibility that AI tooling itself might affect the way that we measure productivity. After all, more of the work is now being done in an automated fashion, meaning that some of the things we measure might no longer be as highly correlated to how much we’re getting done.

Pull requests are a great example of this. We often hear about how many pull requests are now being generated by developers using AI tools, but that doesn’t translate directly into productivity. Not only does that not speak to the quality of the code, but that code still needs to be reviewed by a human developer.

In other words, AI adoption may change the meaning of any given metric because it changes how we work. Fortunately, there’s been some excellent recent scholarship on measuring productivity in software development that also takes into account the impact of AI tooling.

What Does The Research Say? Permalink

We have to be careful here because a lot of the big vendors who sponsor or even author these studies are the same vendors trying to sell AI tooling. So there’s an inherent conflict of interest. For example, a study of GitHub (now Microsoft) Copilot usage, which includes authors from GitHub (and Microsoft), claims that using Copilot resulted in more than a 50% improvement in coding speed. That doesn’t mean the study is irrelevant, but rather that we should maintain a healthy skepticism about the results.

With that caveat out of the way, one of the more interesting papers (from Microsoft Research) proposes a productivity framework called SPACE, an acronym for satisfaction (and well-being), performance, activity, communication (and collaboration), and efficiency (and flow). While the paper isn’t terribly prescriptive, it provides a nice overview of the ways we can measure various aspects of the software development process.

The authors of the SPACE paper caution against relying on a single metric. For example, if an activity metric, like ticket close rate, improves, but satisfaction metrics decline, the apparent productivity gain may be a result of developers simply working harder.

In our next installment in this series, we’ll explore some more specific metrics, like PR turnaround and defect density rates, and how we might effectively measure them across teams and projects.

Until then, let us know what metrics your organization is using (or which ones you think they should be using). Is there a strong evidence-based case for the claim that AI boosts developer productivity? Does your own experience (or that of your colleagues) support such claims?