Chapter 3 — The Productivity Paradox

"An enterprise AI mandate is a process-redesign problem, not a tooling deployment: the gain grows with use rather than arriving overnight, and it relocates work downstream rather than removing it."

— He, Agarwal, Denisov-Blanch, Azaletskiy, Koyejo, Vasilescu, arXiv 2607.01904 [B]

3.1 A gradient, not a contradiction

The literature on AI and developer productivity looks like a shouting match. It is not. It is a well-behaved gradient that nobody plots, because plotting it requires citing studies that disagree with you.

Here are the five best-measured points, all verified against primaries.

+55.8%. Peng, Kalliamvakou, Cihon, and Demirer recruited developers and asked them to implement an HTTP server in JavaScript as quickly as possible. The group with Copilot finished 55.8% faster [A]. This is the number in every vendor deck. The task is greenfield, self-contained, specified in a sentence, and touches no existing code.

+26.08%. Cui, Demirer, Jaffe, Musolff, Peng, and Salz pooled three randomised controlled trials run at Microsoft, Accenture, and an anonymous Fortune 100 company as part of ordinary business. Across 4,867 developers, completed tasks rose 26.08% — with a standard error of 10.3%, which is a confidence interval wide enough to deserve mention every time the point estimate is quoted. Published in Management Science [A]. Less experienced developers adopted more and gained more.

+21%. A Google enterprise trial put 96 engineers on a realistic internal task — a logging service, ten files, 474 lines of code — with and without three AI features. The AI group finished in 96 minutes against 114: 21% faster. And then the honest part, which is why this study deserves more citation than it gets: when the authors controlled for developer-level and task-level factors, the effect held at roughly 21% but lost statistical significance at p < 0.05 [A]. The authors caution explicitly against assuming transfer to other tools, tasks, or environments.

+40%. Heilman and colleagues estimated a within-engineer dose–response across 16,223 Microsoft engineers, finding roughly 40% more pull requests in developers' heaviest-usage weeks, robust under a falsification battery [B].

−19%. METR's randomised controlled trial gave 16 experienced developers 246 real tasks in repositories they had worked in for an average of five years. AI-allowed tasks took 19% longer, CI +2% to +39% [A].

The gradient has an obvious reading and a less obvious one.

The obvious reading: AI helps most where there is least existing context to respect. A blank file has no invariants, no conventions, no five years of decisions your change must not violate. As those accumulate, the balance of work shifts from producing to reconciling, and the tool that accelerates production stops mattering.

The less obvious reading is about instruments. Notice that the two studies with the tightest experimental control — Peng et al. and METR — sit at opposite ends. Rigour does not determine sign; task selection does. Which means an unscrupulous advocate need not fake anything to get any answer they want. They need only choose a task. This is why "is there an RCT?" is the wrong question to ask of this literature, and "what was the task, and whose codebase was it?" is the right one.

3.2 Three instruments, three answers

Before the mechanism, the epistemics, because a reader who has not resolved this will not accept the mechanism.

We have three ways to ask whether these tools help, and they disagree systematically.

The third row is the one almost nobody cites, and it is the most interesting. When METR ran a follow-up, so many developers declined to work without AI that the selection effect forced a redesign of the experiment [B]. These are not survey responses. They are choices with a price attached, and the price was declined.

The temptation is to pick a winner. Resist it, because each instrument is valid for something different and the disagreement is informative.

Stated preference measures experience, and it is wrong about duration. The METR result is not that developers lie. It is that they were 19% slower and could not feel it, after living through it. Something about the experience of delegating — less waiting on your own recall, less friction, more of the session spent reading fluent output — registers as speed regardless of the clock. Taleb's general warning about introspection applies with unusual force: the feeling of a process is not evidence about the process. Chapter 18 takes this apart.

Measured outcome answers a narrow question precisely. METR measured task completion time for experienced developers in familiar repositories. That is exactly what it measured. It did not measure whether those developers could now do tasks they previously would not have attempted, or work on unfamiliar parts of the system, or sustain output across a longer day. Those are different questions and the instrument is silent on them.

Revealed preference aggregates everything the participant values, and cannot tell you what. The developer refusing \$50/hour may be avoiding tedium, or protecting a skill they no longer maintain, or genuinely faster in ways the task-timer missed, or habituated. All of these produce the same refusal. Revealed preference is a total, and totals do not decompose.

So the honest synthesis is not "they were wrong" or "the study was wrong." It is:

These tools reliably improve the experience of the work, reliably improve throughput on well-scoped tasks, and do not reliably improve completion time on expert work in familiar code. All three can be true because they are three different quantities, and only one of them is what an organisation buys.

That last clause is the pivot into the rest of the chapter. An organisation does not buy experience, and it does not buy task-completion time. It buys delivered, working software. Neither of the first two instruments measures that.

3.3 The best-documented case we have

In June 2025 the CTO of a mid-sized commercial B2B software company sent an email to an R&D organisation of several hundred engineers. The goal: double engineering productivity through AI adoption over twelve months. The designated progress metric: merged pull requests per engineer per month.

Researchers from Carnegie Mellon and Stanford were given access to the telemetry. The resulting study covers 802 developers and 196,212 non-bot pull requests across 364 repositories, from January 2024 to April 2026 [B]. The authors describe the site as near-ideal and unusually favourable: young, AI-forward, permissive, provisioning commercial tools with no per-seat caps or token budgets. If large gains exist anywhere, they should appear here.

They did.

• Mean pull requests per active developer rose from 21.2 in the pre-mandate baseline to 44.3 in April 2026 — a 2.09× increase.
• Company-wide pull request volume grew 3.1×.
• Token spend among pull-request authors rose roughly 20×.
• The share of AI-authored pull requests climbed from near zero to about 90%.
• Among the 246 developers present throughout, mean throughput rose 2.3×.
• Controlling for calendar time, the within-developer gain was +0.54 log points (1.72×), and +0.38 (1.46×) under full calendar-month fixed effects — conservative, and still large.

The mandate was met. Per-capita merged pull requests roughly doubled in twelve months.

This is an inconvenient result for the pessimistic reading of §3.1, and it should be stated without softening: at organisational scale, in a favourable setting, the gains were real and they aggregated. Anyone citing METR as proof that AI does not work at scale is doing the same selective citation this book was written to correct.

Now the rest of the table.

And the authors' summary of what happened downstream, which is the sentence this chapter is built on:

"Adoption also restructured code review around automation: per-reviewer load roughly doubled and automated review overtook human review, while merge and revert rates held steady."

Three facts to hold together. Throughput doubled. Per-reviewer load also doubled. And the organisation absorbed that load by substituting machine review for human review — automated bots came to account for roughly 38% of review events and roughly half of all review comments.

The work did not disappear. It moved, and then it was mechanised. The authors say so directly: the gain "relocates work downstream rather than removing it."

3.4 The team boundary is a queue

Why should doubling generation double reviewer load rather than doubling delivery? Because the pipeline is not a sequence of independent stages. It contains a shared server with finite capacity, and that changes the arithmetic entirely.

Start with the weaker bound, which is already sobering. Amdahl's law: if authoring code is a fraction f of total cycle time and you speed that fraction by factor S, overall improvement is 1 / ((1−f) + f/S). Suppose authoring is 30% of cycle time. Doubling authoring speed yields 17.6%. Making authoring instantaneous — infinite speedup — yields 43%, and not one point more. Under this model there is a ceiling and it is low.

But review is not merely another stage. It is a queue with servers, and queues do not degrade linearly.

For a single-server queue, time in system is W = 1/(μ−λ). As arrivals approach capacity, ρ → 1 and W diverges. This is not a metaphor; it is why motorway traffic collapses at a specific density and why an on-call rotation at 90% utilisation feels categorically worse than one at 70%.

The enterprise study reports per-reviewer load roughly doubling — λ per server doubled — while a large-scale telemetry study across 10,000+ developers and 1,255 teams reports the accompanying symptoms: +98% merged pull requests, +91% review time, +154% average PR size, +9% bugs, and no measurable improvement in organisational DORA metrics [D] — vendor data, so graded accordingly, but the shape is exactly what Figure 3.3 predicts and it corroborates the peer-reviewed and preprint sources rather than standing alone.

Credit where due: this mechanism is not original to this book. Farrag names the pattern the Productivity–Reliability Paradox and identifies the review bottleneck as one of two amplifying mechanisms — "review capacity unchanged while PR volume increases 98%, absorbing system-level gains" [B]. What follows below is the part I have not seen stated: the queue admits exactly four responses, and they exhaust the space.

3.5 Four valves, all of them costly

When ρ approaches 1, a real organisation does not diverge to infinity. It does one or more of four things. There is no fifth.

Now read the evidence back against the diagram, and notice that the two organisations we have real data for pulled valves 3 and 4.

Valve 3 was pulled, measurably. Average PR size rose 37% on adoption and a further 33% post-mandate in the enterprise study; the telemetry study reports +154%. Larger batches reduce the item count arriving at review, which relieves λ. They also make each item harder to review well, and DORA identifies working in small batches as one of the seven capabilities that determine whether AI benefits reach the organisation at all [C]. So the organisation relieved queue pressure by moving away from the practice most associated with the outcome it wanted. Nobody decided this. It is what a system under pressure does.

Valve 4 was pulled, decisively. Automated review overtook human review. Bots reached ~38% of review events and ~half of all review comments.

Here is what I find genuinely remarkable about that finding, and it is the bridge to the rest of the book. Nobody in this book's framework told that company to build inferential sensors. They were not following a lifecycle. They had a throughput mandate and a review queue that was saturating, and under that pressure they independently invented the thing Chapter 7 calls an inferential feedback control — a model-based reviewer inserted into the loop because the human one could not scale.

Part II of this book is therefore not a proposal. It is a description of what organisations under queue pressure already do, arrived at by necessity, without naming it, without measuring it, and without deciding how much to trust it. The contribution of a lifecycle is not to invent valve 4. It is to make valve 4 deliberate, measured, and bounded, and to stop valves 2 and 3 from being pulled by default.

3.6 "Merge and revert rates held steady"

The enterprise study reports that merge and revert rates did not move. AI-authored code, reviewed increasingly by machines, shipped without a measurable quality penalty.

Take that seriously. It is real evidence against the alarmist reading, and this book's second commitment requires stating it in its strongest form: on the quality proxies available, at the scale of a year, substituting automated review for human review did not visibly hurt.

Then read what the authors themselves say those proxies are: "AI-generated code now ships under automated review without a measurable penalty on these coarse, short-horizon quality proxies."

Coarse. Short-horizon. Their words, and they matter enormously, because we can situate them against a study by the same lead author.

Hao He is first author both here and on the difference-in-differences study of Cursor adoption across public repositories, whose title states its finding: Speed at the Cost of Quality? How Cursor AI Increases Short-Term Velocity and Long-Term Complexity in Software Projects [A]. Velocity gains: large, transient. Complexity and static-analysis warnings: substantial, persistent.

So one researcher, two datasets, two horizons, two findings that only appear to conflict:

A revert happens within days. Complexity accretion is visible over months. Duplication over quarters. The proxies that stayed flat are precisely the ones whose window closes before the documented damage becomes visible.

Taleb calls the general error the turkey problem: the turkey's confidence in the farmer rises monotonically for a thousand days, and its data series contains no hint of day one thousand and one. The bird is not being stupid. It is doing correct inference on an insufficient window. An organisation observing twelve months of flat revert rates is in exactly that position — not wrong, but reasoning from a window shorter than the phenomenon.

To be fair in both directions: the long-horizon studies are observational on public repositories, weaker designs than the within-developer panel, and one controlled experiment cited in the enterprise study found no maintainability penalty when later developers evolved AI-co-developed code. The quality verdict is genuinely unsettled, and Chapter 4 presents it in full rather than resolving it here.

What is not unsettled is the methodological point, and it is the practical takeaway of this section:

Report the horizon with the metric, or the metric is not a claim. "No quality penalty" means nothing without "measured over twelve months on merge and revert rates." Every dashboard in Chapter 14 carries a window, for this reason.

3.7 The metric that was met

Return to the CTO's email. The mandate was to double engineering productivity. The designated metric was merged pull requests per engineer per month. That metric went from 21.2 to 44.3.

But average PR size rose roughly 37% on adoption and a further 33% after the mandate.

So a pull request in April 2026 was not the same object as a pull request in April 2025. The unit drifted. This does not mean the gain was illusory — more, larger pull requests is unambiguously more code shipped, and the token spend and volume figures confirm real work happened. It means something more specific and more troubling: the organisation hit its target while the unit of review got larger, which is to say while reviewability degraded.

This is Goodhart's law operating exactly as advertised — when a measure becomes a target, it ceases to be a good measure — with the twist that the corruption was not gaming. Nobody split or padded pull requests to make a number. The metric was hit honestly, and hitting it honestly required the batch growth that made the underlying goal harder to achieve.

The enterprise study cites Miller and colleagues on a related failure they name the productivity pressure paradox: a mandate that raises expectations without granting time to build skill undermines the gains it seeks [B]. And Storey's reframing is the sharpest available description of what accumulates: not technical debt exactly, but cognitive and intent debt — the gap that opens when code outpaces a team's capacity to absorb it [D].

Two rules follow, and both are inputs to Chapter 14.

A count is not a measure unless the unit is fixed. Merged pull requests per engineer is a defensible metric only alongside a batch-size guardrail. Chapter 14 pairs every delegation metric with the guardrail that keeps its unit honest, for exactly this reason.

Mandate the constraint, not the output. A mandate on generation raises λ. Since generation was already the elastic part of the system, this is the one intervention guaranteed not to relieve the bottleneck. A mandate to double review capacity — through sensors, through structure, through batch discipline — would have raised μ, and everything downstream of it. Taleb's objection to naive interventionism is precisely this: acting on the visible, tractable part of a system while the binding constraint sits elsewhere, and then attributing the disappointing result to insufficient commitment.

3.8 The synthesis

Everything in this chapter reduces to five statements. Each is supported above; none requires the others to be believed on faith.

1. Individual gains are real and large. +55.8% on scoped tasks, +26.08% across 4,867 developers in three RCTs, +40% in heaviest-usage weeks across 16,223 engineers.
2. They shrink to zero, and can invert, as the work becomes expert work in familiar code. −19%, in the most controlled study of that setting we have.
3. They can nonetheless aggregate. 2.09× per-capita merged pull requests over twelve months in a favourable enterprise setting; 3.1× company-wide volume.
4. But the work is relocated, not removed, and it lands on review. Per-reviewer load roughly doubled in the same study.
5. Every response to that relocation costs something. Lead time, scrutiny, reviewability, or the engineering effort to mechanise review. The organisation studied paid in reviewability (+37% batch size) and mechanisation (automated review overtaking human review), and its short-horizon quality proxies stayed flat over a window too short to detect the effects the long-horizon studies report.

Which yields the sentence this book exists for:

Verification capacity is the constraint. Generation capacity is not. An organisation that increases the second without increasing the first has not become faster; it has moved its bottleneck somewhere less visible.

The remaining twenty-four chapters are about increasing μ deliberately. Part II builds the apparatus. Part III makes it a lifecycle with metrics and gates. Part IV addresses the humans whose judgment the apparatus is supposed to concentrate rather than replace.

And one closing note on why this cannot be solved by hiring reviewers. Review capacity does not scale linearly with headcount: reviewers need context, contexts diverge, and the coordination cost of keeping many reviewers coherent on one codebase is the oldest known result in this field. Brooks published it in 1975 and nothing since has repealed it. If μ cannot be raised much by adding humans, and λ can be raised almost without limit by adding tokens, then mechanising part of verification is not one option among several. It is the only available lever, and the question is not whether to pull it but how to do so without quietly deciding that nobody reads the code any more.