Chapter 18 — Calibrated Trust

"AI-assisted programming is not a faster version of solo coding but a cognitively distinct activity."

— Burelli, Calefato, Grassi, Hristova, Novielli, Romano, Tell [B] EEG, eye-tracking, electrodermal activity and heart-rate variability, within-subjects crossover, two universities

18.1 The inversion, precisely

METR's randomised controlled trial gave sixteen experienced developers 246 real tasks in repositories they had worked in for an average of five years [A].

• Before starting, they forecast that AI would make them 24% faster.
• Measured, AI-allowed tasks took 19% longer — confidence interval +2% to +39%.
• Afterwards, having lived through it, they estimated AI had made them 20% faster.

The third line is the one that matters. The first is a forecast, and forecasts are routinely wrong. The third is a retrospective report about an experience the person just had, and it was wrong by roughly forty percentage points in the same direction as the forecast.

METR's own later survey work puts a number on the general case: in early 2025, people overestimated AI's effect on their time spent on tasks by 40 percentage points on average [B]. And a 2026 survey of 349 technical workers found a median self-reported 1.4–2× change in the value of their work — with METR themselves noting there are reasons to be sceptical of the magnitude, and observing that public estimates from surveys have consistently exceeded those from field experiments and quasi-experiments [B].

One further datum, and it is the one that should worry anyone setting a budget: nearly 89% of engineering leaders reported that developer productivity improved after adopting AI coding tools [B]. The overestimation is not confined to the people doing the work. It is larger, and less checkable, one level up — which is precisely where the spending decisions are made.

18.2 A puzzle worth sitting with

Here is a fact that appears to contradict everything in §18.1.

The 2025 Stack Overflow Developer Survey found developer trust in AI at its lowest recorded level [C]:

• 46% actively distrust the accuracy of AI tool output — up sharply from 31% the previous year
• 33% trust it
• Only 3% report highly trusting it
• Among experienced developers, 2.6% highly trust and 20% highly distrust
• Positive sentiment fell from over 70% in 2023 and 2024 to 60%
• And 84% use or plan to use these tools anyway, up from 76%

So developers are sceptical, increasingly so, and the most experienced are the most sceptical of all. How can a population this distrustful overestimate its own speedup by forty points?

The resolution is that these are different judgements about different objects.

Trust in accuracy asks: is this output correct? Developers have direct, frequent, immediate evidence on this. They see wrong code, they correct it, and their estimate updates — which is exactly what the rising distrust figure shows. This is a well-calibrated judgement, improving over time, because the feedback loop is tight.

Belief about throughput asks: did this whole process take less time than the alternative? The alternative did not happen. There is no feedback loop at all, and §18.3 explains why the substitute signal points the wrong way.

The practical warning for anyone running an engineering organisation:

"Our engineers are appropriately sceptical" is not protection. Scepticism about output accuracy is orthogonal to calibration about throughput. A team can correctly believe the model is often wrong, correct it diligently, and still be systematically wrong about whether the whole arrangement is faster — because knowing the output is unreliable tells you nothing about how long you spent finding out.

18.3 Three mechanisms, none of them a belief

Why does the retrospective estimate survive the experience that should have corrected it?

One: the felt signal tracks effort; the measure tracks duration. The subjective experience of work is dominated by effortfulness rather than elapsed time. Reading generated code is recognition; writing it is recall, and recall is heavier. So a session composed largely of reading, checking, and correcting feels lighter than one composed of authoring — even when it runs longer on the clock. Section 18.4 is the evidence that this is real and measurable rather than plausible.

Two: the counterfactual is unavailable. To know you were slower you would need to know how long the task would have taken otherwise. That world does not exist. So the estimate is not retrieved, it is constructed — and it is constructed from what was salient. The moments the agent produced something impressive are highly salient. The forty minutes spent understanding why its output failed feel like ordinary debugging, which is what work has always felt like, and are not attributed to the tool at all.

This is structurally identical to Chapter 4's problem with debt. That chapter showed technical debt is unmeasurable because it is a claim about a codebase that does not exist. Speed estimation fails for the same reason: both require comparison against a counterfactual, and both therefore admit whatever story is most available.

Three: waiting is not encoded as working. While the agent generates, the developer often switches away. Switched-away time is poorly attributed to the task it was switched away from, so the wall clock and the remembered duration diverge. This one is reasoning rather than evidence; I include it because it is mechanically plausible and because it predicts something testable — that the estimation error should grow with generation latency.

All three push in the same direction, and — this is the important structural point — not one of them is a belief. They are properties of how the experience is generated, encoded, and reconstructed. Which is why Chapter 17 §17.5 classified this shift as behavioural rather than informational, and why the next section is worse news than this one.

18.4 The instrument is disconnected

If introspection is unreliable, the natural next move is to introspect more carefully — to attend to your own effort, notice your own state, and calibrate from the inside with more discipline.

There is now evidence that this route is closed.

Burelli and colleagues ran a multisite, within-subjects crossover study at two universities — Bari and the IT University of Copenhagen — collecting EEG, eye-tracking, electrodermal activity, and heart-rate variability alongside a rubric-based performance score and NASA-TLX self-reported workload across six dimensions. The study was pre-registered as a Stage 1 Registered Report with In-Principle Acceptance at ESEM 2025, which means the hypotheses were fixed before the data existed [B].

Two families of finding.

First, cognitive engagement fell. Under AI assistance the EEG θ/α ratio was lower during the first task and the gaze blink rate was higher during the second — both, in the authors' words, "consistent with reduced cognitive engagement when developers offload generative effort to the model." This is §18.3's first mechanism, measured neurophysiologically rather than argued: the work genuinely is lighter. Notably, the pattern did not differ between undergraduate and graduate students, which Chapter 20 will have to reckon with.

Second — and this is the finding that changes what calibration can mean — the signals stopped tracking performance.

Read carefully, this is a stronger claim than "self-report is biased." A biased instrument can be corrected — you learn the offset and adjust. A decoupled instrument cannot, because there is no stable relationship left to correct for. Under AI assistance, both a physiological measure and self-reported workload lost their association with how well the developer was actually doing.

The authors' conclusion is the epigraph of this chapter and it is the right one: AI-assisted programming is not a faster version of solo coding but a cognitively distinct activity. Their keyword list includes cognitive debt, which Chapter 19 takes up.

Three honest caveats, because this is a load-bearing result. The participants were undergraduate and graduate students, not professional developers, so external validity to experienced practitioners is unestablished — and the METR cohort, whose inversion this explains, were experts. The sample is drawn from two European universities. And the manuscript is a working draft: the version I read carries a visible editorial TODO in its related-work section, so figures and framing may change before publication. The pre-registration is a strong credential; the paper is not yet final, and this book's rule is to say so.

But the direction is clear enough to act on, and it produces the chapter's governing constraint:

You cannot calibrate trust from the inside. Not because people are careless, but because under AI assistance the internal signals that would carry the information have stopped carrying it. Calibration must come from external instruments or it does not come.

18.5 What trust actually is

Given all that, the standard framing of "calibrated trust" — hold beliefs about reliability that match observed frequencies — is not usable. It asks you to tune an instrument that has been unplugged.

So reframe. Ask what trust does rather than what it is.

Trust is not a belief about quality. It is a decision about how much verification to spend.

That is what trust determines operationally: whether you read the whole diff or skim it, whether you run the tests yourself, whether you check the edge case, whether this change needs a second pair of eyes. Every one of those is an allocation of the scarce resource Chapter 3 identified as the binding constraint.

Which yields the definition this book uses:

Miscalibrated trust is misallocated verification capacity.

The reframing earns its place for a practical reason: an allocation is observable. You cannot see someone's belief about model reliability, and after §18.4 they cannot see it either. You can see how long they spent, what they ran, and what they read. Trust becomes an engineering quantity rather than a psychological one.

18.6 The two errors are not symmetric

Standard calibration treats over- and under-confidence as symmetric errors to be minimised. Here they are not, and the asymmetry should be built into policy rather than discovered.

Under-trust costs capacity. You verify things that did not need it. The cost is real, linear, and bounded: some hours, spent on the wrong changes. It shows up immediately in Chapter 14's review-load metric.

Over-trust costs escapes. Something got through. The cost is unbounded in the tail — Chapter 11 established that security failures are Extremistan, where one event dominates the total, and Chapter 4 established that structural damage arrives late and is hard to attribute.

So the loss function is asymmetric, and Taleb's argument applies directly: when one side of an error is bounded and the other has an open tail you cannot estimate, symmetric optimisation is the wrong objective. The correct policy is deliberately biased toward under-trust where blast radius is high, accepting known waste to avoid unbounded exposure.

Which is Chapter 13's blast-radius veto arriving from the psychological side rather than the governance side. That chapter said no amount of harness quality makes a severe-blast-radius change eligible for high autonomy; this one says no amount of accumulated good experience should make it eligible for low verification. Two derivations, same rule.

18.7 What actually calibrates

If introspection is closed and briefings do not work — METR's cohort knew the result and still felt fast — what remains?

The calibration literature, developed for forecasting rather than for engineering, is unusually clear about this: calibration improves through frequent, recorded, scored predictions with fast feedback, and it does not improve from being told one is miscalibrated. Applied here, three practices, in descending order of value and ascending order of cost.

One: the prediction log. Before opening an agent-produced change, record one bit — will this need changes? Then review, and record the outcome.

The prediction log works precisely where the throughput estimate fails. The counterfactual problem of §18.3 does not arise, because you are predicting an observable event rather than an alternative history. The feedback is immediate rather than never. And the record is external, which is what §18.4 established is necessary.

Two: start from the base rate, not the diff. Chapter 14's first-pass acceptance rate, computed per task class, is your team's base rate. Trust in a specific change should begin there and be adjusted, rather than being constructed fresh from how the diff looks — because how a diff looks is exactly the salience-driven signal §18.3 warned about. This is reference-class forecasting applied to review, and it requires the metric to exist, which is why Chapter 14 comes first.

Three: the held-out acceptance set is calibration infrastructure. Chapter 8's partition — human-authored acceptance criteria the agent never sees, run only at the merge gate — was introduced as a defence against circularity. It is also the only mechanism in this book that tells you what got through, which is the raw material calibration needs. Chapter 16's sampling audit does the same job at organisational scale and with the coarse resolution that section was careful about.

And what does not work, stated plainly because it is what most organisations do: telling people about the inversion. Chapter 17 §17.5 classified this shift as behavioural; §18.4 explains why. The information does not reach the faculty producing the estimate, because that faculty is running on signals that have been decoupled from the outcome.

18.8 Measuring trust without asking

Trust as allocation (§18.5) is observable. Four proxies, all derivable from artefacts a team already produces if it followed Chapter 12 §12.8:

One honest caution. Chapter 16 §16.7 rejected attention metrics — time in file, scroll depth — as a way of rebuilding review's authority, on the grounds that they are proxies for effort one layer down and are gameable in an afternoon. The proxies above are the same measurements pointed at a different question, and the objection partly transfers.

The difference is the incentive. Nobody has a reason to game "I trusted this" the way they have a reason to game "I reviewed this carefully", because the first is not what they are evaluated on. That difference is real and it is not permanent: the moment these become performance measures, they become worthless. Use them diagnostically, hold them privately, and do not put them in a performance review.

18.9 The problem one level up

A closing observation that follows from §18.1's least-quoted figure.

Nearly 89% of engineering leaders reported productivity improvements from AI coding tools [B]. Leaders are further from the work, receive their information through reports from people whose own estimates are miscalibrated by forty points, and have less access to any corrective. Their estimate is a summary of summaries, and every step in the chain adds the same directional error.

They are also the people who set throughput mandates — and Chapter 12's seventh prohibition forbids exactly that, because generation is the elastic side and mandating it raises arrival rate against fixed capacity. The best-documented enterprise case ran this experiment: a CTO mandate to double productivity, measured in merged pull requests per engineer, which was met while batch size grew by roughly a third [B].

So the calibration problem is worst where its consequences are largest, and the remedy is the same one this whole chapter argues for. Not better beliefs. Instruments — Chapter 14's five metrics, reported without self-report anywhere in the chain.