Appendix D — Metrics Instrumentation Guide

Report the horizon with the metric, or the metric is not a claim.

Use this appendix to: actually collect what Chapter 14 defines — with the definitional choices that decide whether each number means anything, and the caveats that keep it honest.

Established in: Chapter 14 (definitions, guardrails, gaming modes), Chapter 12 §12.8 (artefacts are the prerequisite), Chapter 26 §26.3 (four are computable from git).

D.0 Before anything: the prerequisite

Chapter 12 §12.8 stated it without hedging: a lifecycle without artefacts has no outer loop. Not a weak one — none.

Every metric below except batch size depends on capturing something that is not captured by default.

   CAPTURE THESE, OR NOTHING BELOW WORKS

   ▸ execution traces        what the agent did, step by step
   ▸ sensor outcomes         which control fired, on what, when
   ▸ review events           who reviewed, when, human vs automated
   ▸ escalations             where the agent stopped and asked
   ▸ authorship attribution  which changes were agent-authored

   Cost: storage and a small amount of wiring.
   Benefit: zero, immediately. Every measurement below, eventually.

   This is why it loses every prioritisation argument (Ch 15 §15.6)
   and why Chapter 26 puts it in the first thirty days, before
   anyone has to argue for it.

Authorship attribution deserves a note because it is easy and everyone skips it. The enterprise study used a repository-level convention — an internal created-by-ai label applied at PR creation — plus bot identity detection and Co-authored-by trailers [B]. Any of the three works. Without one, rework rate and autonomy mix are uncomputable.

D.1 The five to start with

Chapter 14 §14.10's ordering, with the instrumentation for each.

1 · Autonomy mix

Distribution of merged work across A0–A4. The number to ask for first — it reveals whether delegation is real or whether an organisation bought tooling and kept its process.

Read it as: 95% A1 means tooling adopted and process unchanged. That is Chapter 21's Agent Washing, and this metric is the detector.

2 · Rework rate

Percentage of merged agent-authored code changed again within 14 days. The honesty guard on every throughput claim.

   for each merged agent-authored change C:
       lines_C     = lines added or modified by C
       reworked_C  = lines from C modified again within 14 days
                     by any subsequent commit
   rework_rate = Σ reworked_C / Σ lines_C

⚠️ Exclude formatting-only changes, or a formatter run will dominate the number.

3 · Batch size

Median lines per merged change. Free — you already have the data.

This is not a health metric. It is the unit-stability guard for every count in this guide, and Chapter 14 §14.4 explains why: enlarging batches leaves utilisation exactly unchanged while degrading reviewability, blast radius per item, and revert granularity. The enterprise study measured PR size rising ~37% on adoption and a further ~33% post-mandate [B].

Report it beside every count metric. A throughput number without a batch-size number beside it is not a claim.

4 · W and λ → saturation elasticity

The constraint itself. Chapter 14 §14.3.

The definitional choice that decides everything:

   W must be WAIT, not DURATION.

   ✗ WRONG   time from PR open to PR merge
             (that is cycle time, and it moves when service
              time moves, i.e. when PRs get bigger)

   ✓ RIGHT   time from PR open to FIRST HUMAN REVIEW EVENT
             (that is queue wait, which is what rho drives)

Excluding automated review events is essential and not optional. The enterprise study found bots at roughly 38% of review events and half of all comments, and set its own boundary at the first human review for exactly this reason [B]. Count a review event as human if the actor is not in your bot identity list and the comment is not templated.

   λ  = review-demanding items arriving per week
   W  = median time from open to first human review

   saturation elasticity = %Δ W  /  %Δ λ      (quarter on quarter)

   < 1     slack
   1 – 2   loading up
   > 3     on the cliff — additional generation converts to queue,
           not to throughput

You never need to estimate μ, which is the point: reviewer capacity is not headcount × hours, it is the rate at which reviews complete at adequate quality, and adequate quality is precisely what degrades silently under load.

⚠️ Pair W with a scrutiny proxy — human review comments per unit of change, or first-pass acceptance rate. W falling while acceptance rate rises is not capacity relief; it is Chapter 21's Review Collapse.

⚠️ The bands are reasoned, not fitted. Chapter 14 flags this as an open question: anyone with two years of review telemetry could calibrate them properly in an afternoon.

5 · Time-to-control

Median elapsed time from a failure mode's second occurrence to a control that prevents it. The outer loop's own clock.

This is the hard one, and Chapter 14 said so. It requires classifying failures into modes across sessions, which Chapter 27 lists as the single most enabling missing instrument in the field.

What to do until that exists: measure it by hand on a sample of twenty incidents. Tag each with a coarse failure mode, find the second occurrence, find the control. A hand-computed median from twenty incidents is enormously more informative than nothing, and it is the only way to test Chapter 12 §12.7's uncomfortable claim — that a monthly outer loop cannot regulate an inner loop running dozens of times a day.

Read it as: a median measured in quarters means the steering loop is a historian. A median measured in weeks means it is a regulator.

D.2 Harness metrics

Add these once drills are running (Appendix B.5). Before that, harness coverage measures configuration and is itself Harness Theatre.

Harness coverage

   coverage = failure modes with a control that has DEMONSTRABLY FIRED
              in a drill within the review window
              ─────────────────────────────────────────────────────────
              known recurring failure modes

⚠️ The caveat that must accompany every report of this number: it measures diligence, not safety. The denominator is known failure modes — the ones you have already met. It is silent about everything you have not encountered, and it can read 100% while you are fully exposed to a class of failure nobody has imagined. Chapter 14 §14.6 calls this silent evidence in a denominator.

⚠️ And there is no known target. Chapter 27 lists this as a missing measurement: nobody knows whether 40% is negligent or excellent.

Security column (Chapter 11 §11.8): coverage over the read set — for each artefact class an agent consumes, has a drill placed a payload there and verified the outcome? With the asymmetry intact: a failed security drill is no evidence the attack cannot occur.

Sensor firing rate and precision

Per sensor: how often it fires, and what share of firings were real.

A sensor with zero firings in 90 days is a drill candidate, not a success.

Guide contradiction count

Currently unmeasurable. Chapter 14 carries it deliberately with a null value, because a metric with no detector is a standing statement about a missing tool rather than a missing discipline. Both Galster's authors and Böckeler name it independently as a gap. Report it as "no detector exists" rather than dropping the row.

D.3 Economics

Cost per accepted change

   cost = tokens + compute + human review minutes
          ────────────────────────────────────────
          changes merged WITHOUT rework within 14 days

The denominator is deliberate: a change merged and then reworked is not accepted, it is deferred.

⚠️ Human review minutes are the dominant term and the hardest to instrument. Chapter 16 §16.6's finding is that organisations meter the abundant input (tokens) carefully and the scarce one (review minutes) not at all. If you exclude review minutes because they are hard, you have measured the cheap part and called it the cost.

Approximation that is better than omission: (number of reviewers × median time-in-review-tool per change). Weak, gameable, and vastly better than a zero.

Review load per engineer per week

Hours spent reviewing, per person. Chapter 3's true capacity constraint; the enterprise study measured it roughly doubling [B].

⚠️ Do not collect this by asking. Chapter 18 disqualified self-report. Derive it from review event timestamps, and accept that the number is a lower bound.

⚠️ Never put this in a performance review. The moment it becomes a performance measure it becomes worthless, and it will also penalise exactly the apprenticeship review Chapter 20 §20.7 says is being squeezed out.

Context efficiency

Tokens consumed per accepted change. Chapter 6's budget made measurable.

It moves in the right direction for two different good reasons — better payload curation, and graph traversal replacing exploratory search. LocAgent reported an ≈86% cost reduction from that substitution [B].

⚠️ Hold the model constant or normalise. Otherwise the metric measures your procurement decisions.

D.4 Health guardrails

Complexity trend, duplication trend, batch size. Chapter 4 §4.7: "track these as guardrail metrics, not vanity metrics" — you do not optimise them, you watch the shape of the curve.

Each needs a threshold and an owner, or it is a chart. The threshold need not be principled:

If duplication rises for two consecutive quarters, the platform team opens an investigation.

That is a perfectly good rule and vastly better than watching.

D.5 The dashboard: segregate by horizon

Chapter 14 §14.8 names a design failure that is easy to commit and hard to notice: a dashboard with mixed horizons lies by juxtaposition.

The slow metrics are not flat because they are unimportant. They are flat because they are slow, and proximity teaches otherwise.

D.6 What not to instrument

Chapter 14 §14.9, restated as an instrumentation instruction because these are the things a well-meaning platform team will build first.

The last one is the most requested and the only one that cannot be measured at all. It requires comparing against a world that did not happen — the same structure that defeats debt measurement in Chapter 4.