Chapter 27 — Open Problems: An Invitation

A book that ends with answers has usually stopped looking.

27.1 Selection, not enumeration

Every chapter in this book closed with open questions. Collected, they number several dozen, and a list of several dozen is not an invitation — it is a shrug.

So four criteria for what earns a place here.

Something depends on it. Not curiosity. A decision a practitioner must make, or an argument this book makes, rests on the answer.

It is tractable. Stated precisely enough that a competent person could begin on Monday and know what would count as progress.

It is unclaimed. Not "someone is probably working on this."

And solving it would change what practitioners do, not only what researchers know.

That leaves the problems below, and one observation about them that I did not expect when I started assembling this chapter.

27.2 Four kinds of open

"Open problem" is used to cover four situations that differ in the only way that matters — who can address them and with what.

That distribution is the chapter's first finding. The discourse treats agentic engineering as a frontier of hard unknowns. The reality is that one problem is hard and the rest are unbuilt, unrun, or uncollected — which is a far more optimistic position than the field's tone suggests, and a far more embarrassing one.

27.3 The keystone: the behaviour harness

The one genuinely unsolved problem, and the one Part III is waiting on.

The question. How do you verify, mechanically, that generated software does what was actually wanted?

Why it is hard. Chapter 8 §8.7 stated the circularity precisely: code and tests generated from one model of intent prove consistency, not correctness. Their agreement is evidence about the generator's self-consistency and silent about whether its model of intent matched yours. Böckeler's assessment stands — current practice "puts a lot of faith into the AI-generated tests, that's not good enough yet."

What depends on it. More than any other item here.

What would count as progress. Not a solution — a bound. Chapter 8 §8.8's held-out acceptance set does not solve the problem; it converts an unbounded exposure into a bounded one. You remain exposed to what you failed to articulate, and no longer to what the generator failed to infer. Progress looks like more of that: mechanisms that shrink the unarticulated residue.

The most promising direction, and its limit. Chapter 1 §1.6 noted that Meaning A solved a version of this — nobody building an agent as a product lets the system under test author its acceptance criteria; the eval set is built first, by humans, from stakeholder-aligned success metrics. Chapter 8 attempted the transplant. What did not transfer is the statistical logic: an eval set samples a distribution of user requests, while a feature has a particular intent. The held-out set is a gate, not an estimate, and it is small, hand-made, and unrepresentative in a way an eval set is not.

And the bet underneath it. Chapter 5 §5.9 asked what happens if verification becomes as cheap as generation. Every inferential sensor in this book is a wager that it will — that model-based review becomes reliable enough to carry the semantic load. If that wager pays, several failures in this book reverse. If it does not, the behaviour harness stays open indefinitely and Chapter 13's ceiling stays where it is. Nobody should be confident about which, and the honest position is that the field's central economic question is currently a research question about model reliability.

27.4 Four missing instruments

We know what to measure. Nobody has built the thing that measures it.

1. Failure-mode classification across sessions. The question: given execution traces, cluster failures into recurring modes. What depends on it: Chapter 8's steering loop triggers on "whenever an issue happens multiple times" — which requires counting. Chapter 14's time-to-control cannot be computed without it. Chapter 12 §12.7 showed the outer loop must run at machine speed, and recurrence detection is the part that must be automated for that to be possible. What would count: a method that takes a trace store and returns labelled recurrence counts with acceptable precision. It is a clustering problem over semi-structured logs. Who: anyone building agent observability. This is the single most enabling missing tool in the book.

2. Blast-radius computation. The question: given a proposed change, compute its blast radius automatically. What depends on it: Chapter 13's ceiling matrix is currently a conversation. Ceiling violations — the closest thing to a safety metric this book offers — cannot be counted. Chapter 23 §23.6's crossing response requires it in the first hour. What would count: a query answering does this change path reach a credential, mutate production data irreversibly, cross a bounded context, or touch a symbol with large fan-in? Every one of those is a graph question (Chapter 9), and the graph is buildable. Who: whoever builds the code graph. It is a small addition to a thing that must exist anyway.

3. Guide contradiction detection. The question: detect conflicts between guides across layered context files. What depends on it: the only anti-pattern in Chapter 21 with no detector, and Chapter 14 carries the metric with a null value deliberately rather than dropping it. Chapter 7 measured the exposure — CLAUDE.md pointing to AGENTS.md in 301 cases across 2,853 repositories. Its provenance: named independently by Galster's authors and by Böckeler. When two sources reach the same gap from different directions, it is real. What would count: even pairwise conflict detection over natural-language guides, with tolerable false positives. Chapter 8 §8.9's rule stands until then: never let the agent resolve a contradiction, because that is a silent, unlogged, per-invocation decision about which of your stated intentions wins.

4. Unspoofable provenance markers. The question: carry the origin of each payload segment into context in a way injected content cannot forge. What depends on it: Chapter 11 §11.6 showed three of Part II's four rings are permeable to injection because the payload discards provenance that was free at assembly time. Fix this and context, harness, and graph can defend themselves; leave it and only capability scoping holds. Why it is hard: a model that trusts a delimiter can be persuaded the delimiter appeared. This may require architectural support rather than prompt convention — which places it at the boundary between this category and §27.3's. Who: model providers, more than practitioners. It is the item on this list that individual teams cannot solve.

27.5 Four missing experiments

Designable, unrun, and several are a week's work.

1. The contract-inference experiment. Chapter 10 §10.9. Two behaviourally identical versions of a module — one soundly encapsulated, one with an equivalent obligation maintained by convention and absent from the interface. Same task, both arms. Measure the rate of changes that violate the obligation while passing review. What depends on it: the entire outermost ring of Part II. Chapter 10 admits there is no causal evidence that modular codebases produce better agent outcomes, only a mechanism. A null result falsifies the chapter. Cost: a week. I would rather someone ran it than agreed with me.

2. Specification versus structure. Chapter 22 §22.5. Take real brownfield tasks; supply complete specifications for half. If performance recovers, specification dominates; if not, structural legibility does. What depends on it: where an organisation should spend. Chapter 22 argued the two are substitutes for the same Ashby constraint with different cost structures, on the strength of one suggestive comparison — the Google trial's realistic-but-specified +21% against METR's realistic-and-unspecified −19%, with real confounds.

3. Does deeper configuration pay? Galster's authors ask this themselves: "future work should assess whether deeper configuration leads to measurable performance gains." What depends on it: Chapter 7's central normative claim — that the field's 29:1 investment in context files over hooks is a cost artefact rather than an efficient allocation — rests on Chapter 5's authority analysis rather than on a controlled comparison, because that comparison does not exist. If context files simply have a better return per hour, Chapter 7 is wrong.

4. Does deliberate practice work? Chapters 19 and 20. Does attempt-before-delegating, plus a prediction log, actually maintain competence — and does it accelerate juniors enough to compensate for fewer hands-on hours? What depends on it: Chapter 20's entire remedy. The mechanism comes from the MIT ordering effect measured on essay writing with 54 participants; applying it to engineering competence is an inference. And a cohort is running the uncontrolled version of this experiment now, which is the least satisfactory way to find out.

27.6 Three missing measurements

The data exists. Somebody has it and has not published it.

1. The escape rate. Chapter 16 §16.7's sampling audit, run at organisational scale over a year, would produce the first real estimate of how much defect flows through agentic review. It would settle much of Chapter 4's dispute, and it requires no new technique — only the willingness to re-review a random sample deeply and count.

2. A harness coverage target. Chapter 14 §14.6 defines the measurement and cannot supply a target. Nobody knows whether 40% is negligent or excellent. This needs several organisations to measure and compare, and the denominator's dependence on known failure modes may make cross-organisation comparison meaningless — which is itself worth establishing.

3. The failure denominator. Chapter 25 §25.7's survivorship problem. Of organisations that attempted a harness programme, what fraction got one? Every published account is a survivor. Gartner's forecast that over 40% of agentic AI projects will be cancelled by end-2027 [C] 🟡 is the only adjacent estimate, and it is a forecast rather than a measurement.

And one that belongs in a category of its own: the longitudinal study. Chapter 4 could not resolve whether agent-assisted codebases degrade, because the acute proxies (merge, revert) and the chronic ones (complexity, duplication) are measured over incompatible windows and nobody has tracked both on the same codebases for long enough. Three years, instrumented, matched controls. Until it exists, everyone will keep citing whichever horizon suits them, and Chapter 4's central question stays open.

27.7 What this book probably got wrong

A book that graded its evidence should grade its own arguments. These are the four I would attack first, in descending order of how likely I think they are to be wrong.

Chapter 10's structural argument. Information hiding makes local inference sound; leaky abstractions are invisible from the inside; therefore modularity is the substrate of agent control. The mechanism is clean and there is no causal evidence for it. Chapter 10 says so and supplies the falsifying experiment. If §27.5's first experiment returns null, Part II's outermost ring is decoration.

Chapter 16's public-good framing. I argued the harness is non-rivalrous and non-excludable, so no team will fund it. On reflection the argument is strongest for templates, graphs, and platform infrastructure and weakest for per-repository sensors — a team's own blocking hook on its own repository is largely excludable and its benefits largely captured by that team. Chapter 7's cost gradient may therefore be doing more explanatory work than the public-good economics, and I over-generalised.

Chapter 12's clock-speed claim. I asserted a roughly thousand-to-one mismatch between inner and outer loop cadence and used it to argue the outer loop must be mechanised. The inner-loop figure is an estimate and the outer-loop figure is an impression of how organisations run retrospectives. The direction is almost certainly right; the magnitude is unmeasured, and if the real ratio is ten to one the argument weakens considerably.

And the Ashby framing throughout. Requisite variety is a theorem about regulators, applied here to harnesses by analogy, with "variety" never formally measured. It has organised a great deal of this book — Chapter 8's forcing result, Chapter 22's substitutes, Chapter 24's identity — and it is used qualitatively. A reader who thinks the analogy is doing more work than it can bear is entitled to that objection, and I would want to know where they think it breaks.

Two things I would defend more firmly, for symmetry. Chapter 5's account of the withdrawn goods holds up: it explains the observed distribution of practice, predicts the anti-patterns, and was independently corroborated when two engineering organisations turned out to have built the controls it implies. And the evidence-grading discipline earned its cost — nine corrections to the original research notes, and at least one occasion where reading a primary source reversed the conclusion of a section that an automated summary had got backwards.

27.8 What remains true regardless

Chapter 0 closed with a claim, and it is the right place to end.

A harness cannot verify correctness against a requirement that was never articulated. Specification is the last human monopoly.

Twenty-seven chapters later, that sentence has survived everything built on top of it, and §27.3 explains why: the one genuinely unsolved problem in this book is precisely the problem of verifying intent. Every control in Parts II and III operates on stated intent. Where intent is unstated, no apparatus helps, and the failure is silent — because generated code that satisfies an unstated requirement incorrectly is indistinguishable from generated code that satisfies it correctly.

So the book ends approximately where it began, with the human obligation intact and the machinery around it incomplete. That is not a disappointing conclusion. It is the useful one, because it tells you where to stand.

Three things are settled well enough to act on, whatever happens to the rest.

The bottleneck moved. Generation is elastic; verification is not. Nothing in the evidence contradicts this, and the best-documented enterprise case doubled throughput while doubling per-reviewer load. Any strategy that increases generation without increasing verification capacity relocates a bottleneck rather than removing one.

The controls that used to be free must now be bought. Scrutiny, intent, pacing, a trust boundary, structural correction, and apprenticeship were all purchased by the expense of writing code. Nobody removed them; the price that paid for them fell. They are engineered artefacts now, with owners and budgets, or they are absent.

And the work nobody owns is the work that matters most. Six times this book arrived at the same structure — drift detection, the outer loop, the harness, competence maintenance, apprenticeship, noticing the crossing — each with no triggering event, local cost, and diffuse benefit. That category needs a standing mandate and a named owner, and it is also, as it happens, the category an unattended agent can do.

The invitation is straightforward. Most of what remains open in this field is not hard. It is unbuilt, unrun, or uncollected, and much of it is a week of work by someone with the right data or the right repository. If you have a trace store, you can measure recurrence. If you have a graph, you can compute blast radius. If you have two teams, you can run the ordering experiment. If you have an incident log and a permissions history, you can test whether Chapter 21's pathology sequence is real.

The field's problem is not that these questions are difficult. It is that nobody has been asked to answer them, which — one last time — is the same structure as everything else in this book.