Chapter 2 — The Scaffolding Insight

RAG with GPT-4 Turbo: 1.31% resolved. SWE-agent with GPT-4 Turbo: 12.47% resolved.

— Table 1, Yang et al., SWE-agent, NeurIPS 2024 [A] Same benchmark. Same model. Same weights. Nine and a half times the result.

2.1 The experiment

In 2023, SWE-bench asked a question that sounded modest and turned out not to be: can a language model resolve a real GitHub issue? Not write a function from a docstring — resolve an issue, in one of twelve real repositories, such that the maintainers' own unit tests pass.

The first answers were humbling. Claude 2 resolved 1.96% of the benchmark [A]. The framing at the time was that models were not yet good enough, and the implied remedy was to wait for better models.

In May 2024, a group at Princeton published a different remedy. They did not train a model. They built an interface.

Their reasoning began from an analogy that has aged well. Humans do not interact with computers through raw system calls; we use editors, file browsers, and search tools, and the design of those tools measurably determines what we can accomplish. Human–computer interaction is a discipline precisely because interface design has large effects on capable operators. Language models are also operators — differently capable, differently limited — and nobody had asked what interface they need.

They called it an Agent–Computer Interface, and they measured it.

Read the left column first. Retrieval-augmented generation — fetch the likely-relevant files with BM25, hand them to the model, ask for a patch — resolves 1.31% with GPT-4 Turbo. The same model, given a purpose-built environment to work in, resolves 12.47%. On the Lite split the progression is visible in three stages: 2.67% with retrieval, 11.00% with a plain shell and a worked demonstration, 18.00% with the designed interface.

Nothing about the model changed. No fine-tuning, no additional training data, no new checkpoint. What changed was a file viewer, an edit command, a search command, and a policy about what to keep in context.

This is the founding result of harness engineering, and it deserves its status for a reason that is easy to miss. It is not impressive that a number went up. It is impressive that the number went up under a controlled comparison — the one experimental design that licenses a causal claim. Most assertions in this field are before-and-after observations confounded by everything. This one holds the model fixed and varies the environment.

There is a second reading in that table, and it undermines the simplest interpretation. Claude 3 Opus was the better model at retrieval — 3.79% against GPT-4 Turbo's 1.31%, nearly three times as good — and the worse model inside the interface: 10.46% against 12.47%. The ranking inverts. Whatever the interface was providing, it was not a uniform multiplier applied to model quality. It was a fit between a specific interface and a specific model's habits.

Hold that thought. It is the whole chapter.

2.2 Four dials, all non-monotonic

The Princeton group did something more useful than reporting a headline. They ablated their own interface — varied one dial at a time and measured what happened. This is Table 3 of the paper, and it is, I think, the most under-read table in the literature.

Look at what these four rows have in common.

The file viewer. Showing the agent 100 lines at a time scores 18.0%. Showing it 30 lines scores 14.3% — too little to reason with. Showing it the entire file scores 12.7%, which is worse than either. More information, less capability. The curve has a peak and the generous side falls further than the stingy side.

Context retention. Keeping the last five observations scores 18.0%. Keeping the full history scores 15.0%. The agent given complete memory of its own session performs worse than the agent given a sliding window.

The editor. No edit command at all: 10.3%, and the paper explains why — without a purpose-built editor the agent falls back to sed and shell redirection, both of which are error-prone for multi-line changes and, crucially, provide "lack immediate feedback about file updates, making these silent operations potentially confusing for models to interpret." Adding an edit command: 15.0%. Adding a linter that rejects the edit if it would introduce a syntax error: 18.0%.

That last increment is worth pausing on, because it is the first sensor in this book. The linting guardrail does not help the agent write better code. It refuses to apply code that is already broken, which prevents a specific failure cascade: an errant edit introduces a syntax error, the agent sees the broken file, tries to fix it, breaks it further. The paper quantifies the cascade. Across 2,294 task instances, 1,185 (51.7%) of trajectories contained at least one failed edit. Any single edit attempt had a 90.5% chance of eventually succeeding — but after one failed edit, that dropped to 57.2%.

A control that costs milliseconds prevented a failure mode that halved recovery odds. This is the shape of every worthwhile sensor in Part II, and it appeared in the first paper.

2.3 The tool that was worse than no tool

Now the search row, which is the most instructive line in the table and the reason this chapter exists.

• No search capability at all: 15.7%
• Iterative search — results shown one at a time, next and prev to page through, each match displayed with surrounding context: 12.0%
• Summarised search — all matches listed at once, with guidance to refine the query if there are too many: 18.0%

Iterative search is a good interface. It is what Vim does. It is what VS Code does. It is the design that decades of human interface work converged on, because a human paging through matches builds a mental model as they go and stops when they have found what they need.

Agents did not stop. From the paper: "when given a large number of search results, agents tend to look through every match exhaustively, calling next until each result has been inspected. This inefficient behavior can exhaust an agent's cost budget or context window."

The intervention made things worse than the absence of the intervention — 12.0% against 15.7%, a deficit of nearly four points against doing nothing at all.

Taleb calls this iatrogenics: harm caused by the healer. His examples are medical — the treatment whose side effects exceed the disease — and his argument is that iatrogenics is systematically under-counted, because the harm from action is attributed to the illness while the harm from inaction is attributed to the physician. Engineering has the same asymmetry. Nobody is blamed for the linter they did not write. Everybody can point to the tool they built.

Three things follow, and I would rather state them plainly than dress them up.

First, a capability is not a benefit until it is measured. Iterative search was not a bad idea badly built. It was a good idea, well built, imported from a domain where it demonstrably works, that turned out to be harmful in this one. No amount of design reasoning would have caught it. Only the ablation caught it.

Second, harness elements interact with the operator's failure modes, not with the task. The reason iterative search failed is that the model lacked the thing a human brings for free: a sense of enough. A human stops paging when satisfied. The model, having no reliable satiety signal, exhausted the list. The interface was well matched to the task and badly matched to the operator.

Third — and this is the practical consequence — every guide and sensor you add to a harness is a hypothesis, and untested hypotheses accumulate. Chapter 8 will call the resulting condition Harness Theatre and Chapter 14 will propose measuring it. The founding paper of the field contains a worked example of a control that would have looked excellent in a configuration file and was actively destroying performance.

2.4 Failure was not resource-limited

One more finding from the same paper, because it forecloses the most common intervention.

The authors observed that "agents succeed quickly and fail slowly." Successful instances finished with a median cost of \$1.21 across 12 steps. Unsuccessful ones consumed a mean of \$2.52 across 21 steps. And the decisive number: 93.0% of resolved instances were submitted before exhausting the cost budget, against 69.0% of instances overall.

Their conclusion is stated without hedging: "increasing the maximum budget or token limit are unlikely to substantially increase performance."

The failures were not agents that ran out of room. They were agents that had gone wrong early and were spending the remainder of their budget going wronger. The paper's failure taxonomy confirms it — roughly 52% of unresolved instances were classified as incorrect or overly specific implementations, and a further 23.4% as cascading failed edits. Very few were "ran out of time."

This matters because "give it a bigger budget" is the first thing every organisation tries, and it is the intervention most likely to be measured as a cost increase with no corresponding benefit. If most failures are wrong-direction failures rather than insufficient-resource failures, then the corrective is a sensor that detects the wrong direction early, not a larger allowance to continue in it. This is the argument for keeping quality left that Chapter 8 develops, and here it has an empirical basis rather than a rhetorical one.

There is a general principle underneath, and it is one of Taleb's better ones: when a system fails, ask whether it failed from scarcity or from misdirection, because the remedies are opposite. Scarcity is fixed by adding. Misdirection is made worse by adding.

2.5 The counter-evidence

If the chapter stopped here it would be propaganda. The naive inference from §2.1 — interfaces matter, therefore build elaborate interfaces — is false, and two systems demonstrate it.

Agentless, published two months after SWE-agent, asked whether the agent loop was necessary at all. It replaced autonomous exploration with a fixed three-phase pipeline: localise, repair, validate. No planning, no tool selection, no trajectory. On SWE-bench Lite it achieved 27.33% at an average cost of \$0.34 — at the time the highest performance and the lowest cost of any open-source approach, later improved to 32.00% [B].

Set the two side by side, and compute what the papers do not:

(Third column derived: average cost ÷ resolve rate. Arithmetic is mine, inputs are the published figures.)

The elaborate interface bought reach — it resolved issues that retrieval could not touch — but it did so at roughly twice the cost per resolved issue. And a well-designed pipeline with no agent loop at all beat it on both axes at once, resolving half again as many issues for a seventh of the cost each.

mini-swe-agent completes the arc. It is one hundred lines of Python. It has no tools other than bash — it does not even use the models' tool-calling interface. It has no configuration surface to speak of. It scores >74% on SWE-bench Verified [D]. Its authors state the lesson directly: complex agent scaffolding is not a prerequisite for superior performance, and minimalist architectures offering low-level primitives can outperform feature-heavy designs.

So we have three data points and they appear to contradict each other:

• 2024: a purpose-built interface beats a bare shell by 7 points and retrieval by 9.5×.
• 2024: no agent loop at all beats the purpose-built interface on cost and resolve rate.
• 2025: one hundred lines and a bash prompt score above 74%.

2.6 Reconciliation: the harness compensates for deficits

These findings are not in conflict once you ask what the interface was for.

The 2024 ACI was not adding capability. It was compensating for specific model deficits, and the ablation table names each one:

Every one of those is a statement about a 2024-era model. And every one of them is the kind of deficit that subsequent training closes. When the deficit closes, the compensation does not become neutral — it becomes overhead. The scaffold that was carrying the model is now constraining it, occupying context that the model no longer needs spent on its behalf.

This produces the most actionable prescription in the chapter, and it runs against every instinct an engineering organisation has:

A mature harness gets smaller when the model improves.

Taleb's via negativa — the observation that in complex systems, removal is more reliable than addition, because you can be confident about what harm you have eliminated and only hopeful about what benefit you have introduced — is not a metaphor here. It is the operating procedure. Chapter 8 turns it into a practice: every element of a harness should carry the date it was added and the failure it was added for, and should be re-tested against a current model on a schedule. Controls whose failure mode no longer occurs are not free. They cost context, latency, and, worst of all, credibility — a harness full of dead rules teaches everyone that the rules do not matter.

Notice also what did not get smaller across the arc. Agentless kept validation. mini-swe-agent kept the shell and the loop-until-tests-pass structure. What was discarded was compensation for model weakness; what was retained was connection to ground truth — the ability to run the code and find out. Chapter 8 elevates this to a distinction: guides age, sensors persist. A rule telling the model how to edit becomes obsolete. A test that tells it whether the edit worked does not.

2.7 The benchmark is a casino

Everything above rests on SWE-bench. So it is necessary to say plainly what SWE-bench is, and it is not what most people who cite it believe.

Taleb's ludic fallacy is the error of mistaking the tidy, bounded uncertainty of a game for the messy, unbounded uncertainty of the world. In a casino the probabilities are known, the rules are fixed, and the game ends. Outside, none of that holds — yet people trained on games reason about the world as though it were one.

SWE-bench is a casino, in three specific ways.

It supplies ground truth that real work does not have. Every instance comes with the maintainers' own tests: a set that must go from failing to passing, and a set that must stay passing. That is a complete, machine-checkable specification of success, delivered free, before the work begins. Chapter 8's behaviour-harness problem exists precisely because real tasks arrive without this. The benchmark hands out, at the start of every instance, the artefact whose absence is the field's central unsolved problem.

A third of it leaked. The SWE-Bench+ study found that 32.67% of successful patches involved solution leakage — the fix was present in the issue report or its comments — and that a further 31.08% of passing patches were suspicious due to weak tests that could not distinguish a correct patch from a plausible one [B]. A separate study using differential patch testing found 7.8% of patches counted as correct while failing the developers' own test suite [B].

And when it was cleaned, the founding number collapsed. Filtering the problematic instances, SWE-Agent with GPT-4 dropped from 12.47% to 3.97% [B].

That is the same 12.47% in the epigraph of this chapter.

I could have omitted this. The chapter reads better without it. But a book whose second commitment is "no selective citation" cannot cite a number in its epigraph and bury the study that dissolves two-thirds of it, so here it is, in the same chapter, in bold.

2.8 Why the argument survives

Having said that, the argument of this chapter is unharmed, and the reason is worth more than the argument itself, because it generalises to every contaminated measurement you will ever be handed.

Contamination destroys absolute claims. It leaves relative claims standing, provided it is symmetric across the arms.

Solution leakage does not know which interface is reading the issue. If a fix is written in the issue comments, it is equally available to the RAG baseline, the shell-only agent, and SWE-agent. Weak tests are equally weak for all three. The contamination is a property of the instances, and every arm of the comparison ran on the same instances.

So the correct reading is not "SWE-agent resolves 12.47% of real GitHub issues." It never was. The correct reading is:

On a fixed set of tasks, holding the model constant, replacing retrieval with a purpose-built interface multiplied the resolve rate by roughly nine. On the same fixed set of tasks, four separate interface dials each showed an interior optimum, and one tool performed worse than its own absence.

Every clause survives cleaning, because cleaning removes instances from both arms.

This is a habit worth acquiring, because you will need it constantly. Almost every number quoted at you about agent performance is a level. Levels are the fragile kind. When you are told a tool "resolves 70% of issues," the question is not whether the number is honest — it probably is — but what it is a level of, and whether anything held constant between the thing being praised and the thing it is being praised against.

And there is a positive corollary. You can build a useful private benchmark without solving contamination, because you only need it to support relative claims: does this harness change help us, on our tasks, against our previous configuration? Chapter 14 makes this the foundation of harness measurement. Your own tasks are contaminated too. It does not matter, provided both arms run on the same ones.

2.9 What this licenses

Four claims are now on solid ground, and one that is often bundled with them is not.

Licensed. Interface design has a large causal effect on agent capability, independent of the model. Controlled comparison, model held constant, ratio robust to contamination.

Licensed. That effect is non-monotonic in richness. Four dials, four interior optima, one control worse than its own absence.

Licensed. Most failures are misdirection, not scarcity. Successful runs are cheaper and shorter; 93% of successes finish inside budget.

Licensed. The optimal harness shrinks as the model improves. Three systems across two years, each simpler and cheaper than the last, each scoring higher.

Not licensed — and this is the inference the field makes constantly — that a higher benchmark score means a tool will work better on your codebase. SWE-bench instances come with ground-truth tests, are drawn from twelve public Python repositories, and are shaped like bug fixes. If your work involves a private codebase, an unstated specification, or a change that is not a bug fix, the benchmark is measuring a different game. Chapter 21 names the error Benchmark Cargo Cult and it is the ludic fallacy wearing a procurement badge.

The honest summary of what Princeton demonstrated is smaller than the headline and more useful:

The environment an agent works in is a designed artefact with measurable effects, an interior optimum, and no guarantee that adding to it helps.

That sentence is the licence for everything in Part II. Context engineering, harness engineering, graph engineering, and structural engineering are all attempts to design that environment deliberately. They are engineering rather than configuration for exactly one reason: the ablation table. Dials with interior optima require measurement. Dials that only go up require enthusiasm.