AI and the Industrial Revolution

Better Ground, Not a Better Model

You don't get a better agent by buying a better model alone. You get one by preparing the ground it runs on: the tests, the structure, the rules it runs inside.

Sertan Yamaner

— 14 min read
Share
Tea terraces contoured into a misty hillside with a dirt path winding up through the rows, forested mountains and low cloud behind
The terrain is the part you build.

I run a coffee roaster that sits at around 200°C in my kitchen, with an agent in the loop, and it has flagged first crack batch after batch since last November. The version doing that today works, and I still would not trust it as it stands, because the safety lives nowhere in particular.

So I am rebuilding it around one rule: the model that advises the roast must never be able to touch the controls directly. That safety will not come from a smarter model or a cleverer prompt. It comes from the structure around the model, deterministic code that reads the model's suggestion and decides whether to allow it, clamp it to a safe range, or throw it out, before anything reaches a machine that gets very hot in an occupied house.

That is the whole argument in miniature. You do not get a trustworthy agent by buying a better model alone. You get one by preparing the ground it runs on, and by deciding what it is allowed to do once it is there.

The last post ended on a handoff: the technology does not decide the outcome; what you build around the agents does. This is the first half of that answer, the ground.

We will look at what that ground consists of, why tests change what an agent can do, and what the structure still does not fix. Most of the rebuild is already built, and its design is the point here, because that is where the safety lives.

It isn't the prompt, and it isn't more context

Take my prototype. It works, it still roasts my coffee, but it runs on an n8n workflow that gives me no real state machine, no defined behaviour on restart, and no single, testable safety boundary. No prompt and no amount of extra context turns that into something you would trust with the process. What it is missing is structural, and that gap is exactly what the field has been finding names for.

The leverage keeps moving outward, from the words you type toward the whole environment around the agent, and the vocabulary has stretched to keep up with it.

In the middle of 2025 the conversation moved from "prompt engineering" to "context engineering." Tobi Lütke described the skill as providing everything a task needs to be plausibly solvable; Andrej Karpathy called it filling the context window with the right information for the next step. The rename mattered because "prompt" had come to mean typing tricks into a chatbot, while the real work was much larger. By 2026 there was a narrower name for one part of that work, "harness engineering": the guides, deterministic controls, and feedback sensors built around the agent. I use the simpler word "ground" for the whole thing: the context, the codebase, the tests, the feedback loops, and the authority boundaries the agent is allowed to work inside.

Chad Fowler named the deeper pattern "Relocating Rigor": discipline does not vanish when an agent writes the code, it moves closer to the feedback, the tests, and reality. That is the same thing the first post said when it argued the work had moved from writing code to judging which code to trust. Fowler says it one level out: the rigour moves from the keystrokes to the environment around them.

Three nested layers with the outermost emphasised: the code you write, wrapped by the context you curate, wrapped by the terrain you build around the agent

Rigour relocates outward, each layer wrapping the last: from the code you write, to the context you curate, to the terrain, the ground you build around the agent.

Now the part most teams get wrong in the other direction. If the prompt is not the lever, the answer is not simply "give it more context" either.

A Stanford-led study, "Lost in the Middle," showed that models use the beginning and end of a long input well and the middle badly, so the thing you need can end up in the dead zone. Chroma's "Context Rot" report then tested eighteen models and found every one grew less reliable as the input got longer. For coding agents this is an everyday failure mode: searching and backtracking accumulate noise, which degrades what the agent does next. One honest caveat is that Chroma sells a retrieval product; the Stanford result points the same way without that commercial interest, which is why I lean on it.

If it is not the prompt and it is not more context, what is left is the ground itself: the structure, the tests, the boundaries the agent is allowed to work inside.

The ground decides

Here is the cleanest field evidence I know that the environment, not just the model, sets the ceiling.

When Microsoft's coding agent first arrived in dotnet/runtime, a fourteen-million-line legacy system, it could not reach the package feeds, did not know the build commands, and could not test its own changes. Among decided pull requests, its merge rate was 38.1% before the first setup changes. After the team opened the dependency access, added repository instructions, and taught it how to build and test, the rate was 69%. That is an observational result rather than a controlled trial, and the task mix shifted over the period too, but the intervention was not a new model. It was better ground.

Horizontal bar chart: the Copilot Coding Agent's merge rate in dotnet/runtime rose from 38.1% of decided pull requests before repository setup changes to 69% after

Microsoft's observational before-and-after in dotnet/runtime: 38.1% of decided Copilot Coding Agent pull requests merged before the first repository setup changes, against 69% afterwards. The changes gave the agent package-feed access and repository-specific build / test instructions, not a new model. Task selection and use of the agent also evolved over the period, so this is field evidence rather than a controlled causal estimate.

Scale's SWE-bench Pro widens the pattern across repositories. It collected more than 1,800 real tasks from 41 production repositories, and its most useful finding is about variance, not averages. The same models that clear a task in one repository fail almost everything in another: on some repositories every model resolved fewer than one task in ten; on others some models cleared more than half. Same models, same evaluation. The difference was the codebase, its complexity, the kind of problem, and the quality of its documentation.

The model was the constant. The ground was the variable.

The benchmarks tell the same story from the other side. Performance falls further as the work gets harder: on long-horizon tasks that evolve a codebase across many files and steps, even the best resolves only about a quarter (the SWE-EVO benchmark). And the older, curated benchmarks flatter, partly through contamination: OpenAI has stopped using SWE-bench Verified as a measure of frontier capability, because tasks had leaked into training data and were often underspecified. A curated benchmark flatters the agent; real ground exposes it. Treat the score as planning evidence, not a guarantee.

So if you want a better result, the question is not only "which model," it is "what does the ground look like." And one of the most powerful pieces of ground you can lay down is tests.

Tests sharply improve the reliability of what you accept from an agent, for a plain reason: the agent can check its own work, and you can stop treating line-by-line review as the only signal and start reading the aggregate. SWT-Bench measured this directly: keeping only fixes that passed tests written by the agent roughly doubled precision, from about 20% to 48%, though it kept fewer fixes to do it. UTBoost makes the point from the other side: leave the agents and patches untouched, strengthen only the tests, and 345 supposedly passing patches turn out not to fix the issue. Change the tests, and the measured competence of the agents moves underneath them.

A test suite is judgement turned into infrastructure. The first post borrowed a rule from the Industrial Revolution: when an input gets cheap, you do not get the same work done more carefully, you get a flood of it. Cheap attempts flood a codebase the same way, and no single human can read all of it. A test suite is the scarce skill from that post, telling a correct answer from a merely plausible one, written down once and run automatically so it scales past the person who would otherwise have to read every line.

I did not need a benchmark to learn this. Early in the rebuild, on a supervised roast, the advisor was a capable, pinned model giving sound advice, and the batch overshot its target anyway. The model was not the problem; the ground beneath it was. A development guard in my own deterministic code overruled a correct drop recommendation five times as the bean climbed from 197 to 203°C, past its profile ceiling, and the roast ran on to a bitter finish.

The response was not a better model or a cleverer prompt; it was work down in the deterministic ground: an anticipatory trim that eases the heat back before the crack, and a correction to the guard logic itself that I am still working through. That is the whole argument again, learned on a hot machine rather than a leaderboard.

The rebuilt structure has two established parts worth describing.

The first is a single source of truth. The prototype talked to two separate servers, one for detection and one for roaster control, which forced the agent to keep reconciling two different clocks. I have replaced them with one local server that owns the machine and the session, and it has passed driver-level validation against the real hardware.

The second is the harness itself, built and running. A deterministic controller owns the loop; the model returns a typed recommendation that safety code allows / clamps / rejects before any command reaches the hardware. The control tick is set by the thermocouples, not by the model. By construction the model has no direct path to the hardware; every recommendation passes through deterministic policy first, and the safety boundary is a property of that structure, which is mine to design.

Why delegation worked

The split that made the work go fast was strict, and it is worth being precise about who owned what.

I owned the architecture, the machine-learning decisions, the rules the agent had to follow, and the quality gates. The agent owned execution: writing the code, running the training, doing the exports, working through the checklist. On that division, the first stream of the rebuild, the detection and training work, shipped eighteen stories across ten pull requests in two evenings. I would not describe that as magic. It was execution against prepared ground: the architecture, safety rules, and quality gates were not delegated.

The harness is what my half of that split is for: the safety boundaries, the single source of truth, the validate / clamp / reject step. The coding agent could write that boundary, but it did not get to decide what the boundary was, or what counted as done; those stayed mine, enforced through specs, tests, and review. So the split is not a temperament thing, a matter of liking to keep control. It is a property of the ground. Prepare the ground and you can delegate the execution more safely. Leave it unprepared and delegating execution is just how you ship a lot of confident, broken work quickly.

The honest limits

Preparing the ground scales your judgement. It does not remove the need for it, and pretending otherwise is the easy way to write something that sounds clean and is wrong.

Early in 2026, METR had maintainers review agent-written patches that had already passed an automated grader. About half would not have been merged: some for quality, some for breaking unrelated code, and some because they did not fix the problem the tests claimed they had fixed. METR is careful to note that the agents got a single attempt and could not revise after review the way a human contributor would, so this is a limit of that setup, not proof of a hard capability ceiling. Two other studies point the same way. Strengthened tests reduced the leading agent from 79% to 62%, while earlier work found that about 30% of plausible, test-passing patches behaved differently from the developers' own fix.

Passing the tests is not the same as being right, and I have been caught by that in my own build more than once, with a green suite that was quietly checking less than I thought. That is why the human judgement from the first post does not go away.

The roaster keeps me honest in the same direction, because it is unfinished. Loading the beans is a manual step I do by hand, and there are overrides and recovery states. The harness has run a mock vertical slice in continuous integration, the real server in mock-driver mode, and a series of supervised roasts, up to a full roast where the controller carried out the advisor's drop recommendation through the safety box. What is not yet there is repeatability: the post-first-crack control is still being stabilised, and redesigned to pull more of it back into deterministic code. I am not going to call it production-ready, because that consistency is exactly what is still missing.

A crumbling concrete wartime pillbox half-overtaken by dune grass, rusted reinforcing bars showing through the decaying concrete

Neglected ground. With an agent building volume on top of it, bad ground is now more dangerous, not just slower.

There is also a sharper edge. Bad ground is more dangerous now because the agent builds volume on top of it. A weak test suite or vague spec used to slow one person down; now it lets an agent generate plausible, wrong changes at speed. Systems with no tests and no clear contracts are the hardest case, and they are common in established companies. The dotnet/runtime turnaround earlier is the encouraging version of this, a team with the resources to improve its ground after the early failures. Teams under delivery pressure may instead skip that work and ship plausible debt fast.

Where this goes next

Ground is something you prepare inside a codebase. But who is given the time to do it, who is allowed to, and how that work is coordinated when several teams depend on the same ground: those are not codebase questions anymore. They are organisational ones, and they are the next and final post.

For now the practical version is small, and a little against the grain. The model is the part you buy; the ground is the part you build. Preparing it is the slow, unshowy work that nobody applauds, and choosing to fund it instead of just shipping faster is a decision someone in charge has to make on purpose. That decision is what turns raw capability into something you can actually trust.

The first post ended by saying the outcome is still ours to choose. This is one of the places we choose it.

Sources

The context shift

Why more context isn't the fix

  • Liu et al. (Stanford), "Lost in the Middle: How Language Models Use Long Contexts" (2023). https://arxiv.org/abs/2307.03172. Models use the start and end of long inputs better than the middle.
  • Hong, Troynikov & Huber, "Context Rot: How Increasing Input Tokens Impacts LLM Performance," Chroma (14 July 2025). https://www.trychroma.com/research/context-rot. Eighteen models, all degrade (non-uniformly) as input grows, even on simple tasks. Chroma sells a retrieval product and the piece is not peer-reviewed, so read its framing with that in mind.

The ground evidence

  • Scale AI, "SWE-bench Pro" (1,865 real tasks across 41 production repositories; blog 19 Sep 2025, leaderboard maintained). Blog: https://scale.com/blog/swe-bench-pro. Paper: https://arxiv.org/abs/2509.16941. Public leaderboard: https://labs.scale.com/leaderboard/swe_bench_pro_public. Source for the body's per-repository variance ("fewer than one in ten ... more than half"): "Some repositories show consistently low resolve rates across all models (below 10%), while others allow certain models to achieve resolve rates exceeding 50%." The benchmark splits into a published public set (731 tasks across 11 open-access repositories), a commercial set (276 tasks across 18 proprietary repositories), and an unpublished held-out set (858 tasks across 12 repositories). Scale acknowledged unspecified leaderboard issues (news entry dated 18 May), and an external report (issue #93: https://github.com/scaleapi/SWE-bench_Pro-os/issues/93) documents reward hacking via future git history left in the public Docker images. Its absolute scores warrant caution, so the body leans on the structural variance pattern rather than any single number.
  • Stephen Toub (Microsoft), "Ten Months with Copilot Coding Agent in dotnet/runtime" (.NET Blog, 23 Mar 2026). https://devblogs.microsoft.com/dotnet/ten-months-with-cca-in-dotnet-runtime/. Source for the before/after, verbatim: "PRs created before our first setup changes: 38.1% success rate. PRs created after: 69%," where success = merged ÷ (merged + closed), excluding open PRs. The setup changes were package-feed firewall access, a copilot-instructions.md, build and test commands, and steps to avoid build timeouts. Observational, not a controlled trial: task selection and use of the agent also evolved over the period. (The same report also gives a cross-repo comparison, a 77.3% merge rate on a ~105k-line greenfield repo vs 67.9% on dotnet/runtime, with roughly ten times the review comments per 100 lines.)
  • "SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios" (FPT Software AI Center; arXiv:2512.18470, v6, 22 May 2026). https://arxiv.org/abs/2512.18470. Long-horizon, multi-file evolution (48 tasks, 7 Python repos, avg ~21 files/task): even the best model (GPT-5.4 + OpenHands) resolves only 25%. Source for the "about a quarter" line.
  • OpenAI, "Why we no longer evaluate SWE-bench Verified" (early 2026). https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/. Training-data contamination plus underspecified/flawed tasks; recommends migrating to SWE-bench Pro.
  • Mündler et al., "SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents" (NeurIPS 2024). https://arxiv.org/abs/2406.12952. Coverage tracks quality, and tests the agent writes itself, used as a filter, roughly double its precision (≈20% → 47.8%, recall falling to 20%).
  • Yu, Zhu, He & Kang, "UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench" (ACL 2025; arXiv:2506.09289, Jun 2025). https://arxiv.org/abs/2506.09289. Strengthening only the test suites (models and patches held fixed) changed leaderboard rankings for 40.9% of SWE-bench Lite and 24.4% of SWE-bench Verified entries, and exposed 345 patches scored as passing that did not fix the issue.

The honest limits

  • METR, "Many SWE-bench-Passing PRs Would Not Be Merged into Main" (March 2026). https://metr.org/notes/2026-03-10-many-swe-bench-passing-prs-would-not-be-merged-into-main/. About half of test-passing AI patches rejected by maintainers. Important caveat: the agents got one attempt and could not respond to review feedback, so this is not evidence of a hard capability limit.
  • SWE-ABS, "Adversarial Benchmark Strengthening Exposes Inflated Success Rates" (arXiv:2603.00520, 28 Feb 2026). https://arxiv.org/abs/2603.00520. Under strengthened tests, the top agent falls from 78.8% to 62.2% (1st → 5th), and about one in five "solved" patches from the top-30 SWE-bench Verified agents prove semantically incorrect. Source for the body's "79% to 62%" line.
  • Wang, Pradel & Liu, "Are 'Solved Issues' in SWE-bench Really Solved Correctly? An Empirical Study" (arXiv:2503.15223, 19 Mar 2025). https://arxiv.org/abs/2503.15223. The corroborating prior: ≈29.6% of plausible (test-passing) patches behave differently from the developers' ground-truth fix, and 7.8% pass while failing the project's own developer test suite. (A behavioural-divergence rate, not a blanket "wrong" rate.)

The build (what exists, what is in design)

Read more