Harness Engineering: The Engineering Around the Model Determines Whether an Agent Succeeds

Models keep getting smarter, so why do agents still fail so often? The answer lies in the engineering system around the model: the harness. This guide moves from concepts to practical implementation and ends with a checklist you can use directly.

1. Let’s first look at a counter-intuitive fact

The same Claude / Codex model, the same task, changing two sets of "shells", the success rate can jump from 20% to nearly 100% - the model has not changed a word.

This is a repeatedly observed conclusion in the learn-harness-engineering course (see the reference document at the bottom): for a TypeScript/React project, the success rate is about 20% when the model is only given a README; after the "rules, status, verification" shell is completed, the success rate is close to 100%. The model was not changed, only the harness was changed.

OpenAI's case is even more striking (Ryan Lopopolo, February 2026). A small team began with an empty repository in late August 2025. Over roughly five months, humans wrote zero lines of code: Codex generated every line. The result was about one million lines across application logic, infrastructure, tooling, and documentation, plus 1,500 merged PRs. The team grew from three to seven people while per-person throughput rose to 3.5 PRs per day. Rather than writing business code, the engineers focused on designing an environment in which the model could produce reliable work. They estimated that the project took about one-tenth the time of writing the code by hand.

The thing now has a name: Harness Engineering.

One sentence definition:

Agent = Model + Harness. The model is responsible for "smartness", and the harness is responsible for "reliability".

2. What is Harness?

The most common misconception is that a harness is merely a well-written system prompt or a CLAUDE.md file. It is not.

The definitions given by various companies are highly consistent, and the core is the same sentence:

Harness = In addition to model weight, all engineering infrastructure that determines "how much intelligence can be implemented". (LangChain's statement: "every piece of code, configuration, and execution logic that isn't the model itself.")

It includes but goes far beyond prompt: tool definition and execution, file system and sandbox, state persistence, verification mechanism, context management, sub-agent orchestration, deterministic hooks/middleware (compaction, continuation, lint interception and other links that do not rely on model awareness and are enforced by code), observability...

To give the most appropriate analogy:

Think of Agent as a senior engineer joining today. He's smart, but knows nothing about your code base. He needs: project documentation, a running environment, usable tools, knowing the "done" criteria, and shift records. The sum of these things is the harness. The more complete the scaffolding you provide, the closer the ability of this "new person" will be to his true level.

Key insights: Today's Agent failures were mostly not "not smart enough" but "can't see/can't run/don't know when it's over" - these are all harness problems, not model problems. This is the so-called capability–execution gap.

3. Five subsystems of Harness

A complete harness can be disassembled into five subsystems. The following five-point method is synthesized from three sources: WalkingLabs’ learn-harness-engineering course, LangChain’s component panorama, and Anthropic’s long task practice. It is recommended to self-check item by item according to the following table:

A little explanation (the divisions among different companies are not completely consistent): The original five-piece set of the learn-harness-engineering course is Instructions / State / Verification / Scope / Session Lifecycle - "Session Life Cycle" is listed as a separate item. This article changes to a method closer to LangChain/Anthropic: "Tools and environment" (which both emphasize, and most directly determines whether the Agent can run) is made a core item, and "session life cycle" is incorporated into the long task method in Section 4 (initialization/execution separation, clean restart). It doesn't matter how the subsystem is named or how many items it is classified into. It is important not to omit items.

                ┌─ Instructions  What the project is and what constrains it
                ├─ Tools/Env     What the agent can do and whether it can run
   Harness ─────┼─ State         Where the previous session stopped
                ├─ Scope         What this session will do and what counts as done
                └─ Verification  How the agent proves its work is correct

Dismantle them one by one below.

1. Instructions: Put the onboarding guide in the repository

The principle is not "the more you write, the better", but progressive disclosure: a core instruction file of about 100 lines that clearly explains what the project is, the technology stack, the first run command, inviolable constraints, and the document entry. Link out details as needed, rather than gluing an encyclopedia to the model all at once, which would only dilute attention.

Core iron rule: The repository is the only source of truth (repo as the system of record). Agent cannot turn around and ask colleagues like humans. Everything it needs must be visible in the warehouse.

2. Tools/Environment: Give less constraints and more capabilities

• Tool permissions follow least permission instead of banning them all - if the restrictions are too strict, the agent can only rely on guesswork.
• The environment must be self-describing and reproducible: the dependency files, runtime versions, and container configurations are all in place so that the Agent can start serving with one click. In Anthropic's practice, there is a dedicated init.sh responsible for pulling up the development environment.
• An underrated ability: Give Agent a sandbox that can run bash/execute code. Rather than pre-configuring a bunch of narrow tools, "being able to write code and run it yourself" is the key to universal autonomy - it can verify and debug by itself.

3. State: Handover like a shift worker

Long tasks span multiple context window, each new session is a blank slate. Anthropic likens this to "engineers working shifts without handover documentation."

The solution is to put the state on the disk:

• Progress file (claude-progress.txt / PROGRESS.md) records: what has been done, what is being done, and where it is stuck.
• Use git commits and commit messages to leave traceable traces.
• Each session starts by reading the progress and git log, and updates the progress before ending.

Effect: When there is a harness, the Agent can accurately "continue from the last breakpoint"; when there is no harness, it will either redo it or start from the beginning every time.

4. Scope: Do one thing at a time

A common problem with Agents is being greedy for too much—trying to implement too many functions at once, but not finishing any of them.

Anthropic's approach is very straightforward: split the task into 200+ extremely fine-grained feature lists, mark each item with pass/fail, and use JSON instead of Markdown (JSON is structured, and the model is less likely to be rewritten without authorization). Each sessiononly works on one feature, completes it, tests it, submits it, updates the progress, and then moves on to the next one.

5. Verification: Change "good or bad" into "pass or not"

This is the subsystem with the lowest investment and the highest return, and it is also the most easily ignored.

The model has two fatal tendencies:

1. Premature declaration of victory - Saying "done" without testing.
2. Self-praise - Ask it to evaluate its own output, and it will almost always say "excellent", even if it looks like a mess to others.

Countermeasures:

• Explicitly list the verification commands (test/lint/typecheck) in the document so that the Agent can run it every time.
• For web applications, it is required to use browser automation (such as Puppeteer MCP) for end-to-end verification, rather than "just look right". Anthropic's actual measurement has greatly improved the accuracy of "the function is really usable".
• Turn subjective qualityinto an objective standard that can be scored. When Anthropic does front-end work, it splits the subjective judgment of "good or bad" into four evaluable dimensions: design quality, originality, craft, and functionality, and then uses few-shot examples to calibrate reviewers.

4. Advanced methods for long tasks

For short tasks, it is enough to do the above five things well. But when the task needs to run for several hours and span dozens of context window, three advanced modes are needed.

1. Separate initializing Agent and executing Agent

Anthropic's long-term task harness is a dual-role architecture:

• Initializer Agent: The first session lays the foundation - write init.sh, build a progress file, make the first git commit, and generate the 200+ feature list.
• Coding Agent (Execution): Each subsequent session follows a fixed process - read the progress → run the E2E smoke test first → complete a feature → submit → update the progress, must leave a clean state that can be merged when leaving.

2. Separation of generation and evaluation (Generator/Evaluator)

Don't let the same Agent be both a player and a referee. Independent Generators and Evaluators can crack "self-praise". The evaluator only pays for it when the task is beyond the scope of "a single model can reliably complete it alone" - when the task is simple, this set of overhead is a waste.

Anthropic's classic controlled experiment: one sentence prompt makes a retro game.

• Single Agent running alone (solo run): $9, 20 minutes, producing an application that cannot run.
• Three Agent harness (planning + generation + evaluation): $200, 6 hours, to produce a playable game with exquisite UI.

The quality gap can support this investment - the premise is that the task is really difficult.

3. Context reset is better than context compression

When the model feels that "the token is about to run out", context anxiety will occur - ending in a hurry. The intuitive approach is to compress the history into a new window; but Anthropic found that it is better to directly clear the context and start over with a structured handover product (progress file + feature list). This also echoes what LangChain said "Ralph Loop": re-inject prompt in a clean new window to combat context rot (context rot).

5. In-depth reading of the case: How did OpenAI sustain its 1 million lines of code?

OpenAI's "Harness Engineering: Leveraging Codex in an Agent-First World" is currently the most detailed practical record. Taking it apart, almost every piece of experience can be mapped back to the previous five subsystems - but there are a few points that are counter-intuitive enough and worth talking about separately.

1. The role of engineers has changed: don’t write code, build the environment

Early progress was slower than expected. The reason was not that the Codex was not good, but that the environment lacked specifications—the Agent lacked tools, abstractions, and internal structures. So the core work of engineers becomes one sentence: "What capabilities are missing, and how to make them visible and mandatory to the Agent?"

When something goes wrong, the fix is ​​almost never "make the model work harder" but go back and fix the harness. Humans only work at higher levels of abstraction: prioritizing, translating user feedback into acceptance criteria, and validating results. Even the repairs are done by Codex itself.

2. Give a map, not a thousand-page manual

They also tried "a big AGENTS.md" at first, and it failed very typically:

• Context is a scarce resource - large files crowd out tasks, code, and related documents;
• All focus = no focus - Agent degenerates into partial pattern matching;
• Instant rot - quickly filled with expired rules and left unmaintained;
• Difficult to verify - A large file cannot be mechanically checked (coverage, freshness, cross-linking).

Change the method: Treat AGENTS.md as a table of contents, not an encyclopedia. An AGENTS.md of about 100 lines is just a "map", and the real sources of truth are placed in the structured docs/ directory - design documents, execution plans (active/completed/technical debt tracking), generated DB schema, product specifications, *-llms.txt references, etc. Plans are first-class citizens: Use temporary lightweight plans for small changes, and execute plans signed into the warehouse after completion of complex work, with progress and decision logs.

This is Progressive Disclosure: Agent enters from a small but stable entrance and is "taught" where to find the next step. Moreover, mechanical enforcement - specialized linter and CI verify whether the knowledge base is up to date and cross-linked; there is also a "doc-gardening" Agent that regularly scans expired documents and automatically proposes repair PRs.

3. The most important thing to remember: If the Agent cannot be seen, it does not exist.

From the agent's point of view, anything it can't access in-context while running effectively doesn't exist.

That Slack discussion that aligned the team structure? If the Agent cannot retrieve it, it is the same as "newcomers joining the company three months later don't know" and it does not exist. Conclusion: Continuously stuff context into the warehouse - code, markdown, schema, executable plan, these are the only things that the Agent can see.

This leads to a set of counter-intuitive trade-offs: Prefer dependencies and abstractions that the Agent can fully internalize. The so-called "boring" technologies are better modeled because their APIs are stable, composable, and appear frequently in the training set. Sometimes it is more cost-effective to rewrite a subset yourself than to bypass an opaque upstream library. Instead of using the universal p-limit, they hand-write a map-with-concurrency, deeply integrating their own OpenTelemetry, 100% coverage, and fully controllable behavior.

4. Make the application itself readable by the Agent (this is a real bottleneck in throughput)

After the code throughput increases, the bottleneck becomes human QA capabilities. The countermeasure is to make the UI, logs, and indicators directly visible to Codex:

• Each git worktree can independently start an application instance, one instance is allocated for each change;
• Chrome DevTools Protocol is connected to the Agent runtime, and DOM snapshots, screenshots, and navigation skills are created - Codex can reproduce bugs, verify repairs, and reason about UI behavior by itself;
• Each worktree is equipped with a temporary observability stack, and Codex uses LogQL to check logs and PromQL to check indicators. So "Guarantee that the service startup is completed within 800ms" and "There is no span in these four critical links for more than 2 seconds" This kind of prompt becomes executable.

Effect: A single Codex run often lasts for more than 6 hours (usually while the person is sleeping).

5. Enforce invariants, not micromanage

Documentation cannot support the coherence of a fully automatically generated code base, only architectural constraints can. Their slogan is "enforce invariants, not micromanage implementations":

• It requires "Resolve data shape at the boundary", but does not specify how to do it (the model itself likes to use Zod, no one specifies it);
• The entire application is built on a rigid layered architecture: each business domain has a fixed layer (Types → Config → Repo → Service → Runtime → UI), the dependency direction is strictly verified, and cross-cutting concerns (authentication, connectors, telemetry, feature switches) can only be entered from a single Providers interface;
• All use custom linter (also written in Codex, of course) and structured testing mechanical force; customize the error message of lintand directly inject repair guidance into the Agent context.

A highlight: **This kind of architecture was originally needed by hundreds of engineers in the company; with Agent, it became a prerequisite from the beginning - constraints are the prerequisite for "fast but not out of order". ** The principle is "The center enforces boundaries, and local autonomy is allowed": Strictly maintain boundaries, correctness, and reproducibility; within the boundaries, give the Agent full freedom. Code does not always conform to human aesthetics. As long as it is correct, maintainable, and readable by future Agents, it will meet the standard.

6. Throughput changes the merge philosophy

When the Agent throughput far exceeds the attention span of a human being, many traditional engineering specifications are harmful: the warehouse has almost no blocking merge access control, PR is short-lived, and test flake is solved with "rerun once" instead of blocking indefinitely. Because in this kind of system, correction is very cheap and waiting is expensive. - This is irresponsible in a low-throughput environment, but it is often true here.

Going deeper, behind this is the economic core of the entire engineering practice: tokens, computing power, and Agent working hours are no longer scarce. The only thing that is truly scarce is the attention that "requires simultaneous intervention." ** So almost all engineering decisions converge to the same goal - **to save people's simultaneous attention to the extreme. The trade-offs of merging access control, cracking QA bottlenecks, and automatic background cleaning are all essentially saving this item. Once you understand this, all the previous counterintuitive choices will become logical.

7. Entropy and garbage collection: "encoding people's taste once and forcing it permanently"

Fully automatic also brings new problems: Codex will copy the existing patterns in the warehouse, including bad, so it will inevitably drift.

They initially relied on manual cleaning - Every Friday (20% of working hours) was dedicated to cleaning "AI slop", without scaling at all. The real solution is to encode the golden principles into the warehouse and equip it with a periodic cleaning process:

• Prioritize using shared toolkit to converge the invariants in one place instead of hand-writing scattered helpers everywhere;
• Not "YOLO-style" geophysical data - but check at the boundary or use a typed SDK to prevent the Agent from being built on the guessed data shape;
• A set of background Codex tasks regularly scan for deviations, update quality scores, and propose directional refactoring PRs - most of which can be reviewed within one minute and automatically merged.

This mechanism is like garbage collection. Technical debt is a high-interest loan: it is always better to continue to repay it in small amounts than to let it compound and repay it in painful clusters. Human taste only needs to be captured once and then continuously enforced on every line of code.

8. The tipping point of autonomy

As testing, verification, review, feedback processing, and recovery are all coded into the system, this warehouse has recently crossed a threshold: given a prompt, Codex can verify the current status end-to-end → reproduce the bug → record a video of the failure → implement the repair → drive application verification → record a video of the repair → open a PR → respond to feedback from people and agents → detect and fix build failures → Only upgrade to people when judgment is needed → Merge.

But they clearly reminded: This is highly dependent on the specific structure and tool investment of the warehouse. Don't assume that it can be directly generalized - at least not yet.

In one sentence: human discipline has not disappeared; it has moved from the code itself into the harness.

6. Comparing approaches: how far should you go?

According to the investment, it is divided into three levels from light to heavy:

• A. Naked prompt - only give a README and add nothing.
• B. Single Agent + Five Subsystems - Complete the five items in Section 3; one Agent can span sessions.
• C. Multi-Agent complete set - Layer the advanced methods of Section 4 on top of B (initialization/execution separation, generation/evaluation separation, context reset).

How to choose depends on a question: Can an Agent complete this task in one session?

• Yes, and throw it away → A
• To continuously iterate and span multiple sessions → B (in most cases)
• An Agent cannot handle it alone (several hours, spanning dozens of windows, high quality requirements) → C

In summary:

• The optimal solution for most teams is B - the highest input-output ratio and the best cost-effectiveness point. Don’t overlay multiple Agents as soon as you start.
• C is only cost-effective when the task is "beyond the capabilities of a single model alone": more than a few hours, spanning multiple windows, and with high quality requirements (such as complete applications, complex reconstructions). Its overall rating is not higher than B because the set of overhead in simple tasks is purely wasteful - the complexity must match the difficulty of the task.
• A For one-time, disposable exploration only. Anything that requires continuous iteration, stopping at A will pay the price of repeated rework.

A discipline that has been proven repeatedly: After the model is upgraded, go back and perform ablation testing on the harness piece by piece. ** Anthropic discovered on Opus 4.6 that after the model becomes stronger, the original "sprint contract" mechanism can be deleted entirely without a drop in quality and a significant drop in tokens. **Harness does not just increase but does not decrease - as the model adds capabilities, the burden on the harness should be reduced.

7. Ready-to-use harness checklist

When you land, check this list:

• Instructions
  • There is a AGENTS.md / CLAUDE.md with ≤100 lines. Write down the project, technology stack, first run command, and red line constraints.
  • Follow the links for details, don’t pile them up into an encyclopedia.
  • Everything the Agent needs is in the repository (repo = single source of truth)
• Tools / Environment
  • Dependencies, runtime versions, and container configurations are complete, and the environment can be started with one click (init.sh)
  • Provides a sandbox that can run bash/execute code
  • Permissions should be granted based on the least privileges, rather than banning them across the board.
• State
  • There is a progress file to record "finished/in progress/stuck"
  • Reading progress + git log at the beginning of the session, updated before the end
  • Use git to commit and leave traces
• Scope
  • Split the task into a machine-readable feature list (JSON is recommended)
  • Only do one feature per session
• Verification
  • The test / lint / typecheck commands are explicitly listed in the documentation.
  • Web applications have end-to-end validation (browser automation)
  • Subjective quality has been broken down into objective dimensions that can be scored
• Long mission extras
  • Separation of initialization and execution of Agent
  • Separation of generation and evaluation (only if the task is difficult enough)
  • Use context reset + handover products instead of blind compression

8. Why this matter will be important in the long run

Some people may ask: Isn’t the model getting stronger and stronger? When it is smart enough, won’t the harness be useless?

Quite the opposite. The judgments of LangChain and OpenAI are consistent: the stronger the model, the focus of harness is just shifting from "patch-filling the pits of the model" to "optimizing the system for intelligence" - but it will not disappear. Just as models are no longer stupid today, prompt engineering is still important.

There is also a layer of harder coupling that is often overlooked: Today's strong models are trained together with the harness. LangChain pointed out that products such as Claude Code put "model + harness" into the same loop for joint debugging in the post-training stage, and the model is specially optimized to fit the shell it was trained in. This also explains a counter-intuitive phenomenon - putting a strong model into an unfamiliar harness often fails to achieve its true level. The relationship between model and harness is co-evolution, not that one will replace the other sooner or later.

OpenAI’s words pointed out the industry’s turn:

In the agent-first world, the engineer's main job is no longer to write code, but to design an environment that allows the Agent to produce reliable output.

The model is the engine, and the harness is the chassis, transmission and instrument panel. With just an engine, it can't run far and it's not safe.

References

• LangChain — The Anatomy of an Agent Harness: Component panorama of Agent = Model + Harness.
• Anthropic — Effective harnesses for long-running agents: Actual implementation of initializing Agent, feature list, and self-verification.
• Anthropic — Harness design for long-running apps: Generation/evaluation separation, context reset, scoreable quality.
• OpenAI — Harness Engineering: Leveraging Codex in an Agent-First World: One million lines of code and the golden rule.
• WalkingLabs — learn-harness-engineering (Online Handouts): 12 lectures + 6 hands-on projects systems course (source of the Five Subsystems Framework).
• WalkingLabs — awesome-harness-engineering: Collection of resources (Context, Assessment, Guardrails, Orchestration, Benchmark).