Reliability Is a Harness Property
Model quality matters, but it is not the reliability system. The system is the harness: the contracts that decide what an agent sees, what it can do, what it must prove, and when it is forced to repair the run instead of declaring victory.
The Model Is Not the Operating System
Weak agent programs treat reliability as a procurement problem. The run fails, so the team swaps the model, raises the context window, or spends more on inference. Sometimes that helps. It does not create a reliable operating surface by itself.
The practical failure is usually lower than intelligence. The agent was given a vague task. It loaded the wrong context. It trusted stale notes. It called a tool without a contract. It stopped after a plausible answer. It repaired the same local symptom three times because no part of the harness forced a plan reset. Those are system failures.
This is why Greyforge treats harness engineering as the real reliability layer. Capability has to pass through contracts before it becomes dependable work.
What the Research Keeps Saying
The public research arc points in the same direction. Serious evaluation keeps moving away from isolated prompt scoring and toward real environments, execution feedback, tool boundaries, and reproducible checks. The lesson is not that benchmarks are perfect. The lesson is that useful agent evaluation has to look more like systems engineering than trivia grading.
SWE-bench
Real repository issues push evaluation beyond short-form coding tasks.
SWE-agent
Agent-computer interface design changes execution quality.
AgentBench
Long-horizon reasoning and instruction following remain hard under action loops.
AgentDojo
Tool use must be tested for utility and prompt-injection pressure together.
Source-selection method: sources were selected when they expose real repository or computer tasks, explicit tool interfaces, executable scoring, or adversarial tool-use conditions. General model leaderboards and vendor claims were excluded because they do not isolate the harness boundary discussed here.
| Claim | Evidence source | Boundary |
|---|---|---|
| Repository-scale evaluation needs executable environments and testable outcomes. | SWE-bench; SWE-agent | Evidence from benchmark design, not proof that one harness works for every repository. |
| Interface and tool design affect agent performance independently of the underlying model. | SWE-agent; ReAct is discussed in the full dossier | Research evidence; Greyforge's specific contracts are operating doctrine. |
| Utility and adversarial pressure should be evaluated together for tool-using agents. | AgentDojo | Supports the evaluation principle, not Greyforge's exact approval policy. |
| Task, context, tool, verification, recovery, and memory contracts form Greyforge's reliability stack. | Greyforge operating doctrine | A position derived from operations, not a controlled external benchmark result. |
Sanitized Trace-Ledger Example
This abbreviated example shows the evidence shape without exposing a private repository, prompt, path, or implementation detail. A trace is useful because the failed verification remains visible instead of being overwritten by the repaired result.
| Phase | Recorded event | Gate result |
|---|---|---|
| Intake | Change a parser without altering public output | Owner and completion test recorded |
| Context | Load parser contract and focused fixtures | Unrelated memory excluded |
| Action | Patch one owned module | Write set remains bounded |
| Verification | Focused fixture fails on an edge case | Completion blocked; no success claim |
| Recovery | Revise parse boundary and rerun focused checks | Pass recorded with full gate deferred |
The Public Rule
A reliable agent harness is a stack of contracts. The model can still reason, write, inspect, and repair. The harness makes those actions bounded, observable, and reversible enough for real work.
This is the same doctrine behind Memory Quality Without an LLM Judge: make the cheap boundary deterministic before spending a model call on what a gate could have rejected. It also explains why memory continuity and operations control matter so much. An agent that cannot inherit the right state cannot be trusted to finish the right job.
What Stays Behind the Gate
The full edition is not a longer pep talk. It is the operational dossier: failure classes, harness layers, scorecards, trace discipline, budget policy, security pressure, and the minimum reference architecture a serious builder can adapt.
Greyforge will keep public evidence online, but the transferable method belongs in the premium Chronicle layer. That protects the forge from automated extraction while still giving public readers a real thesis they can inspect, cite, and challenge.
Diagnose agent failures as task, context, tool, state, verification, or recovery failures before spending more on a model swap.
- Edition
- Engineering dossier 1.0
- Published
- Feb 18, 2026
- Preview updated
- Aug 27, 2026
- Reading time
- 16 minutes
- Full edition
- 4,300 words
- Status
- current
Literature Synthesis
Secondary class: Position Note
- Research question or engineering problem
- Which system layers determine whether an agent can complete real repository work reliably?
- Principal finding
- Public evaluation work and Greyforge operations both point toward task contracts, environment feedback, tool boundaries, traces, and verification as distinct reliability layers.
- Evidence type
- Selected public research plus Greyforge operating doctrine.
- Method summary
- Select research on real environments, repository tasks, tools, and adversarial conditions, then map its signals separately from Greyforge policy.
- Scope
- Agent harness architecture; not a universal model leaderboard or controlled cross-platform benchmark.
- Limitations
- The source set is selected rather than systematic.
- Greyforge doctrine is operating judgment, not external experimental proof.
- Public source or reproduction note
- SWE-bench paper
- Published
- 2026-02-18
- Last verified
- 2026-08-27
- Status
- current
Who this is for
- Builders of long-running tool-using agents.
- Reviewers designing acceptance, trace, and recovery gates.
- Operators deciding whether to repair a harness or change the model.
Not for
- Readers seeking a model leaderboard, a universal reliability score, or a prompt-only recipe.
Detailed contents
- 01
Reliability is a systems property
- 02
Public research basis
- 03
Six failure classes
- 04
Eight harness layers
- 05
Action and trace ledger
- 06
Verification and loop control
- 07
Recovery contract
- 08
Reliability scorecard
- 09
Model-swap decision rule
- 10
Minimum reference architecture
Named artifacts
Reliability failure taxonomy
Six classes that locate failure below model capability.
Eight-layer harness architecture
Intake, context, tools, ledger, verification, loop, recovery, and memory.
Sanitized trace ledger
A reviewable sequence of evidence, action, result, and unresolved assumption.
Reliability scorecard
Contract and evidence checks for deciding whether a run can close.
Model-swap decision rule
When capability is the bottleneck and when the harness is the defect.
Reliability failures happen below intelligence
A failed run is often described as a model failure because the model produced the visible mistake. That diagnosis is too shallow when the task never had a testable completion contract, the context mixed stale and current evidence, the tool surface granted ambiguous authority, or the run was allowed to close on plausibility instead of proof.
The harness should classify the failure before changing the model. A stronger model cannot repair an undefined target, make contradictory state authoritative, or prove a result when the workflow never required evidence. It may hide those defects longer by producing more convincing intermediate work.
Capability becomes the leading hypothesis only after the task, context, tool, state, verification, and recovery contracts are adequate and the trace still shows a reasoning limit. This ordering prevents procurement from becoming a substitute for systems engineering.
Failure taxonomy
- Task contract
- Completion was vague, contradictory, or impossible to test.
- Context contract
- Evidence was missing, stale, excessive, or internally inconsistent.
- Tool contract
- Authority was too narrow to finish or too broad to remain safe.
- State contract
- The run mixed epochs or trusted derived state over the canonical owner.
- Verification
- Confidence or a partial check was mistaken for completion evidence.
- Recovery
- Repeated fixes continued without a circuit breaker or fresh diagnosis.
Evidence and method
mixed: Literature synthesis across public agent benchmarks and field observation from Greyforge harness operation. Public research claims are separated from Greyforge operating doctrine.
Limitations
- The cited evaluations use different environments and do not form one uniform comparator.
- Private traces and proprietary operating details are withheld from the public preview.
- The scorecard is an engineering framework, not a universal measured reliability score.
Access and updates
Purchase includes lifetime read access to this edition, email-based recovery, and revisions published to the same edition.
Public companion: Read the public Harness edition
The full dossier turns the thesis into a working harness model.
Includes the reliability taxonomy and the eight-layer harness architecture.
Includes the failure ledger, scorecard, and trace review cadence.
Includes the model-swap decision rule: when to upgrade, when to repair the harness, and when to stop the run.
Unlock the full edition
The public preview above is complete enough to evaluate the purchase. Checkout unlocks this full record for the checkout email.
4,300 words
current edition
One-time purchase with email recovery. No recurring subscription.
Reliability Is a Harness Property: The Agent Engineering Dossier
Checkout opens lifetime read access to this Chronicle edition.
Use the address that should own and recover this lifetime unlock.
No subscription. No hosted account required.