Continue reading · Public introduction
The argument in context
Model capability, task difficulty and the surrounding system jointly affect reliability. Greyforge proposes explicit contracts for context, tools and verification, and recommends evaluating their contribution.
The Model Is Not the Operating System
A failed run can motivate a model change, a harness repair or both. Choosing between them requires a diagnosis and a defined comparison.
Missing context, stale state, ambiguous authority and incomplete verification are candidate failure causes. This edition does not measure their frequency relative to model limitations.
The contract model is Greyforge operating doctrine. It does not establish that harness changes generally outperform model upgrades.
Selected Research and Its Limits
The selected papers evaluate agents through repository tasks, interactive environments, tool interfaces and execution feedback. Their results are bounded by their own protocols; they do not validate one universal harness.
SWE-bench
Real repository issues push evaluation beyond short-form coding tasks.
SWE-agent
Agent-computer interface design changes execution quality.
AgentBench
Long-horizon reasoning and instruction following remain hard under action loops.
AgentDojo
Tool use must be tested for utility and prompt-injection pressure together.
Source-selection method: sources were selected when they expose real repository or computer tasks, explicit tool interfaces, executable scoring, or adversarial tool-use conditions. General model leaderboards and vendor claims were excluded because they do not isolate the harness boundary discussed here.
| Claim | Evidence source | Boundary |
|---|---|---|
| Repository-scale evaluation needs executable environments and testable outcomes. | SWE-bench; SWE-agent | Evidence from benchmark design, not proof that one harness works for every repository. |
| Interface choices affected results within the tested SWE-agent settings. | SWE-agent; ReAct is discussed in the full dossier | Scoped research evidence; Greyforge's contracts are proposals, not model-independent findings. |
| Utility and adversarial pressure should be evaluated together for tool-using agents. | AgentDojo | Supports the evaluation principle, not Greyforge's exact approval policy. |
| Task, context, tool, verification, recovery, and memory contracts form Greyforge's reliability stack. | Greyforge operating doctrine | A position derived from operations, not a controlled external benchmark result. |
Illustrative Trace-Ledger Example
This illustrative trace shows a possible record shape. It is not a retained run or measured result. Keeping the failed check visible alongside its repair is the proposed documentation practice.
| Phase | Recorded event | Gate result |
|---|---|---|
| Intake | Change a parser without altering public output | Owner and completion test recorded |
| Context | Load parser contract and focused fixtures | Unrelated memory excluded |
| Action | Patch one owned module | Write set remains bounded |
| Verification | Focused fixture fails on an edge case | Completion blocked; no success claim |
| Recovery | Revise parse boundary and rerun focused checks | Pass recorded with full gate deferred |
The Public Rule
Greyforge recommends explicit task, context, tool and verification contracts. Whether those controls improve reliability or security requires evaluation in the intended environment.
This is the same doctrine behind Memory Quality Without an LLM Judge: make the cheap boundary deterministic before spending a model call on what a gate could have rejected. It also explains why memory continuity and operations control matter so much. An agent that cannot inherit the right state cannot be trusted to finish the right job.
Editorial review · September 5, 2026. No controlled Greyforge reliability, security, cost or model-swap outcome is reported. The sample is an operating position. Read the correction and source boundary.