Harness Engineering > Model Swaps: What Actually Improves Agent Reliability

The Core Reframe

Teams often treat agent improvement like a model shopping problem. The operator view is different: the model is the engine, the harness is the control system. If control is weak, capability leaks through avoidable failures - early exits, spec drift, loop traps, and wasted budget. We learned this firsthand building the Saga assembly line.

The practical question is not "which model is smartest?" It is "what behaviors does the harness reliably enforce under stress?"

What Moves Reliability

1) Verification-gated completion

Most weak runs die at "looks right." Strong harnesses force proof before completion: explicit spec checks, test execution where available, and a mandatory fix loop if verification fails. This replaces confidence-based exits with evidence-based exits.

2) Deterministic context onboarding

Agents burn time rediscovering environment facts. Injecting directory, tool, and constraint context at start cuts that error surface. Less blind search, fewer avoidable path errors, faster convergence. This is the same principle behind the project-based sharding that cut our latency by 99%.

The same discipline applies to memory writes. Before a model is asked to bless another note, a cheap deterministic gate should be allowed to reject obvious residue. Memory Quality Without an LLM Judge is that argument applied to the write path instead of the run loop.

3) Loop intervention heuristics

Agents can get myopic and repeat local edits against a broken plan. Lightweight loop detection (e.g., repeated edits in the same locus) plus forced reconsideration prompts can recover stalled runs.

4) Reasoning budget scheduling

Uniform max reasoning sounds safe but often hurts throughput under time constraints. A staged policy is usually better: heavier planning, moderate build, heavier final verification. The target is correctness per unit time, not just depth.

Trace-Driven Improvement Loop

Reliability work without traces is mostly superstition. The repeatable loop is:

1. capture traces at scale,
2. classify failures by type,
3. patch harness behavior,
4. rerun and compare distribution shifts.

This turns "I think it got better" into measurable engineering.

Operator Checklist

• Add pre-exit verification gate.
• Inject deterministic environment context at run start.
• Detect repeated-edit loops and trigger plan refresh.
• Adopt staged reasoning budgets by phase.
• Track regressions by failure class, not just final score.

Bottom Line

If your agent fails in production, swapping models can help - but harness quality determines whether model capability translates into repeatable execution. Build stronger control loops first. The ForgeClaw v3 rebuild was our hardest lesson in this principle.

Reference inspiration: public post by @Vtrivedy10 on harness engineering and benchmark gains.