What would a fair benchmark for agent architecture look like? [D]

I am working on an evaluation design and would appreciate criticism before running it.

Most coding-agent benchmarks collapse the model and its harness into one score. If a run fails, it is difficult to tell whether the cause was model capability, context assembly, task decomposition, tool design, retry policy, or the acceptance gate. A model can also look worse because the harness truncated its output, or look better because the gate only checked for plausible surface markers.

The experiment I am considering crosses two independent variables:

1. Workflow: one monolithic task versus decomposition into bounded slices with explicit contracts and acceptance criteria.

2. Model policy: frontier-only versus cheapest-capable with escalation after a capability-graded failure.

That produces four cells: frontier monolith, routed monolith, frontier decomposed, and routed decomposed. The frontier-decomposed cell seems especially important because it changes the task architecture while holding the model tier fixed.

I would freeze the original tasks, source revisions, available tools, total retry budget, final acceptance criteria, validator versions, and the verifier. Every cell would be judged against the same final delivered outcome rather than against the persuasiveness of the agent's report.

Proposed primary measures are cost per independently accepted change, false acceptance, false rejection, first-pass accepted yield, verification time, and reproducibility across three fresh runs. Token use, latency, escalation count, and context volume would be secondary measures.

The confound I am least satisfied with is budget normalization. Decomposition changes the task distribution and may create more calls, which is part of the architectural treatment, but giving every slice the monolith's full context or retry budget would subsidize the decomposed condition. A shared system-level budget is cleaner, although it may hide which slices actually needed more capacity.

There are no results yet, so I am not claiming that decomposition or routing wins. I am trying to make the comparison falsifiable before seeing any outcomes.

What would you preregister or change? Would you treat decomposition as part of the system being evaluated, or try to isolate it from model quality more aggressively?