Overview
A test suite can tell a team whether an agent fixed a bug. The team still has to decide whether the agent can see the repository, which edits require approval, how to recover if it takes a wrong turn, and how much time the patch will cost to review. This NeurIPS position paper argues that an evaluation meant to guide adoption should report those conditions alongside the score.
Key Findings
Identify the conditions the score describes
A coding agent combines a model with a system-level harness that supplies tools, state, and execution controls. Figure 1 places that system inside the user's workflow. The organization determines whether the agent may see a repository. The user decides whether it can edit, execute, or act without approval. A reviewer decides whether the result meets the team's standards. A score taken under one set of choices can be hard to interpret when the intended deployment makes different choices.
The proposed reporting instrument groups those choices into four dimensions. The useful distinction is between disclosure about a configuration and measurement on a task. A checkpoint policy can be disclosed once for a configuration. How many times a user rejects a patch or restarts a failed run varies by task.
| Dimension | Disclose for the configuration | What can be measured or audited |
|---|---|---|
| Environment | Deployment mode, repository context, team scale, compliance, and legacy dependencies | Inspect access scope and deployment settings |
| Target users | Intended and actual roles, proficiency, and who supervises or accepts changes | Record who evaluated each task or output |
| Capabilities | Read, edit, execute, and external actions, plus approval checkpoints and recovery options | Record overrides, rejections, and restarts |
| Code quality | Which criteria matter most to the deployment | Use tests for functional semantics, ratings for comprehensibility and extensibility, and operational checks where available |
Appendix E proposes these reporting slots. Some values are directly observable in logs. Others, such as why a reviewer intervened or how comprehensible a patch is, still require judgments and a suitable instrument.
Three studies expose different sources of deployment friction
The position draws on three qualitative studies that examine adoption, oversight, and acceptance in different ways.
| Study | Evidence collected | What it clarifies |
|---|---|---|
| 1. Adoption environment | Survey of 36 US-based IT professionals who used AI coding tools | Practitioners describe where they can share context and which kinds of work they avoid delegating |
| 2. Agent execution and control | Speculative-design interviews with seven software engineers and AI researchers | Practitioners describe the oversight and recovery they would want during agentic work |
| 3. Code acceptance | Seven interviews and a focus group of six additional practitioners across five enterprises, yielding 94 responses | Reviewers describe what makes a change acceptable in a production workflow |
In Study 1, non-sensitive code, accessible documentation, and familiar stacks make assistance easier to use. Proprietary code, strict compliance rules, sparse specifications, legacy frameworks, and difficult debugging can make the same tools poor fits. A laboratory score measured with full repository access cannot tell a team whether its own restricted context is sufficient.
In Study 2, people want to inspect the agent's plan and sources, correct it partway through, and resume after an error without reconstructing all the context. Their workplaces offered limited access to agentic tools at the time. The evidence identifies anticipated requirements rather than measured productivity improvements. A recovery mechanism can be available in the product while unusable in a particular workflow if the configuration fails to retain state.
In Study 3, 11 of 13 participants treated consistency with conventions and transparent business logic as preconditions for trust. Their verification priorities shifted as code moved from drafting to review and deployment. Compile errors can be easy to fix early. Legacy compatibility, resource limits, and service configuration can decide whether otherwise correct code is safe to merge. This explains why counts of successful benchmark tasks leave out work that the eventual deployer must still do.
Read a benchmark result next to a deployment record
Table 2 applies a subset of the reporting instrument to SWE-bench Verified and an illustrative checkout-service workflow. It exposes missing conditions in the comparison without reporting a performance difference between the two.
| Reporting slot | SWE-bench Verified | Illustrative checkout service |
|---|---|---|
| Compliance level | None specified as a requirement of the benchmark | Level unstated. The scenario permits company-wide agent use |
| Agent autonomy | Depends on the submitted scaffold | Plan with approval |
| Checkpoint placement | Depends on the submitted scaffold | Per edit |
| Highest code-quality priorities | No priorities specified in those slots | Comprehensibility and operational fitness |
The rows also show why the paper separates environment from capability. Policy can permit an agent to use a repository while leaving the agent configured for read access only. Likewise, two teams can run the same scaffold while applying different standards to the resulting patch. Reporting both levels makes those differences inspectable.
Translate the reporting proposal into a testable comparison
Appendix E asks whether resolution rates and model rankings change when a harness condition varies while the model and tasks stay fixed. One way to formalize that proposed comparison is to hold the model , tasks , and acceptance rubric fixed, then compare two configurations and .
This is an illustrative operationalization of the paper's proposal. It is not an equation fit to the three studies. A comparison might change context access or approval checkpoints, record failures and interventions for every task, and measure human review and recovery time alongside the acceptance rate. Restricting time measurements to successful patches would hide the cost of failures and could make a setup look easier to adopt than it is.
The paper's evidence is strongest as a case for reporting these conditions jointly. Its studies concern professional software work, and the instrument's proposed categories and example configurations still need validation across teams, tasks, and settings.
Questions & Answers
Q: What would a benchmark author actually add?
A: Report which users and environments the result represents, what they allow the agent to do, when they can intervene, and how they evaluate the resulting code. Appendix E provides a template and examples.
Q: Do the studies show that the template improves adoption?
A: They identify factors practitioners say matter and ground the design of the template. Testing whether the reporting changes deployment decisions or improves adoption is a separate experiment.
Q: Why distinguish code quality from capability?
A: Capability describes what the agent may do under the chosen configuration. Code quality describes what the resulting change has to satisfy before users accept it. Allowing autonomous edits and requiring highly maintainable code are two different decisions.
Future Ideas
Test whether harness conditions change the ranking
Run the same agent and task across several explicitly reported harness settings, varying context access, approval checkpoints, or editing permissions while holding the model and total budget fixed. Ask reviewers from the relevant teams to judge whether they would merge the changes.
Record task completion, review time, recovery effort, and acceptance under each setting. Replicate across teams with different development standards to find which benchmark conclusions transfer and which depend on local conditions.
Sources
- Author-supplied NeurIPS 2026 Position Paper Track manuscript, Figures 1–2, Tables 1–2, Sections 2–7, and Appendix E.
- NeurIPS 2026 accepted-paper index
- NeurIPS 2026 Position Paper Track
Original abstract
As coding agents become increasingly adopted in real-world software development, their evaluation remains largely benchmark-driven and model- or system-centric. Existing evaluations emphasize correctness and task completion, offering limited insight into their utility in deployment. This creates a gap between how coding agents are evaluated and whether and how they are adopted in practice. In real workflows, users make coding agents usable by adapting and constraining their behavior to fit specific tasks and contexts, yet these practices often remain informal and ad hoc. We introduce the user-level harness as a reporting abstraction that captures these practices, including task scope, context, constraints, checkpoints, and workflow adaptations, to specify the conditions under which evaluation results should be interpreted. To motivate key considerations for a user-level harness, we conducted three studies with software practitioners examining contextual con-ditions shaping adoption, expected execution and control properties of agentic systems, and code-quality criteria beyond correctness. Synthesizing these findings with prior literature, we propose four interrelated dimensions of the user-level harness: environment, target users, capabilities, and code quality. We argue that reporting user-level harnesses across these four dimensions alongside coding-agent evaluation results enables more contextually grounded interpretations of agent per-formance, complementing existing benchmark and system-level evaluations. Our studies provide empirical grounding for this conceptual framework. We call on the community to adopt, evaluate, and extend this reporting abstraction across diverse user populations and settings beyond the professional and enterprise contexts.