Evaluation & Judgment / NeurIPS Position Paper Track 2026

What a Coding-Agent Benchmark Leaves Out

Three practitioner studies inform a reporting template for the conditions that determine whether a coding agent is useful at work.

From Benchmark to Adoption: Coding Agent Evaluation Needs User-Level Harness

Dasol Hong, Ji Yong Cho, Bumsoo Kang, Moontae Lee

Overview

A test suite can tell a team whether an agent fixed a bug. The team still has to decide whether the agent can see the repository, which edits require approval, how to recover if it takes a wrong turn, and how much time the patch will cost to review. This NeurIPS position paper argues that an evaluation meant to guide adoption should report those conditions alongside the score.

Key Findings

Identify the conditions the score describes

A coding agent combines a model with a system-level harness that supplies tools, state, and execution controls. Figure 1 places that system inside the user's workflow. The organization determines whether the agent may see a repository. The user decides whether it can edit, execute, or act without approval. A reviewer decides whether the result meets the team's standards. A score taken under one set of choices can be hard to interpret when the intended deployment makes different choices.

The paper places the model and system-level harness inside a larger evaluation scope that also includes environment, target users, capabilities, and code quality.
The paper places the model and system-level harness inside a larger evaluation scope that also includes environment, target users, capabilities, and code quality.

The proposed reporting instrument groups those choices into four dimensions. The useful distinction is between disclosure about a configuration and measurement on a task. A checkpoint policy can be disclosed once for a configuration. How many times a user rejects a patch or restarts a failed run varies by task.

The four dimensions separate the work environment, the people involved, the capabilities they delegate, and the quality criteria used to judge outputs.
The four dimensions separate the work environment, the people involved, the capabilities they delegate, and the quality criteria used to judge outputs.
DimensionDisclose for the configurationWhat can be measured or audited
EnvironmentDeployment mode, repository context, team scale, compliance, and legacy dependenciesInspect access scope and deployment settings
Target usersIntended and actual roles, proficiency, and who supervises or accepts changesRecord who evaluated each task or output
CapabilitiesRead, edit, execute, and external actions, plus approval checkpoints and recovery optionsRecord overrides, rejections, and restarts
Code qualityWhich criteria matter most to the deploymentUse tests for functional semantics, ratings for comprehensibility and extensibility, and operational checks where available

Appendix E proposes these reporting slots. Some values are directly observable in logs. Others, such as why a reviewer intervened or how comprehensible a patch is, still require judgments and a suitable instrument.

Three studies expose different sources of deployment friction

The position draws on three qualitative studies that examine adoption, oversight, and acceptance in different ways.

StudyEvidence collectedWhat it clarifies
1. Adoption environmentSurvey of 36 US-based IT professionals who used AI coding toolsPractitioners describe where they can share context and which kinds of work they avoid delegating
2. Agent execution and controlSpeculative-design interviews with seven software engineers and AI researchersPractitioners describe the oversight and recovery they would want during agentic work
3. Code acceptanceSeven interviews and a focus group of six additional practitioners across five enterprises, yielding 94 responsesReviewers describe what makes a change acceptable in a production workflow

In Study 1, non-sensitive code, accessible documentation, and familiar stacks make assistance easier to use. Proprietary code, strict compliance rules, sparse specifications, legacy frameworks, and difficult debugging can make the same tools poor fits. A laboratory score measured with full repository access cannot tell a team whether its own restricted context is sufficient.

In Study 2, people want to inspect the agent's plan and sources, correct it partway through, and resume after an error without reconstructing all the context. Their workplaces offered limited access to agentic tools at the time. The evidence identifies anticipated requirements rather than measured productivity improvements. A recovery mechanism can be available in the product while unusable in a particular workflow if the configuration fails to retain state.

In Study 3, 11 of 13 participants treated consistency with conventions and transparent business logic as preconditions for trust. Their verification priorities shifted as code moved from drafting to review and deployment. Compile errors can be easy to fix early. Legacy compatibility, resource limits, and service configuration can decide whether otherwise correct code is safe to merge. This explains why counts of successful benchmark tasks leave out work that the eventual deployer must still do.

Read a benchmark result next to a deployment record

Table 2 applies a subset of the reporting instrument to SWE-bench Verified and an illustrative checkout-service workflow. It exposes missing conditions in the comparison without reporting a performance difference between the two.

Reporting slotSWE-bench VerifiedIllustrative checkout service
Compliance levelNone specified as a requirement of the benchmarkLevel unstated. The scenario permits company-wide agent use
Agent autonomyDepends on the submitted scaffoldPlan with approval
Checkpoint placementDepends on the submitted scaffoldPer edit
Highest code-quality prioritiesNo priorities specified in those slotsComprehensibility and operational fitness

The rows also show why the paper separates environment from capability. Policy can permit an agent to use a repository while leaving the agent configured for read access only. Likewise, two teams can run the same scaffold while applying different standards to the resulting patch. Reporting both levels makes those differences inspectable.

Translate the reporting proposal into a testable comparison

Appendix E asks whether resolution rates and model rankings change when a harness condition varies while the model and tasks stay fixed. One way to formalize that proposed comparison is to hold the model MM, tasks Q\mathcal{Q}, and acceptance rubric fixed, then compare two configurations h1h_1 and h2h_2.

ΔR=R(M,h2;Q)−R(M,h1;Q),R(M,h;Q)=#{accepted task outputs under h}∣Q∣.\Delta R = R(M,h_2;\mathcal{Q})-R(M,h_1;\mathcal{Q}), \qquad R(M,h;\mathcal Q)=\frac{\#\{\text{accepted task outputs under }h\}}{|\mathcal Q|}.

This is an illustrative operationalization of the paper's proposal. It is not an equation fit to the three studies. A comparison might change context access or approval checkpoints, record failures and interventions for every task, and measure human review and recovery time alongside the acceptance rate. Restricting time measurements to successful patches would hide the cost of failures and could make a setup look easier to adopt than it is.

The paper's evidence is strongest as a case for reporting these conditions jointly. Its studies concern professional software work, and the instrument's proposed categories and example configurations still need validation across teams, tasks, and settings.

Questions & Answers

Q: What would a benchmark author actually add?

A: Report which users and environments the result represents, what they allow the agent to do, when they can intervene, and how they evaluate the resulting code. Appendix E provides a template and examples.

Q: Do the studies show that the template improves adoption?

A: They identify factors practitioners say matter and ground the design of the template. Testing whether the reporting changes deployment decisions or improves adoption is a separate experiment.

Q: Why distinguish code quality from capability?

A: Capability describes what the agent may do under the chosen configuration. Code quality describes what the resulting change has to satisfy before users accept it. Allowing autonomous edits and requiring highly maintainable code are two different decisions.

Future Ideas

Test whether harness conditions change the ranking

Run the same agent and task across several explicitly reported harness settings, varying context access, approval checkpoints, or editing permissions while holding the model and total budget fixed. Ask reviewers from the relevant teams to judge whether they would merge the changes.

Record task completion, review time, recovery effort, and acceptance under each setting. Replicate across teams with different development standards to find which benchmark conclusions transfer and which depend on local conditions.

Sources

Original abstract

As coding agents become increasingly adopted in real-world software development, their evaluation remains largely benchmark-driven and model- or system-centric. Existing evaluations emphasize correctness and task completion, offering limited insight into their utility in deployment. This creates a gap between how coding agents are evaluated and whether and how they are adopted in practice. In real workflows, users make coding agents usable by adapting and constraining their behavior to fit specific tasks and contexts, yet these practices often remain informal and ad hoc. We introduce the user-level harness as a reporting abstraction that captures these practices, including task scope, context, constraints, checkpoints, and workflow adaptations, to specify the conditions under which evaluation results should be interpreted. To motivate key considerations for a user-level harness, we conducted three studies with software practitioners examining contextual con-ditions shaping adoption, expected execution and control properties of agentic systems, and code-quality criteria beyond correctness. Synthesizing these findings with prior literature, we propose four interrelated dimensions of the user-level harness: environment, target users, capabilities, and code quality. We argue that reporting user-level harnesses across these four dimensions alongside coding-agent evaluation results enables more contextually grounded interpretations of agent per-formance, complementing existing benchmark and system-level evaluations. Our studies provide empirical grounding for this conceptual framework. We call on the community to adopt, evaluate, and extend this reporting abstraction across diverse user populations and settings beyond the professional and enterprise contexts.