准确衡量我们模型的能力对于合理的部署和安全决策至关重要，包括 OpenAI 准备框架（在新窗口中打开） 下的决策。每次发布模型时，我们都会报告各种外部和内部基准测试的结果，以跟踪模型进展。当评估存在影响结果的缺陷时，它们可能对能力产生错误理解，歪曲安全案例并影响研究优先级。
我们近期调查了最广泛使用的编码基准之一 SWE-bench Verified，发现其存在根本性设计和污染问题，并认定该评估不再能提供关于软件开发能力的有意义信号。当时，我们鼓励更广泛的社区转向 SWE-Bench Pro。
SWE-Bench Pro（在新窗口中打开） 旨在通过测试模型在更长周期和更现实的编码任务上的表现来改进 SWE-bench Verified，从而更好地追踪智能体编码能力。与 SWE-bench Verified 类似，任务通过程序化方式从一组公共和私有仓库的功能变更历史中获取。模型需要实现一个解决方案，通过新功能测试，同时不破坏现有功能。在包含 731 个任务的公共子集中，前沿模型在八个月内通过率从 23.3% 提升至 80.3%。
此后，我们对 SWE-Bench Pro 进行了类似的审计，使用数据点分析流水线审查数据集。该流水线审查模型对任务的尝试、任务元数据和失败轨迹，以标记可能的评估缺陷。每个被标记的任务随后通过多个调查智能体轮次进行评估，并由五位经验丰富的软件工程师独立审查，分歧部分升级进行进一步调查。
我们发现数据集中相当一部分任务存在严重问题。我们的数据点分析流水线标记了 200 个（27.4%）有问题的任务，而人工标注活动识别出 249 个（34.1%）。
问题主要分为四类：
- 过于严格的测试1 强制实施提示中未指定的具体实现细节，导致许多功能正确的提交无效。
- 未充分指定的提示2 省略了隐藏测试强制执行且无法合理推断的要求。
- 低覆盖率测试 对请求的功能检查不足，因此不完整的修复可能通过。
- 误导性提示 引导模型走向错误行为或与测试要求相矛盾。
我们的发现指出了策划困难但公平的基准的难度，以及智能体在可扩展数据质量检查中日益增长的实用性。鉴于这些结果，我们估计约 30% 的 SWE-bench Pro 任务存在问题，并建议模型开发者仔细检查结果。
方法论
我们的目标是确保任务失败反映真实的模型局限性，而任务成功反映对提示要求的完整且有效的解决方案。为了检查评估中使用的数据质量，我们创建了一个质量保证流水线，以评估每个数据点是否准确反映模型能力。
初始数据质量流水线标记问题以供审查。我们通过对标记任务进行更深入的智能体辅助审计，以及与经验丰富的工程师合作进行人工标注活动来验证。
初始自动化过滤器审查提供给模型的指令、模型解决任务的尝试以及用于评分这些尝试的测试，以标记可能存在问题或有缺陷的示例。该过滤器标记了 286 个可能有问题任务。随后，我们通过两种方式对该子集进行了更深入的审查：人工监督的智能体审查，通过调查智能体进行广泛检查并做出最终人工判断；以及与经验丰富的软件开发人员合作进行的人工标注活动。
人工监督的智能体审查
每个被标记的问题都通过基于 Codex 的调查智能体进行审计，这些智能体可以访问任务仓库和环境。这有助于他们区分合理的任务模糊性（通常可以通过研究附近代码和仓库约定来解决）与真正的未充分指定。智能体可以运行测试、检查仓库中的文件，并调查模型在任务上的尝试及其常见失败模式。在多次独立重复这些更深入的审计后，研究人员审查摘要，做出最终判断，并标记可能的问题。
人工标注活动
与此同时，我们对标记的子集进行了人工标注活动。我们与经验丰富的软件工程师合作，他们在审查任务前接受了基准目标、问题分类和边缘案例的培训。每个任务由五位工程师审查。
审查员在利用流水线分析或记录作为支持背景之前，根据可见的问题陈述、测试用例和真实参考解决方案（称为黄金补丁）形成独立判断。然后，审查员根据具体证据分配标签和严重性评级，并将分歧或低置信度案例升级进行进一步审查。
人工审查员比调查智能体更倾向于将任务标记为有问题。两种审查路径在类别上也存在一些分歧，但在任何被标记的任务中，“无问题”都不是最常见的人工标签。在智能体流水线标记的类别中，审查员的判断在 74% 的案例中重叠。
与智能体流水线相比，人工审查员也更倾向于为任务选择多个标签，表明他们发现任务以多种方式存在问题，或无法干净地归入单一类别。这表明智能体加审查员流水线导致了保守的标签：它捕捉了人类识别的相同广泛失败模式，同时低估了审查员看到额外或重叠问题的案例。最大的差异在于低覆盖率测试，人类将其选为基准中最常见问题的比例为 9.4%，而智能体流水线为 4.1%。
失败模式
在几个案例中，任务提示规定了特定的实现方式，但隐藏测试用例期望不同的行为。
讨论
我们识别出的问题，加上 SWE-bench Verified 中的类似案例，凸显了严格检查基准测试的重要性。开源仓库中的议题和拉取请求最初是为人类协作而创建的，通常通过维护者和贡献者之间漫长的来回沟通。因此，问题描述、合并代码和单元测试并不总能形成清晰、独立的评估任务，以可靠地评估模型。特别是，拉取请求中包含的测试可能过于严格，因为它们是为了验证特定更改而编写的，而非定义一种与实现无关的任务解决标准。
与此同时，与不久之前相比，现在更容易检测到评估缺陷。随着模型能力的提升，我们可以利用这些模型更深入、更一致地检查提示、测试、补丁、追踪和边缘情况，从而帮助发现那些以前大规模查找成本高昂或不切实际的基准测试问题。
我们希望更广泛的评估社区能够开发出由经验丰富的软件开发者专门为测试模型能力而构建的新基准测试。这种方法可以保持我们衡量模型能力所需的高标准与真实性，并在整个过程中实现更好的人工监督。鉴于本次分析中发现的问题，我们撤回之前关于采用 SWE-Bench Pro 的建议。
最终，评估应通过难以被操纵、易于信任且真正反映模型能力或对齐程度的基准测试，提供有意义的信号。由于这些结果会影响 OpenAI 的部署和安全决策，我们追踪的评估必须有效且具有信息价值。
Accurately measuring our models’ capabilities is important for sound deployment and safety decisions, including decisions under OpenAI’s Preparedness Framework(opens in a new window). With each model release, we report results for a variety of external and internal benchmarks to track model progress. When evaluations have flaws that affect results, they can give a false understanding of capabilities, misrepresenting safety cases and affecting research priorities.
We recently investigated how one of the most widely used coding benchmarks, SWE-bench Verified, had fundamental design and contamination issues, and found that the eval no longer provided meaningful signal on software development capabilities. At the time, we encouraged the wider community to switch to SWE-Bench Pro.
SWE-Bench Pro(opens in a new window) was designed to improve on SWE-bench Verified by testing models on longer horizons and more realistic coding tasks to better track agentic coding capabilities. As in SWE-bench Verified, tasks are sourced programmatically from the history of feature changes in a set of public and private repositories. Models are required to implement a solution that passes new tests for a feature, without breaking existing functionality. On the 731-task public split, frontier models improved from a pass rate of 23.3% to 80.3% in eight months.
We’ve since performed a similar audit on SWE-Bench Pro, reviewing the dataset using a datapoint analysis pipeline. The pipeline reviewed model attempts at the task, task metadata, and failure traces to flag likely evaluation flaws. Each flagged task was then assessed through multiple investigator-agent passes and independently reviewed by five experienced software engineers, with disagreements escalated for further investigation.
We find evidence of breaking issues in a significant portion of the dataset. Our datapoint analysis pipeline flagged 200 (27.4%) broken tasks, while the human annotation campaign identified 249 (34.1%).
The issues primarily fell into four categories:
- Overly strict tests1 enforce specific implementation details not specified in the prompt, invalidating many functionally correct submissions.
- Underspecified prompts2 omit requirements that hidden tests enforce and that are not reasonably inferable.
- Low-coverage tests under check the requested feature, so incomplete fixes can pass.
- A misleading prompt points models toward the wrong behavior or contradicts what tests require.
Our findings point to the difficulty of curating hard but fair benchmarks and the growing utility of agents for scalable data quality checks. In light of these results, we estimate that ~30% of SWE-bench Pro tasks are broken, and advise that model developers carefully examine results.
Methodology
Our aim is to ensure that task failures reflect genuine model limitations, and task successes reflect complete and valid solutions to the prompt requirements. To check the quality of the data used in the evaluation, we created a quality assurance pipeline to assess whether each datapoint accurately reflects model capabilities.
An initial data quality pipeline flags problems for review. We validate with a deeper agent-assisted audit of flagged tasks and a human annotation campaign working with experienced engineers.
An initial automated filter reviews the instructions given to the model, attempts by the model to solve the task, and the tests used to grade these attempts to flag likely broken or problematic examples. This filter flagged 286 potentially broken tasks. We then conducted a deeper review of that subset in two ways: a human-supervised agent review, which conducts extensive checks with investigator agents and a final human judgment; and a human annotation campaign working with experienced software developers.
Human-supervised agent review
Each flagged problem is audited with Codex-based investigator agents that were given access to the task repository and environment. This helps them distinguish reasonable task ambiguity, which can often be resolved by studying nearby code and repository conventions, from true underspecification. The agent can run tests, inspect files in the repo, and investigate model attempts and their common failure modes on the task. After several independent repeats of these deeper audits, a researcher reviewed the summaries, made a final judgment, and labeled the likely issues.
Human annotation campaign
In parallel, we ran a human annotation campaign over the flagged subset. We worked with experienced software engineers who were trained on the benchmark goals, issue taxonomy, and edge cases before reviewing tasks. Each task was reviewed by five engineers.
Reviewers formed an independent judgment from the visible problem statement, test cases, and the ground-truth reference solution (known as the gold patch) before using the pipeline analysis or transcript as supporting context. The reviewers then assigned a label and severity rating based on concrete evidence, and escalated disagreements or low-confidence cases for further review.
Human reviewers were more likely than the investigator agents to mark tasks as broken. There was also some disagreement on categories between the two review paths, but in no flagged task was “not broken” the most common human label. Of the categories the agent pipeline flagged, reviewers’ judgments overlapped in 74% of cases.
Compared with the agent pipeline, the human reviewers were also more likely to select multiple labels for a task, indicating that they found tasks to be broken in multiple ways or did not fit cleanly into a single category. This suggests the agent-plus-reviewer pipeline resulted in conservative labeling: it captured the same broad failure modes humans identified, while undercounting cases where reviewers saw additional or overlapping issues. The largest difference was in low-coverage tests, which humans selected as the most common issue for 9.4% of the benchmark compared with 4.1% from the agent pipeline.
Failure modes
In several cases the task prompt prescribed a specific implementation, but the hidden test cases expected different behavior.
Discussion
The issues we have identified, coupled with similar cases in SWE-bench Verified, highlight the importance of rigorously checking benchmarks. Issues and pull requests from open-source repositories were originally created for human collaboration, often through long back-and-forths between maintainers and contributors. As a result, problem descriptions, merged code, and unit tests do not always line up to form clean, isolated tasks for evaluating models reliably. In particular, tests included in pull requests can be overly strict because they are written to validate a specific change, rather than to define an implementation-agnostic standard for solving the task.
At the same time, evaluation flaws are easier to detect now than they would have been even a short time ago. As model capabilities improve, we can use those models to inspect prompts, tests, patches, traces, and edge cases with much greater depth and consistency, helping surface benchmark issues that were previously costly or impractical to find at scale.
We hope the wider evaluation community will develop new benchmarks built by experienced software developers specifically to test model capabilities. That approach can preserve the high bar and realism we want to measure model capabilities, and allows for better human oversight throughout the process. Given the issues uncovered in this analysis, we retract our earlier recommendation to adopt SWE-Bench Pro.
Ultimately, an eval should provide meaningful signal through benchmarks that are hard to game, easy to trust, and genuinely reflective of model capability or alignment. Because these results inform OpenAI’s deployment and safety decisions, the evals we track need to be valid and informative.
本文内容采集自官方网站，排版和翻译可能与原页面存在差异。
阅读官方全文