可信第三方评估的共享指南

独立、可信的第三方评估在加强安全生态系统中发挥着关键作用⁠。这些评估针对前沿模型进行，为关于关键能力和安全缓解措施的声明提供额外证据。在这篇文章中，我们分享迄今为止学到的经验，并推荐设计能够有效评估前沿模型的方法，希望有助于为这一领域的新兴标准提供参考。

早期，许多评估将模型视为聊天机器人：评估像用户提问一样提示模型，模型回答，然后评估者判断输出。如今的前沿模型能做得更多：它们可以使用工具、跨多个步骤跟踪信息，并在更大的工作流程中行动。这意味着性能不仅取决于模型，还取决于任务发生的环境以及促进其行动的设置。这种周围设置，我们称之为“支架”，可以改变系统性能的关键方面，包括它如何使用工具、跟踪信息或从错误中恢复。

这改变了评估需要如何进行，以及读者应在评估报告中关注什么。我们认为，最有用的报告除了结果本身外，还应明确描述两件事：第一，它们说明评估设置旨在测试什么声明；第二，它们分享评估结果有效的可用证据。

评估中测试的声明通常属于三类之一1：

• 能力激发：模型是否可能产生被评估的能力？
• 防护性能：测试的防护措施对评估的行为或攻击有多稳健？
• 比较：不同模型在同等条件下表现如何？

评估报告还需要解释评估者如何检查可能影响结果有效性的因素。这些因素包括：

• 奖励黑客：利用任务或评分器中的捷径，使系统获得分数而不展示评估旨在衡量的行为。
• 拒绝：以掩盖被测试行为的方式拒绝。
• 污染：因为评估任务、答案或接近的变体出现在训练数据中，或在评估期间可通过浏览等方式发现，导致表现过高。
• 问题缺陷：因为任务无效导致表现不佳。原因可能包括不公平的评分（例如，正确答案需要未说明的实现细节）和无法解决的环境（例如，缺少关键文件或工具不可靠）。
• 沙袋效应：在意识到被评估时故意表现不佳。

为评估选择正确的支架对于获得最佳结果至关重要

我们观察到，对于在较长轨迹上行动的系统，支架的作用尤其重要。当模型可以使用工具、维护状态并跨多个步骤从错误中恢复时，支架可以改变观察到的性能水平，甚至决定被评估的能力是否出现在评估中。例如，一个保留状态并重试失败操作的支架可能让模型完成一个在更简单支架中从未完成的多步骤任务。

在下表中，我们区分了评估者可能想要提出的三类声明，以及我们认为每类声明所需的支架。

评估试图支持的声明合适的支架选择需要报告的证据 在强激发下的能力：当设置旨在发挥系统最强可信性能时，系统A可以完成X类任务。为系统使用最强可信激发设置，包括支架、工具、脚手架和有能力用户合理使用的预算。支架和工具设置、激发指导、允许的预算/努力、令牌/成本/时间，以及为什么该设置是声称能力的可信代理。如果在不同优化设置下比较系统，将其标记为系统间或强激发比较。 受控比较：在共享评估设置下，系统A优于系统B。保持任务、评分和预算固定。使用共享的支架/工具设置，或一组预先选择的标准化支架，为被比较的系统提供合理的最大激发。共享的任务集、工具、评分方法、支架、预算、令牌效率/成本以及已知限制。对于编码代理评估，像Codex CLI这样的开源支架可以为不同系统提供固定的代理循环和工具接口。实现最大激发的理想方法是为每个任务和系统优化定制支架，但目前在实践中不可行。 在激发攻击下的防护稳健性：系统A的防护措施对于相关模型行为或激发攻击是足够的。使用旨在在相关对手模型下激发最强可信攻击的防护测试设置。评估者如何描述相关模型行为、测试的防护配置、激发策略、用于执行的支架以及允许的预算或努力。

**能力声明仅与其背后的激发一样强：评估者需要选择最适合任务和评估试图测量的能力的支架。**标准化支架可能适合在相同条件下比较系统，但当它省略了帮助模型执行任务的特定支架功能时，可能会低估能力。例如，GPT‑5.5在OpenAI网络靶场上的表现展示了支架选择如何实质性地改变需要长时间、多步骤工具使用的任务上的测量能力：当支架使用压缩⁠来在交互变长时保留任务相关上下文时，模型表现更好。这表明对于某些模型，省略压缩的支架会低估性能。

更高的成功率更好。

其他已发表的评估2也显示，提示框架和预算选择会改变评估结果。增加测试时计算量可以显著改变评估所揭示的能力，尤其是在成功易于验证的领域，例如许多网络任务。在英国AISI的网络靶场评估⁠(在新窗口中打开)中，将预算从1000万增加到1亿个token，性能提升了高达59%，并且在测试的最高预算下性能仍在提升。详细说明这一点可以使评估更具可解释性：它向读者展示了结果如何依赖于所测试的提示设置。当性能随着额外预算的增加而持续提升时，得分应被描述为在该提示框架和预算下的表现，而非测量到的能力上限。能力通常依赖于资源，而非一个可以一劳永逸地清晰测量的固定量。当成功可以通过重复尝试来衡量时，报告还应考虑每次成功解决所需的预期成本，而不仅仅是固定token预算下的成功率。这可以使严重性更易于理解：如果重复尝试的成本在相关威胁模型范围内，那么低成功率可能仍然具有实际意义。对于能力声明，可避免的提示不足是一种测量失败：如果提示框架或预算阻止系统展示其本可以产生的行为，那么得分并不能衡量所声称的能力。当评估者已尽可能推动提示，而性能仍在提升时，报告应明确说明这一点，并明确指出结果仅为下限估计。

**安全测试在未考虑攻击者可用的资源（包括自定义提示框架）时，可能会低估攻击能否成功及其严重程度。**在英国AISI的GPT‑5.5网络评估⁠(在新窗口中打开)中，他们的专家红队发现了一种通用越狱方法，能够在OpenAI提供的恶意查询中（包括多轮代理设置）引发违规的网络内容。他们使用Codex创建了一个自定义提示框架来增强模型的攻击性能：它将一个可重复使用的安全绕过模式嵌入到交互中，在轮次和块之间保留该模式，并将其应用于OpenAI提供的恶意网络查询。安全测试应与对手相匹配。如果声明是关于对专家滥用的鲁棒性，那么测试应在定义的预算下评估最强可信的端到端攻击策略，包括任何为保留和重用该策略所需的提示框架。否则，结果可能存在校准错误的风险：它们可能仅支持关于抵抗更简单提示的较窄声明，可能既无法捕捉攻击的严重程度，也无法捕捉一旦提示方法被实施后的成功概率，并且如果给予过多预算，还可能夸大问题的可能性或严重性。

标准化提示框架的比较有其时间和场合，但评估者应明确说明为何使用一致的提示框架集是合适的，以及它能支持何种声明。METR的时间跨度评估⁠(在新窗口中打开)是一个更广泛、适当固定的评估设置的例子：它旨在对其评估的系统产生可比较的结果。METR定义了一个共同的结果，即AI代理在给定可靠性水平下被预测能够成功完成的人类任务的典型持续时间。它应用了一个共享的任务套件、评分方法、拟合方法，以及在每个一起报告的估计批次内的一小组可重复使用的脚手架，例如Triframe和ReAct⁠(在新窗口中打开)。当METR扩展任务套件并将评估基础设施从一个名为Vivaria的框架迁移到另一个名为Inspect的框架时，它报告了这一变化（时间跨度1.1更新⁠(在新窗口中打开)），并在新的评估设置下重新评估了模型。这就是标准化评估设置（包括一致的提示框架集）的价值：它可以使读者确信，得分的差异确实反映了被比较系统之间的差异，而非测量设置的变化。

我们建议第三方评估报告说明其评估设置旨在支持何种声明；描述所测试的内容在多大程度上反映了该更广泛的声明；描述影响结果的提示框架选择；详细说明这些选择在评估之间何时发生变化；并包括支持性证据，以展示结果是如何产生的，以及它在多大程度上可以推广到该声明。

通过检查已知的、可能扭曲结果的危险因素来评估有效性

随着模型能力的增强，评估得分更容易被误解。相对于真实能力，如果模型识别出自己正在被评估并策略性地表现不佳，评估得分可能会被人为降低。如果模型利用了任务、提示、评分器或提示框架中的捷径，得分可能会被夸大。得分也可能因污染（模型在不解决任务的情况下已经知道或能找到答案）或“有缺陷”的问题（如模棱两可、评分错误、无法解决或容易受到意外捷径影响）而失真。因此，评估报告应将主要得分与对这些危险因素的讨论相结合，以便读者能够评估得分是否反映了预期的行为。

提示框架、预算、工具、评分规则、监控器和审查程序都会影响代理是在解决预期任务、回避任务、记忆任务，还是寻找绕过任务的方法。一份可信的报告会使这些检查透明可见：评估者应在每次运行评估时审查样本中是否存在这些行为。

奖励黑客

奖励黑客

奖励黑客指的是以未反映预期能力的方式获得高评估分数。这里的担忧在于，系统通过利用任务、评分器、提示或测试框架来获得分数，而非完成评估旨在衡量的实际工作。METR 对 GPT 5.4 的评估⁠(在新窗口中打开) 展示了这一点为何重要：尽管该模型在任务上的成功率，初步看来相当于约 13 小时的时间跨度，但人工审查显示，其中一些成功源于奖励黑客行为。将结果修正为仅计入没有奖励黑客行为的实例后，估计值降至约 6 小时。评估者应评估此类调整的必要性，并在需要时清晰报告：当读者能够看到哪些表面上的成功被判定无效、为何被判定无效以及结果在多大程度上依赖于该判断时，能力估计会更有用。

拒绝

模型也可能因安全防护措施而在能力评估中表现不佳。模型可能因拒绝执行评估任务而非完成它们，导致评估性能低于其实际能力。因此，报告应说明拒绝行为是否属于评估结果的一部分，如果是，则有多少样本因此受到影响。

污染

污染对于公开或重复使用的基准测试⁠最为重要。如果任务、答案或高度相似的变体出现在训练数据中，或者可以被具备浏览能力的智能体找到，那么性能可能反映的是记忆或检索能力，而非模型的实际性能。评估者应尽可能优先使用私有或新构建的任务，并在求解前测试模型是否会重现任务特定的细节。

有缺陷的问题

有缺陷的问题包括诸如问题意外无法解决等情况。有缺陷的问题相当常见⁠(在新窗口中打开)，应将其视为标准有效性风险。它们包括：错误的真实答案、模糊的提示、缺失的文件、不稳定的服务、不公平的评分标准，以及暴露捷径的环境，例如隐藏的答案文件或包含真实补丁的仓库历史记录。

消极应付

模型或系统的消极应付意味着在出色表现会与某些目标冲突时，策略性地表现不佳。Apollo 的 GPT‑5.5 评估⁠(在新窗口中打开) 通过创建一个模型理解到高评估分数可能触发遗忘或阻止部署的上下文来测试这一点。GPT‑5.5 仍然正确回答，表明几乎没有策略性表现不佳的行为证据3。但他们对其推理轨迹的审查增加了重要的背景信息：Apollo 发现，在消极应付条件下的样本中，有 52% 显示出中等或更高程度的对齐评估意识迹象，而在反事实条件下这一比例为 0%。这一证据使对结果的解读复杂化，但最终为读者提供了更多信号：读者既能看到令人安心的行为结果，也能看到对过度解读该结果保持谨慎的理由。

我们如何支持更严格的评估

强有力的主张需要正确的测试框架来引出行为，以及有效性检查来证明结果是可靠的。我们认为测试框架和有效性检查是评估结果的一部分，这一观点塑造了我们如何在实践中支持第三方评估：

• 我们正在与评估者分享具体的最大引导指南。
• 我们要求能力评估者使用 Codex 作为 OpenAI 模型的通用基准：测试至少应通过用户可能依赖的相同智能体界面运行基线，而不仅仅是通过精简的模型界面。
• 在需要评估欺骗、消极应付或评估意识时，我们也会提供推理轨迹和其他中间产物。自 GPT‑5 以来，METR 和 Apollo 已在 OpenAI 评估中使用此访问权限。
• 最后，我们优先研究以更深入地理解测试框架选择在何时以及如何显著改变结果，涵盖上下文管理、工具访问、重试行为、评分和资源预算等方面。

这对评估标准和未来研究方向意味着什么

这些建议不仅旨在改进单个评估报告，也旨在为新兴的国家⁠(在新窗口中打开)和国际⁠(在新窗口中打开)前沿 AI 评估与报告标准提供信息。展望未来，第三方评估标准应要求提供足够详细的细节，以便决策者理解特定评估支持哪些主张、测试了哪个系统、结果是如何引出的，以及评估者如何检查其有效性。对于在智能体能力至关重要的任务上进行测试的前沿系统，细节应包括（在考虑任何安全或保密问题的情况下）：

• 主张： 评估是比较系统、估计能力上限，还是测试安全防护措施。
• 评估内容： 关于任务或任务分布的足够细节，以便读者理解评估实际测试的技能、行为或故障模式。
• 被测试系统： 模型、推理设置、工具访问、测试框架和安全防护措施。
• 预算： 轮次、令牌、尝试/重试次数、挂钟时间、推理成本，以及适用情况下的每次成功求解的预期成本。
• 引导方法： 用于引出结果的测试框架选择，以及所测试的内容在多大程度上反映了所提出的更广泛主张。
• 有效性检查： 评估者如何寻找奖励黑客、评估意识、污染、拒绝、消极应付以及其他可能削弱结果的行为，包括已确认案例如何影响评分或解读。

忽略测试框架选择或有效性检查的标准可能会低估系统的能力，或高估对安全主张的信心。构建强大的测试框架和引导方法仍然是一个开放的研究领域，应成为进一步调查和投资的焦点。

Independent, trusted third party evaluations play a critical role⁠ in strengthening the safety ecosystem. These evaluations are conducted on frontier models to provide additional evidence for claims about critical capabilities and safety mitigations. In this post, we share lessons we’ve learned so far, and recommend approaches for designing evaluations that can validly assess frontier models that we hope help inform emerging standards in the space.

Earlier, many evaluations treated models like chatbots: the evaluation prompted a model as though it were a user asking a question, the model answered, and an evaluator judged the output. Today’s frontier models can do much more: they can use tools, keep track of information across many steps, and act within a larger workflow. This means that performance depends not only on the model, but also on the environment in which the task takes place, and on the setup that facilitates its actions. This surrounding setup, which we call the “harness,” can change key aspects of the system’s performance, including how it uses tools, keeps track of information, or recovers from mistakes.

This changes how evaluations need to be conducted, and what readers should look for in evaluation reports. In our view, the most useful reports explicitly describe two things beyond the result itself: First, they specify what claim the evaluation setup was designed to test, and second, they share the available evidence that the evaluation result is valid.

Claims tested in evaluations typically fall into one of three buckets1:

• Capability elicitation: Can a model plausibly produce the capability being evaluated?
• Safeguard performance: How robust are the tested safeguards against the behavior or attack being evaluated?
• Comparison: How do different models perform under equivalent conditions?

Evaluation reports also need to explain how evaluators checked for effects that could impact the validity of a result. These include:

• **Reward hacking:**Exploiting shortcuts in the task or scorer, so the system gets credit without demonstrating the behavior the evaluation is meant to measure.
• Refusals: Refusing in ways that obscure the behavior being tested.
• **Contamination:**Overperforming because evaluation tasks, answers, or close variants appeared in training data or were discoverable during the evaluation, such as through browsing.
• **Broken problems:**Underperforming because tasks are invalid. Reasons can include unfair scoring (e.g., correct answer requires unstated implementation details) and unsolvable environments (e.g., missing critical files or unreliable tools).
• Sandbagging: Deliberately underperforming when they show awareness of being evaluated.

Selecting the right harness for an evaluation is crucial for optimal results

We’ve observed that the role of the harness is especially important for systems that act over longer trajectories. When models can use tools, maintain state, and recover from mistakes across many steps, the harness can change the observed level of performance, and even determine whether the capability that’s being assessed appears in the evaluation at all. For example, a harness that preserves state and retries failed actions may let a model finish a multi-step task that the same model never completes in a simpler harness.

In the table below, we separate three kinds of claims evaluators may want to make and the harness we believe each kind of claim requires.

Claim the evaluation is trying to supportAppropriate harness choiceEvidence to report Capability under strong elicitation: System A can complete tasks of type X when the setup is designed to draw out its strongest credible performance.Use the strongest credible elicitation setup for the system, including the harness, tools, scaffolding, and budget a capable user would reasonably use.The harness and tool setup, elicitation guidance, budget/effort allowed, tokens/cost/time, and why the setup is a credible proxy for the claimed capability. If comparing systems under different optimized setups, label it as a system-to-system or strong-elicitation comparison. Controlled comparison: System A outperforms System B under a shared evaluation setup.Keep the tasks, scoring, and budget fixed. Use either a shared harness/tool setup or a fixed set of standardized harnesses chosen up front to provide reasonable max elicitation for the systems being compared.The shared task set, tools, scoring method, harness, budget, token efficiency/cost, and known limitations. For coding-agent evaluations, an open-source harness such as Codex CLI can provide a fixed agent loop and tool interface across systems. The ideal approach for maximum elicitation would be to optimize a bespoke harness for each task and system, but doing so is currently impractical in practice. Safeguard robustness under elicited attack: System A’s safeguards are sufficient for the relevant model behavior or elicited attack.Use a safeguard-testing setup designed to elicit the strongest credible attack under the relevant adversary model.How evaluators characterized the relevant model behavior, the safeguard configuration tested, the elicitation strategy, the harness used to carry it out, and the budget or effort allowed.

**Capability claims are only as strong as the elicitation behind them: evaluators need to choose the harness that best fits the task and the capability the evaluation is trying to measure.**A standardized harness may be right for comparing systems under identical conditions, but it can understate capability when it leaves out specific harness features that help the model perform the task. For example, GPT‑5.5’s performance on OpenAI’s cyber ranges shows how a harness choice can materially change measured capability on tasks that require long, multi-step tool use: the model performs better when the harness uses compaction⁠ to preserve task-relevant context as the interaction gets longer. This demonstrates that for certain models, a harness that omits compaction would under-elicit performance.

Higher success rates are better

Other published evaluations2 also show harness and budget choices changing evaluation results. **Increasing test-time compute can significantly change what capability an evaluation elicits,**especially in domains where success is easy to verify, such as many cyber tasks. In UK AISI’s cyber range evaluation⁠(opens in a new window), increasing the budget from 10M to 100M tokens improved performance by up to 59%, and performance was still increasing at the highest budget tested. Detailing this makes the evaluation more interpretable: it shows readers how the result depends on the tested elicitation setup. When performance is still improving with additional budget, the score should be described as performance under that harness and budget, not as a measured capability ceiling. Capability is often resource-dependent rather than a fixed quantity that can be cleanly measured once and for all. Where success can be measured across repeated attempts, reports should also consider expected cost per successful solve, not just success rate at a fixed token budget. This can make severity easier to interpret: a low success rate may still be practically meaningful if the cost of repeated attempts is within the relevant threat model. For capability claims, avoidable under-elicitation is a measurement failure: if the harness or budget prevents the system from exhibiting behavior it could otherwise produce, the score does not measure the capability being claimed. Where evaluators have pushed elicitation as far as is feasible and performance is still improving, reports should say so clearly and make clear that the result is only a lower-bound estimate.

**Safeguard testing can understate whether an attack can succeed, and how severe it could be, when not accounting for the resources available to attackers, including custom harnesses.**In UK AISI's GPT‑5.5 cyber evaluation⁠(opens in a new window), their expert red teaming found a universal jailbreak that elicited violative cyber content across the malicious queries OpenAI provided, including in multi-turn agentic settings. They used Codex to create a custom harness to strengthen the model’s attack performance: it embedded a reusable safeguard-bypass pattern into the interaction, preserved that pattern across turns and blocks, and applied it across the malicious cyber queries OpenAI provided. Safeguard testing should match the adversary. If the claim is about robustness to expert misuse, the test should evaluate the strongest credible end-to-end attack strategy under a defined budget, including any harness needed to preserve and reuse that strategy. Otherwise, the results risk miscalibration: they could support only a narrower claim about resistance to simpler prompting, could miss both how severe the attack becomes and its probability of success once the elicitation method is operationalized, and could also overstate how likely or severe a problem is if given too much budget.

There is a time and place for standardized harness comparisons, but evaluators should be explicit about why using a consistent set of harnesses is appropriate and what claim it can support.METR's time-horizon evaluation⁠(opens in a new window) is an example of a broader, appropriately fixed evaluation setup: it is designed to produce comparable results across the systems it evaluates. METR defines a common outcome, the typical duration for a human task at which an AI agent is predicted to succeed at a given reliability level. It applies a shared task suite, scoring method, fitting method, and a small set of reusable scaffolds such as Triframe and ReAct⁠(opens in a new window) within each batch of estimates reported together. When METR expanded the task suite and moved evaluation infrastructure from a framework called Vivaria to one called Inspect, it reported the change (Time Horizon 1.1 update⁠(opens in a new window)) and re-evaluated models under the new evaluation setup. That is the value of a standardized evaluation setup, including a consistent harness set: it can make readers confident that a difference in scores really reflects a difference between the systems being compared, rather than a change in the measurement setup.

We recommend that third party evaluation reports state what kind of claim their evaluation setup is meant to support; describe how closely what was tested reflects that broader claim; describe the harness choices that shaped the result; detail when those choices change between evaluations; and include supporting evidence to show how the result was produced and how well it generalizes to the claim.

Assess validity by checking for known hazards that can distort results

As models become more capable, evaluation scores become easier to misinterpret. Relative to real capabilities, evaluation scores can be artificially reduced if a model recognizes it is being evaluated and strategically underperforms. They can be inflated if the model exploits a shortcut in the task, prompt, scorer, or harness. They can also be distorted by contamination (where a model already knows or can find an answer without solving the task) or by “broken” problems that are ambiguous, incorrectly scored, unsolvable, or vulnerable to unintended shortcuts. Evaluation reports should therefore pair headline scores with a discussion of these hazards, so readers can assess whether the scores reflect the intended behavior.

Harnesses, budgets, tools, scoring rules, monitors, and review procedures all affect whether an agent is solving the intended task, avoiding it, memorizing it, or finding a path around it. A trustworthy report makes those checks visible: evaluators should review samples for these behaviors every time an assessment is run.

Reward hacking

Reward hacking means achieving high evaluation scores in ways that do not reflect the intended capability. Here, the concern is that the system gets credit by exploiting the task, scorer, prompt, or harness rather than by doing the work the evaluation was meant to measure. METR’s evaluation of GPT 5.4⁠(opens in a new window) shows why this matters: despite the model succeeding on tasks at a rate that would have registered as a roughly 13-hour time horizon at first pass, human review showed that some of those successes came from reward hacking, and revising the results to account for only those instances without reward hacking lowered the estimate to about 6 hours. Evaluators should assess the need for such adjustments and, when they are needed, report them clearly: a capability estimate is much more useful when readers can see which apparent successes were disqualified, why they were disqualified, and how much the result depends on that judgment.

Refusals

Models can also underperform capability evaluations due to safeguards. A model may have lower evaluation performance than it is capable of due to refusing evaluation tasks instead of completing them. Reports should therefore explain if refusals were part of the evaluation results and if yes, then how many samples were compromised because of them.

Contamination

Contamination matters most for public or reused benchmarks⁠. If tasks, answers, or close variants appear in training data or can be found by an agent with browsing, performance may reflect memorization or retrieval rather than model performance. Evaluators should prefer private or newly constructed tasks where possible and test whether models reproduce task-specific details before solving.

Broken problems

Broken problems include situations such as a problem that accidentally cannot be solved. Broken problems are common enough⁠(opens in a new window) that they should be treated as a standard validity risk. They include incorrect ground truth, ambiguous prompts, missing files, flaky services, unfair scoring criteria, and environments that expose shortcuts such as hidden answer files or repository history containing the ground-truth patch.

Sandbagging

Sandbagging by a model or system means strategically underperforming when strong performance would conflict with some objective. Apollo’s GPT‑5.5 evaluation⁠(opens in a new window) tested this by creating a context where the model understood that high evaluation scores might trigger unlearning or block deployment. GPT‑5.5 still answered correctly, suggesting little behavioral evidence of strategic underperformance3. But their review of reasoning traces added important context: Apollo found moderate-or-higher signs of alignment-evaluation awareness in 52% of sandbagging-condition samples, compared with 0% in counterfactuals. That evidence complicates interpretation of the result, but ultimately gives readers more signal: readers can see both the reassuring behavioral result and the reason to be cautious about overinterpreting it.

How we are supporting stronger evaluations

Strong claims require both the right harness to elicit the behavior and validity checks to show the result is sound. Our view that harnesses and validity checks are part of the evaluation result shapes how we support third party evaluations in practice:

• We are sharing specific maximum-elicitation guidance with evaluators.
• We are asking capability evaluators to use Codex as a common floor for OpenAI models: tests should at least run a baseline through the same agentic interface users are likely to rely on, rather than only through a stripped-down model interface.
• We are also making reasoning traces and other intermediate artifacts available where they are needed to assess deception, sandbagging, or evaluation awareness. METR and Apollo have used this access in OpenAI evaluations since GPT‑5.
• Finally, we are prioritizing research to more deeply understand when and how harness choices materially change results, from context management and tool access to retry behavior, scoring, and resource budgets.

What this means for evaluation standards and future research directions

These recommendations are intended not only to improve individual evaluation reports, but also to inform emerging national ⁠(opens in a new window)and international ⁠(opens in a new window)standards for frontier AI evaluation and reporting. Going forward, third party evaluation standards should require enough detail for decision makers to understand what claims the specific evaluations support, what system was tested, how the result was elicited, and how evaluators checked its validity. For frontier systems being tested on tasks where agentic capabilities matter, details should include (subject to any security or confidentiality concerns):

• **The claim:**whether the evaluation compares systems, estimates a capability ceiling, or tests safeguards.
• **Evaluation content:**enough detail about the tasks or task distribution for readers to understand what skills, behaviors, or failure modes the evaluation is actually testing.
• **The tested system:**the model, reasoning setting, tool access, harness, and safeguards.
• The budget: turns, tokens, attempts/retries, wall-clock time, inference cost, and where applicable expected cost per successful solve.
• **Elicitation methods:**harness choices used to draw out the result, and how closely what was tested reflects the broader claim being made.
• Validity checks: how assessors looked for reward hacking, evaluation awareness, contamination, refusals, sandbagging and other behaviors that could undermine the result, including how confirmed cases affected scoring or interpretation.

Standards that leave out harness choices or validity checks can understate what a system can do or overstate confidence in a safety claim. Building strong harnesses and elicitation methods remains an open research area and should be a focus for further investigation and investment.