GPT‑5.6 Sol 尝试解决 ARC-AGI-3 基准测试中谜题的加速视频，左侧为官方测试框架，右侧为我们的 Responses API 测试框架（保留推理过程并支持压缩）。在_该游戏_（在新窗口打开）排行榜上，没有任何前沿模型能通过第一关之后的关卡。使用我们的框架，GPT‑5.6 Sol 通过了全部六关。
GPT‑5.6 Sol 已解决数学领域长期存在的开放性问题，例如圈双覆盖猜想（在新窗口打开），并通关了《宝可梦 火红》等游戏。但在 ARC-AGI-3（一个 2D 益智游戏基准测试）中，GPT‑5.6 Sol 仅得 7.8%，而 GPT‑5.5 几乎无法进行游戏，得分仅为可怜的 0.4%。
是 2D 益智游戏对我们的模型异常困难吗？还是另有原因？
基准测试很少孤立地衡量 AI 模型。它们同样衡量关于 API 设置、测试框架设计和提示词等不易察觉的选择。在 ARC-AGI-3 的案例中，我们发现启用我们在 ChatGPT 和 Codex 中使用的两项 API 设置——保留推理过程和压缩——在公开任务集上将得分提高了三倍，并将输出 token 减少了 6 倍。
使用官方测试框架，GPT‑5.6 Sol 在 ARC-AGI-3 公开集上得分为 13.3%。启用保留推理过程和压缩后，其得分为 38.3%。得分衡量的是相对人类行动效率（_RHAE_（在新窗口打开））——一种将模型性能与人类基线进行比较的指标。根据_官方游戏日志_（在新窗口打开），我们估计人类测试者的平均得分为 48%。模型不会被告知评分方式，也无法在整个过程中看到自己的分数——行动仅返回每帧的文本表示和当前所在关卡。
ARC-AGI-3
ARC-AGI-3 是一个旨在衡量 AI 智能体学习和推理能力的基准测试。智能体探索不熟悉的 2D 游戏，并在没有明确指令的情况下推断其运作方式。您可以在 arcprize.org/tasks（在新窗口打开） 试玩 25 个演示游戏。
ARC-AGI-3 使用了一个故意设计得通用的测试框架，不包含工具或特殊功能。ARC 的理由是，简单的测试框架能使模型缺陷更明显，并使模型之间的比较更公平。相比之下，商业开发者会针对每个模型的特性和怪癖优化测试框架。
Ethan Mollick_展示_（在新窗口打开）了 Codex 中的 GPT‑5.6 Sol 通关《杀戮尖塔 2》中随机生成的每日挑战，该游戏发布于 GPT‑5.6 Sol 的知识截止日期之后。
但当我们深入探究时，发现模型的许多困惑并非源于模型本身，而是源于测试框架中的设置。
首先，我们注意到每次游戏动作后，所有私有推理过程都被丢弃。这意味着每次动作时，GPT‑5.6 Sol 都被要求重新理解游戏，无法记住之前的思考。模型仍然可以看到过去动作的记录和简短的附带注释，但无法看到导致这些动作的计划、洞察或想法。
其次，我们发现测试框架使用了滚动截断窗口，导致随着历史记录增长，较早的动作变得不可见。因此，GPT‑5.6 Sol 不仅无法记住过去的思考，还在逐渐失去对过去动作的记忆。
测试框架的这两个特性——丢弃推理过程和滚动截断——有助于解释为什么 GPT‑5.6 Sol 难以随时间学习。
智能体在记住所做之事时表现最佳
我们的模型经过训练，会在输出回复或工具调用之前，通过私有推理消息进行思考。这些私有思考消息会作为对话历史的一部分被保留。如果对话变得过长，我们会对其进行总结并继续。
这是我们的模型被训练的方式，也是它们在 ChatGPT 和 Codex 中部署的方式。为了更好地匹配我们的生产环境，我们使用 Responses API（在新窗口打开） 实现了 ARC-AGI-3 测试框架。我们的 API 使上下文管理变得简单：对于 GPT‑5.6，传递之前的响应 ID 会自动在工具调用和轮次之间保留推理过程。
保留推理过程后，我们注意到两个重大变化。首先，GPT‑5.6 Sol 在每次动作前思考的时间减少了，因为它不再需要每轮都从头解释游戏。其次，当它能够记住过去的想法时，GPT‑5.6 Sol 在随时间学习和采用连贯策略方面表现更好。
下一个改进来自用 压缩（在新窗口打开） 替换滚动截断，这是 Responses API 中的另一项设置。
ARC-AGI-3 测试框架通过滚动截断来处理上下文限制。当对话上下文超过 175,000 个字符时，最早的消息会被丢弃。
滚动截断有两个缺点。首先，模型会丢失早期的观察和动作。其次，它在大部分任务中都在更满的上下文窗口中运行，这可能会轻微损害性能。
当我们在 ARC-AGI-3 上启用压缩时，GPT‑5.6 Sol 能够更好地在更长的运行过程中保留对每个游戏所学到的内容，并以更少的输出 token 获得更高的分数。
为了说明保留推理过程和启用压缩的效果，以下动画展示了 GPT‑5.6 Sol 在使用不同测试框架解决一系列 ARC-AGI-3 谜题时，其 175K 上下文窗口的变化情况。
中间两列展示了每个测试框架如何不同地使用模型上下文窗口。凭借对过去更好的记忆，GPT‑5.6 Sol 每次动作思考更少，进展更快。注意：我们的实现使用了 175,000 个 token 的限制而非字符，但这最终非常相似，因为绝大多数文本是动作网格，我们的分词器以 1:1 的比例对其进行分词。
保留推理过程和压缩共同使 GPT‑5.6 Sol（最大）能够以 6 倍更少的输出 token 获得大约 3 倍的分数。
结论与建议
我们希望这些实验能提醒大家，评估很少孤立地衡量模型——它们同样衡量关于 API 设置、测试框架设计和提示词等不易察觉的选择。这并非我们第一次因公开基准测试的低分感到惊讶，然后发现评估运行器使用了丢弃推理消息的通用测试框架。
如果您是希望最大化性能的 API 开发者，我们建议使用与我们产品中部署相同的设置：
- 使用我们的 Responses API，而非旧版 Chat Completions API
- 保留推理过程
- 采用压缩处理
若您正在比较模型，我们建议依赖采用上述设置的评估方案，这些设置最贴近 ChatGPT 和 Codex 中的实际使用场景。
我们感谢 ARC 多年来在 AGI 评估方面的创造性工作，以及他们的分析启发我们对此进行更深入的审视。
A sped-up video of GPT‑5.6 Sol attempting to solve puzzles in the ARC-AGI-3 benchmark, with the official harness (left) and our Responses API harness (right), which retains reasoning and enables compaction. On the leaderboard for_this game_(opens in a new window), no frontier model solves any level beyond the first. With our harness, GPT‑5.6 Sol solves all six.
GPT‑5.6 Sol has solved longstanding open problems in mathematics like the cycle double cover conjecture(opens in a new window) and beaten games like Pokémon FireRed. But on ARC-AGI-3, a benchmark of 2D puzzle games, GPT‑5.6 Sol scored just 7.8%, and GPT‑5.5 could barely play the games at all, scoring a paltry 0.4%.
Were 2D puzzle games unusually difficult for our models? Or was something else going on?
Benchmarks rarely measure AI models in isolation. They also measure less visible choices about API settings, harness design, and prompting. In the case of ARC-AGI-3, we discovered that turning on two API settings we use in ChatGPT and Codex—retained reasoning and compaction—tripled scores and cut output tokens by 6x on the public task set.
With the official harness, GPT‑5.6 Sol scored 13.3% on the ARC-AGI-3 public set. With retained reasoning and compaction, it scored 38.3%. Scores measure Relative Human Action Efficiency (_RHAE_(opens in a new window))—a metric comparing model performance to a human baseline. Based on_official gameplay logs_(opens in a new window), we estimate the average human tester scored 48%. Models are not told how they will be scored, and cannot see their score throughout—actions only return a text representation of each frame and what level they are on.
ARC-AGI-3
ARC-AGI-3 is a benchmark designed to measure how well AI agents learn and reason. Agents explore unfamiliar 2D games and infer how they work without explicit instructions. You can play 25 demo games at arcprize.org/tasks(opens in a new window).
ARC-AGI-3 uses an intentionally generic harness, without tools or special features. ARC’s reasoning was that a simple harness makes model shortcomings more visible and makes model comparisons more fair. Commercial developers, by contrast, optimize harnesses for each model’s features and quirks.
Ethan Mollick_shows_(opens in a new window)GPT‑5.6 Sol in Codex beating a randomized daily challenge in Slay the Spire 2, a game released after GPT‑5.6 Sol’s knowledge cutoff.
But as we looked deeper, we discovered much of the model’s confusion was not inherent to the model itself, but due to settings in the harness.
First, we noticed that after each game action, all private reasoning was discarded. This meant that with each action, GPT‑5.6 Sol was asked to figure out the game anew, unable to remember its past thinking. The model could still see a record of past moves and brief accompanying notes, but it could not see the plans, insights, or thoughts that led to them.
Second, we saw that the harness used a rolling truncation window, causing older actions to become invisible as the history grew. So not only was GPT‑5.6 Sol unable to remember its past thinking, it was losing memory of its past actions too.
Together, these two features of the harness—discarding reasoning and rolling truncation—helped explain why GPT‑5.6 Sol was struggling to learn over time.
Agents do best when they remember what they’ve done
Our models are trained to think with private reasoning messages before they output replies or tool calls. These private thinking messages are retained as part of the conversation history. If a conversation grows too long, we summarize it and continue.
This is how our models are trained, and also how they are deployed in ChatGPT and Codex. To better match our production setup, we implemented the ARC-AGI-3 harness with our Responses API(opens in a new window). Our API makes it easy to manage context: for GPT‑5.6, passing the previous response ID automatically retains reasoning across tool calls and turns.
With reasoning retained, we noticed two big changes. First, GPT‑5.6 Sol spent less time thinking before each action, because it no longer had to interpret the game from scratch every turn. Second, when it was able to remember its past thoughts, GPT‑5.6 Sol was much better at learning over time and employing coherent strategies.
The next improvement came from replacing rolling truncation with compaction(opens in a new window), another setting in the Responses API.
The ARC-AGI-3 harness addresses context limits with rolling truncation. When the conversation context exceeds 175,000 characters, the oldest messages are discarded.
Rolling truncation has two drawbacks. First, the model loses earlier observations and actions. Second, it spends much of the tasks operating with a fuller context window, which can slightly impair performance.
When we enabled compaction on ARC-AGI-3, GPT‑5.6 Sol was better able to preserve what it had learned about each game across longer runs, and achieved a higher score with fewer output tokens.
To illustrate the effect of retaining reasoning and enabling compaction, here’s an animation showing GPT‑5.6 Sol’s 175K context window as it solves a series of ARC-AGI-3 puzzles with each harness.
The two central columns depict how the model context window is used differently by each harness. With better memory of its past, GPT‑5.6 Sol thinks less per action and proceeds much faster. Note: our implementation uses a limit of 175,000 tokens instead of characters, but this ends up being quite similar, as the vast majority of text is action grids which are tokenized at a 1:1 ratio by our tokenizer.
Together, retaining reasoning and compaction allow GPT‑5.6 Sol (max) to achieve roughly 3x the score with 6x fewer output tokens.
Conclusion and recommendations
We hope these experiments serve as a reminder that evals rarely measure models in isolation—they also measure a bundle of less visible choices about API settings, harness design, and prompting. This isn’t the first time we’ve been surprised by low scores on a public benchmark and then discovered that the eval runner was using a generic harness that dropped reasoning messages.
If you’re an API developer trying to maximize performance, we recommend using the same settings that we deploy in our own products:
- Use our Responses API, not our legacy Chat Completions API
- Retain reasoning
- Use compaction
And if you’re comparing models, we recommend relying on evals that use the settings above, which best match real-world use in ChatGPT and Codex.
We are grateful to ARC for their years of creative work on AGI evaluation, and for their analysis that inspired us to take a closer look here.
本文内容采集自官方网站，排版和翻译可能与原页面存在差异。
阅读官方全文