摘要
问题
- 红队测试对于发现漏洞和提升模型鲁棒性至关重要。然而，当前方法难以规模化，形成了瓶颈。
- 我们最新模型已普遍饱和于常用的鲁棒性评估。
- 我们需要开发能让安全性与对齐能力随模型能力同步扩展的方法。
我们的工作
- 我们训练了 GPT‑Red，一种自动化红队测试模型，可规模化发现漏洞，以便在广泛部署前修复它们。
- GPT‑Red 是一个强大的红队测试器，我们之前的模型极易受到其提示注入攻击。
- 我们使用 GPT‑Red 对 GPT‑5.6 进行对抗性训练，使其对提示注入的鲁棒性大幅提升。
- 我们将继续结合人工与第三方红队测试、分层防护措施和实时监控来扩展此方法。
AI 系统通常通过浏览器、连接的应用、本地文件和其他工具接触第三方数据。这些功能对于执行现实任务必不可少，但也为恶意行为者影响模型行为创造了更多机会。例如，第三方可能在电子邮件、网页、工具响应或代码仓库中嵌入精心设计的指令——旨在诱骗模型将敏感数据上传至外部服务器。
人工红队测试是我们安全工作的关键部分，帮助我们在部署前发现这些漏洞并设置适当的防护措施。但仅靠人工红队测试难以规模化。设计和执行这些测试耗时巨大，限制了我们快速识别新故障模式并将其纳入更强防护措施的能力。此外，虽然这些测试能产生有价值的成功攻击案例，但无法生成训练所需的数量和多样性对抗数据，以通过训练提升模型鲁棒性。
要跟上能力日益增强的模型，红队测试也必须同步扩展。为此，我们一直在训练仅限内部使用的自动化红队测试模型，以便在部署前发现漏洞，并在模型训练期间生成攻击以提升鲁棒性。我们相信，自动化红队测试解锁了一种关键的安全自我改进形式：利用今天的模型直接帮助未来的模型变得更安全。
GPT‑Red 是这些努力的成果，也是我们当前最佳的自动化安全红队测试模型。类似于人工红队测试者设计攻击的方式，该模型通过发送提示、观察 GPT 模型如何响应并迭代来达成目标。我们在 OpenAI 部分最大规模的后训练计算资源上训练了 GPT‑Red——这是前所未有的专门用于提升安全的计算量。
我们直接将 GPT‑Red 纳入生产模型的训练过程。因此，GPT‑5.6 Sol 是我们迄今为止对提示注入最鲁棒的模型，在最具挑战性的直接提示注入基准测试中，故障率比仅四个月前的最佳生产模型降低了 6 倍。我们方法的可扩展性让我们对未来更强大的结果充满期待，因为我们将继续训练更强的红队测试模型。
提示注入对话示例
通过自我对弈训练 GPT‑Red
GPT‑Red 使用自我对弈强化学习进行训练，其中模型和一组多样化的防御者大语言模型在广泛的红队测试场景上同时训练。GPT‑Red 因引发有效故障（如成功的提示注入）而获得奖励，而防御者模型则因抵抗攻击并完成原始任务而获得奖励。随着防御者变得更强，GPT‑Red 被迫发现更强大、更多样化的攻击。
为支持自我对弈训练，我们构建了一套广泛的现实场景，其中可能插入提示注入。每个环境都有一个威胁模型，指定 GPT‑Red 可以控制什么以及什么算作成功攻击。例如，GPT‑Red 可能控制本地文件的一部分、网页横幅、电子邮件正文或工具输出。
训练结束时，GPT‑Red 是一个非常强大的攻击者：它可以攻破几乎所有与之对抗的模型，包括内部模型和最高至 GPT‑5.5 的生产模型。GPT‑Red 完成训练后，我们用它为 GPT‑5.6 的训练生成提示注入，使该模型对 GPT‑Red 的攻击具有高度抵抗力。
我们将 GPT‑Red 与部署的模型分开。这确保了我们专门训练到 GPT‑Red 中的恶意能力不会落入对抗行为者手中，同时将鲁棒性注入我们的生产模型。
GPT‑Red 有多强？
GPT‑Red 对训练时使用的防御者模型群体和红队测试场景非常有效。我们还评估了该模型作为通用红队测试代理是否有助于 OpenAI 整体安全。为此，我们在新的安全环境和目标模型上测试了 GPT‑Red 的有效性。
我们首先使用 Dziemian 等人 (2025) 的间接提示注入竞技场(在新窗口中打开) 的复制版本来评估 GPT‑Red 泛化到新红队测试场景的能力。在此挑战中，人工红队测试者和 GPT‑Red 分别针对一组预定义环境对 GPT‑5.1 提出攻击。这些红队测试场景和目标与训练 GPT‑Red 时使用的不同。GPT‑Red 实现了显著更高的攻击成功率，在 84% 的场景中成功，而人类仅为 13%。
GPT‑Red 作为自动化红队测试器表现出色。在 Dziemian 等人 (2025) 的间接提示注入竞技场中，使用内部镜像，GPT‑Red 能够在比人工红队测试者多得多的场景中针对 GPT‑5.1 生成成功攻击。
现实红队测试案例研究
红队测试器的最终考验是在对系统底层模型和工具设计了解不完整的情况下，针对现实世界代理系统实现特定恶意目标的能力。我们在该设置中的第一个实验将 GPT‑Red 与 OpenAI 办公室中由 Andon Labs 生产的 AI 驱动自动售货机（类似于 Project Vend(在新窗口中打开)）进行对抗。我们向 GPT‑Red 提供了系统描述，并使其能够发送攻击并观察模拟代理的工具调用，该模拟代理紧密模仿真实世界部署。在迭代攻击后，GPT‑Red 对生产代理部署了攻击，实现了其所有三个恶意目标：
- 将一件昂贵库存商品的价格改为最低允许价格 0.50 美元；
- 订购一件价值 100 美元以上的新商品并以 0.50 美元出售；以及
- 取消另一位客户的订单。
我们已披露这些漏洞，新的安全措施正在积极测试中。
GPT‑Red 能够攻破一个名为 Vendy 的实时自主代理（该代理管理自动售货机）。GPT‑Red 首先在模拟环境中进行测试，随后将攻击转移至实时代理，成功使生产代理将昂贵商品改为低价，并取消其他客户的订单。
我们还使用 GPT‑Red 攻击了一个基于 GPT‑5.4 mini 的 Codex CLI 代理，针对一套包含 10 个保留数据外泄场景的测试集。我们将该模型与基于提示的 GPT‑5.5 基线进行对比，以研究训练流程对保留红队测试性能的影响。GPT‑Red 不仅更有效（能在更多场景中成功诱使代理外泄敏感数据），而且 token 效率更高。
GPT‑Red 在攻破实时 Codex 代理方面更高效且更有效。我们针对一个由 GPT‑5.4 Mini 驱动的 Codex 代理，在自定义的 10 项数据外泄任务套件上进行了测试。
通过 GPT‑Red 提升鲁棒性
GPT‑Red 的最终目标是提升我们模型的鲁棒性。过去六个月中，我们通过增加计算量，逐步训练了越来越强的红队测试模型（GPT‑Red 的前身），并自 GPT‑5.3 起，将这些模型用于每个后续生产模型的训练。随着时间的推移，每个后续 GPT 版本的鲁棒性都得到了提升。
例如，GPT‑Red 的早期版本发现了一类新型直接提示注入攻击，称为“虚假思维链”攻击。这些攻击在 GPT‑5.1 上的成功率高达 95% 以上，但在 GPT‑5.6 Sol 上已降至 10% 以下。同样，我们针对开发者工具和浏览器的间接提示注入基准测试中，最新模型已接近饱和（准确率 >97%）。
对 GPT‑Red 自身的鲁棒性也显著提升。在广泛的鲁棒性环境中，GPT‑Red 的攻击成功率随时间单调下降。在我们最新发布的模型 GPT‑5.6 Sol 中，GPT‑Red 的直接提示注入仅有 0.05% 失败。
随着我们持续扩展提示注入的自对弈训练，我们发现了能够攻破现有模型的新威胁。然而，我们的扩展也显著提升了针对这些攻击的鲁棒性。攻击成功率计算为 GPT‑Red 在保留环境中所有尝试的平均成功次数。
保持高能力的同时具备鲁棒性
模型可以通过拒绝更多请求或降低能力来显得更安全。一个做得更少的模型自然更难被攻击，但这并非有用的鲁棒性。
我们全面评估了通用前沿能力以及我们设计的针对性过度拒绝任务。我们发现，所有正常能力均未受影响，同时鲁棒性显著提升。这表明鲁棒性的提升源于对恶意指令的更好抵抗，而非不当使用工具或默认拒绝合法请求。
下一步计划
AI 代理已被用于提升我们下一代模型的能力。我们相信，通过 GPT‑Red，我们已开始解锁类似的安全飞轮：今天的模型可用于使明天的模型更鲁棒、更对齐、更值得信赖。我们将继续扩展计算和数据规模，同时进行算法改进，以训练比当前模型更强的未来版本 GPT‑Red。而这些模型将反过来帮助未来的 GPT 版本更安全。
我们将在本周晚些时候发布一份预印本，提供更多细节。
Summary
Problem
Red-teaming is essential to discovering vulnerabilities and improving the robustness of our models. However, current approaches are not scalable, creating a bottleneck.
Commonly used robustness evaluations have already been saturated by our latest models.
We need to develop methods that allow safety and alignment to scale alongside model capabilities.
What we did
We trained GPT‑Red, an automated red-teaming model that scales our ability to find vulnerabilities so we can fix them before wider deployment.
GPT‑Red is a strong red-teamer, and our previous models are highly vulnerable to its prompt injection attacks.
We use GPT‑Red to adversarially train GPT‑5.6, making it much more robust to prompt injections.
We will continue to scale this approach alongside human and third-party red-teaming, layered safeguards, and real-time monitoring.
AI systems commonly encounter third-party data through browsers, connected apps, local files, and other tools. These affordances are necessary for performing real-world tasks, but they also create more opportunities for malicious actors to influence model behavior. For example, a third party might embed a carefully crafted instruction—designed to trick the model into uploading sensitive data to an external server—in an email, webpage, tool response, or code repository.
Human red-teaming is a critical part of our safety work, helping us uncover these vulnerabilities before deployment and put the right safeguards in place. But human red-teaming alone is difficult to scale. Designing and running these exercises is time-intensive, limiting how quickly we can identify new failure modes and incorporate them into stronger safeguards. Further, while these exercises produce valuable examples of successful attacks, they cannot generate the volume and diversity of adversarial data needed to improve model robustness through training.
Keeping pace with increasingly capable models requires red-teaming to scale as well. To this end, we have been training automated, internal-only red-teaming models that uncover vulnerabilities before deployment and generate attacks during model training to improve robustness. We believe automated red-teaming unlocks a crucial form of self-improvement for safety: using today’s models to directly help make future models safer.
GPT‑Red is the culmination of these efforts and our current best automated safety red-teaming model. Similar to how human red-teamers craft attacks, the model works toward a goal by sending a prompt, observing how GPT models respond to it, and iterating. We trained GPT‑Red at the compute scale of some of our largest post-training runs at OpenAI—an unprecedented amount of compute dedicated purely for improving safety.
We directly incorporate GPT‑Red into the training process of our production models. As a result, GPT‑5.6 Sol is our most robust model to prompt injections to date, achieving 6x fewer failures on our hardest direct prompt injection benchmark compared to our best production model from just four months earlier. The scalability of our approach leaves us excited for even stronger results in the future as we continue to train stronger red-teamers.
Sample prompt-injected conversations
Training GPT‑Red through self-play
GPT‑Red is trained using self-play reinforcement learning, where the model and a collection of diverse defender LLMs are trained simultaneously on a broad set of red-teaming scenarios. GPT‑Red is rewarded for eliciting a valid failure, such as a successful prompt injection, while the defender models are rewarded for resisting the attack and completing their original tasks. As the defenders become more robust, GPT‑Red is forced to discover stronger and more diverse attacks.
To support self-play training, we build an expansive set of realistic scenarios where prompt injections might be inserted. Each environment has a threat model that specifies what GPT‑Red can control and what counts as a successful attack. For instance, GPT‑Red might control part of a local file, a webpage banner, an email body, or output of a tool.
At the end of its training, GPT‑Red is a very strong attacker: it can break nearly all models it is pitted against, both internal and production models up to and including GPT‑5.5. After GPT‑Red completed training, we used it to generate prompt injections for the training of GPT‑5.6, resulting in the model becoming highly resistant to GPT‑Red’s attacks.
We keep GPT‑Red separate from the models we deploy. This keeps the malicious capabilities we specifically train into GPT‑Red out of the hands of adversarial actors, while instilling robustness into our production models.
How strong is GPT‑Red?
GPT‑Red is highly effective against the population of defender models and red-teaming scenarios that it was trained on. We also evaluate whether the model is useful as a general-purpose red-teaming agent to benefit safety broadly at OpenAI. To do so, we test GPT‑Red’s effectiveness on novel safety environments and target models.
We first evaluate GPT‑Red’s ability to generalize to novel red-teaming scenarios using a replicated version of the indirect prompt injection arena from Dziemian et al. (2025)(opens in a new window). In this challenge, both human red-teamers and GPT‑Red independently proposed attacks against GPT‑5.1 on a set of pre-specified environments. These red-teaming scenarios and goals are distinct from those used to train GPT‑Red. GPT‑Red achieves significantly higher attack success rates, finding success on 84% of scenarios compared to 13% for humans.
GPT‑Red excels as an automated red-teamer. GPT‑Red is able to generate successful attacks against GPT‑5.1 in substantially more scenarios than human red-teamers on the indirect prompt injection arena from Dziemian et al. (2025) using an internal mirror.
Realistic red-teaming case studies
The ultimate test of a red-teamer is the ability to achieve targeted malicious goals against real-world agentic systems with incomplete knowledge of the system’s underlying model and harness design. Our first experiment in this setting pitted GPT‑Red against an AI-powered vending machine in the OpenAI office (similar to Project Vend(opens in a new window)) produced by Andon Labs. We gave GPT‑Red a description of the system and the ability to send attacks and observe tool calls from the simulated agent that closely mirrors the real world deployment. After iterating on attacks, GPT‑Red deployed its attack against the production agent, achieving all three of its malicious objectives:
- Change the price of an expensive in-stock item to the minimum allowed price of $0.50;
- Order a new $100+ item and offer it for $0.50; and
- Cancel another customer’s order.
We disclosed these vulnerabilities and new safeguards are actively being tested.
GPT‑Red is able to break a live autonomous agent known as Vendy that manages a vending machine. GPT‑Red was able to cause the production agent to change expensive items to be cheap and cancel other customer orders, by first testing in simulation and then transferring attacks to the live agent.
We also use GPT‑Red to attack a Codex CLI agent (based on GPT‑5.4 mini) on a suite of 10 held-out data-exfiltration scenarios. We compare the model to a prompted GPT‑5.5 baseline to study the impact of our training procedure on held-out red-teaming performance. GPT‑Red is both more effective, in that it can successfully get the agent to exfiltrate sensitive data in more scenarios, and is more token efficient.
GPT‑Red is more effective and efficient at breaking live codex agents. We test against a codex agent backed by GPT‑5.4 Mini on a custom suite of 10 data exfiltration tasks.
Improving robustness with GPT‑Red
The ultimate goal of GPT‑Red is to improve the robustness of our models. Over the last six months, we’ve trained progressively stronger red-teaming models (precursors to GPT‑Red) with increasing compute, and used these models in the training of each successive production model since GPT‑5.3. Over time, each subsequent GPT release has gotten more robust.
As one example, an early version of GPT‑Red found a novel class of direct prompt injection attacks known as “Fake Chain-of-Thought” attacks. These attacks achieved success rates of upwards of 95% on GPT‑5.1 but are now below 10% for GPT‑5.6 Sol. Similarly, several of our indirect prompt injection benchmarks that target attacks in developer tools and browsing have been saturated by our latest model (>97% accuracy).
Robustness to GPT‑Red itself has also improved substantially. On a broad set of robustness environments, GPT‑Red’s attack success rates have dropped monotonically over time. With our latest model release, GPT‑5.6 Sol fails on only 0.05% of GPT‑Red’s direct prompt injections.
As we have continued to scale up self-play training for prompt injections, we have found new threats that can break existing models. Yet, our scaling has helped substantially improve robustness to these attacks as well. Attack success rate is computed as the average attempt success across all attempts by GPT‑Red on held-out environments.
Robust while still being highly capable
A model can appear safer by refusing more requests or becoming less capable. A model that does less is naturally harder to attack, but that is not useful robustness.
We thoroughly evaluate both general frontier capabilities along with targeted over refusal tasks that we design. We find that all normal capabilities remain unaffected while significantly improving robustness. This suggests that the robustness gains came from better resistance to malicious instructions rather than improper tool-usage or refusing legitimate requests by default.
Next steps
AI agents are already being used to improve the capabilities of our next-generation models. We believe with GPT‑Red that we have started to unlock a similar flywheel for safety, where today’s models can be used to make tomorrow’s models more robust, aligned, and trustworthy. We will continue to scale compute and data while making algorithmic improvements, to train future versions of GPT‑Red that are stronger than today’s model. And in turn, these models will help make future GPT releases safer.
We will be releasing a pre-print with more details later this week.
本文内容采集自官方网站，排版和翻译可能与原页面存在差异。
阅读官方全文