科学数据很少附带使用说明。研究人员必须判断某个模式反映的是生物学现象还是噪声，数据能否支撑所提出的问题，以及每个结果应如何影响后续操作。人工智能智能体正日益能够执行复杂的分析，但真正的科学研究不仅依赖于回忆事实或遵循预设的工作流程，更在于做出这些高阶判断。
今天，我们推出 GeneBench-Pro——一个具有挑战性的研究级基准测试，用于检验模型能否处理现实计算生物学所需的那种高度依赖判断的分析。它基于 GeneBench（在新窗口中打开） 进行了扩展，涵盖了基因组学、定量生物学和转化医学中更困难、更贴近实际的任务，捕捉了计算生物学研究中复杂性、迭代性和模糊性的特点。
迄今为止，对于使现实计算研究变得困难的系统级判断能力，鲜有令人信服的评估。这些能力包括处理模糊性、修正假设、选择正确的分析路径，以及判断结果何时可用于决策。由于这些技能难以形式化，因此也难以严格评估，即便这些方面的弱点正日益制约着人工智能的整体表现。
GeneBench-Pro 旨在精确衡量这些更高层次的能力。在 GeneBench-Pro 中，我们将“研究品味”定义为塑造分析过程的判断链：数据能支持哪些问题，早期诊断应如何改变模型或估计目标，以及初始计划何时需要修订。每个 GeneBench-Pro 问题都会为模型提供一个真实且杂乱的数据集、简要的实验背景，以及一个与下游决策相关的目标估计量。要正确回答，模型必须探索数据、选择合适的分析方法、进行迭代实验过程，并给出最终答案。
数据集构建
在生物学领域，数据生成（例如基因组测序）的成本已大幅下降，一些研究人员现在认为（在新窗口中打开） 限制因素不再是样本采集，而是下游计算与分析。GeneBench-Pro 旨在评估解决这一瓶颈的进展，包含 129 个问题，涵盖计算生物学中广泛的场景和方法。
领域图谱：10 个领域、21 个子领域的 129 个问题
点击上方圆点了解基准测试问题。
此图谱展示了 GeneBench-Pro 的广度。请访问 案例研究页面 以更详细地探索 10 个代表性问题。
GeneBench-Pro 的设计也旨在避免常见的基准测试缺陷。许多长周期生物学基准测试围绕杂乱的历史数据集构建多步骤问题，其中可能不存在单一正确的分析路径。一个智能体可能选择某个合理的阈值，而另一个智能体可能选择不同但同样合理的选项，这反映的更多是基准测试创建者的任意选择，而非模型性能的根本差异。反之亦然：如果问题对数值不够敏感，智能体可能在分析中犯下根本性错误，却仍能得出及格的结果。
为避免这些失败模式，每个 GeneBench-Pro 问题都是合成构建的：我们了解完整的因果结构，并直接模拟数据生成过程。这使我们能够调整每个问题的复杂性，确保主观分析选择上的合理差异仍能产生可接受的数值结果，并通过消融研究验证看似合理但错误的分析会失败。然后，我们通过详细的痕迹分析来审计问题草案，检查信息泄露和意外的解决路径。这让我们确信，得出正确答案取决于选择正确的分析路径，而非利用捷径或匹配作者的任意偏好。
我们将 129 个 GeneBench-Pro 问题中的 82 个发送给了外部领域专家，包括研究生、博士后研究员、产业界科学家和教授。评审员评估了每个问题的真实性、目标答案是否可识别，以及方法和估计量是否恰当。反馈被用于改进问题。
评估与评分
每个 GeneBench-Pro 问题都是一个独立的科学分析。智能体可以访问一个隔离的工作空间，其中包含简短的提示、数据文件以及标准的生物信息学工具栈，包括 Python、科学计算库和基础基因组学软件包（如 PLINK 2.0），尽管这些问题并不需要特定领域的工具。
由于我们控制着完整的数据生成过程，我们可以根据已知目标确定性地评判正确性，从而避免了基于标准评分标准评估中出现的模型选择变异和冗长效应。
每个问题还附带丰富的元数据，包括预期的分析结构、附加的数据文件、详细的多页案例研究以及专家评审结果。我们将在 Hugging Face（在新窗口中打开） 上完全开源 10 个具有代表性的 GeneBench-Pro 问题，并提供 交互式网页界面 供浏览。最后，我们将在近期向 Artificial Analysis（在新窗口中打开） 提供一个包含 50 个问题的子集，用于独立的第三方基准测试。
结果
我们最强的模型 GPT‑5.6 Sol 在最高推理水平下达到了 28.7% 的通过率（启用 Pro 模式时为 31.5%）。这比我们开始构建原始 GeneBench 时有了显著提升；当时，我们最前沿的模型 GPT‑5 得分低于 5%。该基准测试上的进展表明，前沿模型正在快速进步，即使在较不具象的系统级科学推理方面也是如此。按照目前的速度，该基准测试可能在今年年底前达到饱和。
结果还显示了扩展测试时计算的影响。在最低推理水平下，GPT‑5.6 Sol 的通过率仅为个位数。在最高推理水平下，GPT‑5.6 Sol 解决的问题数量几乎是 GPT‑5.2 的六倍，而使用的 token 数量约为其三分之二。
跨模型家族的对比表明，在定量不确定性下的高级科学推理任务中，GPT 模型属于最强系统之列。GPT‑5.6、GPT‑5.5 与 GLM 5.2 等领先开源模型之间的性能差距，远超我们根据编程基准测试（在新窗口打开）进行外推时的预期，这表明开源模型更专注于编程能力，而非广泛的推理能力。
我们在开发过程中使用了前沿 GPT 模型来评估和强化问题。因此，我们怀疑 GeneBench-Pro 可能对 GPT 模型存在偏向，不利于其他模型家族。然而，竞品模型最多只能与对应 GPT 模型在发布时的性能持平，且往往差距显著。
考虑到 GeneBench-Pro 问题的难度，这些评估结果——GPT‑5.6 Sol (Pro) 高达 31.5%——令人瞩目。在一项调查中，我们的评审员估计，一个典型的 GeneBench-Pro 问题需要人类专家大约 20 到 40 小时才能完成。按保守的每小时 200 美元计算，单个问题的人力成本高达数千美元。当前的人工智能代理仍不可靠，无法取代人类专家，但成本差距巨大——每个问题的推理成本仅为几美元。这意味着，即使以现有能力实现部分自动化，也能创造可观的经济和科学价值。
尽管如此，前沿模型仍无法解决超过三分之一的问题，这表明改进空间巨大。模型能在具有挑战性的问题上取得部分进展，但难以完成推理闭环。这种失败模式与人类专家和新手之间的差异如出一辙。专家利用经验来构建问题框架并调整方法，而新手虽能做出观察，却难以将其融入问题的更广泛背景中。
要实现近乎完美的性能，需要既能可靠衡量进展，又能识别模型失败之处的评估方法。像 GeneBench-Pro 这样的基准测试，有助于将模糊的能力缺陷转化为可诊断和可改进的具体问题。
如果代理能够可靠地自动化这类分析，它们将显著加速科学发现。人类遗传学证据已是靶点优先排序和转化研究跟进的核心，因为具有遗传学支持的机制更有可能最终获批成为治疗方法。
与此同时，测序成本已大幅下降，生物样本库规模的数据集如今以前所未有的广度将分子、表型和健康记录信息关联起来。限制因素正从数据生成转向将信息转化为可操作的洞见。能够持续执行目前由人类专家团队完成的分析的模型，可通过加速假设筛选、靶点跟进以及数据生成与决策之间的迭代周期，从而变革工业研究。
GeneBench-Pro 是一项初步尝试，旨在评估经验丰富的科学家所具备的良好科学判断中涉及的更抽象技能。这些技能使他们能够凭直觉识别最有前景的初步分析，在数据与初始假设相悖时迭代和修正思路，并得出下游临床、学术或商业决策所依赖的结论。
我们预计，随着模型能力的提升，那些在更高抽象层面探测模型能力的基准测试（而非仅仅测试书本知识或执行常规分析的能力）将变得愈发有用。
Scientific data rarely arrive with instructions. Researchers must decide whether a pattern reflects biology or noise, whether the data can support the question being asked, and how each result should change what they do next. AI agents are increasingly capable of executing complex analyses, but real scientific research also depends not simply on recalling facts or following a predefined workflow but also on making these higher-order judgments.
Today, we’re introducing GeneBench-Pro—a challenging, research-level benchmark for testing whether models can handle the kind of judgment-heavy analysis that real-world computational biology requires. It expands on GeneBench(opens in a new window) to cover harder, more realistic tasks across genomics, quantitative biology, and translational medicine, capturing the complexity, iterative nature, and ambiguity of scientific research in computational biology.
To date, there have been few convincing assessments of the system-level judgment calls that make real-world computational research difficult. These include handling ambiguity, revising assumptions, choosing the correct analysis path, and knowing when a result is decision-ready. Because these skills are difficult to formalize, they are also difficult to assess rigorously, even as weaknesses in them increasingly constrain overall AI performance.
GeneBench-Pro is designed to precisely measure these higher-level capabilities. Within GeneBench-Pro, we define “research taste” as the chains of judgment calls that shape an analysis: which questions the data can support, how early diagnostics should change the model or estimand, and when an initial plan needs to be revised. Each GeneBench-Pro problem gives the model a realistic and messy dataset, brief experimental context, and a target estimand tied to a downstream decision. To answer correctly, the model must explore the data, choose an appropriate analytical approach, engage in an iterative process of experimentation, and supply a final answer.
Dataset construction
In biology, the cost of data generation (e.g., genome sequencing) has fallen dramatically, and some researchers now argue(opens in a new window) that the limiting factor is no longer sample collection but downstream computation and analysis. GeneBench-Pro is built to assess progress in addressing that bottleneck, with 129 questions covering a broad range of computational biology settings and methods.
Domain Atlas: 129 problems in 10 domains and 21 sub-domains
Click on a dot above to learn about a benchmark problem.
This atlas provides a preview of the breadth of GeneBench-Pro. Visit the case studies page to explore 10 representative questions in more detail.
GeneBench-Pro is also designed to avoid common benchmark failures. Many long-horizon biology benchmarks construct multi-step questions around messy historical datasets, where there may be no single correct path through the analysis. An agent might choose one defensible cutoff, while another might choose a different but equally defensible option, reflecting the arbitrary choices made by the benchmark creator more than any fundamental differences in model performance. The reverse can also happen: if a problem is too numerically insensitive, an agent can make fundamental errors in an analysis and still produce a passing result.
To avoid these failure modes, each GeneBench-Pro problem is built synthetically: we know the full causal structure and directly simulate the data-generating process. That enables us to tune the complexity of each problem, ensure that reasonable differences in subjective analytical choices still produce accepted numerical results, and verify (through ablation studies) that plausible but incorrect analyses fail. We then audit problem drafts through detailed trace analyses to check for information leakage and unintended solution pathways. This gives us confidence that getting the right answer depends on choosing the correct analytic pathway and not on exploiting a shortcut or matching an arbitrary author preference.
We sent 82 of the 129 GeneBench-Pro questions to external domain experts, including graduate students, postdoctoral researchers, industry scientists, and professors. Reviewers assessed each problem’s realism, whether the target answer was identifiable, and whether the methods and estimators were appropriate. Feedback was used to improve problems.
Evaluation and grading
Each GeneBench-Pro problem is a self-contained scientific analysis. Agents receive access to an isolated workspace with a short prompt, data files, and a standard bioinformatics stack including Python, scientific computing libraries, and basic genomics packages like PLINK 2.0 (although the problems do not require domain-specific tooling).
Because we control the full data-generation process, we can grade correctness deterministically against known targets, avoiding model-choice variability and verbosity effects found in standard rubric-based evaluation.
Each problem also comes with rich metadata, including the intended analysis structure, attached data files, a detailed multi-page case study, and expert review outcomes. We are fully open-sourcing 10 representative GeneBench-Pro questions on Hugging Face(opens in a new window), with an interactive web interface for browsing them. Finally, we will provide a 50-question subset to Artificial Analysis(opens in a new window) for independent, third-party benchmarking in the near future.
Results
Our strongest model, GPT‑5.6 Sol, attains a pass rate of 28.7% at the highest reasoning level (31.5% with Pro mode enabled). That is a sharp increase from when we began building the original GeneBench; at that time, our best frontier model, GPT‑5, scored below 5%. Progress on this benchmark suggests that frontier models are improving quickly, even on less tangible, systems-level scientific reasoning. At the current pace, this benchmark may be saturated by the end of the year.
The results also show the impact of scaling test-time compute. At the lowest reasoning level, GPT‑5.6 Sol only achieves a single-digit passrate. At the highest reasoning level, GPT‑5.6 Sol solves nearly six times as many questions as GPT‑5.2 does while using about two-thirds as many tokens.
Comparisons across model families suggest that GPT models are among the strongest systems at high-level scientific reasoning under quantitative uncertainty. The performance gap between GPT‑5.6, GPT‑5.5 and leading open-source models such as GLM 5.2 is significantly larger than we would expect when extrapolating from coding benchmarks(opens in a new window), indicating that open-source models are more specialized for coding than for broader reasoning ability.
We used frontier GPT models to evaluate and harden problems during development. As such, we suspected GeneBench-Pro might be biased against GPT models relative to other model families. However, competitor models at best matched the performance of the corresponding GPT model at the time of release, and tended to fall short considerably.
These evaluation results—as high as 31.5% on GPT‑5.6 Sol (Pro)—are striking given the difficulty of the GeneBench-Pro questions. In a survey, our reviewers estimated that a typical GeneBench-Pro problem would take a human expert around 20–40 hours to complete. At a conservative $200 per hour, that puts the human labor cost of a single problem in the thousands of dollars. Current AI agents are still too unreliable to replace human experts, but the cost gap is large, with inference costs at only several dollars per problem. That means even partial automation at current capabilities could create meaningful economic and scientific value.
Still, the fact that frontier models still solve fewer than a third of these problems shows that there is substantial room for improvement. Models can make partial progress on challenging problems, but they struggle to close the inferential loop. This failure pattern mirrors the contrast between human experts and novices. Experts use their experience to frame the problem and adapt their approach, while novices make observations but struggle to integrate them into the broader context of the problem.
Achieving near-perfect performance will require evaluations that both reliably measure progress and identify where models still fail. Benchmarks like GeneBench-Pro can help to turn a vague capability deficiency into something we can diagnose and improve.
If agents can reliably automate this class of analysis, they could significantly accelerate scientific discovery. Human genetic evidence is already central to target prioritization and translational follow-up, because mechanisms with genetic support are much more likely to lead to approved treatments.
Meanwhile, sequencing costs have plummeted, and biobank-scale datasets now link molecular, phenotypic, and health-record information at unprecedented breadth. The limiting factor is shifting from data generation to turning the information into actionable insights. Models that can consistently perform analyses now handled by teams of human experts could transform industrial research by accelerating hypothesis triage, target follow-up, and the iteration cycle between data generation and decision-making.
GeneBench-Pro represents an initial effort to evaluate the more abstract skills involved in good scientific judgment possessed by experienced. These skills allow them to intuit and identify the most promising initial analyses, iterate and revise their thinking when data contradict initial assumptions, and arrive at conclusions upon which downstream clinical, academic, or business decisions may depend.
We anticipate that as model capabilities advance, benchmarks that probe model abilities at these higher levels of abstraction will become increasingly useful, beyond those that simply test book knowledge or the ability to execute routine analyses.
本文内容采集自官方网站，排版和翻译可能与原页面存在差异。
阅读官方全文