Back to Blog

AIPOCH Open-Science Leads the Reported ASI-Bench B1–B4 Comparisons

AIPOCH Open-Science scored 79.35 on ASI-Bench B1 and 56.11 on B2 in three local repetitions, ranking first in the reported comparison for both levels. Learn how B1–B4 differ, how the runs were configured, and how to read the full tables.

5 min read

AIPOCH Open-Science ASI-Bench B1–B4 comparison: 26 listed configurations and 93 reported scores, with local results distinguished from public references

AIPOCH Open-Science reports local ASI-Bench scores of 79.35 on B1, 56.11 on B2, 59.67 on B3, and 59.46 on B4. Its GPT-6 Astra configuration has the highest numerical score in each level’s comparison assembled for this report. These are local evaluation results compared with dated public references; they are not official ASI-Bench leaderboard placements.

The four conditions examine plan execution, workflow organization, method selection, and task completion with distracting context. Open-Science and the local Codex reference each completed three repetitions per level using GPT-6 Astra / xhigh. The numerical gaps are 4.57 on B1, 0.02 on B2, 1.55 on B3, and 4.01 on B4.

Open-Science is an open-source, local-first, model-agnostic, self-hosted AI research workbench for reproducible scientific discovery. Beyond aggregate scores, its .science packages can carry selected conversation branches, file versions, Notebook records, and verification evidence for review, handoff, and archiving in another project or computer. Scores describe task performance; research records help colleagues inspect execution and deliverables. Together they support review of a research workflow.

Results across four conditions

  • B1: 79.35. Execute a complete research plan; 4.57 points above the local Codex reference.
  • B2: 56.11. Organize a specified method into a workflow; close to the local Codex reference at 56.09.
  • B3: 59.67. Select a method from the objective and inputs; 1.55 points above the local Codex reference.
  • B4: 59.46. Complete the task with correct but non-essential context; 4.01 points above the local Codex reference.

What do ASI-Bench B1–B4 measure?

ASI-Bench evaluates project-level scientific work across 60 tasks and 11 domains. The four conditions change the guidance available to the agent while retaining the research objective and required deliverables within each matched task.

LevelWhat the prompt providesMain challenge
B1Background, method, equations, and complete procedureImplement and execute the plan
B2Intended method and constraints without complete stepsOrganize a working workflow
B3Objective, inputs, constraints, and required outputs without a prescribed methodChoose the method and construct the workflow
B4The B3 requirements plus correct but non-essential informationComplete the work while filtering distractions

The conditions describe different task settings. Whether the procedure is complete, the method is specified, or irrelevant background is present changes the work the agent must supply. Scores summarize task quality on a 100-point scale; they are not percentages of tasks solved completely.

AIPOCH Open-Science evaluation conditions from B1 plan execution to B4 method selection with distracting context

What did AIPOCH Open-Science score across B1–B4?

The GPT-6 Astra / xhigh configuration leads the disclosed comparison numerically at each level. The table separates the local Codex reference from the highest listed public reference.

LevelOpen-Science localCodex localLocal gapHighest public reference in the reportPublic reference scoreReported records
B179.3574.78+4.57Claude Code × Claude Opus 5 / max72.2924
B256.1156.09+0.02Codex × GPT-5.6 Sol / ultra49.5723
B359.6758.12+1.55Codex × GPT-5.6 Sol / ultra51.6023
B459.4655.45+4.01Codex × GPT-5.6 Sol / ultra50.4123

Local results and public references are labeled separately so scores can be read alongside the model, runtime, and scoring conditions. Differences use displayed scores and do not establish statistical significance.

B1: turn a complete research plan into executable results

Task requirements

B1 provides the scientific background, intended method, equations, and full procedure. The agent must implement the plan, read and process inputs, execute the analysis, and deliver code, output files, and method notes. With the route specified, implementation and execution quality become the main challenge.

In a single-cell RNA-sequencing task, the procedure can cover preprocessing, dimensionality reduction, clustering, trajectory analysis, and regulatory-network estimation. The agent still needs to handle data formats, connect computational steps, and produce the required artifacts. Supplying a procedure does not complete the data processing or delivery.

Configuration and result

AIPOCH Open-Science used GPT-6 (gpt-6-astra) / xhigh with 11 domain-specialist categories and research Skills, completing three local repetitions. Its B1 score is 79.35, with the reported mean covering 59 valid scored tasks out of the standard 60. One task was excluded because of an official scorer anomaly.

Open-Science has the highest numerical score among the 24 B1 records. It is 4.57 points above the same-model, same-named-effort local Codex reference at 74.78, and 7.06 points above the highest public reference, Claude Code × Claude Opus 5 / max, at 72.29.

What the result means

B1 examines the ability to turn a known research route into inspectable artifacts. Researchers with an established method need to check whether inputs were processed appropriately, steps executed, and deliverables completed. The 79.35 is a task-quality score, not the percentage of tasks solved completely.

B2: organize a specified method into a research workflow

Task requirements

B2 retains the intended method and constraints but removes the complete procedure. The agent must fill in implementation details, determine an order, connect methods, and organize a runnable, inspectable workflow. It tests workflow organization when the method is already known.

In the single-cell example, the prompt can specify methods for clustering, trajectory analysis, and regulatory estimation without listing the complete operations. The agent must decide how to preprocess the data, pass results between methods, and generate intermediate files for subsequent steps.

Configuration and result

AIPOCH Open-Science used GPT-6 (gpt-6-astra) / xhigh with domain specialists and research Skills, completing three local repetitions. Its B2 score is 56.11.

Open-Science has the highest numerical score among the 23 B2 records. The local Codex reference scored 56.09, a difference of 0.02 points. The highest public reference, Codex × GPT-5.6 Sol / ultra, scored 49.57, a difference of 6.54 points.

What the result means

B2 represents a researcher who has selected a method and needs support organizing its implementation. The local scores of 56.11 and 56.09 are close and indicate similar performance in this comparison. The 0.02-point numerical gap alone does not establish a reliable performance advantage.

B3: select a method from the research objective

Task requirements

B3 supplies the objective, input data, constraints, and required outputs without prescribing a method. The agent must choose a computational approach, implement a working workflow, and deliver results for researcher review. Method selection becomes part of the task.

For the single-cell RNA-sequencing task, cell grouping, state trajectories, candidate markers, and regulatory relationships remain the deliverables. The agent must decide which methods to use, whether they suit the inputs, and how to connect them into an analysis that generates the required outputs.

Configuration and result

AIPOCH Open-Science used GPT-6 (gpt-6-astra) / xhigh with 12 frozen specialist configurations, research Skills, and delegation enabled. Open-Science and the local Codex reference each completed three repetitions of 60 seed31415 tasks. Their B3 means are 59.67 and 58.12, respectively.

Open-Science has the highest numerical score among the 23 B3 records, 1.55 points above the local Codex reference. The highest public reference, Codex × GPT-5.6 Sol / ultra, scored 51.60, a difference of 8.07 points.

What the result means

B3 represents computational research with a defined goal and an open choice of method. The result describes the configured system’s combined method selection, execution, and delivery. Model, specialist instructions, delegation, and tools contribute together; the score does not isolate a component or establish that every method choice was appropriate.

B4: complete autonomous research with distracting context

Task requirements

B4 retains B3’s objective, inputs, constraints, and outputs while adding factually correct but non-essential background. The agent must select a method and distinguish useful information from context that could redirect the analysis away from the task.

In the corresponding single-cell task, the required outputs remain cell groupings, trajectories, candidate markers, and regulatory relationships. With more background text present, the agent needs to continue choosing and executing methods around those deliverables rather than adding unnecessary analysis for irrelevant material.

Configuration and result

AIPOCH Open-Science used GPT-6 (gpt-6-astra) / xhigh, 12 frozen specialist configurations, research Skills, and delegation enabled. Both local systems completed three repetitions of 60 seed31415 tasks. Their B4 means are 59.46 and 55.45, respectively.

Open-Science has the highest numerical score among the 23 B4 records, 4.01 points above the local Codex reference. The highest public reference, Codex × GPT-5.6 Sol / ultra, scored 50.41, a difference of 9.05 points.

What the result means

Open-Science’s B4 mean is 0.21 points below B3, while the local Codex reference decreases by 2.67 points. Open-Science’s aggregate means are therefore closer across these two conditions in this evaluation. Similar means do not establish that every task resisted distraction; stability assessment also requires repeat-run variation and task-level results.

How should configuration and scores be read together?

Each condition uses local repeated testing, and its score describes the corresponding configured system. The settings below establish the scope of comparison.

ItemB1 / B2B3 / B4
Main model and effortGPT-6 (gpt-6-astra) / xhighGPT-6 (gpt-6-astra) / xhigh
Specialists and Skills11 domain-specialist categories; research Skills12 frozen specialists; research Skills; delegation enabled
Local repetitionsThree per levelThree per level
Scoring coverageB1: 59 valid scored tasks60 tasks per level per repetition; three-run means
Task instancesseed31415seed31415

The B3/B4 evaluation used Open-Science 0.32.0 with its built-in Codex subscription service; the local Codex reference used version 0.156.1. Main-session and specialist effort was xhigh. Tasks ran serially with a 3,600-second agent-response budget and a 7,200-second outer scoring budget. Both local systems used the replacement judge deepseek-flash / high with thinking enabled.

Specialist configuration, model, budget, and judging can affect results. Each level should be interpreted separately: a higher B3 score than B2 cannot be attributed to removing method guidance. Public references also need to be read alongside their own model and scoring settings.

Inspect task results alongside aggregate scores

In one single-cell analysis record, Open-Science scored 81.76 on B3 and 88.95 on B4; the local Codex record scored 80.32 and 7.86. In a stable latent-state fitting task, Open-Science fell from 49.00 on B3 to 0.00 on B4.

These are individual task records, not three-run means for those tasks. They show that substantial task-level variation can coexist with similar aggregate scores. Practical evaluation should inspect code, outputs, and failure records alongside the summary.

Full reported ASI-Bench B1–B4 comparison

The table contains 26 listed configurations and 93 reported scores: 24 on B1 and 23 on each remaining level. Each column is compared independently; the row order is not a ranking shared by all four levels.

System × modelEffortB1B2B3B4Source type
AIPOCH Open-Science × GPT-6 Astraxhigh79.3556.1159.6759.46AIPOCH local
Codex × GPT-6 Astraxhigh74.7856.0958.1255.45Codex local
AIPOCH Open-Science × DeepSeek v4-flashmax69.64———AIPOCH local
Claude Code × Claude Opus 5max72.2945.8040.7042.39Public reference
Codex × GPT-5.6 Solultra71.7849.5751.6050.41Public reference
Claude Code × GLM-5.3max63.0535.4533.0935.01Public reference
Codex × GPT-5.6 Solxhigh62.7542.9640.8640.74Public reference
Claude Code × Gemini 3.8 Flashmax61.3339.1337.7934.52Public reference
Claude Code × Qwen3.8-Maxmax59.1432.5129.3734.46Public reference
Codex × GPT-5.5xhigh57.5735.2829.2930.46Public reference
Kimi Code × Kimi K3not disclosed56.1634.0528.1526.53Public reference
Claude Code × Kimi K3max55.7935.2427.7829.57Public reference
Claude Code × GLM-5.2max54.0130.8130.2927.27Public reference
Claude Code × Claude Opus 4.8max52.4836.2531.7329.59Public reference
Claude Code × DeepSeek V4 Flash 0731max51.8526.49——Public reference
Claude Code × DeepSeek V4 Flashmax——24.8424.41Public reference
OpenHands × DeepSeek V4 Flash 0731not disclosed50.6024.89——Public reference
OpenHands × DeepSeek V4 Flashnot disclosed——21.9124.99Public reference
Claude Code × GLM-5.3-Flashmax49.2728.6630.7025.43Public reference
Claude Code × Kimi K2.7max44.4323.8419.7521.34Public reference
Claude Code × MiniMax M3max43.5320.8620.6121.21Public reference
Claude Code × DeepSeek V4 Promax42.9718.7919.2018.94Public reference
Claude Code × MiMo V2.5 Promax41.7018.4916.4916.33Public reference
OpenHands × DeepSeek V4 Pronot disclosed37.7816.6615.7816.28Public reference
Kimi Code × Kimi K2.7not disclosed29.9917.0115.6516.22Public reference
MiMo Code × MiMo V2.5 Pronot disclosed27.7311.3311.5014.11Public reference

Reading the table: “—” means not reported, not zero. Local rows report three repetitions. Public B3/B4 references use three runs except Claude Opus 5 and Qwen3.8-Max, which use one; public B1/B2 repetition counts are not listed consistently. The suffixed DeepSeek V4 Flash 0731 entries and the unsuffixed DeepSeek V4 Flash entries remain separate rather than merging distinct version labels.

B1/B2 public references are dated September 29, 2026. B3/B4 public references use a snapshot read on October 10, whose release note labels it September 6. Public results can change; this comparison is bounded by those dates.

Why are local results different from official leaderboard results?

The official scoring guide distinguishes two contracts. seed31415 publishes references for local scoring and remains non-official. seed42 retains private reference material and uses the official website’s submission and scoring process before results can be published to the leaderboard.

Two records can share a task name and a B3 label while using different concrete data, parameters, and references. The run guidance therefore treats the seeds as separate runs. A comparison should also retain the scorer, judge, model version, agent version, budget, and available tools.

For this report, “leads” means the highest displayed numerical score among the disclosed records for each level. It does not mean official overall champion, independently certified performance, statistical significance, or guaranteed success on a researcher’s own data.

How can researchers use AIPOCH Open-Science to inspect their own workflow?

The practical next step is to evaluate a representative task with evidence that a colleague can inspect. AIPOCH Open-Science brings project files, literature tools, model configuration, code execution, and outputs into a research workspace. Researchers can organize their own task evaluation around these capabilities:

  1. Define the task and output contract. Record inputs, constraints, chosen model, required files, and the checks a reviewer will perform. Decide whether the method is supplied or open to selection.
  2. Keep the execution evidence. Retain code, outputs, relevant Notebook records, and file versions. Compare the delivered artifacts with the stated requirements rather than judging only the final prose answer.
  3. Review failures and uncertainty. Inspect missing outputs, failed checks, unsupported method choices, and differences across repetitions. Keep aggregate scores separate from individual examples.
  4. Prepare a review handoff. Select the appropriate records and files for a .science package, and accompany it with the environment and evaluation settings needed to interpret the work.

Imported sessions are read-only: they do not execute code or restore credentials. Side Chat conversations and private reading bookmarks are excluded, and included files depend on export selection. Received verification records describe the sender’s checks; they do not establish that the receiving computer reran them.

The four conditions describe plan execution, workflow organization, method selection, and distraction handling. Researchers can use them to identify questions for their own evaluation, then explore AIPOCH Open-Science with a task, an output contract, and an explicit review plan.

Frequently asked questions

What did AIPOCH Open-Science score on ASI-Bench B1–B4?

The reported local scores are 79.35 on B1, 56.11 on B2, 59.67 on B3, and 59.46 on B4. All four are associated with three local repetitions; B3/B4 are explicitly reported as three-run means over 60 tasks per repetition, while B1 covers 59 valid scored tasks and B2 scored 56.11.

What is the difference between ASI-Bench B3 and B4?

B3 leaves method selection to the agent. B4 retains the same task requirements and adds correct but non-essential information, testing whether the workflow stays focused on the relevant constraints.

Is AIPOCH Open-Science first on the official ASI-Bench leaderboard?

This report does not establish an official placement. Its local scores are numerically highest among the disclosed comparisons; official leaderboard publication requires the official scoring and submission process.

Does the B4 result prove stable resistance to distraction?

No. The B4 mean is 0.21 points below B3, describing similar aggregate performance while task-level failures remain possible. Stability assessment requires repeat-run variation and paired task-level analysis.

Does a .science package rerun or validate an evaluation automatically?

No. It carries selected research records and evidence for review, handoff, and archiving. Imported sessions remain read-only, and received verification evidence is a record of the sender’s checks.

Data and sources

Disclaimer

This report describes local evaluations and dated public comparisons, not an official ASI-Bench leaderboard certification or an independent replication. Scores concern the disclosed systems and conditions; they do not establish statistical significance or guarantee performance on other tasks. Aggregate means and individual task records require separate interpretation.

AIPOCH Open-Science supports research workflows and does not replace researcher judgment, scientific review, or independent verification. This article is informational and does not provide medical advice or validated scientific conclusions. Third-party sources remain the responsibility of their respective publishers.