
AIPOCH Open-Science reports local ASI-Bench scores of 79.35 on B1, 56.11 on B2, 59.67 on B3, and 59.46 on B4. Its GPT-6 Astra configuration has the highest numerical score in each level’s comparison assembled for this report. These are local evaluation results compared with dated public references; they are not official ASI-Bench leaderboard placements.
The four conditions examine plan execution, workflow organization, method selection, and task completion with distracting context. Open-Science and the local Codex reference each completed three repetitions per level using GPT-6 Astra / xhigh. The numerical gaps are 4.57 on B1, 0.02 on B2, 1.55 on B3, and 4.01 on B4.
Open-Science is an open-source, local-first, model-agnostic, self-hosted AI research workbench for reproducible scientific discovery. Beyond aggregate scores, its .science packages can carry selected conversation branches, file versions, Notebook records, and verification evidence for review, handoff, and archiving in another project or computer. Scores describe task performance; research records help colleagues inspect execution and deliverables. Together they support review of a research workflow.
Results across four conditions
- B1: 79.35. Execute a complete research plan; 4.57 points above the local Codex reference.
- B2: 56.11. Organize a specified method into a workflow; close to the local Codex reference at 56.09.
- B3: 59.67. Select a method from the objective and inputs; 1.55 points above the local Codex reference.
- B4: 59.46. Complete the task with correct but non-essential context; 4.01 points above the local Codex reference.
What do ASI-Bench B1–B4 measure?
ASI-Bench evaluates project-level scientific work across 60 tasks and 11 domains. The four conditions change the guidance available to the agent while retaining the research objective and required deliverables within each matched task.
| Level | What the prompt provides | Main challenge |
|---|---|---|
| B1 | Background, method, equations, and complete procedure | Implement and execute the plan |
| B2 | Intended method and constraints without complete steps | Organize a working workflow |
| B3 | Objective, inputs, constraints, and required outputs without a prescribed method | Choose the method and construct the workflow |
| B4 | The B3 requirements plus correct but non-essential information | Complete the work while filtering distractions |
The conditions describe different task settings. Whether the procedure is complete, the method is specified, or irrelevant background is present changes the work the agent must supply. Scores summarize task quality on a 100-point scale; they are not percentages of tasks solved completely.

What did AIPOCH Open-Science score across B1–B4?
The GPT-6 Astra / xhigh configuration leads the disclosed comparison numerically at each level. The table separates the local Codex reference from the highest listed public reference.
| Level | Open-Science local | Codex local | Local gap | Highest public reference in the report | Public reference score | Reported records |
|---|---|---|---|---|---|---|
| B1 | 79.35 | 74.78 | +4.57 | Claude Code × Claude Opus 5 / max | 72.29 | 24 |
| B2 | 56.11 | 56.09 | +0.02 | Codex × GPT-5.6 Sol / ultra | 49.57 | 23 |
| B3 | 59.67 | 58.12 | +1.55 | Codex × GPT-5.6 Sol / ultra | 51.60 | 23 |
| B4 | 59.46 | 55.45 | +4.01 | Codex × GPT-5.6 Sol / ultra | 50.41 | 23 |
Local results and public references are labeled separately so scores can be read alongside the model, runtime, and scoring conditions. Differences use displayed scores and do not establish statistical significance.
B1: turn a complete research plan into executable results
Task requirements
B1 provides the scientific background, intended method, equations, and full procedure. The agent must implement the plan, read and process inputs, execute the analysis, and deliver code, output files, and method notes. With the route specified, implementation and execution quality become the main challenge.
In a single-cell RNA-sequencing task, the procedure can cover preprocessing, dimensionality reduction, clustering, trajectory analysis, and regulatory-network estimation. The agent still needs to handle data formats, connect computational steps, and produce the required artifacts. Supplying a procedure does not complete the data processing or delivery.
Configuration and result
AIPOCH Open-Science used GPT-6 (gpt-6-astra) / xhigh with 11 domain-specialist categories and research Skills, completing three local repetitions. Its B1 score is 79.35, with the reported mean covering 59 valid scored tasks out of the standard 60. One task was excluded because of an official scorer anomaly.
Open-Science has the highest numerical score among the 24 B1 records. It is 4.57 points above the same-model, same-named-effort local Codex reference at 74.78, and 7.06 points above the highest public reference, Claude Code × Claude Opus 5 / max, at 72.29.
What the result means
B1 examines the ability to turn a known research route into inspectable artifacts. Researchers with an established method need to check whether inputs were processed appropriately, steps executed, and deliverables completed. The 79.35 is a task-quality score, not the percentage of tasks solved completely.
B2: organize a specified method into a research workflow
Task requirements
B2 retains the intended method and constraints but removes the complete procedure. The agent must fill in implementation details, determine an order, connect methods, and organize a runnable, inspectable workflow. It tests workflow organization when the method is already known.
In the single-cell example, the prompt can specify methods for clustering, trajectory analysis, and regulatory estimation without listing the complete operations. The agent must decide how to preprocess the data, pass results between methods, and generate intermediate files for subsequent steps.
Configuration and result
AIPOCH Open-Science used GPT-6 (gpt-6-astra) / xhigh with domain specialists and research Skills, completing three local repetitions. Its B2 score is 56.11.
Open-Science has the highest numerical score among the 23 B2 records. The local Codex reference scored 56.09, a difference of 0.02 points. The highest public reference, Codex × GPT-5.6 Sol / ultra, scored 49.57, a difference of 6.54 points.
What the result means
B2 represents a researcher who has selected a method and needs support organizing its implementation. The local scores of 56.11 and 56.09 are close and indicate similar performance in this comparison. The 0.02-point numerical gap alone does not establish a reliable performance advantage.
B3: select a method from the research objective
Task requirements
B3 supplies the objective, input data, constraints, and required outputs without prescribing a method. The agent must choose a computational approach, implement a working workflow, and deliver results for researcher review. Method selection becomes part of the task.
For the single-cell RNA-sequencing task, cell grouping, state trajectories, candidate markers, and regulatory relationships remain the deliverables. The agent must decide which methods to use, whether they suit the inputs, and how to connect them into an analysis that generates the required outputs.
Configuration and result
AIPOCH Open-Science used GPT-6 (gpt-6-astra) / xhigh with 12 frozen specialist configurations, research Skills, and delegation enabled. Open-Science and the local Codex reference each completed three repetitions of 60 seed31415 tasks. Their B3 means are 59.67 and 58.12, respectively.
Open-Science has the highest numerical score among the 23 B3 records, 1.55 points above the local Codex reference. The highest public reference, Codex × GPT-5.6 Sol / ultra, scored 51.60, a difference of 8.07 points.
What the result means
B3 represents computational research with a defined goal and an open choice of method. The result describes the configured system’s combined method selection, execution, and delivery. Model, specialist instructions, delegation, and tools contribute together; the score does not isolate a component or establish that every method choice was appropriate.
B4: complete autonomous research with distracting context
Task requirements
B4 retains B3’s objective, inputs, constraints, and outputs while adding factually correct but non-essential background. The agent must select a method and distinguish useful information from context that could redirect the analysis away from the task.
In the corresponding single-cell task, the required outputs remain cell groupings, trajectories, candidate markers, and regulatory relationships. With more background text present, the agent needs to continue choosing and executing methods around those deliverables rather than adding unnecessary analysis for irrelevant material.
Configuration and result
AIPOCH Open-Science used GPT-6 (gpt-6-astra) / xhigh, 12 frozen specialist configurations, research Skills, and delegation enabled. Both local systems completed three repetitions of 60 seed31415 tasks. Their B4 means are 59.46 and 55.45, respectively.
Open-Science has the highest numerical score among the 23 B4 records, 4.01 points above the local Codex reference. The highest public reference, Codex × GPT-5.6 Sol / ultra, scored 50.41, a difference of 9.05 points.
What the result means
Open-Science’s B4 mean is 0.21 points below B3, while the local Codex reference decreases by 2.67 points. Open-Science’s aggregate means are therefore closer across these two conditions in this evaluation. Similar means do not establish that every task resisted distraction; stability assessment also requires repeat-run variation and task-level results.
How should configuration and scores be read together?
Each condition uses local repeated testing, and its score describes the corresponding configured system. The settings below establish the scope of comparison.
| Item | B1 / B2 | B3 / B4 |
|---|---|---|
| Main model and effort | GPT-6 (gpt-6-astra) / xhigh | GPT-6 (gpt-6-astra) / xhigh |
| Specialists and Skills | 11 domain-specialist categories; research Skills | 12 frozen specialists; research Skills; delegation enabled |
| Local repetitions | Three per level | Three per level |
| Scoring coverage | B1: 59 valid scored tasks | 60 tasks per level per repetition; three-run means |
| Task instances | seed31415 | seed31415 |
The B3/B4 evaluation used Open-Science 0.32.0 with its built-in Codex subscription service; the local Codex reference used version 0.156.1. Main-session and specialist effort was xhigh. Tasks ran serially with a 3,600-second agent-response budget and a 7,200-second outer scoring budget. Both local systems used the replacement judge deepseek-flash / high with thinking enabled.
Specialist configuration, model, budget, and judging can affect results. Each level should be interpreted separately: a higher B3 score than B2 cannot be attributed to removing method guidance. Public references also need to be read alongside their own model and scoring settings.
Inspect task results alongside aggregate scores
In one single-cell analysis record, Open-Science scored 81.76 on B3 and 88.95 on B4; the local Codex record scored 80.32 and 7.86. In a stable latent-state fitting task, Open-Science fell from 49.00 on B3 to 0.00 on B4.
These are individual task records, not three-run means for those tasks. They show that substantial task-level variation can coexist with similar aggregate scores. Practical evaluation should inspect code, outputs, and failure records alongside the summary.
Full reported ASI-Bench B1–B4 comparison
The table contains 26 listed configurations and 93 reported scores: 24 on B1 and 23 on each remaining level. Each column is compared independently; the row order is not a ranking shared by all four levels.
| System × model | Effort | B1 | B2 | B3 | B4 | Source type |
|---|---|---|---|---|---|---|
| AIPOCH Open-Science × GPT-6 Astra | xhigh | 79.35 | 56.11 | 59.67 | 59.46 | AIPOCH local |
| Codex × GPT-6 Astra | xhigh | 74.78 | 56.09 | 58.12 | 55.45 | Codex local |
| AIPOCH Open-Science × DeepSeek v4-flash | max | 69.64 | — | — | — | AIPOCH local |
| Claude Code × Claude Opus 5 | max | 72.29 | 45.80 | 40.70 | 42.39 | Public reference |
| Codex × GPT-5.6 Sol | ultra | 71.78 | 49.57 | 51.60 | 50.41 | Public reference |
| Claude Code × GLM-5.3 | max | 63.05 | 35.45 | 33.09 | 35.01 | Public reference |
| Codex × GPT-5.6 Sol | xhigh | 62.75 | 42.96 | 40.86 | 40.74 | Public reference |
| Claude Code × Gemini 3.8 Flash | max | 61.33 | 39.13 | 37.79 | 34.52 | Public reference |
| Claude Code × Qwen3.8-Max | max | 59.14 | 32.51 | 29.37 | 34.46 | Public reference |
| Codex × GPT-5.5 | xhigh | 57.57 | 35.28 | 29.29 | 30.46 | Public reference |
| Kimi Code × Kimi K3 | not disclosed | 56.16 | 34.05 | 28.15 | 26.53 | Public reference |
| Claude Code × Kimi K3 | max | 55.79 | 35.24 | 27.78 | 29.57 | Public reference |
| Claude Code × GLM-5.2 | max | 54.01 | 30.81 | 30.29 | 27.27 | Public reference |
| Claude Code × Claude Opus 4.8 | max | 52.48 | 36.25 | 31.73 | 29.59 | Public reference |
| Claude Code × DeepSeek V4 Flash 0731 | max | 51.85 | 26.49 | — | — | Public reference |
| Claude Code × DeepSeek V4 Flash | max | — | — | 24.84 | 24.41 | Public reference |
| OpenHands × DeepSeek V4 Flash 0731 | not disclosed | 50.60 | 24.89 | — | — | Public reference |
| OpenHands × DeepSeek V4 Flash | not disclosed | — | — | 21.91 | 24.99 | Public reference |
| Claude Code × GLM-5.3-Flash | max | 49.27 | 28.66 | 30.70 | 25.43 | Public reference |
| Claude Code × Kimi K2.7 | max | 44.43 | 23.84 | 19.75 | 21.34 | Public reference |
| Claude Code × MiniMax M3 | max | 43.53 | 20.86 | 20.61 | 21.21 | Public reference |
| Claude Code × DeepSeek V4 Pro | max | 42.97 | 18.79 | 19.20 | 18.94 | Public reference |
| Claude Code × MiMo V2.5 Pro | max | 41.70 | 18.49 | 16.49 | 16.33 | Public reference |
| OpenHands × DeepSeek V4 Pro | not disclosed | 37.78 | 16.66 | 15.78 | 16.28 | Public reference |
| Kimi Code × Kimi K2.7 | not disclosed | 29.99 | 17.01 | 15.65 | 16.22 | Public reference |
| MiMo Code × MiMo V2.5 Pro | not disclosed | 27.73 | 11.33 | 11.50 | 14.11 | Public reference |
Reading the table: “—” means not reported, not zero. Local rows report three repetitions. Public B3/B4 references use three runs except Claude Opus 5 and Qwen3.8-Max, which use one; public B1/B2 repetition counts are not listed consistently. The suffixed DeepSeek V4 Flash 0731 entries and the unsuffixed DeepSeek V4 Flash entries remain separate rather than merging distinct version labels.
B1/B2 public references are dated September 29, 2026. B3/B4 public references use a snapshot read on October 10, whose release note labels it September 6. Public results can change; this comparison is bounded by those dates.
Why are local results different from official leaderboard results?
The official scoring guide distinguishes two contracts. seed31415 publishes references for local scoring and remains non-official. seed42 retains private reference material and uses the official website’s submission and scoring process before results can be published to the leaderboard.
Two records can share a task name and a B3 label while using different concrete data, parameters, and references. The run guidance therefore treats the seeds as separate runs. A comparison should also retain the scorer, judge, model version, agent version, budget, and available tools.
For this report, “leads” means the highest displayed numerical score among the disclosed records for each level. It does not mean official overall champion, independently certified performance, statistical significance, or guaranteed success on a researcher’s own data.
How can researchers use AIPOCH Open-Science to inspect their own workflow?
The practical next step is to evaluate a representative task with evidence that a colleague can inspect. AIPOCH Open-Science brings project files, literature tools, model configuration, code execution, and outputs into a research workspace. Researchers can organize their own task evaluation around these capabilities:
- Define the task and output contract. Record inputs, constraints, chosen model, required files, and the checks a reviewer will perform. Decide whether the method is supplied or open to selection.
- Keep the execution evidence. Retain code, outputs, relevant Notebook records, and file versions. Compare the delivered artifacts with the stated requirements rather than judging only the final prose answer.
- Review failures and uncertainty. Inspect missing outputs, failed checks, unsupported method choices, and differences across repetitions. Keep aggregate scores separate from individual examples.
- Prepare a review handoff. Select the appropriate records and files for a
.sciencepackage, and accompany it with the environment and evaluation settings needed to interpret the work.
Imported sessions are read-only: they do not execute code or restore credentials. Side Chat conversations and private reading bookmarks are excluded, and included files depend on export selection. Received verification records describe the sender’s checks; they do not establish that the receiving computer reran them.
The four conditions describe plan execution, workflow organization, method selection, and distraction handling. Researchers can use them to identify questions for their own evaluation, then explore AIPOCH Open-Science with a task, an output contract, and an explicit review plan.
Frequently asked questions
What did AIPOCH Open-Science score on ASI-Bench B1–B4?
The reported local scores are 79.35 on B1, 56.11 on B2, 59.67 on B3, and 59.46 on B4. All four are associated with three local repetitions; B3/B4 are explicitly reported as three-run means over 60 tasks per repetition, while B1 covers 59 valid scored tasks and B2 scored 56.11.
What is the difference between ASI-Bench B3 and B4?
B3 leaves method selection to the agent. B4 retains the same task requirements and adds correct but non-essential information, testing whether the workflow stays focused on the relevant constraints.
Is AIPOCH Open-Science first on the official ASI-Bench leaderboard?
This report does not establish an official placement. Its local scores are numerically highest among the disclosed comparisons; official leaderboard publication requires the official scoring and submission process.
Does the B4 result prove stable resistance to distraction?
No. The B4 mean is 0.21 points below B3, describing similar aggregate performance while task-level failures remain possible. Stability assessment requires repeat-run variation and paired task-level analysis.
Does a .science package rerun or validate an evaluation automatically?
No. It carries selected research records and evidence for review, handoff, and archiving. Imported sessions remain read-only, and received verification evidence is a record of the sender’s checks.
Data and sources
- Local results: AIPOCH Open-Science ASI-Bench B1–B4 local evaluations. B1/B2 results are dated September 29, 2026; B3/B4 results are included in the October 2026 evaluation summary.
- Tasks and rules: official ASI-Bench repository, run guidance, and scoring guide.
- Public results: ASI-Bench leaderboard, using the dated snapshots identified above.
- Product scope: AIPOCH Open-Science repository, product page, and
.sciencepackage documentation.
Disclaimer
This report describes local evaluations and dated public comparisons, not an official ASI-Bench leaderboard certification or an independent replication. Scores concern the disclosed systems and conditions; they do not establish statistical significance or guarantee performance on other tasks. Aggregate means and individual task records require separate interpretation.
AIPOCH Open-Science supports research workflows and does not replace researcher judgment, scientific review, or independent verification. This article is informational and does not provide medical advice or validated scientific conclusions. Third-party sources remain the responsibility of their respective publishers.