AIPOCH Open-Science Benchmarks

Benchmarks for agentic scientific research

AIPOCH Open-Science has completed six tests on two scientific-agent benchmarks. Together they cover the main stages of research work: locating the papers a question depends on, assembling a complete literature set, executing a given research protocol, choosing methods autonomously, and staying on course when the task description is padded with irrelevant information.

On every one of the six tests, Open-Science posts the highest score among the records we compiled. Each test below is described in terms of what it measures, how we ran it, and what the number does and does not mean, followed by every comparison record in full. Separate submissions from the same system are kept as separate rows.¹

AstaBench: literature search

AstaBench, from Ai2, spans literature understanding, code execution, data analysis and scientific discovery across 11 sub-benchmarks and more than 2,400 samples. We ran its two literature-retrieval tasks, which test whether an agent can find the work a research question depends on.

Broken down by query type, specific-paper and metadata queries are already strong. Semantic queries, which make up most of the split, are where the most headroom remains.

ASI-Bench: research execution

ASI-Bench contains 60 project-level research tasks across 11 disciplines. Each task is a small research project: the agent has to process data, write and run code, and deliver results that can be checked. Levels B1 to B4 keep the same task and progressively withdraw guidance, so the gap between levels shows how much an agent depends on being told what to do.

All ASI-Bench scores are task-quality scores out of 100, not the share of tasks fully correct. Our runs use the locally available task set while the public leaderboard uses the official one, so our primary comparison is Codex run locally with the same model (GPT-6 Astra), with public leaderboard records as secondary context.

Adding distractors cost Open-Science 0.21 points on average (59.67 to 59.46), against 2.67 points for local Codex (58.12 to 55.45). Across the field, most entries move by a few points in either direction. A near-flat mean says the two levels behave alike on average in this run; it does not mean that no individual task was affected.

How we ran the evaluation

Each benchmark runs with its own tasks, grader and metric, and we report each in that metric rather than converting to a common scale. AIPOCH runs were repeated where noted and the reported value is the stated mean or result; the table below lists the exact scope of each run.

What the results say

Literature retrieval. Open-Science reliably locates a target paper and performs strongly on specific-paper and metadata lookups. Semantic topic search, the largest share of PaperFinding, is the clearest area for improvement.

Research execution. The lead is largest where the work is most concrete (B1, +4.57) and where the task is most cluttered (B4, +4.01). On B2 the two harnesses are level, which tells us that turning a list of methods into a working pipeline is where our harness adds least today.

Same model, different harness. On ASI-Bench, Open-Science and Codex ran the same model on the same tasks, so the gap between them reflects the harness rather than the model.

Availability

Open-Science is AIPOCH's open-source harness for scientific research: model-agnostic, auditable and yours to run. Install the desktop app, read the documentation, or browse the source on GitHub. Medical research skills for the agent are available in the Skills Hub, and every skill can be audited before deployment with MedSkillAudit.

@misc{aipoch2026openscience,
  title        = {AIPOCH Open-Science: Benchmark Results on AstaBench and ASI-Bench},
  author       = {{AIPOCH Research}},
  year         = {2026},
  month        = oct,
  organization = {AIPOCH},
  howpublished = {\url{https://github.com/aipoch/open-science}}
}