Back to Blog

AIPOCH Open-Science Scores 46.30% in a Local AstaBench SUPER-Expert Run

AIPOCH Open-Science reports a 46.30% mean output-match score across all 45 AstaBench SUPER-Expert test tasks. See the local result, scoring method, and report-snapshot comparison.

5 min read

AIPOCH Open-Science recorded a 46.30% mean output-match score across all 45 tasks in a local run of the AstaBench SUPER-Expert test. The result is numerically above every scored official entry in the leaderboard snapshot included in our report. Our run has not been submitted to the official leaderboard, so the comparison is not an official first-place ranking.

AIPOCH Open-Science local AstaBench SUPER-Expert result and the top three submitted records in the report snapshot

What does AstaBench SUPER-Expert test?

SUPER-Expert tests whether an agent can set up an existing ML or NLP research repository, run the requested experiment, and return its output, using the 45 expert tasks from SUPER as an AstaBench test split; the result therefore speaks to research-code execution rather than generic code generation or every stage of scientific research.

The 2024 SUPER paper describes 45 end-to-end problems with expert-annotated solutions from ML and NLP research repositories. The AstaBench v0.5.4 test configuration (checked in 2026) maps SUPER_Expert_test to astabench/super_test and names output_match/mean as its primary metric.

In a typical task, the agent must work from an existing research project: install dependencies, prepare data, adjust code or configuration, run training or inference, and report the requested result. The AstaBench task implementation compares the returned output with a reference answer; structured answers can receive partial credit. The headline metric is a mean output-match score, not a binary pass rate.

AIPOCH Open-Science also supports what happens after execution: a .science package can carry selected conversation branches, file versions, Notebook records, and verification evidence into review or handoff. Package transfer is outside the SUPER-Expert score.

What did AIPOCH Open-Science achieve in the local run?

AIPOCH Open-Science returned scored results for all 45 SUPER-Expert tasks and reached 46.30% on the primary output_match/mean metric with DeepSeek Flash at Max reasoning; 45/45 describes scoring coverage rather than a count of fully solved tasks, and the run remains a local result without an official leaderboard placement.

The AIPOCH result graphic (2026) reports 46.30%, 45 of 45 tasks with scored results, and the local model setup. The AstaBench v0.5.4 configuration (checked in 2026) identifies output_match/mean as the test's primary metric.

MeasureReported local resultHow to read it
Test split45 tasksFull SUPER-Expert test set
Scoring coverage45/45 (100%)Every task received a score; this is not a solved-task count
Primary score46.30%Mean output_match score; partial credit can apply
Per-task answer submission100%An answer was provided for every task; this does not mean a leaderboard submission
Agent and modelAIPOCH Open-Science · DeepSeek Flash / MaxConfiguration reported for this run
Leaderboard statusLocal evaluationNot submitted to the official leaderboard

The same AstaBench scorer reports output match separately from whether an answer was submitted. That distinction matters here: 100% scoring coverage and 100% per-task answer submission do not mean 100% task success.

How does the local result compare with the official leaderboard snapshot?

In AIPOCH's report snapshot, the 46.30% local result is numerically above all 29 scored official entries, including the highest submitted score of 43.88%; the difference is about 2.4 percentage points, but the local row is a comparison entry with no official rank, and the snapshot is not a live leaderboard claim.

The full AIPOCH comparison graphic (2026) displays one local result alongside all 29 scored official entries reproduced in the report. The AIPOCH summary graphic (2026) shows the local score beside the three highest submitted scores in that snapshot.

Position in report snapshotAgent / modelMean output_matchStatus
Local comparisonAIPOCH Open-Science · DeepSeek Flash / Max46.30%Local; no official rank
Official #1ReAct · claude-opus-4-743.88%Submitted record in snapshot
Official #2ReAct · claude-sonnet-4-642.37%Submitted record in snapshot
Official #3ReAct · GPT-5 (2025-08)41.11%Submitted record in snapshot

The complete comparison below includes every scored official record in the report, rather than only the top three. Its numbers 1–30 show score order across the combined comparison; they do not assign AIPOCH Open-Science an official leaderboard rank. The official AstaBench leaderboard may change after the report snapshot.

Complete AstaBench SUPER-Expert score comparison with one AIPOCH Open-Science local result and 29 scored official records

What does this result mean for research teams?

The local SUPER-Expert result is a focused signal about AIPOCH Open-Science's ability to work through research-code tasks, not a blanket measure of scientific correctness; teams can use the benchmark as a starting point, then inspect the inputs, outputs, and execution evidence on a repository they know before trusting a workflow for their own research.

The 2024 SUPER paper frames the expert set around realistic setup and execution challenges in research repositories. The AIPOCH Open-Science product page (checked in 2026) describes a workspace for scientific agents, code execution, and traceable artifacts. The AIPOCH .science package guide (checked in 2026) explains how selected conversation branches, files, and evidence can be carried into a review or handoff.

A .science package can help a colleague inspect the selected research record after a run. An imported record is read-only, and received verification evidence describes checks made by the sender rather than proving that the recipient reran them. The SUPER-Expert score does not evaluate this handoff feature. Explore AIPOCH Open-Science and try it on a research repository your team already understands.