Claude vs GPT-5.6: How AI Agents Define "Reproduce a Paper"

GPT-5.6 sol had just launched, and I wanted to try it on a real scientific task. I already had one: a bioinformatics paper I had previously reproduced with Claude Science. I gave both agents the same goal and expected differences in speed or implementation. Instead, they had different ideas about what it meant to reproduce the paper: Opus 4.8 mostly recovered and continued, while GPT-5.6 sol began auditing the paper and created its own evidence boundaries and stopping rules.

A Real Task for a New Model

When GPT-5.6 sol was released, I did not want to test it with another small coding benchmark or a carefully prepared demo. I wanted to see what it would do with a scientific task that was long, messy, and full of the kinds of incomplete details that make real bioinformatics work difficult. I already had one: a breast-cancer bioinformatics paper I had previously reproduced with Claude Science.

That earlier run had gone far beyond paper Q&A. Claude read the paper, reconstructed the methods, built the environment, downloaded TCGA and GEO data, ran the analyses, generated figures, retried failed steps, used its built-in reviewer, selected substitutes when tools were unavailable, and produced a final report.

So when GPT-5.6 sol came out, reusing the same paper felt like the obvious test. I gave it the same core goal: reproduce all of the bioinformatics analyses in the paper and generate the corresponding figures. I did not ask it to be more conservative than Claude. I did not tell it to create evidence classes, build a risk register, or stop when a result could not be reproduced exactly. I mostly wanted to see how a newly released model would handle a task I already understood.

I expected differences in coding style, environment setup, package failures, execution speed, and even how the final figures would look. What I did not expect was that the two agents would interpret the word “reproduce” differently, which became the most interesting result of the test.

This is Not a Clean Model Benchmark

Before comparing the two runs, there is an important limitation. This was not a controlled experiment designed to isolate model capability.

The first run used Claude Opus 4.8 inside Claude Science. The second used GPT-5.6 sol inside Codex. The two systems have different execution environments, context-management strategies, review structures, and ways of organizing long tasks. They also did not end up with an identical frozen input package: the Claude run proceeded from the paper PDF, while the GPT-5.6 sol run located and downloaded the official supplement. Public database snapshots may have changed between the runs as well. The comparison unit is therefore not simply Claude versus GPT. It is Claude + Claude Science versus GPT-5.6 sol + Codex, observed on the same scientific target.

But the target itself did not change. Both agents were asked to reproduce the paper's bioinformatics analyses and generate the figures. That makes the difference in how they defined “done” worth examining, even if it does not support a universal claim that one base model is better at science.

The table below summarizes what I observed in these two runs. It should be read as a comparison of this Claude Science run and this GPT-5.6 sol + Codex run, not as a permanent product ranking.

Dimension	Claude Opus 4.8 + Claude Science	GPT-5.6 sol + Codex
User goal	Reproduce all bioinformatics analyses and generate the corresponding figures.	The same core goal. No additional request for evidence classes, sensitivity branches, or stopping rules.
Starting material	The reproduction proceeded from the paper PDF without the official supplement or frozen historical database exports.	Started from the same paper and independently located, downloaded, checksum-verified, and audited the official supplement.
Initial task framing	Built an end-to-end execution plan focused on reconstructing and completing the analysis chain.	Split the task into a methods audit, figure-to-analysis map, and resource-availability audit before large-scale execution.
Default behavior when blocked	Retried, repaired, or selected a nearby substitute so the workflow could continue.	First classified whether the block allowed a substitution, only an approximation, or required the result to be gated.
Handling a result mismatch	Adjusted the DEG threshold to move the output closer to the paper's reported count.	Preserved results from valid and sensitivity branches even when none matched the published count.
Handling unavailable methods	Replaced pRRophetic with oncoPredict and continued with an exploratory drug-sensitivity analysis.	Located the historical pRRophetic archive, rejected the incomplete download, and marked the numerical panels as an evidence boundary.
Treatment of historical database state	Current tools or substitutes were used where practical, with limitations described later.	Explicitly separated current reconstruction from paper-time historical reproduction and froze versioned inputs where possible.
Statistical assumption checks	The built-in reviewer checked completed and failed tasks, triggered retries, and moved unresolved issues into limitations.	Statistical checks were also used as gates; the failed proportional-hazards test prevented the nomogram, calibration, and DCA from being generated.
Review structure	Reviewer was native to the agent loop and primarily acted on task outcomes.	The agent created methods matrices, risk registers, figure contracts, completion matrices, and audit scripts as concrete review objects.
Evidence status	Exact, approximate, substituted, and missing elements were mainly explained in the final narrative.	Evidence state was attached at panel level as exact/frozen, current reconstruction, near-reproduction, evidence boundary, or gated.
Sensitivity analysis	Used alternatives mainly to recover the intended workflow or approximate the paper.	Maintained separate branches for aliquot profiles, unique biological samples, corrected methods, and diagnostic interpretations.
Figure strategy	Produced a broad set of clean figures and kept visual progress moving across the paper.	Used a consistent R figure system with plot-source tables and SVG/PDF/TIFF/PNG exports, while also allowing intentionally blank or gated panels.
Reproducibility artifacts	Preserved the session, reviewer results, generated outputs, and final reproduction report.	Preserved scripts, frozen inputs, checksums, package cache, session information, orchestration plans, run logs, panel status, and export-integrity checks.
Main strength in this case	Strong execution momentum, recovery, and broad end-to-end coverage.	Strong method audit, explicit evidence boundaries, sensitivity analysis, and willingness to stop unsupported downstream work.
Main tradeoff in this case	Successful recovery could blur the boundary between reproduction and a scientifically useful substitute analysis.	The project became much heavier and less approachable than the interface most researchers would want to use directly.

What the Claude Science Run Proved

The Claude Science run was impressive for a straightforward reason: it kept a difficult scientific workflow moving.

The agent parsed the paper, planned the reproduction, set up R and conda environments, installed packages, fetched public data, ran differential expression, clustering, pathway, immune, drug-sensitivity, LASSO, Cox, nomogram, calibration, and decision-curve analyses, and generated a report. Each task could be checked by Claude Science's built-in reviewer. Failed steps were retried, repaired, or replaced; unresolved problems were moved into the limitations section.

This matters. A great deal of scientific work lives between the method paragraph and the final plot: dependency conflicts, unavailable services, inconsistent identifiers, long downloads, failed APIs, missing metadata, and code that works only after several rounds of debugging. Claude Science showed that a strong model inside a persistent scientific harness could work through much of that messy middle.

The run also revealed a particular default behavior. When a tool was unavailable or an analysis could not be executed as written, Claude usually tried to keep the downstream workflow moving by retrying the step, repairing the environment, or choosing a nearby substitute method. That is a valuable agent capability, and in many engineering tasks it is exactly what we want.

The harder question is whether the result still means the same thing after the agent changes the method to keep the analysis moving.

How Does GPT-5.6 sol Interpret "Reproduce the Paper" Differently?

The GPT-5.6 sol run began differently, even though I had not changed the scientific target.

Before running the large analyses, it split the paper into an exact-method audit, a figure-to-analysis map, and an availability check for public data, supplements, code, and historical database outputs. During that initial pass, it flagged several problems in the paper itself: DESeq2 was described after conversion to TPM, the stated GSEA gene-set-size limit was incompatible with the gene sets reported in the supplement, and several survival hazard ratios conflicted with their confidence intervals or the direction of the plotted curves.

It then made a decision I had not explicitly requested. It would preserve the published workflow where possible, but it would also run corrected sensitivity analyses rather than silently reproducing a likely methodological error.

That decision gradually became project structure. GPT-5.6 sol created a methods-reproduction matrix, a reproducibility risk register, figure contracts, frozen input manifests, panel-level status records, export-integrity checks, and an explicit completion audit. The resulting project separated five kinds of output:

Status	Meaning in the reproduction
Exact or frozen evidence	The source object was present in the paper, supplement, or a frozen input and could be checked directly.
Current reconstruction	The analysis could be recomputed with current public data, but could not be presented as the paper-time state.
Near-reproduction or approximation	The direction or numerical result was close, but cohort, snapshot, or method differences remained.
Evidence boundary	Some labels or structure could be preserved, but the numerical analysis could not be independently regenerated.
Gated	Required inputs were missing, the method was unspecified, or a statistical assumption failed, so the downstream result was not generated.

I had not put these categories in the prompt. They emerged from how GPT-5.6 sol decided what could honestly be described as reproduced.

When the DEG Count Did Not Match

The clearest difference appeared in the differential-expression analysis. The paper reported using |log2FC| > 0.5 and padj < 0.05, but that stated threshold did not reproduce the reported 3,492 differentially expressed genes.

In the Claude Science run, the agent inferred that the paper may have omitted part of its preprocessing and increased the threshold to |log2FC| > 2 so the resulting gene count would be closer to the published number. I can understand the reasoning. When a method is incomplete, trying plausible alternatives is often the only way to learn what the authors may have done.

But that move changes the optimization target. The workflow is no longer asking, “What does the stated method produce?” It is asking, “What method gets us closer to the stated result?” Those are different reproduction claims.

GPT-5.6 sol did not tune the threshold toward 3,492. It ran a statistically valid raw-count DESeq2 analysis and obtained 20,156 significant rows. It also ran sensitivity branches using a corrected unique-sample count matrix and a diagnostic rounded-TPM interpretation; neither reproduced the published count. Instead of treating that mismatch as an obstacle to remove, the project treated it as a result to preserve.

The supplemental files revealed another issue. The official lists contained 2,002 drug-resistance genes and 495 mitochondrial-related genes. Given their overlap, the DRG-only region in Figure 1B should be 1,656, not the printed 1,669. That discrepancy could not be repaired by changing an analysis parameter because it was an arithmetic inconsistency inside the published figure.

In the first run, the workflow changed the method to get closer to the paper. In the second, the mismatch itself became part of the reproduction.

Recovery and Substitution Are Not the Same as Reproduction

The second difference appeared when the original tool was unavailable.

The paper used pRRophetic for drug-sensitivity prediction. In the Claude Science run, pRRophetic could not be obtained, so Claude replaced it with oncoPredict, a related package from the same group, and continued the analysis. That was a reasonable recovery strategy, and it produced a useful exploratory result. It was also no longer a reproduction of the original computational method.

GPT-5.6 sol found the official historical pRRophetic 0.5 archive, a file of roughly 493 MB, but the transfer remained incomplete. The project explicitly excluded the partial archive from analysis. It preserved the paper's 20 drug labels and qualitative claims, but marked the numerical panels as an evidence boundary instead of filling them with predictions from another package.

This does not mean substitution is always wrong. A substitute can be scientifically useful, especially when the goal is exploration rather than historical reproduction. The workflow simply needs to answer three questions: does the substitute measure the same biological concept, how much does it change the claim, and should the output still be called a reproduction?

TIDE exposed a related but different problem. Claude abandoned the analysis because the original web service was unavailable, even though a Python route may still have existed. That suggests a tool-discovery gap. GPT-5.6 sol took a different position: even if a current implementation were available, the paper had not preserved the version, normalized input, submission settings, or sample-level export needed to reconstruct the paper-time result. That is a provenance gap.

One failure is “the agent did not find a usable implementation.” The other is “a usable implementation today would still not recreate the historical analysis.” A scientific workflow should know the difference.

Some Figures Should Not Be Generated

The strongest contrast came from the clinical model. In the Claude Science run, the agent generated a nomogram, calibration curve, and decision-curve analysis. The outputs looked clean, but the path to them contained unresolved issues: TNM variables were extracted but not used in the Cox model, ER and PR were not included as in the paper, and the DCA was shown only at three years without a preserved reason for that choice.

Claude Science did have a reviewer. The problem was not the absence of review. In this run, review mostly happened after a task had been attempted: pass it, retry it, find a substitute, or record a limitation and continue.

GPT-5.6 sol reconstructed the declared multivariable Cox model and tested its assumptions before producing the downstream clinical artifacts. The global proportional-hazards test failed with p = 0.000436. The project therefore gated Figure 14C-E: no nomogram, no calibration plot, and no DCA from that invalid model. Figure 14A/B remained available, but the visually persuasive downstream products were not generated.

On a completion dashboard, the second reproduction looks less complete. Scientifically, it is more informative because it preserves the reason those figures should not exist.

That changed how I think about scientific-agent output. A clean figure is not a neutral artifact. A survival curve looks authoritative. A nomogram looks clinical. A DCA curve looks decision-ready. Once generated, the visual form itself can lend confidence to an analysis whose assumptions have already failed.

Sometimes the correct output is not another plot. It is a gate with a reason.

What Reproduced, What Changed, and What Could Not Be Resolved

It would be easy to tell this story as “GPT found more problems,” but that would miss an important part of the reproduction.

Using a frozen current cBioPortal cohort, the second run analyzed 975 patients, with 331 in the altered group and 644 in the unaltered group. The reconstructed result was HR 0.519 (95% CI 0.324–0.831) with a log-rank p = 0.00545. The paper reported 335 altered and 642 unaltered patients, HR 0.526 (95% CI 0.318–0.869), and p = 0.00618.

Elsewhere, the differences remained meaningful. Consensus clustering produced groups of 774 and 344 rather than the reported 865 and 253; the paper had not released assignments, preprocessing details, or the random seed, so the project did not force the published sizes. The paper reported GSEA with maxGSSize = 10, while its own supplementary table contained retained gene sets of size 31 to 436, so the published parameter and output could not both be correct. A current ESTIMATE reconstruction produced immune-related directions that conflicted with the paper, and the same directions persisted in the unique-sample sensitivity branch. Several Figure 13 HR, CI, p-value, and curve annotations were internally inconsistent, so the project recomputed mutually consistent statistics instead of copying them.

These are different outcomes: close reproduction, current-data conflict, internal paper contradiction, missing historical state, and unsupported downstream model. They should not collapse into one label called “partially reproduced.”

The Review Process Needs Explicit Evidence and Stopping Rules

One lesson from the comparison is that adding an AI review step is not enough by itself.

Claude Science already has an AI reviewer in the agent loop. That is a real strength. It can catch failed tasks, trigger retries, and push unresolved issues into the report. GPT-5.6 sol in Codex did something different in this run: it created explicit objects for the review process to act on. Required inputs, assumptions, evidence classes, sensitivity branches, figure contracts, and stop conditions became part of the project before the final report was written.

The difference is not simply that one review model was smarter. The quality of the review process depends on the model's judgment, the criteria it is evaluating, the evidence it can inspect, and whether it has the power to prevent a downstream artifact from being generated.

This is why the same user goal can lead to different review policies. One system can review whether a requested task completed successfully. Another can first ask whether that task remains scientifically valid under the available evidence.

GPT-5.6 Sol + Codex Is Not the Final Answer Either

The second project was more explicit, but it was also much heavier.

It left behind dozens of scripts, cached packages, frozen inputs, manifests, contracts, status matrices, logs, diagnostic figures, and gated outputs. That is useful for auditability, but it is not the interface most researchers want to face. A biologist should not need to understand a directory tree and several status enumerations just to learn whether a result can be trusted.

The important decisions also still required judgment. The workflow had to understand why TPM-to-DESeq2 is problematic, whether oncoPredict can substitute for pRRophetic, whether a proportional-hazards violation should stop the downstream model, and how a current database snapshot differs from a historical reproduction. Structure can preserve and test those judgments; it cannot remove the need for scientific reasoning.

And again, this was one paper and two agent stacks, not a controlled benchmark. The result does not prove that GPT-5.6 sol will always be more conservative, or that Claude will always prioritize completion. Different harnesses, tool access, data snapshots, and future versions could produce different behavior.

What this run does show is that a strong model can create a much more rigorous evidence structure even when the user only asks it to reproduce the analyses. That capability is valuable, but relying on the model to reinvent the structure every time is not a product strategy.

What This Changes for Open Science

This comparison has become useful while we are building open science, which is still a work in progress. We are planning to build an interface allowing users to access GPT-5.6 sol as the reasoning model inside open-science and build the scientific workflow around it.

Researchers should not need to discover the right prompt for getting an evidence-aware reproduction. “Reproduce all analyses” should not sometimes mean “keep going until every figure exists” and sometimes mean “audit the method, preserve conflicts, and stop when the evidence ends,” depending on which model happens to interpret the request.

GPT-5.6 sol showed that a strong model can notice methodological contradictions, design sensitivity analyses, distinguish a current reconstruction from a historical reproduction, and decide that a downstream figure should be gated. Those are exactly the high-judgment tasks where I want the model to remain involved. But it should not have to reconstruct the surrounding evidence policy from scratch in every session.

That policy should be native to open science.

For each analysis node, the system should know the required inputs, method assumptions, expected artifacts, sensitivity checks, common failure modes, and stop conditions. Exact, directional, approximate, substituted, unsupported, and gated should be first-class result states rather than adjectives added by the model near the end of a report. A reviewer should be able to block a downstream artifact when its statistical assumptions fail. The original claim, executed method, substitution, mismatch, and final claim boundary should remain connected in one inspectable record.

The division of work I now have in mind is fairly simple. GPT-5.6 sol can handle the flexible scientific judgment: reading incomplete methods, noticing contradictions, choosing reasonable sensitivity analyses, interpreting mismatches, and calibrating the claim. Open-science can provide the persistent layer around that judgment: reusable bioinformatics nodes, provenance, evidence states, review contracts, artifacts, and handoff between sessions.

In other words, the lesson is not that workflow matters more than the model. This reproduction worked because GPT-5.6 sol made strong scientific decisions. The opportunity for open-science is to make those decisions easier to review, reuse, and compound across many papers instead of leaving them inside one successful Codex session.

That is the layer we are exploring with open-science: a model-agnostic scientific workflow that can route high-judgment tasks to stronger models such as GPT-5.6 sol, while preserving how each decision became a result and where the evidence ends.

Conclusion

I started this because GPT-5.6 sol had just launched and I wanted to try it on a real task. I expected a comparison of speed, coding style, and execution quality.

What I got instead was a disagreement over what it means to reproduce a paper faithfully.

Claude Science showed me how far an agent could keep a difficult workflow moving. GPT-5.6 sol in Codex showed me that sometimes the scientifically correct result is a mismatch, a downgrade, or no figure at all.

A useful scientific agent should know how to finish an analysis. A trustworthy one also needs to know when finishing it would make the result worse.