A large benchmark gain invites a smaller, more revealing question: which problems improved? In AIPOCH Open-Science, a public Weight-Space Programming case opens that question through a .science package containing saved inputs, analysis, figures and verification records.
The immediate backdrop is the researchers’ November 2026 report: GPT-6 Astra edited 131,072 weights in Qwen3.5-0.8B, approximately 0.016% of the model, and the reported MATH-500 score rose from 7.9% to 48.1% with a budget of 50 forward passes. The experiment allowed practice problems and bounded forward-pass feedback, but no gradient training. “No training” therefore does not mean “no examples or feedback.”
The Open-Science case investigates the saved score records, rather than repeating the weight-editing experiment. Here, a problem improves when more of its four saved answers carry a correct flag after the edit. Of the 25 shared problems, 18 record a higher count, seven stay at zero, and none record a lower count.
A fair comparison starts with the same problems
The Weight-Space Programming case supports a comparison of 25 shared math problems because the available baseline covers only a small subset of the edited model’s records, making matching by exact problem ID essential before an average can describe a change in performance across the same questions.
The Hugging Face rerun files (2026) contain 25 baseline problems and 500 edited-model problems, with four correctness flags per problem in each file. The case’s inclusion analysis (2026) retains 475 edited-only problems with missing baseline values, rather than assigning invented before-and-after changes.
Start with inclusion_table.csv in the package. The table makes the denominator visible: all 25 baseline IDs have edited counterparts, producing 100 recorded samples per version for the paired analysis. Missing baseline records remain missing, not zero.
Next, open paired_problem_counts.csv. For each problem, the workflow adds four supplied binary grader flags and subtracts the baseline count from the edited count. A change of +3 means three additional flags marked correct among four saved samples; the change does not identify three matched generation trajectories.
| Quantity | Value | What the number describes |
|---|---|---|
| Authors’ original experiment | 48.1% | Reported edited-model MATH-500 score with 50 forward passes |
| Hugging Face rerun, complete edited file | 51.0% | 1,020 correct flags across 500 problems × 4 samples (2,000 flags) |
| Matched case subset | 47% | 47 correct flags among 100 samples on 25 shared problems |
| Baseline on the same subset | 5% | Five correct flags among 100 samples on the same 25 problems |
On the same 25 problems, the count rises from 5/100 to 47/100, a gain of 42 percentage points. The original experiment and the full rerun both cover MATH-500, but their scores come from separate evaluations. The records do not establish why the scores differ. The 25 IDs come from the rerun publisher’s baseline smoke test; their representativeness for all 500 problems is unknown.
Original case output: paired_problem_dumbbell.png, with AIPOCH interpretation outside the chart. Each row preserves one problem-level comparison; rings and dots at zero retain the seven ties.
Eighteen improvements still leave seven zero-to-zero ties
The paired Weight-Space Programming results show gains across many shared questions, but the distribution of correct-count changes reveals a more useful picture than the average alone: improvement varies from one to four additional correct flags, while seven problems receive zero correct flags in all four samples from each version.
The case’s change distribution (2026) records seven problems at +1, three at +2, three at +3 and five at +4. The edited score file (2026) gives an edited count of at least one on 18 of the shared problems and a count of four on eight.
The distribution explains exactly where the average gain comes from:
7 × 1 + 3 × 2 + 3 × 3 + 5 × 4 = 42 additional correct flags.
The five problems with the largest gain contribute 20 additional correct flags. The seven with the smallest positive gain contribute seven more. The seven zero-to-zero ties contribute none. Open correct_count_change_distribution.csv to follow the counts back to their problem IDs.
Original case output: correct_count_change_distribution.png. Empty negative-change bins remain visible. No observed decline among 25 paired IDs does not establish an absence of side effects on 475 unpaired IDs.
The coverage figure asks a complementary question: how demanding should the recorded-success threshold be?
| Required correct samples per problem | Baseline | Edited |
|---|---|---|
| At least one of four | 3/25 (12%) | 18/25 (72%) |
| At least two of four | 1/25 (4%) | 12/25 (48%) |
| At least three of four | 1/25 (4%) | 9/25 (36%) |
| All four | 0/25 (0%) | 8/25 (32%) |
Original case output: recorded_coverage_thresholds.png. Each threshold counts problems meeting a different requirement among the same four saved answers. These descriptive rates do not establish the probability of success on a future attempt.
A reader interested in occasional success may start with the first threshold. A reader interested in consistency across the four saved attempts should inspect the last. Neither threshold establishes reliability on new questions, and even four correct flags do not replace independent review of the underlying answers.
AIPOCH Open-Science connects the headline to inspectable evidence
The AIPOCH Open-Science package makes the Weight-Space Programming case useful as an evidence-reading exercise by keeping inclusion decisions, numerical outputs and response-review limits together, so readers can follow a particular problem through the saved analysis without mistaking an imported record for a newly executed experiment.
The published case (2026) identifies verified_delivery.zip as the final bundle containing five original inputs, 11 CSV tables, six figures and a rebuild script. The official research-package guide (verified 2026) explains that imported sessions are read-only and received verification records describe the sender’s checks, rather than proving a rerun on the receiving computer.
A useful first question is: what remains unknown about a problem whose count stayed at zero? Follow test/algebra/1349.json from paired_problem_counts.csv into response_evidence_review.csv and response_review_decisions.json. The count establishes zero correct flags in both versions. The response review adds recorded observations about repetition and a path-interpretation assumption, while keeping possible explanations separate from evidence.
That distinction matters. The attachment contains six response texts across only three problem IDs. The saved text is not securely linked to a particular sample index in the correctness arrays; stop reasons and generated-token counts are also unavailable. Repetition alone cannot demonstrate a token-limit hit or establish why a weight edit changed a score. The case does not independently prove the original researchers’ proposed loop-repair mechanism.
For a compact reading route:
- Check inclusion:
inclusion_table.csvshows which problems can be compared. - Trace the result:
paired_problem_counts.csvandcorrect_count_change_distribution.csvconnect averages to individual IDs. - Inspect uncertainty:
response_review_decisions.jsonrecords observations, interpretations and gaps. - Read the recorded checks:
fresh_process_verification.jsoninsideverified_delivery.zipdescribes the saved rebuild verification.
The archived verification reports matching CSV outputs and byte-identical regenerated figures. Those checks concern consistency of the analysis and delivery, not independent correctness labels or a causal explanation of model behavior. For this article, the package was inspected statically and the matched totals were recalculated from supplied arrays; neither model inference nor the archived rebuild script was run.
Open the public case, download its .science package, and import the package into AIPOCH Open-Science using the documented import flow. Start with one unchanged problem and follow the evidence as far as the record permits.
Is Weight-Space Programming the same as fine-tuning?
In the researchers’ experiment, a language model directly edited weights without gradient descent or optimizer fitting. Practice examples and forward-pass feedback were allowed. The Open-Science case analyzes saved outcomes of that work; the case does not perform a new weight edit.
Does zero observed decline mean the edit has no downside?
No. The result covers 25 matched problems. The other 475 edited-model problems have no baseline counterpart in the supplied files, and the saved records do not establish broader side effects or general mathematical improvement.
Will importing the package rerun the experiment?
No. A .science package can carry selected conversation branches, file versions, Notebook records and verification evidence. Included files depend on export selection; imported history is read-only and does not execute code or restore credentials. The official guide describes how to use Fork in a compatible desktop app to create a writable session. Side Chat conversations and private reading bookmarks are excluded.