I think the result itself is quite meaningful:
The short version is:
- I would not treat the +5.5 points as “nothing.” The checkpoint clearly changed a lot.
- I also would not yet read it as “GPQA reasoning improved by 5.5 points.” There is a fairly large answer-selection shift mixed into the result.
- Before spending more CPU time, I think the highest-information next step is actually very cheap: extract a few aggregate counts from the logs you already have.
- I would still continue with the stock-Llama + safer-LR experiment. I would just treat that as a new recipe branch, separate from the question of what caused the old Dolphin behavior.
1. About the 24.7% base score
I would not assume that 24.7% is necessarily the canonical Dolphin 3.0 GPQA-Diamond baseline.
There is a useful historical reference point: the public Open LLM Leaderboard detailed run for cognitivecomputations/Dolphin3.0-Llama3.1-8B recorded 0.32828 on leaderboard_gpqa_diamond, i.e. about 32.83% on the 198 Diamond questions:
Open LLM Leaderboard — Dolphin3.0-Llama3.1-8B detailed results
That does not mean your 24.7% is wrong.
It is not an apples-to-apples reproduction. The leaderboard run is from an older evaluation stack, while your run reports gpqa_diamond_zeroshot task version 2.2. GPQA’s lm-eval task has changed since then; in particular, v2.2 removed an older answer-preprocessing regex that could damage valid bracketed answer text:
lm-eval GPQA task changelog
The current zeroshot contract is here:
lm-eval GPQA zeroshot task
Also, the GPQA 4.36 number visible on the Dolphin model card is easy to misread: that is a leaderboard-normalized score, where the four-choice random baseline is mapped to zero. It is not “4.36% raw Diamond accuracy.” Hugging Face documents that normalization separately:
Open LLM Leaderboard score normalization
So if somebody wants to reproduce your 24.7%, I would try to match the evaluation contract before focusing on the CPU model:
same exact model weights / revision
same tokenizer
same lm-eval revision
same GPQA task version
same chat-template setting
same processed answer permutation
same dtype / backend / batch settings
then compare hardware
Hardware can affect exact reproducibility. PyTorch explicitly says that complete reproducibility is not guaranteed across releases/platforms, or even CPU versus GPU with the same seeds:
PyTorch reproducibility notes
So I would not say “hardware cannot matter.”
I just would not make the recycled CPU the first explanation for the pattern you are seeing, especially because the base and checkpoint were evaluated in the same local environment.
2. Is +5.5 points significant, or does it mean nothing?
I think there are actually three different meanings of “significant” here.
A. Did fine-tuning materially change the model’s behavior?
Yes. Very clearly.
The interesting part is not only:
Base: 49 / 198 correct = 24.7%
Checkpoint: 60 / 198 correct = 30.3%
Net: +11 correct
The prediction distribution also moved from:
Base Checkpoint
A 40 115
B 20 13
C 61 30
D 77 40
So A increased by 75 predictions.
That alone proves that at least 75 items moved from a non-A prediction to A. There is no way to obtain a net +75 in A without at least 75 such transitions.
You also found that at least 95/198 top-1 answer labels changed, plus 46 wrong → different-wrong cases.
So this was not a tiny perturbation that happened to flip eleven borderline questions. The model’s multiple-choice decision behavior moved substantially.
B. Is +11 correct on 198 questions already strong statistical evidence of better accuracy?
I would call it suggestive, but not strong by itself.
Because the same 198 questions were evaluated before and after tuning, the useful comparison is paired.
From the paired results:
wrong → right: 30
right → wrong: 19
An exact two-sided McNemar test on those discordant pairs gives roughly p = 0.15.
That is not a magic verdict, and “not below 0.05” absolutely does not mean “there is no effect.” It just says that, on a 198-item benchmark, 30 gains versus 19 losses is not yet especially strong evidence if the claim is specifically “general accuracy increased.”
For reference:
statsmodels McNemar test documentation
So I would distinguish:
The model definitely changed.
from:
We have already established a robust +5.5-point improvement in general reasoning.
The first statement looks strong to me. The second still needs a little decomposition.
C. Does this demonstrate +5.5 points of better reasoning?
That is the part I think is still open.
At the moment I would keep at least three mechanisms on the table:
real content-sensitive improvement
+
answer-label / option-position selection shift
+
broader fine-tuning drift or forgetting
They are not mutually exclusive.
The result could contain some of all three.
3. The A = 40 → 115 movement is probably the highest-value clue
This is the part I would investigate before doing another long full fine-tune.
There is prior work showing that LLMs can develop preferences for the option IDs themselves — A/B/C/D — and that moving answer contents between positions can change MCQ performance.
A particularly relevant paper is:
Large Language Models Are Not Robust Multiple Choice Selectors — ICLR 2024
They call this selection bias: the model can assign different prior preference to option tokens such as A/B/C/D independently of the semantic answer contents.
There is also SFT-specific work showing that multiple-choice symbol-selection bias can persist during supervised fine-tuning:
Strengthened Symbol Binding Makes Large Language Models Reliable Multiple-Choice Selectors — ACL 2024
I would treat those papers as evidence that this is a real class of failure mode — not as evidence that they have already explained your checkpoint.
Your A increase could still partly reflect genuine improvement if many of those extra A predictions happened on questions whose correct target was A.
That is why the next tiny diagnostic is so useful.
4. Before another training run, I would extract only these aggregates
No GPQA questions, answer text, prompts, or raw logs need to be posted.
The GPQA dataset card specifically asks users not to reveal benchmark examples online, so aggregate statistics are a nice way to investigate this without exposing the benchmark:
GPQA dataset card
First, I would check whether the processed target letters were identical between the base and checkpoint evaluations:
base/checkpoint target-letter mismatches: ? / 198
Ideally that is:
0 / 198
Then just print:
Gold labels:
A = ?
B = ?
C = ?
D = ?
and two 4×4 tables:
Base: target × prediction
Pred A Pred B Pred C Pred D
Gold A
Gold B
Gold C
Gold D
Checkpoint: target × prediction
Pred A Pred B Pred C Pred D
Gold A
Gold B
Gold C
Gold D
That is probably the cheapest high-information diagnostic available now.
No new training required.
No new model download required.
No public GPQA examples required.
From those 32 cells you can immediately see whether the A increase is mostly:
- correct movement toward gold A,
- incorrect movement from gold B/C/D into A,
- or a mixture.
How I would read those two matrices
Case 1 — B/C/D gold rows all start flowing into A
For example, if all three of these jump:
P(pred=A | gold=B)
P(pred=A | gold=C)
P(pred=A | gold=D)
then I would take the global A-selection explanation much more seriously.
That would mean the A = 115 total is not just the consequence of a gold distribution that happens to contain many A answers.
At that point, raw accuracy alone would hide quite a lot of internal movement.
Per-gold-label recall would be more informative:
Recall(A)
Recall(B)
Recall(C)
Recall(D)
Case 2 — most of the added A predictions are on gold-A questions
That would weaken the simple “the model just likes A” interpretation.
The marginal A count could look alarming while actually containing a substantial amount of genuine correction.
In that case I would put more weight on real content-sensitive improvement.
Case 3 — all four recalls improve, but A preference also increases
This would be a perfectly plausible mixed result:
real GPQA improvement
+
a newly exposed/amplified answer-label prior
Those can coexist.
An unwanted selector effect does not automatically invalidate every improvement produced by the checkpoint.
Case 4 — A recall improves while B/C/D recall collapse
Then I would be much more cautious about interpreting the headline 30.3% as broad reasoning improvement.
The aggregate score could be hiding a fairly large tradeoff between classes.
This is also why the ICLR selection-bias work looks at behavior conditioned on the correct option rather than only counting how often each option was predicted.
5. I would keep your new stock-Llama experiment
I do not think the findings above are a reason to stop the project or throw away the next experiment.
I would just separate two questions that are easy to accidentally merge.
Branch A — recipe development
stock Llama 3.1 8B
+ safer/progressive LR
+ your Spanish data
Question:
"Can we build a more stable/better recipe?"
That seems completely reasonable to continue.
But there is a different question:
Branch B — causal diagnosis of the old Dolphin result
same Dolphin base
same data
same order
same optimizer/batch/eval contract
change only LR/schedule
Question:
"Did the aggressive update cause the old drift?"
Changing both the base model and the learning-rate strategy is useful for Branch A, but it cannot isolate Branch B.
That is not a problem as long as the two experiments are interpreted separately.
And given your CPU constraint, I would absolutely not say that Branch B is mandatory. If the practical goal is learning how to make a good model rather than writing a causal study of the old checkpoint, it may not be worth repeating a long full fine-tune just to answer that historical question.
So my default order would be:
1. Existing logs: target consistency + 4×4 matrices
cost: almost zero
2. If A-selection still looks suspicious:
same checkpoint, controlled answer-order test
cost: evaluation only
3. Continue the new Llama + safer-LR recipe
cost: training
4. Only if you specifically care about proving the old LR mechanism:
same-Dolphin LR-only control
That gets the cheap information first without blocking the experiment you actually want to run.
About the learning-rate hypothesis
I think your suspicion about the learning rate is reasonable at the broad-drift level.
Fine-tuning can cause catastrophic forgetting or degradation of capabilities learned during pretraining. This is well documented in LLM tuning work:
Revisiting Catastrophic Forgetting in Large Language Model Tuning — Findings of EMNLP 2024
There is also recent work explicitly connecting learning-rate behavior with forgetting during fine-tuning:
Fine-Tuning Without Forgetting via Loss-Adaptive Learning Rates
So a statement like this seems reasonable:
An aggressive full-parameter update could plausibly cause broad drift/forgetting and expose or amplify behavior that was weak in the base model.
What I do not think the evidence supports yet is:
A high learning rate specifically caused the model to prefer Option A.
That second causal step is still missing.
As an engineering reference point only, torchtune’s current Llama 3.1 8B full-finetuning example uses AdamW with lr: 2e-5:
torchtune Llama 3.1 8B full-finetune configuration
I would not treat 2e-5 as a universal “correct” LR. Dataset size, effective batch size, sequence length, optimizer, schedule, model state, and training objective all matter.
It is just a useful reminder that full-parameter post-training recipes usually treat LR as a sensitive recipe choice rather than something with one universally safe number.
If you want to test the Option-A hypothesis directly
After the 4×4 matrices, the cleanest test is conceptually simple:
Keep the checkpoint fixed and move the same answer contents to different A/B/C/D positions.
If the semantic answer stays the same but the chosen answer changes substantially with the letter/position, that is much more direct evidence of an option-ID effect than comparing two separately trained models.
You do not necessarily need all 24 possible permutations.
A cheap smoke test could use two controlled arrangements.
A stronger version could use four cyclic arrangements so that each semantic answer content occupies A, B, C, and D once.
The ICLR paper above uses answer-content permutation as exactly this kind of diagnostic/debiasing tool.
One lm-eval / Datasets gotcha
I would be careful about implementing this by changing only the random seed.
The GPQA preprocessing in lm-eval shuffles the choices, and Hugging Face Datasets can cache the result of a Dataset.map() transform.
The Datasets documentation explicitly says that previous transforms may be loaded from cache and that load_from_cache_file=False or disabling caching forces the mapping function to run again:
Hugging Face Datasets — cache management
So this:
same checkpoint + seed 0
same checkpoint + seed 1
is not by itself proof that you actually evaluated two different answer permutations.
A safer contract would be:
same checkpoint
same lm-eval/task revision
same scoring
same prompt/template
force GPQA preprocessing to recompute
change only the answer permutation
then verify:
the processed target mapping really changed
Even better, if you write a tiny local task variant, make the permutation a deterministic function of something like:
(item_id, permutation_seed)
rather than relying on global RNG state.
This is only a warning for a future permutation experiment. It is not a claim that Datasets caching caused the A=115 behavior in your current checkpoint.
A few smaller interpretation notes
I would call the current observation a selection/decision shift before calling it calibration
In the strict ML sense, calibration is about whether predicted confidence corresponds to empirical correctness probability.
Right now the strong evidence is:
- changed top-1 choices,
- changed answer-label distribution,
- large concentration on A.
So terms like:
- selection shift,
- option-ID preference,
- decision-distribution shift,
are a little safer than “calibration failure” until you also inspect choice probabilities/margins.
Calibration itself can be tested later, but I would put that below the 4×4 table in priority.
I would put direct GPQA leakage fairly low on the list for this particular fine-tuning corpus
From your description, the Spanish training material was created independently and was not assembled from GPQA/MMLU science questions.
That does not mathematically prove zero overlap, and it says nothing about what may have existed in the original model’s pretraining/tuning data, but direct leakage from this new corpus would not be my first explanation for the A=40→115 movement.
The selector/evaluation decomposition is cheaper to check first.
I would also be cautious with the moral_disputes drop
A subject-level MMLU decrease can be useful evidence that the model changed, but I would not use it by itself as evidence that safety constraints were “removed.”
MMLU is also multiple choice, so some of the movement could in principle share the same selector/option-ID effects seen in GPQA.
If that particular result becomes important, the same cheap decomposition — gold-label distribution, prediction-label distribution, per-label recall — would be useful there too.
Where this leaves the original result
My current interpretation would be:
24.7% → 30.3% is worth investigating and worth continuing from.
It is not just meaningless benchmark noise, because the checkpoint’s behavior changed very substantially.
At the same time, I would not yet compress everything that happened into the sentence:
“The model gained 5.5 points of GPQA reasoning ability.”
The much more interesting signal right now may be the A-direction redistribution.
Fortunately, determining whether that is mostly:
real improvement
selection bias
forgetting/drift
or a mixture
does not require another multi-day CPU training run.
The target-consistency count, gold A/B/C/D counts, and two 4×4 matrices from the existing logs should already tell you a lot.
Then you can choose the branch that matches what you actually want to learn:
- want a better model? Continue the stock-Llama + safer-LR recipe.
- want to understand the A behavior? Do a fixed-checkpoint permutation test.
- want to prove whether LR caused the old Dolphin drift? Run the same-Dolphin LR-only control when/if the CPU cost is worth it.
That seems like a good place to keep the experimentation moving without throwing away the interesting result you already have.
: