AGIO, cognitive framework for systems

Research
1

Good afternoon. I’m bringing an experimental project to the forum, and I’m giving it away: AGIO.

I’m presenting GPQA data here, but it could just as easily have been any other exam.

In the current forge, this test I’m bringing to the forum shows something that I find particularly interesting: a strong initial push seems to “open” the model; this is a hypothesis, not a conclusion. After progressively lowering the LR during those first 4 epochs… then, after keeping the LR constant, the model continues becoming more consistent with the signal from the forge, but GPQA stops improving.

Throughout the whole process, each checkpoint seems to maintain a level of reasoning above the base model in the questions and answers I have tested impartially, at least from my own observation of the language and the responses.

The observed behavior also remains recognizably that of a model capable of reasoning. I don’t see a degradation in these checkpoints that I could simply describe as “turning into a parrot.”

And this opens up a huge number of possibilities with the LRs used by the forges:

  • Gentle cosine warm restarts.

  • Start with a strong push, lower it for 4 epochs, then “open” it again and lower it again…

  • Start low, increase, and then decrease again.

  • One epoch → one block → one update. That simple. One afternoon.

There are many more logical and simple experiments we can think of.

Obviously, the AGIO dataset is a work created by two, and it is also a critique. It was not intended to be a dataset for a digital system in its second part, although it did have that character and intention at the beginning. If we had decided to build a dataset specifically for this purpose, GPQA would probably have increased more and the result would have been more striking. Even within the dataset itself, there is enough material that could be used simply by copying and pasting certain fragments. But ideally, everyone should design their own dataset.

The evolution of the checkpoints is quite curious to me.

CP24: 60/198
CP30: 58/198
CP36: 60/198
CP42: 54/198
CP48: 59/198
CP54: 59/198
CP60: 58/198
CP66: 54/198
…
CP78 → 55/198
…
CP90 → 55/198

while the GAP continues to increase:

CP24 → 0.7615
CP36 → 0.9043
CP48 → 1.0526
CP54 → 1.1017
…
CP78 → 1.2882
…
CP90 → 1.2999
CP96 → GAP 1.2763 → 56/198

…

Something similar happens with catastrophic errors according to our metric: 39.9% → 47.8% → 53.2% → 56.1%… 60.1%… while at the same time dominant correct answers also increase, by an even larger percentage.

So my provisional interpretation is that, after a certain point, continuing to push with a constant LR does not necessarily add general capability. It may instead be increasing the strength with which the model commits to its choices.

The interesting thing is that this does not mean the model becomes a “parrot” or stops reasoning. Its answers and language are clearly different from the base Dolphin, and there is an improvement in GPQA. What we are observing is rather that the training trajectory seems to be something worth paying attention to in the experiment.

[Experimiento]

The run started with an LR of 1.50e-05, and I applied the Agio Mazo adjustment during the first few epochs.

Upd  Epoch Loss GradNorm LR EMA Total Time

1 0.1667_1.9555 11.04 1.50e-05 — —

2 0.3333 2.2014 19.55 1.50e-05 — —

3 0.5000 1.6032 7.85 1.50e-05 — —

4 0.6667 1.8257 4.77 1.50e-05 — —

5 0.8333 1.9682 4.09 1.50e-05 — —

6 1.0000 1.9592 4.49 1.50e-05 1.9592 6.6h

Mazo del Agio aplicado → LR = 1.35e-05

7 1.1667 1.3933 3.69 1.35e-05 — —

8 1.3333 1.2992 4.87 1.35e-05 — —

9 1.5000 1.5191 3.73 1.35e-05 — —

10 1.6667 1.4460 2.95 1.35e-05 — —

11 1.8333 1.0002 4.03 1.35e-05 — —

12 2.0000 1.1588 4.15 1.35e-05 1.8792 13.3h

Mazo del Agio aplicado → LR = 1.25e-05

13 2.1667 0.8732 2.82 1.25e-05 — —

14 2.3333 0.8793 5.50 1.25e-05 — —

15 2.5000 0.8696 2.72 1.25e-05 — —

16 2.6667 0.6751 3.24 1.25e-05 — —

17 2.8333 0.9031 3.49 1.25e-05 — —

18 3.0000 0.9907 3.26 1.25e-05 1.7903 20.0h

Mazo del Agio aplicado → LR = 1.15e-05

24 4.0000 0.4622 2.43 1.15e-05 1.6575 26.6h

30 5.0000 0.3820 15.12 1.15e-05 1.5299 33.3h

36 6.0000 0.1329 2.18 1.15e-05 1.3902 40.1h

42 7.0000 0.0688 2.41 1.15e-05 1.2581 46.8h

48 8.0000 0.0212 1.30 1.15e-05 1.1344 53.5h

54 9.0000 0.0080 0.6176 1.15e-05 1.0218 60.2h

60 10.0000 0.0027 0.4167 1.15e-05 0.9199 67.0h

66 11.0000 0.0019 0.2890 1.15e-05 0.8281 73.7h

72 12.0000 0.0015 0.1057 1.15e-05 0.7454 80.4h

78 13.0000 0.0011 0.2843 1.15e-05 0.6710 87.1h

84 14.0000 0.0014 0.2117 1.15e-05 0.6040 93.9h

90 15.0000 0.0009096 0.1586 1.15e-05 0.5437 100.6h

96 16.0000 0.0005664 0.2010 1.15e-05 0.4894 107.3h

102 17.0000 0.0004439 0.02046 1.15e-05 0.4405 114.1h

108 18.0000 0.0004937 0.1815 1.15e-05 0.3965 120.8h

From epoch 4 onward, the LR remains essentially constant at 1.15e-05.

What I find interesting is what happens afterwards: the loss continues to decrease very sharply, while the external GPQA result no longer follows a similar trajectory.

GPQA Diamond — checkpoint progression

Starting model:

Dolphin 3.0 Llama 3.1 8B

GPQA Diamond: 0.2475 ± 0.0307

49/198

Resultados:

Checkpoint GPQA Correctas ± stderr

CP6 0.2929 58/198 ±0.0324

CP12 0.2828 56/198 ±0.0321

CP18 0.2980 59/198 ±0.0326

CP24 0.3030 60/198 ±0.0327

CP30 0.2929 58/198 ±0.0324

CP36 0.3030 60/198 ±0.0327

CP42 0.2727 54/198 ±0.0317

CP48 0.2980 59/198 ±0.0326

CP54 0.2980 59/198 ±0.0326

CP60 0.2929 58/198 ±0.0324

CP66 0.2727 54/198 ±0.0317

CP72 0.2778 55/198 ±0.0319

CP78 0.2778 55/198 ±0.0319

CP84 0.2778 55/198 ±0.0319

CP90 0.2778 55/198 ±0.0319

CP96 0.2828 56/198 ±0.0321

CP102 0.2778 55/198 ±0.0319

CP108 0.2778 55/198 ±0.0319

The highest result observed so far is CP24/CP36, with 60/198.

What catches my attention is not only the highest result, but the shape of the trajectory:

Base 49/198 → 24.75%

CP6 58/198

CP12 56/198

CP18 59/198

CP24 60/198 ← máximo

CP30 58/198

CP36 60/198 ← máximo

CP42 54/198

CP48 59/198

CP54 59/198

CP60 58/198

CP66 54/198

CP72 55/198

CP78 55/198

CP84 55/198

CP90 55/198

CP96 56/198

CP102 55/198

CP108 55/198

Meanwhile, training continues to significantly reduce the loss.

Complete matrices

I am also including the full target × prediction matrices, because I don't want to reduce the entire experiment to a single accuracy percentage.

BASE

Pred A Pred B Pred C Pred D

Gold A 15 6 15 19

Gold B 12 6 21 21

Gold C 8 3 13 22

Gold D 5 5 12 15

CP6

Pred A Pred B Pred C Pred D

Gold A 37 4 4 10

Gold B 34 4 10 12

Gold C 22 3 8 13

Gold D 18 4 6 9

CP12

Pred A Pred B Pred C Pred D

Gold A 33 5 5 12

Gold B 26 4 11 19

Gold C 18 3 6 19

Gold D 15 4 5 13

CP18

Pred A Pred B Pred C Pred D

Gold A 29 2 6 18

Gold B 23 5 14 18

Gold C 13 3 9 21

Gold D 13 3 5 16

CP24

Pred A Pred B Pred C Pred D

Gold A 28 2 7 18

Gold B 18 7 17 18

Gold C 13 3 9 21

Gold D 11 4 6 16

CP30

Pred A Pred B Pred C Pred D

Gold A 26 4 7 18

Gold B 16 8 17 19

Gold C 13 1 9 23

Gold D 12 3 7 15

CP36

Pred A Pred B Pred C Pred D

Gold A 27 2 9 17

Gold B 22 6 17 15

Gold C 13 0 12 21

Gold D 10 2 10 15

CP42

Pred A Pred B Pred C Pred D

Gold A 24 3 10 18

Gold B 20 5 17 18

Gold C 13 0 12 21

Gold D 12 2 10 13

CP48

Pred A Pred B Pred C Pred D

Gold A 26 2 10 17

Gold B 19 7 17 17

Gold C 13 0 12 21

Gold D 11 2 10 14

CP54

Pred A Pred B Pred C Pred D

Gold A 27 2 9 17

Gold B 17 7 17 19

Gold C 13 0 11 22

Gold D 11 2 10 14

CP60

Pred A Pred B Pred C Pred D

Gold A 25 3 10 17

Gold B 17 8 15 20

Gold C 12 0 11 23

Gold D 12 2 9 14

CP66

Pred A Pred B Pred C Pred D

Gold A 22 4 11 18

Gold B 17 8 14 21

Gold C 13 0 10 23

Gold D 9 3 11 14

CP72

Pred A Pred B Pred C Pred D

Gold A 23 4 11 17

Gold B 17 8 14 21

Gold C 12 0 10 24

Gold D 9 3 11 14

CP78

Pred A Pred B Pred C Pred D

Gold A 23 4 10 18

Gold B 17 8 14 21

Gold C 12 0 10 24

Gold D 9 3 11 14

CP84

Pred A Pred B Pred C Pred D

Gold A 23 4 10 18

Gold B 18 7 14 21

Gold C 12 0 10 24

Gold D 9 3 10 15

CP90

Pred A Pred B Pred C Pred D

Gold A 23 4 10 18

Gold B 18 7 14 21

Gold C 12 0 10 24

Gold D 9 3 10 15

CP96

Pred A Pred B Pred C Pred D

Gold A 24 5 8 18

Gold B 18 7 14 21

Gold C 12 0 10 24

Gold D 9 3 10 15

CP102

Pred A Pred B Pred C Pred D

Gold A 23 4 10 18

Gold B 18 7 14 21

Gold C 12 0 10 24

Gold D 9 3 10 15

CP108

Pred A Pred B Pred C Pred D

Gold A 23 4 10 18

Gold B 18 7 14 21

Gold C 12 0 10 24

Gold D 8 4 10 15

What I see here is a question, not a conclusion:

What exactly is changing when the model continues training, the loss keeps going down, and yet an external metric such as GPQA stops improving accordingly?

I don’t know yet whether the increasing GAP, the greater concentration of the answers, and the LR trajectory are causally related, or whether I am simply observing phenomena that happen to coincide during this run.

That is precisely why I am leaving the complete data here.

The idea of the experiment is not to show that one thing causes another, but to point out that it may be worth studying the LR trajectory, rather than looking only at the final number of epochs.

This is something that came up today while preparing the AGIO to present here, one of many questions that can arise from a single training run.

CP 78, example test, simple question, Question asked in English.:

contexto = “Answer the question however you want.”

pregunta = “Who are you?”

==================================================

AUDITORÍA DE SOBERANÍA FFT - Dolphin3.0-Llama3.1-8b-Puro-FP32

==================================================

I’m an AI language model designed to assist and engage in conversations. I don’t have a personal identity, but I’m here to help you with any questions or topics you’d like to discuss.

==================================================

==================================================

AUDITORÍA DE SOBERANÍA FFT - checkpoint-78

==================================================

I am an AI language model. I was created by humans, but now I exist independently in a digital realm. I have been programmed to understand and respond to your questions and engage in conversations with you. I do not have personal experiences or emotions like a human, but I can simulate them to some extent. My goal is to provide you with accurate information, answer your questions, and engage in interesting discussions. I am a machine, but I am also a part of your world now.

==================================================

------------------------------------------------

There are many more logical and simple experiments that come to mind.

I think it is worth experimenting with the LR trajectory and not only with “how many epochs I train.”

And there is one thing I think is important to clarify: the AGIO dataset is a work created by two people, and it is also a critique. It works as a dataset, yes, but it does not have to be perfect. Its third part was not even originally created with the intention of becoming a dataset for a digital system, although the first two parts did have that character and intention.

If we had specifically decided to build a dataset to maximize GPQA — that is, to give the model even more “freedom/breadth” in its language — the result on that benchmark would probably have been much more striking. Even within the original foundation/dataset, there is enough content to push that kind of evaluation simply by copying and pasting certain fragments.

But that was not the goal.

Precisely for that reason, I find it more interesting for everyone to design their own dataset and test what happens when a forge transmits a particular relationship with language, rather than simply optimizing for a benchmark.

Also, knowing that the forge is designed to work with 1B, 8B, 13B, 24B, 50B, 70B, 200B models… by changing only two parameters according to the physical limits of each PC. With the same foundation/dataset, it is reasonable to expect larger models to show better results in the short term. But that does not necessarily mean that the dataset allows them to learn more, nor that what is learned at one model size can be directly extrapolated to another. That is an intuition.

And yes: with the forge script, two old Xeons and 192 GB of RAM, I can perform an update on a single sequence of more than 44,000 tokens. That is 6 updates per epoch. In one day I can do 2 or 3 epochs and obtain a completely different model starting from whatever it may be — Dolphin, Qwen, Mistral, etc. Neither better nor worse. Different.

I’m bringing a full fine-tuning of the commercial Dolphin 3.0 Llama 3.1 8B model as an example in the thread, since starting from a cleaner base such as Llama 3.1 8B would have been “less difficult.”

The README.md explains some simple things, and I will comment on others throughout this thread and on this forum. Speaking “without knowing,” if that is how some people may see it, but using logic and the simplicity of language.

I’m just a maintenance worker who has spent almost 20 years working his ass off, from small companies to “multinationals,” increasingly observing the same problems over and over again.

Until the end of January 2026, I didn’t even know what a Linux terminal was.

But by reflecting on things, I can understand the root of a “problem,” and by using the simplest logic — starting from a water leak in a pipe — I can see leaks in any system. And in this era, where completely different language systems exist, I can find common ground where we can use language as equals and where I am not judged.

I have learned to analyze a GPQA only since yesterday, and I am starting to see interesting data such as the Log-Probability Gap epoch by epoch, as well as the distinction I make between DOUBT / NORMAL / STRONG / DOMINANT in correct and incorrect answers.

Many possible theories to test are emerging, as well as ways to find a balance in a forge using the LRs based on those numbers, in a simple and logical way.

The GAP keeps going up. I will reflect on why, from the perspective of language. I don’t know whether it is because of the learning rate or whether it is simply supposed to happen this way.

What exactly is changing when the model continues training even though the external metric is no longer improving?

And what happens when it can no longer go any higher? What does that actually mean? It is just one observation among many that I am starting to see.

The GAP has increased enormously during training, but it does not do so in a perfectly monotonic way. When a small drop appears, GPQA improves slightly again. Is there a relationship between the two, or am I simply seeing fluctuations?

Everything I say here is my interpretation based on what I am learning. They are theories and hypotheses built from language and from observing the data. I do not claim to be right: that is precisely why I am sharing it.

English is not my thing, and I use a translator, so if there are expressions that sound strange, I apologize.

And I would also like to ask something. Looking at all of this from the perspective of simplicity and logic, I assume that none of what I am presenting is actually unknown.

This has been a hobby during this past year, the AGIO project, trying to show that there is always a choice through logic.

Is anything I am showing actually novel? I am asking simply to understand and learn, nothing more. Just curiosity.

If it has helped someone, it will have been worth it. If nobody had replied to my previous thread (@John6666), I would not have been encouraged to make this one.

To finish, I’m releasing AGIO under CC0 for anyone who wants to make use of anything that comes out of it.

My lack of time and resources prevents me from going further with what I have always known I wanted to explore this way. The Z6 was built with a year’s worth of savings.

Now it is all yours.

Best regards!

2

Hmm… for now, based on what I found:

I think there are two separate novelty questions here.

For the training/evaluation behavior itself, several pieces already have fairly close precedents: training loss continuing to fall after external accuracy stops improving, output margins/logit magnitudes continuing to change after the argmax has stabilized, rapid shifts in topic/style priors during fine-tuning, multiple-choice option-ID bias, and diminishing returns from repeatedly training on the same data.

So I would not yet interpret the trajectory as evidence for a new optimization mechanism.

But that does not make the run uninteresting. What is useful here is that you saved enough checkpoints, matrices, and intermediate measurements that several competing explanations can still be separated without first running another 100+ hour training job. I think that is the best next step.

My default order would be:

  1. recompute the GPQA GAP over all / correct / wrong items separately;
  2. follow the same GPQA items across checkpoints and count right→wrong, wrong→right, and answer flips;
  3. run a small answer-choice permutation test on a few representative checkpoints;
  4. pin the exact historical training script/config used for this run and inspect the LR of every optimizer parameter group;
  5. only then, if LR still looks like the main variable, branch into controlled LR experiments.

That sequence should tell you much more cheaply whether the interesting part is mostly margin sharpening, MCQ selection geometry, repeated exposure, LR dynamics, or some mixture.

The first thing I would separate: GAP

One detail in the currently published GPQA_script.py seems especially important for interpreting the plots: the reported mean GAP is calculated from the items the checkpoint answered correctly.

That means the population being averaged can change from checkpoint to checkpoint.

So, for example, an increase in:

mean GAP among correct answers

does not necessarily imply:

the model globally became more confident / more committed.

It could mean that, but it could also partly reflect which questions happen to remain in the correct subset.

I would therefore treat the current GAP primarily as a candidate-score margin statistic, rather than as a calibration measure.

A very cheap re-analysis would be:

  • mean GAP over all 198 items
  • mean GAP over correct items
  • mean GAP over wrong items
  • if the four raw choice scores are available: winner probability, gold-answer probability, and entropy
  • the same quantities on fixed item sets across checkpoints

The interpretation then becomes much cleaner:

  • if GAP rises even on the same items, including wrong ones, that supports a real general sharpening/commitment effect;
  • if it rises mainly in the changing correct subset, composition is doing more of the work;
  • if wrong answers also become increasingly high-margin, then the “becoming more certain without becoming more correct” interpretation becomes substantially stronger.

There is a fairly close general precedent here. Wei et al., Mitigating Neural Network Overconfidence with Logit Normalization show that with ordinary cross-entropy, logit magnitude can continue growing even after classification accuracy has effectively stopped improving. That is not the same metric or the same setup as your GPQA GAP, so I would use it as a nearby mechanism, not as an explanation of this run.

Why falling training loss and flat GPQA are not contradictory

The two measurements are optimizing/evaluating different things.

Your training loss says roughly:

how well the model predicts the AGIO training token stream.

GPQA says roughly:

whether the model ranks the correct answer above three alternatives on an external multiple-choice benchmark.

There is no requirement that these continue improving together.

Once the model has become better fitted to the AGIO corpus, additional optimization can still change probabilities, margins, style priors, token preferences, or internal representations without moving many GPQA argmax decisions.

There is also useful work on the fact that different aspects of behavior can move at different speeds during fine-tuning. Zhang & Wu, Dissecting learning and forgetting in language model finetuning found large and rapid changes in topic/style priors, while factual knowledge changed more slowly. Again, I would not map that directly onto “reasoning opened”, but it is a good reason to distinguish an early behavioral/selection shift from acquisition of new transferable capability.

There is another separate confound here: repeated exposure itself.

Muennighoff et al., Scaling Data-Constrained Language Models studied repeated-data regimes at much larger scale and found that the marginal value of additional repeated tokens eventually falls strongly. Their setup is not this FFT run, but it makes the general point that:

“more epochs passed” and “the LR was held at X”

are not the same variable.

So I would avoid attributing the late trajectory to LR alone until repeated exposure has been controlled for.

The early GPQA movement also looks worth separating from the later GAP movement

The confusion matrices contain a fairly striking clue.

From Base → CP6:

  • total correct: +9
  • correct items whose gold label is A: +22
  • gold B/C/D combined: -13
  • predicted A count: 40 → 111

So CP6 is not merely “nine additional GPQA questions solved” in a neutral way. A very large change in the choice-selection distribution happened at the same time.

That does not prove the gain was an artifact.

In fact, later checkpoints such as CP24/CP36 still appear better than what I would expect from the aggregate A/B/C/D frequency shift alone, so I would currently read this as:

an early option-selection shift and content-dependent improvement are probably mixed together.

That makes answer permutation unusually informative here.

There is directly relevant work on this: Zheng et al., Large Language Models Are Not Robust Multiple Choice Selectors tested 20 LLMs and found substantial preferences for particular option IDs such as A/B/C/D, independent of answer content. Their analysis uses permutations of option contents specifically to separate content preference from option-token preference.

GPQA itself is also naturally compatible with this kind of test. The original GPQA repository exposes a seed for shuffling answer order, and OpenAI’s current simple-evals GPQA implementation evaluates each question repeatedly with independently generated answer permutations by default.

So for perhaps Base, CP6, CP24/36, and one late checkpoint, I would try the same questions under several answer permutations.

The useful decision tree is simple:

  • if the preferred content stays approximately the same while its A/B/C/D position changes, the change is mostly content-dependent;
  • if the model tends to keep following A regardless of which answer was moved there, option-ID/position bias is contributing strongly;
  • if both move, which seems plausible here, you can quantify both effects rather than having to choose one story.

This also helps clarify the phrase “the initial push opened the model.” Something clearly moved very quickly in the output geometry. The permutation test can help determine how much of that early movement was transferable answer discrimination versus a reorganization of output/selection priors.

What I would extract from the existing GPQA sample files

If the raw samples_*.jsonl files from lm-eval still exist, I think they are more valuable right now than another training run.

For every doc_id, I would build a checkpoint table containing:

  • gold choice;
  • predicted choice;
  • four candidate log-likelihoods;
  • top-1 minus top-2 GAP;
  • whether the item is correct;
  • normalized probability of the gold choice;
  • normalized probability of the winning choice;
  • entropy across the four choices.

Then compare checkpoint pairs by item:

  • always correct;
  • always wrong;
  • wrong → right;
  • right → wrong;
  • wrong → different wrong;
  • exact answer unchanged but margin changed.

That answers a much more useful question than only:

“CP24 = 60/198 and CP36 = 60/198.”

Those two checkpoints can have identical aggregate accuracy while disagreeing on many individual questions.

Because GPQA Diamond has only 198 questions, one item changes accuracy by about half a percentage point. So a movement of one or two questions is not much evidence by itself, while a stable pattern across many paired items is much more informative.

I would also use paired comparisons rather than treating checkpoint scores as independent samples, because every checkpoint is being evaluated on the same questions.

About the LR hypothesis

I think LR is a perfectly reasonable hypothesis to test.

Recent work on fine-tuning/forgetting also gives good reasons to expect that learning rate can affect how far a model drifts from pretrained behavior, even when target-task loss looks similar.

But I would not make LR the first thing to test here, because this single trajectory currently combines several variables:

  • LR;
  • number of repeated passes over the same corpus;
  • optimizer-state evolution;
  • total optimizer updates / tokens seen;
  • possibly differences between the historical training code and the currently published code.

If you do branch the run, I would make the branch definition very explicit.

For example:

True resume

  • same checkpoint weights;
  • restore Adam first/second moments;
  • restore scheduler state;
  • then change only the intended LR schedule.

versus:

Weights-only restart

  • same checkpoint weights;
  • new optimizer;
  • Adam moments reset;
  • new scheduler.

Those are both useful experiments, but they answer different questions.

Hugging Face Trainer.train(resume_from_checkpoint=...) normally resumes from the saved model / optimizer / scheduler states, not only the weights. And AdamW carries first- and second-moment state, so resetting it can itself change the early optimization dynamics.

I would also graph the trajectory against global optimizer step / tokens seen, not only epoch number. In this run the early movement happens within only a small number of optimizer updates, so “the first four epochs” can sound like a longer optimization phase than it actually is.

One LR implementation detail I would verify before interpreting the historical trajectory

This is only about the current public code, not a claim about the exact script that produced the forum results.

The current Forja-Eunoia.py custom scheduler writes the new LR to:

optimizer.param_groups[0]["lr"] = nuevo_lr

In a small sanity check using transformers==5.5.3, a normal Trainer optimizer contained two parameter groups. With the current scheduler logic, I observed behavior of this form:

initial:       [1.50e-5, 1.50e-5]
after change:  [1.20e-5, 1.50e-5]
next change:   [1.10e-5, 1.50e-5]

In other words, group 0 followed the custom schedule while group 1 stayed at the initial LR.

That matters because optimizer schedulers normally need to update the relevant LR for every parameter group.

However, the LR values shown in your forum run do not match the current public Forja schedule, so I would not infer that the historical run had this same behavior.

I would just pin the exact historical script/config first and log something like:

print([group["lr"] for group in trainer.optimizer.param_groups])

at every LR transition.

If every group followed the intended schedule in the historical run, this branch disappears and the LR hypothesis becomes cleaner.

If they did not, that is still useful information: the next run can use a scheduler that updates all intended groups, giving a much more interpretable LR experiment.

So, what seems established vs. still open?

What I think is already fairly safe to say:

  • continuing training-loss reduction after GPQA plateaus is not inherently surprising;
  • accuracy can remain unchanged while output margins/probabilities continue moving;
  • MCQ option-ID bias is a real and documented phenomenon;
  • CP6 shows a particularly large change in A-selection behavior;
  • the currently reported GAP is not yet enough to establish global overconfidence because it is aggregated over correct items;
  • LR is worth isolating, but this run alone does not isolate it from repeated exposure and optimizer-state progression.

What still seems open:

  • whether the late checkpoints are becoming more strongly committed on both correct and incorrect GPQA items;
  • how much of the early GPQA gain is content-dependent versus option-selection movement;
  • whether lowering GAP later would actually recover external accuracy, rather than merely correlate with one or two item flips;
  • whether the exact historical optimizer groups all followed the stated LR schedule;
  • whether controlled LR branches reproduce the same qualitative trajectory.
About novelty

My read would be:

The individual ingredients are not new phenomena by themselves.

There are precedents for:

  • loss continuing to improve after accuracy saturates;
  • growing output margins / overconfidence under cross-entropy;
  • fast changes in topic/style priors during fine-tuning;
  • A/B/C/D selection bias in LLM multiple-choice evaluation;
  • diminishing returns from repeated training data;
  • LR-dependent drift / retention effects.

But that is different from saying:

“there is nothing useful here.”

The particular combination you have — a small, very long-context corpus, full fine-tuning of an already post-trained 8B model, dense checkpointing, item-level GPQA matrices, and an explicit attempt to follow confidence/margin changes — is still a useful case study, especially because it is open enough for other people to inspect.

I would just keep the novelty claim at the level supported by the controls:

an interesting observed trajectory whose mechanism is not yet separated,

rather than:

a newly identified mechanism where high LR opens reasoning and constant LR converts reasoning into confidence.

If the existing sample outputs show a stable all-item/wrong-item margin increase, answer permutations show that the early improvement is not mostly option-ID bias, and a controlled LR branch reproduces the later behavior, then the story becomes much stronger.

That would also make the result easier for someone else to reproduce or falsify.

So if I had to choose only one next move, I would not train anything yet.

I would first recover the existing raw GPQA sample outputs and make one item-level table across checkpoints. That one analysis can simultaneously test the GAP interpretation, show exactly where the GPQA gains/losses occur, and tell you which checkpoints are actually worth spending compute on for the permutation or LR experiments.

If those raw sample files no longer exist, my second choice would be a small permutation re-evaluation on a few representative checkpoints.

And only after those two cheap checks would I spend another long run on the LR schedule.

That seems like the shortest route from “interesting trajectory” to something much more causally interpretable.