Hmm… how about something like this?:
I think the random-matrix result is actually a very useful hinge here, but I would read it slightly differently.
Rather than:
training is not involved because an untrained matrix already shows the effect
I think it gives you something more useful experimentally:
the random matrix is a null model for the part of the phenomenon that comes “for free” from a high-dimensional linear decision partition.
Then the trained-model question becomes much sharper:
Where does a real model depart from that null, and does that departure come from the learned output head, the hidden states, or their compatibility?
I tried a small version of that using Pythia-160M, because Pythia is unusually convenient here: it publishes 154 checkpoints across one training run specifically for learning-dynamics work.
The result was more structured than I expected.
A random initialization really does look like the null
I kept the same frozen 64 Pythia target IDs, two prompts, sampled block outputs 0 / 4 / 7 / 11, and a 17° target-row rotation, then checked:
step0
step512
step5000
step20000
step80000
step143000
The overall direct-head target-rank-1 rates were:
| checkpoint |
target @ 17° |
same-score random residual |
same-score toward blocker |
same-score away blocker |
median first crossing |
| step0 |
100.0% |
100.0% |
60.7% |
100.0% |
8.65° |
| step512 |
43.6% |
43.9% |
12.7% |
64.8% |
18.51° |
| step5000 |
96.9% |
97.5% |
34.4% |
99.0% |
10.51° |
| step20000 |
98.4% |
98.4% |
45.7% |
99.4% |
9.77° |
| step80000 |
99.4% |
99.4% |
35.2% |
100.0% |
9.94° |
| step143000 |
27.5% |
27.7% |
25.0% |
78.7% |
27.42° |
The wrong-target and angle-matched random-tangent controls were 0% at every checkpoint.
For comparison, a deliberately simple iid-Gaussian head null with Pythia’s scale,
d = 768
V = 50,304
theta = 17°
gives approximately
P(target becomes rank-1) ~= 0.99993
rough critical angle ~= 9.7°
while actual Pythia step0 gave:
target rank-1 = 100%
median first crossing = 8.65°
So, at least for this assay, Pythia’s initialization behaves remarkably like the generic random-head baseline.
What happens during training is the interesting part.
The departure from the null is strongly depth-dependent — and not monotonic
At step0:
block 0 100%
block 4 100%
block 7 100%
block 11 100%
By step512:
block 0 100.0%
block 4 46.1%
block 7 21.9%
block 11 6.25%
The median first-rank-1 angle simultaneously moves from roughly 9° at the shallowest sampled block to about:
17.5°
21.6°
24.5°
at the deeper ones.
Then it mostly recovers:
step5000:
100.0 / 100.0 / 99.2 / 88.3%
step20000:
100.0 / 100.0 / 100.0 / 93.8%
step80000:
100.0 / 99.2 / 100.0 / 98.4%
and the final checkpoint reorganizes again:
step143000:
block 0 2.34%
block 4 1.56%
block 7 6.25%
block 11 100.00%
with median first crossings around 30° for the earlier sampled block outputs, but only 2.82° at the last block.
So I don’t think the useful conclusion is simply:
training makes the geometry better
or:
training destroys the random geometry
It looks more like:
initialization:
generic high-dimensional geometry makes target capture easy
very early training:
deeper states move sharply away from that regime
middle training:
broad direct-head accessibility returns
late training:
accessibility becomes strongly localized near the end of the computation
That seems like a much more specific place for “training” to enter the story.
And because this is plain Pythia pretraining rather than an RLHF trajectory, it also separates the broad “training/RLHF caused it” objection a little: you do not need RLHF to produce a very large change in these rates.
The remaining question is what training changed.
The output head/state compatibility seems to matter too
I did one additional small check because Pythia’s output head itself moves a lot during training.
For the frozen target panel, the mean same-token cosine between output-head rows was only roughly:
step0 vs step20000 ~0.28
step20000 vs final ~0.25
step0 vs final ~0.07
So I took the actual post-final-LayerNorm state from:
step0
step20000
step143000
and crossed each with the LM head from those same three checkpoints.
I explicitly checked the boundary first: each checkpoint’s own post-final-LN state passed through its own lm_head reconstructed the model’s final logits with max error 0.0 in this run.
The 3×3 target-rank-1 result at 17° was:
| post-final-LN state |
step0 head |
step20000 head |
final head |
| step0 |
100.0% |
100.0% |
15.6% |
| step20000 |
97.7% |
93.8% |
6.25% |
| final |
100.0% |
77.3% |
100.0% |
The corresponding median first-crossing angles were:
| post-final-LN state |
step0 head |
step20000 head |
final head |
| step0 |
8.37° |
8.80° |
25.22° |
| step20000 |
11.23° |
12.50° |
34.00° |
| final |
9.48° |
13.84° |
2.82° |
That asymmetry is pretty striking:
random-like step0 head:
permissive across all three checkpoint states
final trained head:
easy with the matched final state
difficult with step0 / step20000 states
I would call that a strong state/readout compatibility signal.
I would not call it a proof of semantic co-adaptation yet, because checkpoint representations can undergo distributed basis changes as training proceeds. A global basis-alignment control could potentially explain some of the cross-checkpoint mismatch.
But it does make the next question much narrower:
Is the final head merely expressed in a different global basis, or has training produced more specific state↔readout co-adaptation?
And that is a much nicer question than “geometry or training?”
The exact fixed-score result also looks stronger now, not weaker
The other thing I would keep from the corrected harness is the residual control.
For this Pythia trajectory I used the literal decomposition
v0 = gamma*s + rho*r
q perpendicular to s
q perpendicular to r
v(eps)
= gamma*s
+ rho * (cos(eps)*r + sin(eps)*q)
so the residual rotations preserve:
norm
target projection
raw target logit
to floating-point error.
Across the run the maximum observed errors were around:
norm error <= 1.2e-7
target-score error <= 1.2e-7
|q dot s| <= 2.2e-8
|q dot r| <= 2.9e-8
The competitor effect survives cleanly.
For example, at step512, block 7:
target endpoint rank-1 successes: 28
toward-blocker destroys: 27 / 28
and at the final checkpoint, where the earlier blocks are mostly outside rank 1, moving away from the blocker rescues many failures:
block 0:
94 / 125 failures rescued
block 4:
82 / 126 failures rescued
block 7:
86 / 120 failures rescued
So I think the exact correction has turned this from an implementation question into a fairly useful result:
even at fixed norm and fixed target logit, competitor-relative residual orientation can dominate rank.
That fits quite naturally with the raw LM-head decision-cell picture rather than competing with the target-tangent story.
The target tangent is still a principled direction for increasing target alignment; it is simply not the complete objective when rank depends on every other row too.
So my default next route would now be fairly small
I probably would not spend GPU time on another five-family sweep yet.
The current evidence already separates quite a lot:
generic linear-head geometry
-> visible at initialization
training
-> creates large quantitative departures
depth
-> changes where the target arc remains accessible
output-head evolution
-> large
state/readout compatibility
-> strong checkpoint-specific signal
competitor geometry
-> matters even when target logit is fixed
If you want one additional mechanism check later, I think the highest-information one would be a small basis-alignment test rather than more models.
For example:
early/mid checkpoint representation
|
+-- fit one orthogonal Procrustes alignment
|
+-- repeat the state/head swap
Then:
compatibility mostly restored
-> global representation-basis drift explains much of the mismatch
large mismatch remains
-> stronger evidence for more specific state/readout co-adaptation
But I would regard that as a follow-up branch, not something v2 needs before it can say anything useful.
Pythia itself was designed for exactly this kind of learning-dynamics analysis and exposes 154 checkpoints, so even if you want more resolution, a few extra checkpoints inside the same training run seem more informative to me than adding another unrelated architecture. The Pythia repository documents that checkpoint design.
There is also a recent public codebase, Learning to Read Out, studying unembedding dynamics across pretraining and using cross-checkpoint readout swaps. It is a different question, but the separation between “representation is available” and “the current readout can express it” seems very relevant here.
Why the random-matrix result looks like a genuine null rather than a trained effect
The author’s random test was:
d = 256
V = 2000
theta = 17°
target tangent 89.1%
wrong target 0.0%
random tangent 0.0%
toward blocker 21.9%
away blocker 95.3%
For iid Gaussian output rows, a rough calculation is already enough to see why target steering should work.
On a unit hidden-state direction, an unrelated random output row has a score on the order of a standard normal variable, while the largest of V competitors is approximately on the scale
sqrt(2 log V)
The target row has norm on the scale
sqrt(d)
so after a target-directed angular rotation theta, its score contribution is roughly
sqrt(d) * sin(theta)
The crude crossing estimate is therefore
theta_critical
~= asin(sqrt(2 log(V) / d))
For:
d = 256
V = 2000
that is about 14°.
A more explicit finite-dimensional iid calculation, including the random starting target cosine and target-row norm, gives me approximately:
P(target rank-1 at 17°) ~= 0.883
versus the observed:
57 / 64 = 0.891
That agreement is close enough that I would regard the synthetic result as a successful null-model canary.
The useful signal in a trained LM is then not just:
does target > wrong/random?
because random geometry already gives that.
It is things like:
How far does the trained reach rate depart from the matched null?
At which layers?
At what point in training?
Does preserving real head norms/covariance close the gap?
Does the gap follow the head or the hidden state?
There is some prior work suggesting that a richer null may eventually matter because real output embeddings are not iid. For example, Understanding Token Probability Encoding in Output Embeddings reports learned token-probability/frequency structure in output embeddings, including structure that appears during pretraining.
So if the simple Pythia trajectory becomes a main result, a natural null ladder would be:
iid rows
->
random directions with empirical row norms
->
covariance-matched head
->
real state + randomized head
->
real head + randomized/aligned state
But I would only climb that ladder when the previous level actually fails to explain something.
How I would interpret the depth result
I think the depth result is more informative than the aggregate family percentage.
The final Pythia checkpoint in this small probe is approximately:
direct raw-head target-arc reach @17°
block 0 2.3%
block 4 1.6%
block 7 6.3%
block 11 100.0%
This is important because it suggests the trained phenomenon is not simply a property of a fixed linear head considered in isolation.
The same final head sees very different state geometry depending on where the state comes from.
I would still keep the scope explicit:
intermediate block output
->
direct final-head readout
is an accessibility diagnostic, not:
intervene at block l
->
run all remaining nonlinear blocks
->
measure final generation
That distinction has useful precedent.
Predicting Where Steering Vectors Succeed deliberately applies the model’s unembedding to intermediate states and reports that this kind of linear accessibility is predictive of steering success across layers.
At the same time, Tuned Lens is a useful reminder that raw intermediate unembedding is not identical to the model’s eventual downstream prediction; learned translators can give substantially better intermediate prediction.
So I would label this branch something like:
intermediate-layer raw-head accessibility
rather than an end-to-end intermediate-layer steering result.
The especially interesting question is now:
why does training make accessibility concentrate late?
Potentially cheap diagnostics are:
target tangent vs shortest-cell direction
active blocker identity
active blocker churn
decision margin
head-row norm/covariance
state/readout alignment
rather than another broad success-rate table.
Where I think this now connects to the steering/output-geometry literature
I would probably position the related work by what part of the question it covers, rather than as a list of similar steering methods.
Rotation / angular intervention
Spherical Steering explicitly uses norm-preserving geodesic rotation toward a steering direction.
That makes rotation itself useful prior art, while the distinctive part here can stay focused on:
individual vocabulary targets
LM-head rank
competitor decision geometry
angular reachability
rather than needing rotation itself to be novel.
A Geometric Account of Activation Steering through Angle–Norm Decomposition is also useful because it argues for separating angular and radial components instead of hiding both inside one additive coefficient.
That seems compatible with making theta the canonical intervention magnitude here.
Output decision regions
Low-Rank Softmax Can Have Unargmaxable Classes in Theory but Rarely in Practice is a different question, but it gives a clean prior connection for the idea that an LM head defines real argmax/decision regions whose existence depends on the output rows.
That is exactly the part target-logit ascent alone does not determine.
Intermediate-state accessibility
Predicting Where Steering Vectors Succeed is probably the closest bridge for the layer-wise direct-unembedding branch.
Training dynamics of the readout
Pythia was explicitly constructed to expose learning dynamics through dense training checkpoints.
The public Learning to Read Out code is also interesting here because it studies how the unembedding changes during pretraining and includes cross-checkpoint readout-swap analyses.
I would not claim that any of these already answer the token-rank question. They mostly give useful neighboring coordinates:
rotation
angle/norm
argmax geometry
linear accessibility
readout learning dynamics
The current token-specific rank experiments sit at their intersection.
A few zero-GPU cleanup items I would prioritize before another large sweep
Since the empirical effect has survived the stricter controls, I think synchronization is now higher value than more breadth.
In particular I would try to make one canonical chain read cleanly from paper to code to CSV:
1. Canonical intervention:
theta on the target-row great circle
2. Historical provenance:
alpha=.3 in the normalized-row implementation
maps to a target-dependent theta <= atan(.3)
3. Mathematical guarantee:
target-row / target-logit ascent
4. Empirical claim:
rank ascent / rank-1 reach / no-mid-arc-loss
under the tested conditions
5. Exact residual control:
q perpendicular to both s and r
6. Intermediate-layer tables:
labeled as direct-head accessibility
7. Run metadata:
exact model/tokenizer revision
torch / transformers
dtype / quantization
backend
all RNG seeds
prompt and target-panel hashes
The distinction in point 3/4 seems especially worth keeping because the two-dimensional counterexample already makes it very clean:
target logit can increase
while a competitor increases faster
So the target-gradient identity is still useful; it just proves the one-row objective rather than the full multi-row rank objective.
For the existing cross-family table I would also keep the current target IDs frozen for provenance rather than silently changing the sample.
If you later want a fresh vocabulary-generalization result, I would make that a separate experiment:
define eligible printable-token pool first
then uniformly sample from that pool
rather than mixing a sampling change into a contract-correction rerun.
Small note on the silicon / numerical-reproducibility side
I think the clean separation you made here is basically the useful one:
algebraic endpoint identity
!=
finite-precision/runtime reproducibility
I could not find support for the specific “about ±3° from thermal noise” number.
There are real numerical reproducibility issues worth pinning. PyTorch’s reproducibility notes explicitly warn that results are not guaranteed to be identical across releases/platforms, and some CUDA algorithms/backends can be nondeterministic or numerically different.
So if this ever needs a robustness table, I would test concrete contracts:
fp32 / bf16 / fp16
quantized vs unquantized
backend / attention implementation
deterministic algorithms where available
model revision
rather than a generic hardware-noise model unless someone supplies one.
That seems orthogonal to the geometric identity itself.
Where I land now
I think the random-matrix result and the training objection can both be true, and the Pythia trajectory makes the distinction fairly concrete:
generic high-dimensional linear geometry
explains a large part of the phenomenon at initialization
training
creates large, non-monotonic, depth-specific departures
competitor geometry
still controls rank even when target logit is fixed
the trained readout
becomes strongly compatible with its matched trained state
So I would not frame the next version as needing to choose between:
"it is geometry"
vs
"it is training"
A cleaner decomposition seems to be:
generic geometry gives the baseline;
training determines how the actual state/readout system
departs from that baseline.
That also leaves a very concrete place for future controls without making them prerequisites for the current result.
If I were choosing the default path from here, I would probably:
keep the exact fixed-score result
+
keep the random matrix as the null
+
use the Pythia training trajectory as the learned-deviation example
+
synchronize the canonical paper/code contracts
and stop there for v2 unless one of those results specifically calls for another experiment.
That seems to preserve the original geometric observation while giving the learned part of the model a much more precise, testable role.
