I did a quick experiment in Colab and was able to reproduce a similar phenomenon:
My main takeaway is that the observation looks worth separating into a few smaller questions rather than treating it as one all-or-nothing claim about “Fourier phase.”
In a small K-only probe, I also found conditions where a simple raw 4-bit K quantizer immediately changed the generation trajectory, while a Fourier-domain perturbation that quantized magnitude and preserved phase kept the reference trajectory. So at least qualitatively, I do see the same kind of robustness effect.
What I am less sure about is which part of the construction is responsible for it. There seem to be several effects mixed together:
observation
|
+-- Fourier basis itself?
+-- transform/preconditioning in general?
+-- magnitude vs phase/direction precision?
+-- polar vs Cartesian representation?
+-- where the fixed bit budget is allocated?
+-- what error is actually visible through attention?
+-- interaction with RoPE?
I think the highest-information next step is therefore not a much larger benchmark yet. A few small controls can distinguish most of these branches.
The three controls I would try first
1. Hold the total rate fixed while moving bits between magnitude and phase.
For example, for a direct-angle representation:
mag4 + phase8
mag5 + phase7
mag6 + phase6
mag7 + phase5
mag8 + phase4
This separates two claims that are easy to accidentally merge:
- phase precision matters;
- most of the available bits should go to phase.
In my small probe, lowering phase precision too far definitely hurt, so I think the first statement has some empirical support. But at a fixed total rate, 4+8 was not consistently the best allocation: 5+7 or 6+6 often did better depending on the model and metric.
So I would currently phrase the interesting hypothesis more like:
Magnitude and phase appear to have asymmetric sensitivity, but the optimal rate allocation is still an empirical question.
This also connects fairly naturally to recent KV-cache work on non-uniform bit allocation. RateQuant explicitly treats the distortion-vs-rate curve as quantizer-dependent, and the very recent AATC treats KV compression as an attention-aware transform-coding/rate-allocation problem.
2. Add one transform/representation control at roughly matched rate.
I would not make this a large ablation grid. Even something like
FFT polar
FFT Cartesian (real/imag)
Hadamard
would tell a lot.
The reason is that there is now a fairly strong adjacent literature showing that the coordinate system presented to the quantizer is itself a design variable. Codec-Gauge directly compares raw, random, Hadamard, DCT and learned cache-coordinate transforms; NOVA-KV derives transforms from attention-product distortion rather than treating the original KV coordinates as privileged.
My own tiny probe pointed in the same direction: DCT/Hadamard/random-orthogonal controls were surprisingly competitive, and even with the FFT held fixed, changing the coefficient representation from polar-style to real/imag changed the error substantially.
That would give a clean branch:
If FFT stays clearly better at matched rate:
-> a specifically Fourier-domain effect becomes more plausible.
If several transforms behave similarly:
-> the larger finding may be transform/preconditioning robustness.
If FFT is fixed but polar vs Cartesian differs strongly:
-> representation/codebook geometry is an important part of the effect.
None of those outcomes makes the original observation less interesting; they just locate it more precisely.
3. Add one error metric that attention can actually “see.”
K reconstruction error alone may not rank these methods correctly.
Even one of the following would be enough for a first pass:
QK-logit error
attention KL / JS
top-attended-token flips
attention-output error
fixed-sequence NLL
Recent work increasingly makes this distinction explicit. NOVA-KV defines distortion in terms of attention products, Block-GTQ allocates key bits based on their contribution to future RoPE query-key logits, and AATC derives an attention-aware distortion objective.
In my probe I saw several cases with quite similar K-space reconstruction error but very different attention/final-logit perturbation. So this seems especially relevant here.
I would do those three before asking the method to survive a much bigger LongBench/RULER-style evaluation. They are cheap, and almost every possible result tells you why the method is working.
What I actually tested
This was deliberately a small sanity check, not a reproduction of the 40-prompt table.
The main run used:
- T4;
- FP32 for the final diagnostic run;
- SmolLM2-135M and Pythia-160M;
- cached/post-RoPE K-only interventions;
- V left unchanged;
- fake quantization rather than packed storage;
- a few fixed texts;
- one-step attention/logit diagnostics;
- one short 16-token greedy-generation smoke test on SmolLM2.
For the generation smoke, I got:
| K intervention |
Matching reference tokens |
| baseline |
16 / 16 |
| simple raw 4-bit |
0 / 16 |
| full FFT + mag4 + exact phase |
16 / 16 |
| full FFT + mag4 + quantized cos/sin at 8 bits each |
12 / 16 |
| rFFT + mag4 + direct 8-bit angle |
16 / 16 |
| Hadamard + 6-bit coefficients |
12 / 16 |
I would not read much into the exact ranking from one 16-token sample. The useful part for me was simply that the qualitative phenomenon survived an independent small probe: the raw low-bit perturbation could be destructive while transform-domain variants remained much more stable.
That is why I think this is worth mechanism controls rather than dismissal.
Fixed-rate magnitude/phase allocation was more informative than just sweeping phase bits
One thing I found particularly useful was keeping the rate fixed.
For an rFFT + magnitude + direct angle control, I held the spectral payload at 12 bits per unique complex coefficient:
| Allocation |
SmolLM2 K NRMSE |
SmolLM2 attention JS |
SmolLM2 final-logit rel. RMSE |
| mag4 + phase8 |
0.0344 |
0.00091 |
0.0182 |
| mag5 + phase7 |
0.0215 |
0.00037 |
0.0130 |
| mag6 + phase6 |
0.0294 |
0.00059 |
0.0100 |
That is a very small sample, but it illustrates why I think a matched-rate sweep is valuable.
It confirms the intuitive part — phase cannot simply be made arbitrarily coarse — while also showing that:
phase is important
does not automatically imply
4 bits magnitude + 8 bits phase is the optimal allocation
Pythia made this even clearer: the best allocation depended on whether I ranked methods by K reconstruction, attention distortion, or final-logit distortion.
That seems consistent with a wider pattern in KV quantization: sensitivity is heterogeneous across model, layer, head, K/V side, and quantizer. KVTuner uses layer-wise sensitivity, RateQuant shows that even the distortion-rate curve depends on the quantizer, and TurboAngle uses asymmetric norm/angle precision and layer-specific allocation.
There are also two different works called PolarQuant that are relevant search neighbors, though neither is the same construction as Fmag:
So I think “angular/directional information deserves different treatment from radial information” has useful neighboring precedent, without implying that the exact Fourier magnitude/phase decomposition here has already been done.
Why I would control the basis and the complex representation separately
There are actually two different questions here:
Which transform?
and
How are transformed coefficients represented/quantized?
They can be separated.
For the first:
raw
FFT
DCT
Hadamard
fixed random orthogonal
is enough to test whether FFT is exceptional or whether transforming the coordinates generally makes the quantizer happier.
Codec-Gauge is a useful connection here because it explicitly treats KV channel basis as a compression variable and evaluates random/Hadamard/DCT/PCA-type controls around fixed compression backends.
For the second, hold FFT fixed and compare something like:
magnitude + phase
vs
real + imaginary
In my small probe, this second distinction was surprisingly large. A same-payload full-FFT Cartesian control reconstructed the cache much more accurately than the literal magnitude/cos/sin representation.
That does not mean Cartesian is necessarily the better practical codec. It only means that part of the observed rate-distortion behavior may come from the coefficient representation/codebook rather than Fourier phase alone.
This is also why I would be cautious about importing the classic image-processing intuition that “Fourier phase carries structure” too literally. In an image, the transformed axes have a natural spatial ordering. A Transformer head dimension does not obviously have the same semantics.
I tried random channel permutations as a sanity check. Across multiple permutation seeds I did not get evidence that the native channel order was uniquely favorable to the FFT. What did remain true was that similar K reconstruction errors could produce quite different model-visible errors.
So at this point I would treat “the native channel ordering contains a special Fourier geometry” as an interesting possible outcome of a control, rather than an assumption needed to explain the current result.
The finite-phase implementation and the physical codec may be worth separating
There is one representation detail that I think is easiest to handle by separating the experimental question from the storage question.
The exact-phase experiment is conceptually clean:
z = fft(K)
mag = quantize(abs(z), 4)
z_hat = mag * exp(1j * angle(z))
K_hat = ifft(z_hat)
That tests the basic observation very directly.
For finite phase, the README-style version quantizes something equivalent to:
cos_q = quantize(cos(angle), p)
sin_q = quantize(sin(angle), p)
separately.
That is a perfectly reasonable fake-quant experiment, but it has two interpretation consequences:
cos_q and sin_q are two stored quantities if taken literally as a codec;- independent quantization generally moves the point off the unit circle, so it perturbs effective magnitude as well as angle.
So I would distinguish:
"p-bit phase precision" as an experimental idea
from
"p physical bits for phase" as a storage representation
rather than making one stand in for the other.
A particularly clean physical-codec candidate seems to be:
real K
-> rFFT
-> 4-bit magnitude index
-> one 8-bit phase-angle index
-> irFFT
Because K is real-valued, the FFT is Hermitian-symmetric. PyTorch’s torch.fft.rfft explicitly stores only the non-redundant one-sided spectrum.
For head_dim = 64, that gives 33 unique bins. Ignoring metadata for a moment:
(4 magnitude bits + 8 phase bits) * 33 / 64
= 6.1875 payload bits per original K scalar
Relative to BF16’s 16 bits, that is about a 61% payload reduction. As the head dimension grows, the one-sided-spectrum overhead approaches one half, so the asymptote is about:
6 bits / original scalar
=> 62.5% below BF16
So I do not think the ~62% number needs to be discarded. Rather, it becomes much easier to interpret if the intended storage format is stated explicitly.
For example:
scientific probe:
full FFT + magnitude quantization + exact phase
finite-phase ablation:
explicitly defined fake-quant representation
physical codec:
rFFT + packed magnitude index + packed angle index + metadata
Those can all support the same research direction without needing to be the same representation.
One more scope detail: if this is currently a K-only codec, that percentage is a K-cache figure. A full KV-cache memory figure additionally needs a V representation. If Fmag is intended for both K and V, then specifying the V path separately would make the whole-cache number unambiguous.
Why attention-visible distortion seems especially useful here
I would avoid relying too heavily on raw K MSE/NRMSE as the mechanism test.
Attention only observes K through its interaction with Q, and different error directions can have very different effects even at nearly identical Euclidean error.
A cheap diagnostic ladder would be:
K reconstruction error
|
v
QK-logit error
|
v
softmax / attention-distribution error
|
v
attention-output error
|
v
fixed-sequence NLL / final logits
You do not need all of these. Even one extra level is informative.
This also avoids having to lean too hard on the explanation that softmax “normalizes” magnitude scaling. Multiplying K changes the scale of the QK logits, and multiplicative logit scaling generally changes the softmax distribution. So I would leave the softmax mechanism open and measure the downstream distortion directly.
NOVA-KV is particularly relevant: it formulates KV quantization as transform coding where the distortion is error in the attention products rather than ordinary cache reconstruction error.
AATC similarly derives an attention-aware distortion and allocates rate around that objective.
Block-GTQ is another useful comparison because under RoPE it explicitly decomposes a key’s future logit contribution into 2-D frequency blocks and spends more bits on the blocks that matter more to QK accuracy.
My small experiments made this distinction fairly concrete. I had cases where K reconstruction errors were very similar while attention and final-logit errors were materially different. Pythia was an extreme example: relatively small perturbations could produce large changes in individual attention distributions while the final-logit difference remained much smaller.
That seems like useful evidence that the “right” distortion measure is itself part of the research question.
Pythia may be a more interesting architecture control than it first appears
One architecture detail may be worth updating before drawing a rule from the Pythia result.
The published EleutherAI/pythia-160m config has:
{
"rotary_pct": 0.25,
"use_parallel_residual": true
}
So Pythia-160M is not a clean “learned absolute position embeddings instead of RoPE” control; it uses partial RoPE.
I actually think that makes the failure case more useful, because the architecture branch can become:
full RoPE
vs
partial RoPE
and/or
pre-RoPE K
vs
post-RoPE/cached K
rather than simply:
RoPE
vs
no RoPE
There is precedent for treating this boundary as important. KVQuant explicitly uses pre-RoPE key quantization to avoid some of the quantization difficulty introduced after rotation.
And Block-GTQ goes further by treating the native 2-D RoPE blocks themselves as non-uniformly sensitive quantization units.
That gives another possible low-cost control:
FFT/global head-dim representation
vs
native RoPE 2-D pairs/blocks
I would not infer from one Pythia result that partial RoPE, LayerNorm, parallel residuals, or any other single architecture feature is the cause. There are too many family differences at once.
But Pythia looks like a useful partial-RoPE branch for narrowing that question.
K-only diagnostics and a full KV codec are different stages
I do not think a K-only experiment is a problem by itself.
For mechanism isolation it can actually be useful, because it asks a simpler question:
How much can I perturb cached keys before the attention routing changes?
What I would keep separate is the later claim about an end-to-end KV-cache codec.
There is a long-standing reason to be cautious about assuming that the same rule should apply to K and V. KIVI found different distributional behavior and used per-channel quantization for K but per-token quantization for V. More recent methods also often find K/V asymmetry or layer-specific sensitivity.
So a clean progression could be:
Stage 1:
K-only mechanistic probe
Stage 2:
K codec + independent V baseline/control
Stage 3:
joint K+V memory/quality accounting
Stage 4:
packed serving implementation
There is no need to do all four before the first result is interesting.
A small reproducibility note
I may simply be following a different experiment path/version, so I would treat this as a navigation note rather than a result issue.
The algebraic_kv_tests.py path I followed behaves like a small fixed-prompt, K-only diagnostic, while the forum post reports the larger 40-prompt comparison and the finite-phase tables.
If the 40-prompt / Std4-3-2 / phase-6-8 results came from another runner or commit, pinning that exact script plus its raw output would make the result substantially easier for other people to reproduce or extend.
Something as simple as:
experiments/
reproduce_hf_table.py
prompts.json
results.csv
plus the model revision / Transformers version would probably be enough.
Again, it is entirely possible that this already exists somewhere I missed; I am mostly mentioning it because the result seems interesting enough that a single canonical reproduction path would be useful.
Related work I found nearby
I did not find the exact construction
head_dim FFT
-> low-bit Fourier magnitude
-> preserved/high-precision Fourier phase
in the papers I checked.
I did find several nearby branches that seem useful for positioning/searching:
Polar / angular KV representations
These are not Fourier-phase methods, but they are close to the broader idea that radial and angular information need not receive the same representation or precision.
Basis / transform choice
- Codec-Gauge — treats the cache coordinate basis itself as a post-training compression variable.
- NOVA-KV — attention-preserving transform coding.
- AATC — attention-aware transform coding and rate allocation.
RoPE-aware allocation
- Block-GTQ — allocates key bits over native 2-D RoPE frequency blocks according to attention-logit sensitivity.
- KVQuant — among other things, explicitly separates pre-RoPE key quantization.
FFT already appearing in adjacent KV quantization work
- VidKV is a VideoLLM method and a rather different setting, but it does combine FFT with ultra-low-bit key-cache quantization for selected channels.
Because of that, I would probably avoid a broad claim such as “the first use of FFT/Fourier transforms in KV-cache quantization.”
But I have not found the exact Fmag magnitude/phase factorization above, so there may still be a much narrower novelty claim there. I would just phrase it as an exact-method literature-search question rather than a settled novelty judgment.
I would leave kernel work until after the representation question is settled
If the goal eventually becomes a production cache, I would separate three numbers:
1. algorithmic/fake-quant quality
2. physical packed bytes
3. end-to-end latency / throughput
They do not automatically move together.
The current Hugging Face KV-cache documentation explicitly notes that a quantized cache can reduce memory while hurting latency at short context lengths when enough VRAM is available.
For Fmag there is an additional transform/decode cost, so eventually the useful systems question becomes something like:
saved memory bandwidth
versus
FFT / inverse-FFT + unpack/dequant overhead
possibly with fusion.
But I would not make that the next blocker. First I would establish which representation and which distortion objective are actually worth implementing. A CUDA/Triton kernel for the wrong codec would be much more expensive than another tiny matched-rate control.
So if I had to reduce all of this to one default next route, it would be:
1. Keep the basic Fmag observation.
2. Compare magnitude/phase allocations at fixed total rate.
3. Add one non-Fourier transform and one FFT Cartesian control.
4. Measure one attention-visible distortion metric.
5. If the effect survives those controls, then choose the physical rFFT/phase representation.
6. Only then spend effort on K+V packing and kernels.
That route seems relatively cheap, and every branch gives useful information:
- if FFT remains special, the Fourier interpretation gets stronger;
- if several transforms work, the finding broadens into a transform/preconditioning result;
- if angle/direction consistently deserves more rate, the phase story gets stronger;
- if optimal allocation varies by model/layer, that points toward adaptive precision;
- if K-space error and attention-space error rank methods differently, the attention geometry becomes part of the mechanism;
- if the compact
rFFT + magnitude + angle version retains the effect, the observation also has a fairly clean path toward a real codec.
That seems like a good position to be in: the initial observation does not need to be weakened in order to test the alternatives — the controls mostly tell you what kind of finding it is.
