HI. I don’t know much about endorsement, but for now, I’ll share some feedback on the parts that stood out to me:
The main thing I like about the current version is that it no longer asks one experiment to prove several different things at once. The separation between state steering, semantic transmission, payload recovery, and downstream readability / write site makes the results much easier to reason about.
So I probably cannot be very useful on the endorsement mechanics themselves, but on the technical side my current read would be:
- Regime A looks like a real, reproducible, sign-sensitive residual-state transition, but the evidence currently fits a prompt-conditioned, direction-selective transition better than a general or concept-specific “semantic inverse.”
- Regime B looks more interesting to me as a channel-design problem: the receiver-side oracle establishes that continuous ordered payloads can be readable, while the actual Qwen3 → Llama bridge is currently closer to partial semantic transmission than robust identity-preserving payload recovery.
- The strongest engineering result may actually be the decomposition into representation bandwidth + bridge quality + landing site + readout protocol, rather than the existence of a cross-model map by itself.
If I were choosing a default path from here, I would probably keep that separation, tighten the positioning, and improve the Regime-B reproduction surface before spending a lot more compute on breadth.
Regime A: I would keep the phenomenon, but narrow what the noun means
Your current paper already says something close to what I think the controls support:
a sign-sensitive residual-state transition, not a universal semantic inverse.
That distinction seems important.
The particularly informative controls for me are still:
- negative adapted direction: 5/5 on the selected panel;
- positive sign: 0/5;
- tested random direction: 0/5;
- coordinate permutation: 0/5;
- unrelated adapted direction: 4/5;
- bias-only can reach the stove basin at shifted gains;
Wv alone does not reach the same object reading on the tested grid;- exact Householder reflection about the external adapter direction does not reproduce the signed-addition effect.
That combination makes “ordinary random magnitude perturbation” too weak an explanation, but it also makes “the semantic inverse of this particular concept” too strong at the moment.
A description such as prompt-conditioned, direction-selective residual basin transition seems to preserve the interesting empirical object without claiming the mechanism before the controls identify it.
I also ran a small sanity check against the pinned public setup because the single-random-direction control seemed like an especially cheap place to get more information.
Using the public Qwen/Nomic revisions, float32, layer 4, the original fireplace prompt, greedy decoding, and the five published gains:
- the public target and seed-0 control outputs reproduced;
- the target direction was 5/5;
- among 128 predeclared isotropic random directions, 1/128 reached 5/5 and 4/128 reached at least 4/5;
- the 5/5 random case was a real fire-pit/object reading in the raw text, not just a keyword-scoring accident;
- among the 12 public adapted concept directions I checked on the same locked prompt, Glub-Tub was the only 5/5 direction;
- changing the prompt frame away from the fireplace setup largely removed or qualitatively changed the target effect.
I would not treat those gain cells as independent statistical samples, and the simple keyword scorer becomes unreliable once the prompt changes, so I would not turn those numbers into a formal significance claim.
But as a sanity check they suggest a useful middle ground:
not:
"any random direction does this"
and not yet:
"only the concept's semantic inverse does this"
more like:
"there is a direction-dependent residual landscape;
this adapted direction is an unusually reliable route
into one prompt-conditioned basin,
but it is not the only possible route"
That still seems worth studying.
The recent work on steering strength is also relevant here because non-monotonic behavior as steering strength changes is not unique to this experiment. That makes the narrow gain islands interesting observations, but I think the direction × prompt × gain interaction carries more mechanistic information than non-monotonicity by itself.
Why the Regime-A controls seem to point this way
One thing I found useful was separating three questions:
1. Is the intervention doing something structured?
Probably yes.
The sign control, permutation control, the reproducible gain windows, and the fact that most random directions do not reproduce the same five-cell pattern argue against reducing everything to “the residual was perturbed.”
2. Is the useful direction unique to the target concept?
Not established yet.
The unrelated adapted direction is particularly important here because 4/5 is much harder to dismiss than the original single random control.
The broader adapted-direction check also gave an interesting shape: Glub-Tub was the strongest direction under the locked prompt, but several other learned directions could still move the output toward object/device-like regions. The ordering also did not look like a simple monotonic function of cosine similarity to the Glub-Tub direction.
So there may be structure in the learned adapter family that is not captured by one cosine.
3. Is the “object” basin mostly created by the direction or by the prompt?
The prompt seems to be a major part of the effective operator.
For example, the target effect largely disappeared when the fireplace cue was replaced by ordinary-home or cold-water framing. A rare random direction that was extremely effective on the fireplace prompt also largely lost that effect after the prompt change.
That makes me think of the empirical object less as:
direction -> fixed semantic antipode
and more as:
prompt state
× injected direction
× sign
× gain
× downstream nonlinear dynamics
-> reachable generation basin
This is not necessarily a less interesting result. It is just a different object.
A compact way to distinguish the interpretations later, if you still want to pursue Regime A, would be:
if target >> unrelated adapted + random across several prompt frames:
stronger evidence for direction/concept specificity
elif target + several adapted directions work mainly under the same cue-rich prompt:
prompt-conditioned adapter-family susceptibility gains weight
elif rare random directions can also reproduce the basin:
basin access is direction-selective but not unique to the learned semantic direction
I would probably stop expanding the random null now unless Regime A becomes the main paper claim. The information gain from another few hundred random directions seems smaller than the information gain from the Regime-B side.
Regime B: I think this becomes clearer if the channel is split into interfaces
For me the useful decomposition is something like:
| Interface |
Question |
| source representation |
What information is actually present in the source vectors? |
| bridge |
What survives translation into the target coordinate system? |
| landing site |
Does the translated vector land in a region the receiver knows how to process? |
| ordering / binding |
Are identity, relation, and token/fragment order preserved? |
| readout |
Can the receiver access the payload under this prompt/protocol? |
| use |
Can the recovered information affect an answer or inference? |
This seems compatible with your own address / state / payload / speech separation rather than replacing it.
The strongest thing the oracle gives you, in my view, is a receiver-side channel ceiling:
under some ordered target-space writes and readout protocols, a frozen Llama can recover and use information that was not present as visible text.
That is already useful.
But I would keep that statement very explicitly separate from:
a Qwen3 representation can be translated through the current ridge bridge into a robust, identity-preserving Llama payload.
The public actual-bridge examples still look more like semantic transmission:
fire burning hamster
and:
A wolf is a fire breathing dragon.
There is clearly memory-specific material getting through, but the nonce identity/binding is not yet surviving the same way as the target-space oracle.
That makes the next problem much more concrete: not “can vector memory exist?” but “what kind of bridge + landing state preserves the information the receiver actually needs?”
That is a much easier problem to compare against existing work.
Regime B: oracle, identity/binding, readout, and scoring boundary
There are a few evidence levels here that I would keep visibly separate.
Oracle channel ceiling
The Worb-glob oracle is especially informative because the same payload behaves differently depending on the readout:
readback can produce the complete Worb-glob statement;finish can recover the predicate/content;ask can return essentially nothing.
So I would distinguish:
payload is readable under a suitable decoder/readout
from:
payload is transparently accessible from arbitrary natural-language queries
The first is supported much more strongly than the second.
That is not just a prompt-engineering nuisance; I think readout protocol is part of the channel interface.
Identity/type/property should probably stay separate
The nonce controls are useful, but I noticed something worth making explicit in the evidence labels.
For example, a stored oracle HIT can produce:
A penguin that glows.
for a proposition whose full nonce identity includes “drivel snib.”
Likewise another row preserves beekeeper/night information while changing the nonce identity.
Those are good demonstrations that content survives, but under the strict definition in your current Methods they are not the same thing as preserving every identity/type/property binding conjunctively.
The current Methods criterion is actually the cleaner one:
exact payload recovery:
all required frozen content groups
in one generation
semantic transmission:
meaningful memory-specific components survive,
but the full proposition is not required
I would keep the older receipts rather than rewriting history, but perhaps make their scoring provenance explicit:
legacy_verdict
current_strict_rescore
rubric_version
That would eliminate a lot of possible confusion for later readers.
The reversed-order controls reinforce the same point. Some content words survive reversal while the relationship/binding becomes wrong. That is useful evidence in favor of treating content direction and ordered binding as separate channel properties.
Matched vs cross-memory
The oracle cross-memory controls are some of the cleanest evidence in the set.
Correct memory + matching question succeeds, while wrong-memory and blank conditions do not simply produce the other memory’s content.
That is stronger evidence for content-specific transmission than merely comparing “vector present” versus “no vector.”
There is a related evaluation lesson in the recent causal audit When Does Latent Communication Pay?: a large effect from inserting a latent object is not automatically evidence that the correct example-specific information in that object caused the gain. Their matched/mismatched/zero/random style of control seems like a useful general pattern here too.
For the actual bridge, if you find a setting above the current floor where readback is somewhat successful, the highest-information comparison might simply be:
correct mapped memory
mismatched mapped memory
blank
norm/moment-matched random
order / coordinate controls
If the matched memory separates cleanly there, that would say more than adding a much larger benchmark.
Rank curve
I think the rank curve is one of the strongest parts of the paper.
At low rank the Qwen3 representation can retain very strong pairwise retrieval geometry while token identity, proper names, and relations reconstruct much more slowly.
That gives a quantitative reason not to use one word such as “semantic preservation” for all of these interfaces.
The only wording distinction I would keep is:
the rank curve directly measures retrieval/reconstruction properties at each rank;
not:
the rank curve directly measures reasoning capacity at each rank.
The latter would require a downstream behavioral experiment across the rank sweep.
The write-site result may be more central than it first looks
The input-marker versus live-residual result seems important because it says the bridge cannot be evaluated independently of where the vector lands.
A vector can be geometrically “close” to the intended target representation and still land in a part of state space that the next block immediately re-expresses or disrupts.
That makes me think the practical channel is closer to:
source representation
-> bridge
-> calibration
-> landing site
-> downstream transformer dynamics
-> readout
rather than just:
source vector -> regression -> target vector
This is also where some nearby work becomes useful.
xRAG is a close conceptual neighbor on the “dense embedding becomes directly readable LM input” side: the retriever and LM are frozen and only the modality bridge is trained. Its goal is compressed RAG rather than ordered nonce identity/relation transfer, but it is a useful precedent for treating a retrieval embedding as something other than an address. The xRAG code is also public.
On the cross-model side, Activation Space Interventions Can Be Transferred Between Large Language Models transfers activation-space interventions across Llama/Qwen/Gemma families, and Transferring Linear Features Across Language Models With Model Stitching uses affine residual-stream mappings to transfer probes, SAE features, and steering vectors.
So I would probably avoid making “an affine map between model spaces can work” carry too much of the novelty burden. There is already a fairly substantial neighboring literature for that.
What looks more distinctive here is the channel decomposition:
- compact address/retrieval geometry;
- state steering;
- ordered payload;
- identity/relation bandwidth;
- landing-site lifetime;
- downstream readout.
That is also consistent with the caution from Functional Alignment Can Mislead: successful functional alignment does not by itself imply that two representations contain equivalent information.
Nearby work I would probably connect, without changing your origin story
I would keep your current experiment-driven origin story if that is historically accurate.
I would just separate:
“where these experiments came from”
from:
“where a reader should place them relative to current work.”
A short “Nearby / related work” map would probably be enough.
Cross-model activation / feature transfer
These help delimit what an affine/learned cross-model correspondence establishes and what it does not.
Dense embeddings as LM-readable features
This seems particularly relevant to the “vector is memory, not merely an address” framing, while still differing substantially in objective and payload structure.
Latent model-to-model communication
Interlat explicitly studies heterogeneous latent communication.
StateBridge is interesting from a different angle because it treats input-space compatibility as a first-class engineering issue: centering/whitening, orthogonal alignment, norm calibration, vocabulary anchoring, then injection through inputs_embeds.
Its current public release is scoped to homogeneous agents sharing pretrained weights, with heterogeneous sender/receiver transfer left as future work, so I would not cite it as a solved Qwen3→Llama case. But its alignment/calibration ablations are a useful comparison for your landing-site problem.
If you eventually compare bridges, a small matrix such as:
ridge
orthogonal / Procrustes
+ norm calibration
+ optional vocabulary anchoring
with both:
static reconstruction metrics
and
behavioral readback / binding metrics
could be much more informative than optimizing cosine alone.
One thing I would probably improve before adding another large experiment: Regime-B replayability
The public receipts are useful; I could follow quite a lot from them.
But while following the Regime-B path I could not locate several of the execution/scoring artifacts named in the logs in the currently linked public repository/archive — for example the soft-slot runner and some of the E10 raw/preregistration/scorer material.
If those already live somewhere else, this may be nothing more than a missing link.
If not, I think a very small reproduction bundle would give unusually high value for the effort.
Something like:
1. current soft-slot injection runner
2. bridge-building / split-construction script
3. exact endpoint revisions + quantization/settings
4. frozen rubric/scorer version
5. E10 preregistration + raw rows + summary
6. one command reproducing:
- one oracle cell
- one actual-bridge cell
No need to redistribute model weights.
That would let somebody distinguish:
source representation issue
bridge issue
landing-site issue
readout issue
scoring/rubric issue
without having to reconstruct the experimental pipeline from receipts.
I also think explicitly versioning the evaluation rule would help:
semantic transmission
identity-preserving payload recovery
binding/order recovery
downstream use
That seems especially useful because your current paper already has a stricter and clearer definition than some of the earlier stored HIT labels.
If you want one more high-information Regime-B experiment later
I would not start with a giant benchmark.
I would first find one actual-bridge configuration that is clearly above floor but below oracle, then hold everything else fixed.
A. Pairing specificity
correct memory
wrong memory
blank
matched random
coordinate permutation
reversed order
Question:
Is the receiver using information specific to the mapped source payload, rather than merely reacting to a continuous prefix?
B. Separate the payload dimensions
Instead of one semantic score:
nonce identity
entity/type
predicate/relation
order/binding
full conjunctive proposition
downstream use
This would make a failure much more informative.
For example:
type + predicate survive
identity fails
is a very different engineering problem from:
identity survives
relation swaps
or:
everything is present in an oracle readback
but natural queries cannot access it
C. Bridge versus landing site
If a geometry-preserving/calibrated bridge improves static metrics:
Does readback improve?
Does identity improve?
Does reversed-order separation improve?
Does inference improve?
If static reconstruction improves but behavior does not, I would stop optimizing the bridge and look at landing/readout.
D. Connect write lifetime to behavior
Your layer traces already show that input-slot writes and live residual writes have very different survival.
A small behavioral comparison using the same payload and norm at a few write sites could connect that mechanistic observation to actual readability:
input slots
block 1 residual
block 4 residual
block 8 residual
Then measure payload reconstruction / binding / inference rather than cosine alone.
That would make “landing site is part of the channel” substantially stronger.
If I had to prioritize
My ordering would probably be:
- Keep the current state/payload/readout separation.
- For Regime A, use the conservative headline: local prompt-conditioned, direction-selective state transition unless later factorial evidence supports something stronger.
- For Regime B, keep oracle ceiling and actual cross-model bridge explicitly separate.
- Add a compact Related Work section so readers can distinguish this from xRAG, model stitching, transferred steering, and latent-agent communication.
- Make the Regime-B replay path/rubric version easier to follow.
- Only then spend more compute, preferably on one pairing-specific or bridge-vs-landing experiment that actually changes the claim boundary.
I think that route preserves the original idea rather than shrinking it: “ontological inversion” can remain the historical/name-of-experiment piece, while the more general result becomes a cleaner question about what different continuous interfaces can carry between frozen models, at what bandwidth, and under what landing/readout conditions.
That, to me, is the part that now looks most reusable beyond the individual Glub-Tub example.
