Is the q_proj/v_proj default worth revisiting? Measurements across four models

[Acedo]

The target_modules default that most tutorials and configs use, query and value projections, comes from the original LoRA paper. I have been measuring how well it holds when the objective is retaining a previous capability rather than accuracy on the new task, and I wanted to leave the numbers here in case they are useful to anyone thinking about defaults.

The measurement compares, before any training, the rank-r gradient subspaces of two tasks module by module, and selects where they overlap least. On Llama-3-8B fine-tuning code to prose, over ten paired seeds with the same number of adapted modules in both arms, HumanEval retention goes from 54.1% with the q,v default to 75.6% with the measured placement, 9 of 10 seeds. The selected set contains no v_proj at all, which is half of the default.

Three things I would rather state myself:

The type ordering reproduces across Llama-3-8B, Mistral-7B, Qwen2.5-7B and TinyLlama-1.1B, with o_proj first and v_proj last or absent. The depth pattern does not: two of those four concentrate in deep layers and two do not, so a fixed rule cannot know which case it is in.

A cheap rule over module type and layer depth ranks well (AUROC 0.83 to 0.92) but selects a different set, about a third of the modules differ. Whether that changes retention is a run in progress, pre-registered, and I will post it either way.

One honest caveat: the implementation is AGPL, so it is not something you could pull into an Apache-2.0 project. I am not proposing an integration, just leaving the numbers.

Anyone can reproduce the measurement on their own model in a free Colab, which prints a target_modules list ready to paste into LoraConfig:

Google Colab

Code and per-seed data: https://github.com/BiomeMakers/TIM-OmegaS
Paper: https://doi.org/10.5281/zenodo.21999659

[John6666]

For now, after running a few experiments in Colab, I think I found something along these lines:

---

Short answer: yes — I think the q_proj / v_proj default is worth revisiting. But I would separate two questions:

1. Is q/v a universal optimum? Probably not.
2. Does the current TIM scalar score already provide a universally calibrated replacement? My small tests do not support that stronger claim yet.

What I found instead is, to me, a more interesting middle ground: the TIM module-level diagnostic seems to contain useful signal, but placement quality can depend strongly on interactions between selected modules, enough that simply averaging/adding independent module scores can miss a very bad configuration.

That suggests a relatively cheap workflow:

TIM diagnostic
    ↓
small shortlist of candidate placements
    ↓
verify actual PEFT targets + trainable parameter budget
    ↓
tiny local / one-swap neighborhood check
    ↓
compare retention at matched new-task acquisition
    ↓
choose the final placement

In other words, I would not throw away the measurement idea. I would probably separate measurement, candidate generation, configuration validation, and final selection.

There is also an important distinction depending on what “better placement” is supposed to optimize:

Maximum downstream task performance
    → all-linear is an important baseline

Small adapter / fewer adapted modules / serving efficiency
    → selective placement is directly the optimization problem

Retention-aware or continual adaptation
    → compare old-task retention at comparable new-task learning
       and placement/interference diagnostics become especially relevant

That distinction matters because the original LoRA q/v choice was not arbitrary. The original LoRA work, under its GPT-3 attention-only fixed-parameter-budget ablation, found q+v to be a strong configuration, and the official LoRA repository describes q/v as a “simple yet effective setup”. But the same repository explicitly says LoRA can be applied to other subsets and that the optimal configuration is likely to vary by architecture and task.

Later work makes that boundary even clearer. QLoRA found, for its LLaMA-7B Alpaca experiment, that standard q/v LoRA did not match the 16-bit baseline and that adapting all linear Transformer-block layers was important for matching it. Current PEFT LoRA consequently supports explicit targets as well as target_modules="all-linear". So I would treat q/v as a useful historical baseline, rather than as something that needs to be defended as a theorem.

For a recent independent example that target choice still matters materially, Amazon’s Nova 2.0 Lite target-module ablation found quite different behavior among qkv, o_proj, feed-forward targets and combinations across tasks. That does not validate TIM — it is a different model and objective — but it does support the broader premise that module placement is a real optimization variable.

A few cheap checks also seem worth doing before making the selector itself more complicated:

• compare actual LoRA trainable parameters, not only the number of targeted modules;
• record the modules that PEFT actually adapted;
• evaluate acquisition and retention together, rather than retention alone;
• keep generated baselines and published/stored baselines distinguishable if they are not literally identical.

For the first two, current PEFT guidance for custom models already exposes useful sanity checks:

peft_model.print_trainable_parameters()
print(peft_model.targeted_module_names)

I found these surprisingly useful, because a configuration that looks structurally matched can still differ in actual LoRA capacity or in the modules PEFT really attached to.

What I tested in Colab

Scope first

This is not a reproduction of the four-model headline experiment.

I used:

• HuggingFaceTB/SmolLM2-135M
• a fixed small synthetic code/prose task pair
• three seeds for the GPU comparisons
• a five-depth panel at layers 0, 7, 14, 22, 29
• q_proj, k_proj, v_proj, o_proj
• the same general TIM score construction used by the implementation I was testing
• explicit checks of PEFT targets and actual trainable parameters

So I would read the results as a controlled sanity experiment about the selector and placement behavior, not as evidence that the same numerical effect must occur on Llama-3, HumanEval, CodeSearchNet/OpenWebText, etc.

1. The module diagnostic itself was reasonably stable

Before training anything, I repeated the small TIM measurement on two independent sample blocks.

On the later calibrated run, module-score agreement was approximately:

• Spearman: 0.8075
• Kendall: 0.6316

So, at least in this toy setup, “the TIM score is just random sample noise” did not look like the best explanation.

I also did an earlier parameter-matched training comparison between:

• a stable TIM-selected placement,
• an o_proj placement,
• a budget-matched q/v placement,
• and a simple Task-B-only gradient-salience placement.

Under a first-crossing control for Task-A learning, the selected TIM placement was encouraging: it beat the matched q/v arm on the main retention/final-A/final-B readouts across the three toy seeds, and its retention was better than the simple o_proj and Task-B-salience alternatives.

I would call that positive selector evidence, but narrow evidence. It does not by itself tell us what the numerical TIM score means globally.

2. So I tested the stronger score claim directly

The next question was:

If TIM score is being used as a placement-quality quantity, do progressively higher-scoring placements actually forget progressively less?

To make that harder to explain away with simple capacity differences, I enumerated 390 candidate placements and selected five score quantiles while holding the gross placement structure fixed.

Every selected placement had:

• 6 target modules
• exactly 2 o_proj + 1 q_proj + 1 k_proj + 2 v_proj
• coverage of all five panel layers
• exactly 46,080 actual PEFT trainable parameters
• the same data and optimizer setup
• the same three seeds
• Task-A checkpoints selected by a common earliest first-crossing criterion before Task B

The result was:

score band	mean TIM placement score	mean Task-A forgetting ΔNLL
very low	0.450847	-0.0385
low-mid	0.482286	-0.0809
middle	0.494613	+1.2799
high-mid	0.508072	-0.0878
very high	0.528385	-0.0004

The expected direction, if “higher score = safer retention”, would be a negative score/forgetting correlation.

Observed Spearman was +0.10.

For that expected negative direction, the exact one-sided permutation p-value was 0.6083.

With only five placements this is obviously not a high-powered statistical calibration study, so I would not interpret this as “TIM is disproven”. The narrower statement seems much safer:

The tested module-level diagnostic was reasonably stable, but its simple placement-level scalar aggregation was not a monotonic predictor of forgetting in this matched small placement space.

Those two observations are compatible. A signal can be useful for generating candidates without its numerical value being a calibrated global utility function.

3. The strange part was the middle placement

The middle placement was:

layer 0   q_proj
layer 7   o_proj
layer 7   v_proj
layer 14  o_proj
layer 22  v_proj
layer 29  k_proj

In the original calibration its Task-A forgetting ΔNLL was:

seed 11   +1.3542
seed 37   +1.2379
seed 83   +1.2476
mean      +1.2799

The other score-quantile placements were near zero or slightly negative.

Task-B training loss still fell normally, so this did not look like a simple “training failed to run” case.

At that point, the obvious structural clue was the colocated o_proj + v_proj at layer 7. But that still leaves several explanations:

• o7 itself might be problematic;
• v7 itself might be problematic;
• the specific o7 + v7 pair might be problematic;
• any duplicated placement at layer 7 might be problematic;
• the destination topology of the other modules might matter;
• or the six-module configuration might have a higher-order interaction that cannot be assigned to one module.

A full combinatorial sweep seemed unnecessary, so I tried the smallest local intervention I could think of.

4. Strict one-swap neighborhood

I reran the center plus six neighbors.

Each neighbor changes exactly one of the two layer-7 modules while preserving:

• 6 modules,
• 2o + 1q + 1k + 2v,
• all five depth locations represented,
• LoRA rank/alpha,
• optimizer and data,
• 46,080 actual trainable parameters,
• the three seeds,
• and seed-local Task-A first-crossing matching.

The pathological center also reproduced qualitatively in this second experiment:

middle forgetting ΔNLL
seed 11   +1.2671
seed 37   +0.7483
seed 83   +1.0657
mean      +1.0270

The exact magnitude moved because the seven-arm experiment recomputed the seed-local common reachable Task-A target, but the large center pathology remained in every seed.

Then the neighborhood did this:

placement	score	Δscore vs middle	mean Task-A forgetting ΔNLL	mean Task-B acquisition ΔNLL
middle	0.494613	0	+1.0270	+0.0581
o7 → o0	0.459956	-0.034657	-0.0378	+0.1546
o7 → o22	0.464996	-0.029617	-0.0158	+0.1847
o7 → o29	0.465893	-0.028719	-0.0498	+0.1903
v7 → v0	0.487957	-0.006656	+0.0207	+0.1127
v7 → v14	0.492006	-0.002607	+0.1098	+0.0762
v7 → v29	0.507622	+0.013010	-0.0378	+0.1413

Every one of the six strict neighbors reduced forgetting substantially relative to the center for all three paired seeds.

I would not call that “18 independent successes” — those paired comparisons are strongly related — but the directional pattern is difficult to miss.

5. It was not simply “the neighbor learned Task B less”

This was important to check because LoRA retention can otherwise be deceptive.

The paper LoRA Learns Less and Forgets Less gives a good general warning here: less forgetting can accompany less learning of the target distribution.

That is why I would avoid interpreting retention without an acquisition measure.

In this one-swap experiment, however, the rescue does not look like that trivial trade-off.

The pathological middle arm had mean Task-B acquisition of only about 0.058 NLL, while the neighbors were around 0.076–0.190 NLL.

So the neighbors generally:

• learned Task B more, and
• forgot Task A dramatically less.

That looks more like escaping a bad placement/configuration region than merely reducing plasticity to preserve Task A.

I still would not infer a specific mechanism from that result alone, but it makes “they simply did not learn B” a poor explanation here.

6. What seems localized — and what does not

The result makes the exact layer-7 configuration a strong suspect, but I would keep the wording narrow.

It is not enough to say “putting o_proj and v_proj in the same layer is generally bad.”

For example:

• o7 → o22 creates an o22 + v22 pair and rescues the pathology;
• v7 → v14 creates an o14 + v14 pair and also rescues it.

So generic o+v co-location is not sufficient to explain this toy result.

What the experiment really says is closer to:

Something about the particular layer-7 configuration, or its interaction with the rest of this placement, matters a lot.

A useful next localization experiment if this phenomenon itself becomes interesting would be a tiny layer-7 pair-identity panel:

o7 + v7     current pathological center
q7 + v7
q7 + o7
k7 + v7
k7 + o7

with the same type-count / layer-coverage / parameter-budget controls.

That would distinguish “specifically o7+v7” from “two adapters at layer 7 are problematic”.

But I would regard that as a research extension, not something needed before TIM is useful.

7. Where simple additivity becomes questionable

The cleanest counterexample to a simple additive interpretation was v7 → v29.

Its mean scalar placement score moved up:

middle       0.494613
v7 → v29    0.507622

yet its Task-A forgetting moved from approximately:

+1.027  →  -0.038

So the local behavioral improvement was enormous while the additive score changed in the opposite direction from what a simple “higher score = less forgetting” interpretation would predict.

That is why I currently think the useful distinction is:

module-level diagnostic signal
        ≠
complete configuration-level utility function

The first can be useful even if the second needs an interaction term.

There is some broader theoretical precedent for being cautious about additivity here. The recent paper Understanding and Guiding Layer Placement in Parameter-Efficient Fine-Tuning of Large Language Models explicitly treats layer coupling as a separate quantity, with approximately additive layer contributions arising in a weak-coupling regime. I would not claim that paper explains this particular layer 7 observation — different setup, theory and objective — but it gives a useful language for the general issue: independent per-location measurements need not compose additively when adaptation sites interact.

How I would separate TIM's roles

The result above makes me think TIM may become easier to reason about if four roles are kept separate.

1. Measurement

Measure the Task-A / Task-B gradient geometry at individual candidate modules.

This is where the distinctive part of TIM lives.

The small repeated-block experiment gave me some confidence that there is a reproducible signal here.

2. Candidate generation

Use the measurement to shrink the placement search space.

This does not require claiming that the scalar score is perfectly calibrated.

For example, the output could be:

top candidate
+
a few near-top structurally different alternatives

rather than:

highest mean score = final answer

3. Configuration validation

Before doing the expensive run:

• verify actual adapted modules,
• verify the actual trainable parameter budget,
• perturb the proposed placement locally,
• and run a short matched-acquisition check.

The one-swap result makes this step look particularly high-information.

You do not need a huge search. Four to six deliberately chosen neighbors may already reveal whether the top candidate is sitting on a sharp interaction pathology.

4. Final decision

Pick the operating point using the quantity actually relevant to the use case.

For continual adaptation I would prefer something like:

new-task acquisition
vs.
old-task forgetting / retained old-task performance

rather than ranking placements by forgetting alone.

If the goal is ordinary single-task SFT performance instead, I would change the baseline set rather than forcing a continual-learning objective onto it.

The baseline depends on the goal

I think this distinction could prevent several comparisons from becoming unnecessarily confusing.

If the goal is maximum downstream performance

Include an all-linear configuration.

This is the natural lesson from QLoRA: in its LLaMA-7B Alpaca experiment, q/v-only LoRA did not match the tuned 16-bit baseline, while LoRA on all linear Transformer-block layers did.

That does not imply all-linear is always best. It means it is an important performance-oriented reference point.

If the goal is sparse adapters or inference efficiency

Then all-linear may be the wrong target entirely.

Now the real question becomes:

Which small subset buys most of the useful adaptation?

This is where a placement selector is directly valuable.

A recent example is the Amazon Nova 2.0 Lite target-module study, where o_proj, qkv and feed-forward combinations produced materially different accuracy/latency trade-offs across tasks.

Again, that particular result is model-specific. I mainly cite it because it makes the optimization problem concrete: target selection can matter precisely because adding adapters everywhere is not free at serving time.

If the goal is retention-aware adaptation

Then TIM has a particularly interesting niche because it explicitly compares Task-A and Task-B geometry rather than only asking “where is Task B salient?”

That is a different problem from:

• static weight geometry,
• ordinary new-task gradient salience,
• rank allocation,
• or simply adapting every linear layer.

Current PEFT even contains a different automatic target selector, KappaTune / find_kappa_target_modules, based on weight condition numbers. That might make a useful low-integration-cost comparator if you ever want one.

I would not add a zoo of selectors immediately. One contrastive selector is probably enough to answer the useful question:

Is Task-A/Task-B interference geometry adding information beyond a task-agnostic/static placement heuristic?

A few implementation/reproducibility notes that looked cheap to fix or record

These are secondary to the main result, but they were cheap checks and may help make future comparisons cleaner.

1. Same target-module count does not necessarily mean the same LoRA parameter budget

For the Llama-3 configuration I checked, the stored TIM and conventional placements both contained 31 target modules, but because Llama-3 uses GQA, the projection shapes are not all equal.

At rank 8, I calculated:

TIM placement          1,933,312 LoRA trainable parameters
conventional placement 1,662,976 LoRA trainable parameters

So the suggested placement has roughly 16% more LoRA parameters despite having the same number of targeted modules.

I would not interpret this as “therefore the reported result disappears”. It only means that a pure placement interpretation becomes cleaner if actual LoRA capacity is either:

• matched,
• or at least reported beside module count.

This is one reason I like leaving print_trainable_parameters() in the experiment log.

2. Parameter names and PEFT module targets are slightly different contracts

The diagnostic naturally produces names like:

model.layers.7.self_attn.q_proj.weight

but PEFT’s normal target_modules path targets modules such as:

model.layers.7.self_attn.q_proj

In the PEFT version I tested, passing the .weight form through the normal target_modules route did not behave like targeting the module name.

Current PEFT LoRA documentation also explicitly distinguishes:

• target_modules for modules such as nn.Linear;
• target_parameters for cases where an nn.Parameter itself needs to be targeted.

So I would make that conversion an explicit boundary in the code and then assert the result:

peft_model.print_trainable_parameters()
print(peft_model.targeted_module_names)

That turns a potentially silent targeting mistake into a cheap runtime failure.

3. Published and regenerated baselines can be named separately

In the revision I tested, the fresh conventional q/v constructor did not reconstruct the stored/published conventional placement exactly, even though both were q/v-style placements.

That can be made unambiguous without changing the method at all by keeping artifacts named along the lines of:

published_conventional
generated_conventional

and recording the exact target list beside each result.

That also makes future comparisons easier if baseline generation logic changes.

4. A small amount of provenance buys a lot

For each reported arm I would keep:

model revision
PEFT version
seed
rank / alpha
exact requested targets
actual targeted_module_names
actual trainable parameter count
placement JSON / hash
Task-A matching criterion
Task-B training budget

Most of this costs almost nothing and makes placement experiments much easier for another person to reproduce.

If I were choosing follow-up work by information gain

I would roughly order it this way.

Tier 0: nearly free

Using existing runs:

• report actual LoRA parameter counts;
• record exact targeted_module_names;
• put acquisition and retention in the same table;
• plot TIM placement score against measured forgetting;
• distinguish stored/published and freshly generated baselines.

Tier 1: small GPU cost — probably my default

For a TIM-selected candidate:

1. keep the candidate;
2. construct a handful of same-budget one-swap neighbors;
3. match new-task acquisition approximately;
4. compare old-task retention;
5. only take survivors into the expensive evaluation.

This is the part that my Colab result changed my mind about most. A tiny local perturbation exposed much more information than another broad score sweep probably would have.

Tier 2: only if the selector itself is the research question

Add one comparator, not ten.

For example:

• PEFT’s KappaTune selector,
• a simple Task-B gradient/Fisher salience score,
• or another cheap static selector.

Then the interesting result is not merely “placement A beats q/v”, but:

Does the Task-A/Task-B relation measured by TIM contain useful information beyond simply selecting modules that look adaptable or important for Task B?

Tier 3: only if the interaction effect reproduces elsewhere

Then it could be worth extending the score itself:

placement utility
≈
module terms
+
pair / topology terms
+
possibly higher-order terms

But I would not start there.

The strict one-swap result is strong enough to justify checking whether this interaction behavior reproduces on another task/model, but not strong enough to justify building an elaborate interaction model immediately.

So my current interpretation would be:

q_proj/v_proj is still a perfectly reasonable baseline, and it has a real empirical history behind it. But I would not treat it as an architecture-independent optimum.

I also would not interpret my small experiment as a rejection of TIM. Quite the opposite: I saw a reasonably stable module-level signal and one encouraging selected-placement result. The main boundary I found was that the simple scalar/additive placement interpretation was much weaker than the underlying diagnostic signal, and a very small local intervention exposed a large configuration effect.

That makes me think a practical version of the idea could be:

TIM for interference-aware candidate generation, followed by a cheap configuration check, with the final choice made on an acquisition–retention operating point.

That seems to preserve the useful part of the idea while avoiding the need to assume that independently measured module scores compose perfectly.

[Acedo]

Thank you for actually running it. That is more than I expected from a forum post, and the parameter-budget point is a hit. You are right, and it is worse than a nitpick.

I matched the arms on module count, which is not the thing that needs matching. Llama-3-8B uses grouped-query attention, so q_proj and o_proj are 4096x4096 while k_proj and v_proj are 4096x1024. Running the numbers on the exact lists in the repo, at r=8:

• measured placement (17 o_proj, 10 q_proj, 4 k_proj): 1,933,312 trainable parameters
• conventional (16 q_proj, 15 v_proj): 1,662,976

So the arm I report as better carries 16.3% more trainable capacity. Equal module count, unequal budget. The paper does not state that and it should; the numbers are above for anyone reading this thread.

One piece of luck: the control arm I designed and pre-registered uses the published type composition (17/10/4) with layers drawn at random, one draw per seed, precisely because the paper reports composition but not depth. That arm is parameter-matched to the measured one by construction, so this confound does not reach it. Not foresight, but I will take it.

On composition, I think you have found the real boundary of the method and I do not have a defence. The scores are measured per module, independently, and then I take a top quartile as if they added up. Your one-swap result says they do not. The diagnostic can carry signal while the selection rule built on top of it is not optimal, and those are separable claims that I conflated. Your framing, TIM as interference-aware candidate generation followed by a cheap configuration check rather than as a final selector, is more defensible than mine, and I would rather adopt it than argue for the stronger version.

The two sanity checks are cheap and worth having, and I will add them when I next touch the harness: targeted_module_names and print_trainable_parameters() are a real verification, where I had a string check on a log line. Same for reporting acquisition alongside retention rather than retention alone. Retention is a ratio, so it can flatter an arm that simply learns the new task less, and that should be visible rather than inferred.

Your historical framing is also more accurate than mine: q+v came from a fixed-budget attention-only ablation, and QLoRA already found that matching the 16-bit baseline needed all linear layers. I had been treating the default as unexamined, when the record is that it was examined under conditions that no longer hold. That is a better sentence than the one in my paper.

What I would ask, if you are willing: which base model and which task pair, and what the one-swap actually changed. The collapsed sections did not come through in the email notification. Even the raw numbers would help, and I would report them as yours. A configuration effect large enough to show up in a small test is exactly the kind of thing that should bound the claim, and at the moment I only have your description of it.

The retention control is designed and the module lists are frozen. Whether it gets run depends on compute I have not committed yet. If I run it, the result goes here whatever it says.

[Acedo]

Your comment made me run the control arm I had pre-registered and not yet run. Ten seeds, finished today. It answers both of your points, so here are the numbers first.

The control is the placement a reader of the paper could actually build without measuring anything. The paper reports the type composition of the selected set (17 o_proj, 10 q_proj, 4 k_proj) but not its depth distribution, so such a reader has the composition and has to pick layers blind. That is the arm: same composition, layers drawn at random, one draw per seed, frozen before any result existed.

HumanEval retention on Llama-3-8B, code to prose, 10 paired seeds, 31 adapted modules in every arm:

• measured placement: 0.7557
• conventional q,v: 0.5405
• same composition, random depths: 0.3923

Paired difference, measured minus random-depth: +0.363, sd 0.172, t = 6.7, winning in 10 of 10 seeds. Pre-registered indistinguishability threshold was 0.066.

Why this is a cleaner test than what the paper had. The two arms share type composition, so they also share the trainable parameter budget exactly. The only thing that differs is which layers. The entire gap is therefore attributable to depth selection, which is precisely the information the probe provides and the paper does not print. Measuring beats reading the paper, by a wide margin.

Your parameter-budget point is right about the published comparison, and worth stating plainly: under grouped-query attention, q_proj and o_proj are 4096x4096 while k_proj and v_proj are 4096x1024, so at r=8 the measured set is 1,933,312 trainable parameters against 1,662,976 for q,v. Equal module count, 16.3% unequal budget. It does not reach the new comparison, but the paper should say it, and it will.

On acquisition, which you were right to ask for: the random-depth arm reaches higher HumanEval after task A (0.312 vs 0.274) and retains far less. So the measured arm’s advantage is not a ratio flattered by an arm that learned less. It learned less and retained more.

One result I did not expect: the reader’s placement is also worse than plain q,v, by 0.148 in 8 of 10 seeds. Publishing a composition without its depth distribution is worse than publishing nothing, which is a concrete argument for shipping the full placement rather than a summary of it.

I also re-ran three seeds of the measured arm in today’s environment, since the published arms were run in August on different hardware. The gap holds within a single environment (+0.337, 3 of 3). Per-seed values do not transfer across machines, though: seed 42 went from 0.533 to 0.805 with identical code and seed. Distributions reproduce, individual seeds do not, and that goes in the reproducibility notes.

Where your critique still stands, because today’s result does not touch it: this shows the measured set beats what a reader can construct, not that the top-quartile rule is optimal. Independently measured scores composing additively is an assumption I make and you tested; your one-swap result says it does not hold exactly. Candidate generation plus a cheap configuration check is a better description of what this tool is good for than the one I gave.

Twenty per-seed JSON files, the frozen module lists and the pre-registration go in the repo. If you are willing to share which model and task pair you used and what the one-swap changed, I will run it against this setup and report the result as yours.

[Acedo]

TIM: Adapter Placement by Task Interference Mapping | Zenodo update

[John6666]

Hi. For now, I’ve uploaded the Notebook here:

SmolLM2 strict one-swap reproduction — executed Notebook

---

Short version

The new same-composition control in v12 changes the picture in a useful way.

My read now is:

1. The measurement/configuration side of TIM has substantially stronger evidence than it did before. With the same 17 o_proj / 10 q_proj / 4 k_proj composition and therefore the same LoRA parameter budget, the measured configuration still beats the reader/random-depth configuration by a large margin.

2. That does not automatically validate the scalar aggregation/top-quartile selector. The SmolLM2 one-swap reproduction below keeps module count, projection-type composition, LoRA settings, actual PEFT parameter count, seeds, and the Task-A matching rule fixed, yet every one of six strict one-module neighbors reduces forgetting relative to the center. More importantly for the score-ordering question, 5/6 of those rescues move the summed score downward.

3. I also tried transporting the local test to an independent Llama-3.2-3B setup. That produced a useful boundary rather than a simple win: a lower-scoring one-swap retained an advantage after matching the original Task-A evaluation state, but the advantage did not transfer to a separate untouched family of Task-A functions. So I would currently treat the Llama result as evidence of local/configuration sensitivity, not as a general retention-only effect.

That leaves a workflow that seems fairly practical to me:

TIM measurement
    ↓
candidate / shortlist generation
    ↓
verify actual PEFT targets + trainable capacity
    ↓
small same-budget local configuration check
    ↓
control the pre-B Task-A state
+
verify Task-B acquisition
    ↓
choose the final acquisition/retention operating point

In other words, I think the new control makes the measurement side more convincing, not less. The remaining question is how much work the current scalar/top-quartile selection rule can safely do by itself.

What I get from the new v12 control

For the new same-composition comparison, I get the following from the v12 results rather than the earlier forum速報 values:

retention ratio
--------------------------------
measured             0.721616
reader/random-depth  0.392303
paired difference   +0.329313
measured wins        10/10

I think that is a meaningful positive control.

Both arms use:

17 o_proj
10 q_proj
 4 k_proj

so this comparison no longer has the GQA-dependent LoRA-capacity mismatch that complicated the original measured-vs-q/v comparison.

The narrower claim I would attach to it is:

In this Llama-3-8B code→prose experiment, the full measured layer/configuration pattern contains useful information that is not recoverable from the projection-type histogram alone.

That seems quite well supported now.

I would still stop short of calling Table 3 a clean isolation of an independent “depth score”, because the measured and reader arms differ in the whole layer/configuration pattern: exact layers, clustering/co-location, and whatever interactions come with training those modules together all move at once.

So to me it is best read as a positive control for the full measured configuration, not yet for additive per-module utility.

There is also a denominator issue with the retention ratio. The Task-A state before B is not identical between the two arms, so I would put substantial weight on the absolute after-B result as a robustness check:

absolute HumanEval after Task B
--------------------------------
measured             0.195732
reader/random-depth  0.118293
paired difference   +0.077439
measured wins        10/10

That is the part I find especially reassuring. The effect is not only a ratio artifact.

The reader arm also reaches somewhat higher Task-A capability before B (~0.312 vs ~0.273), so the simple explanation “the measured arm retained more only because it learned less Task A” does not fit the observed direction. I would still keep the absolute after-B result alongside the ratio rather than treating the ratio alone as the endpoint.

The exact one-swap reproduction you asked for

The Notebook at the top is a small self-contained reproduction on HuggingFaceTB/SmolLM2-135M.

This is deliberately not presented as a Llama-3/HumanEval reproduction. It is a local configuration test where the invariants are cheap enough to check explicitly.

The tasks are also simpler than the Llama experiment:

Task A:
deterministic notebook-generated synthetic Python/code

Task B:
deterministic notebook-generated synthetic explanatory prose

The center has six LoRA targets:

layer 0   q_proj
layer 7   o_proj
layer 7   v_proj
layer 14  o_proj
layer 22  v_proj
layer 29  k_proj

Every arm keeps exactly:

2 o_proj
1 q_proj
1 k_proj
2 v_proj

LoRA rank     8
alpha        16
dropout       0

actual trainable PEFT params:
46,080

Only one o_proj or v_proj at layer 7 is moved in each neighbor; the other five modules remain unchanged.

ΔNLL below is:

Task-A NLL after Task B
-
Task-A NLL at the selected pre-B checkpoint

so lower is better retention, and a negative number means the Task-A evaluation NLL actually improved after the Task-B phase.

arm	mean TIM score	seed 11 ΔNLL	seed 37 ΔNLL	seed 83 ΔNLL	mean ΔNLL
center	0.494613	+1.267124	+0.748269	+1.065674	+1.027022
o7→o0	0.459956	-0.023831	-0.058221	-0.031418	-0.037823
o7→o22	0.464996	+0.036202	-0.048561	-0.034949	-0.015769
o7→o29	0.465893	-0.027973	-0.068364	-0.053038	-0.049792
v7→v0	0.487957	+0.053136	-0.000974	+0.009932	+0.020698
v7→v14	0.492006	+0.234694	+0.016926	+0.077803	+0.109807
v7→v29	0.507622	-0.030963	-0.035499	-0.046803	-0.037755

All six neighbors had lower forgetting than the center in all three paired seeds.

One score-direction detail is worth being precise about here.

If the intended direction is:

higher TIM score
→
lower forgetting

then v7→v29 is actually direction-consistent:

score:
0.494613 → 0.507622

forgetting:
+1.027022 → -0.037755

I would therefore not use that arm as the score-ordering counterexample.

The more diagnostic observation is the other side of the table:

Five of the six rescues lower the summed TIM score and nevertheless reduce forgetting.

For example:

o7 → o0

score:
0.494613 → 0.459956

mean forgetting:
+1.027022 → -0.037823

That is why I still hesitate to read the summed score as a locally monotone configuration-level utility function. The underlying module measurement can contain useful signal while the mapping

independent module scores
        ↓
sum / rank
        ↓
top configuration

loses information about joint training.

There is another useful sanity check here: the rescue is not explained by every neighbor simply refusing to learn Task B.

The center’s mean Task-B acquisition is about:

+0.0581

while the six neighbor means are roughly:

+0.0762 ... +0.1903

So in this run the neighbors generally learned more of B while forgetting much less A. I would not call acquisition exactly matched, but the result is not a trivial “retain A by doing less B” trade.

Exact SmolLM2 protocol and controls

The reference Notebook run uses:

model:
HuggingFaceTB/SmolLM2-135M

seeds:
11, 37, 83

LoRA:
r = 8
alpha = 16
dropout = 0

learning rate:
5e-4

batch:
2

max sequence length:
128

Task-A trajectory:
up to 100 update steps
evaluated every 2 steps

Task-B training:
50 update steps

Task-A first-crossing control

Rather than comparing arms at an arbitrary fixed Task-A step, each seed gets a common reachable Task-A target.

The procedure is:

for the seven current arms:

1. find each arm's best reachable Task-A matching NLL
2. take the maximum of those seven minima
3. add a small 0.005 margin
4. cap at the untrained-base matching NLL
5. for each arm, choose the earliest checkpoint
   that reaches that common target
6. start Task-B training from that checkpoint

The target is recomputed from the current seven-arm panel; a target from an older/different arm set is not reused.

That was important because a fixed number of A updates can otherwise turn a placement comparison into a mixture of:

placement
+
different amount of Task-A learning
+
different starting point for Task B

Exact one-module changes

o7→o0
model.layers.7.self_attn.o_proj
→ model.layers.0.self_attn.o_proj

o7→o22
model.layers.7.self_attn.o_proj
→ model.layers.22.self_attn.o_proj

o7→o29
model.layers.7.self_attn.o_proj
→ model.layers.29.self_attn.o_proj

v7→v0
model.layers.7.self_attn.v_proj
→ model.layers.0.self_attn.v_proj

v7→v14
model.layers.7.self_attn.v_proj
→ model.layers.14.self_attn.v_proj

v7→v29
model.layers.7.self_attn.v_proj
→ model.layers.29.self_attn.v_proj

The Notebook verifies the actual adapter targets and trainable count before training, rather than assuming that the requested strings produced the intended PEFT adapter.

PEFT itself documents both useful runtime checks:

peft_model.print_trainable_parameters()
print(peft_model.targeted_module_names)

See the PEFT custom-model / target verification guide.

The TIM values in this Notebook are embedded from the preceding SmolLM2 diagnostic run. The public Notebook reproduces the downstream one-swap training/control experiment; it does not pretend to recompute the gradient-overlap diagnostic itself.

How I would connect the v12 and one-swap results

I do not see these as contradictory results.

They answer different questions.

v12 same-composition control
    ↓
Does measuring/selecting the full layer configuration
contain useful information beyond the type histogram?

Answer:
apparently yes, strongly in this experiment.

versus:

strict one-swap neighborhood
    ↓
Does the scalar score give a reliable local ordering
of nearby jointly-trained configurations?

Answer:
not in this tested SmolLM2 neighborhood.

So the combined picture I would use is:

useful diagnostic signal
        ≠
fully calibrated configuration utility

That distinction also seems compatible with the direction the TIM-OmegaS repository is moving: preserve the measurement, publish the actual configuration, verify what PEFT really targeted, and treat downstream configuration validation as its own step.

Independent Llama-3.2 transport: useful, but the control changes the claim

I also wanted to see whether the local placement phenomenon was purely a SmolLM2 toy-model curiosity, so I ran a separate transport experiment on unsloth/Llama-3.2-3B.

This is not the author-side Llama-3-8B/HumanEval setup, so I would keep it secondary to the SmolLM2 reproduction and to v12.

Still, it produced a useful sequence of results.

I first measured all 28 o_proj locations with the same TIM-style diagnostic and froze the candidate arms before looking at their sequential-training outcomes.

The seven-site center was:

C:
L17, L18, L19, L21, L23, L24, L26

and the lower-scoring strict one-swap was:

B:
L3, L17, L19, L21, L23, L24, L26

change:
L18 o_proj → L3 o_proj

Scores:

C = 0.937183
B = 0.936186

The Task-B protocol was separately calibrated and then frozen at:

LR    = 3e-5
steps = 24

First precommitted 10-seed run

For the precommitted primary endpoint:

B - C forgetting

the result over ten entirely fresh seeds was:

mean B-C:
-0.004455 NLL

B wins:
10/10

bootstrap 95% CI:
[-0.005890, -0.003438]

and all 40 final runs in the four-arm panel had positive Task-B acquisition.

At first glance that looks like a strong independent local transport.

But there was an important control problem: B was already better than C on the held-out Task-A evaluation before Task B:

pre-B A_eval:
mean B-C ≈ -0.03705 NLL

B better:
10/10

and the pre-B gap and later forgetting gap were strongly associated:

r ≈ 0.917

So I did not think it was safe to call that a pure retention effect.

Pre-B matched C/B control

I then reduced the experiment to only the frozen C and B arms, left the Task-B protocol unchanged, and selected the pre-B checkpoints using the original held-out Task-A evaluation NLL itself.

Across another ten precommitted seeds, the matching quality was:

mean |B-C| pre-B balance NLL:
0.003378

max |B-C|:
0.009279

On that matched evaluation family, the B advantage survived:

B-C forgetting:
-0.002477 NLL

B wins:
10/10

bootstrap 95% CI:
[-0.002916, -0.001967]

So the original v5 result cannot be explained only by the large pre-B scalar gap that was visible in the first run.

However, I also kept a second Task-A evaluation set completely out of checkpoint selection. This used different synthetic code-function families.

On that untouched set:

pre-B mean B-C:
+0.002619 NLL

B better before B:
3/10

and after Task B:

B-C forgetting:
+0.000040 NLL

B wins:
3/10

bootstrap 95% CI:
[-0.000286, +0.000321]

That is essentially a null result for the broader Task-A family.

So I would summarize the Llama transport fairly narrowly:

The lower-scoring one-swap retains a consistent advantage on the Task-A evaluation family used to match the pre-B state, but that advantage does not generalize to a separate untouched family of Task-A code functions.

That supports local placement/configuration sensitivity, but it does not support a broad claim that B is intrinsically a better retention placement.

Why I am keeping the Llama claim narrow

There are two slightly different questions hidden in the matched-control result.

1. Is the v5 result merely a consequence of B starting Task B from a better measured Task-A state?

The matched control argues against that simple explanation.

The original pre-B difference was large:

~ -0.03705 NLL

After explicit matching, the remaining average absolute pre-B gap on the balance set was only:

0.003378 NLL

yet B still won the matched-family forgetting endpoint in 10/10.

So there is residual local structure after that control.

2. Does that residual structure generalize beyond the evaluation family involved in checkpoint selection?

Here the result is much weaker.

The untouched code-function family gives:

mean B-C forgetting:
+0.000040

95% CI:
[-0.000286, +0.000321]

with only 3/10 B wins.

That is why I would not upgrade the result to a general retention claim.

There is also an important design distinction here:

checkpoint-selection set
≠
independent final generalization set

The primary matched endpoint is informative about degradation conditional on closely matching that observed Task-A state, but it is not the cleanest independent generalization endpoint because the same evaluation family participates in checkpoint selection.

The untouched set protects against that, but it also introduces a second change: it uses different Task-A function families rather than merely another sample from exactly the same generator family.

So one still cannot distinguish perfectly between:

A. finite-set / selection-conditioned specificity

and:

B. a real placement effect that is specific
   to one Task-A subdistribution

If that distinction ever becomes important, the cheapest discriminator I can see is a cross-fitted same-family holdout:

A_match
    → checkpoint selection only

A_holdout_same_family
    → primary independent retention endpoint

A_holdout_different_family
    → secondary transport/generalization endpoint

The first two would be independently generated from the same function family; the third would retain the current harder distribution shift.

That is essentially the standard reason to protect a final holdout from selection. The broader issue is discussed in work on adaptive holdout reuse, although this little experiment is obviously much narrower than that literature.

One more subtlety: matching observable pre-B NLL does not make the internal optimization histories identical. In the matched run, B generally reached the matched Task-A region slightly earlier than C. So the causal object here is closer to:

placement
→ learned state with comparable observed Task-A NLL
→ subsequent Task-B behavior

than:

identical internal state
+ one isolated placement perturbation

For this forum question I think the former is already useful, but I would keep that boundary explicit.

What seems established vs what still looks open

At this point I would separate the claims roughly like this.

Fairly well supported in the current evidence

1. Projection type alone is not enough in the Llama experiment.

The v12 reader/random-depth control has the same 17o/10q/4k composition and budget, but performs much worse than the measured configuration.

2. The actual module × layer configuration matters.

That means the reproducibility artifact should be the exact placement, not only a type histogram.

3. TIM’s diagnostic contains useful placement information.

The v12 control is a strong reason not to dismiss the measurement as noise.

4. Configuration-level behavior can be locally sensitive.

The SmolLM2 strict neighborhood is hard to reconcile with the idea that moving one selected site while preserving type/budget must produce a small or score-monotone change.

5. Acquisition and retention need to be reported together.

A placement that forgets less only because it barely learned B is not the same result. The SmolLM2 neighbors and the Llama runs both make this worth checking explicitly. This is also consistent with the broader learning/forgetting trade-off discussed in Biderman et al., “LoRA Learns Less and Forgets Less”.

Still separate questions

1. Global scalar calibration

Does a higher configuration score reliably predict lower forgetting across configuration space?

The current data do not establish that.

2. Additivity

Do independently measured per-module values compose approximately additively once several LoRA sites are optimized jointly?

The current experiments show reasons to test this rather than assume it, but they do not map the full interaction structure.

3. Top-quartile optimality

The v12 result shows that the measured configuration can be much better than a same-composition random-depth one. It does not show that the current top-quartile rule is the best same-budget configuration.

4. Cross-model/task transport

The independent Llama-3.2 experiment suggests some local behavior transports, but the fresh Task-A-family null result is a useful warning against turning that into a universal rule.

And none of this substitutes for a direct near-score perturbation around the exact Llama-3-8B/HumanEval configuration if that particular claim is the target.

The lowest-cost routes from here

I see three reasonable branches depending on what one wants to establish.

If the practical goal is simply to get a good adapter placement

I would use:

TIM diagnostic
→ shortlist
→ test a small same-type / same-budget neighborhood
→ keep acquisition + retention together
→ choose the operating point

This gets most of the value of the diagnostic without requiring the scalar score to be globally calibrated.

If the goal is to validate the scalar/top-quartile selector itself

The higher-information test is not another distant random placement. It is a local panel around the selected configuration:

C  = selected center
A  = one frozen same-type swap
B  = another frozen same-type swap
AB = both swaps

With all choices frozen before outcomes, that panel can distinguish several cases cheaply:

single swap violates score ordering
    → local scalar-ranking problem

A and B small, AB large
    → configuration interaction

A, B and AB follow the predicted direction
and AB ≈ A + B
    → local support for approximate additivity

nothing exceeds run noise
    → no detectable local problem in that neighborhood

That would tell us more about the selection rule than simply adding more random baselines.

If the goal is a general retention claim

Then I would protect an independent Task-A holdout from checkpoint selection and keep it from the same generator/task family first.

Only after that would I use a deliberately shifted Task-A family as the transport endpoint.

That cleanly separates:

state matching
from
same-distribution retention
from
cross-distribution retention

which turned out to matter in the Llama-3.2 probe.

Reproducibility details I would preserve

A few small things seem worth keeping as part of the experimental contract.

Publish the exact placement

Not just:

17 o_proj / 10 q_proj / 4 k_proj

but the complete module paths/layers.

The new reader-control result itself shows why.

Verify what PEFT actually adapted

I would keep both:

peft_model.print_trainable_parameters()
print(peft_model.targeted_module_names)

The PEFT documentation explicitly recommends these checks for verifying trainable parameters and adapted layers.

This is particularly useful when target selection is expressed with module-name patterns or custom lists.

Record the actual trainable parameter budget

Same number of modules does not imply same LoRA capacity when the matrix shapes differ.

This matters especially with GQA architectures.

Keep calibration and confirmatory seeds separate

For the Llama-3.2 experiment I used the Task-B calibration run only to freeze:

LR = 3e-5
24 steps

before the fresh confirmatory seeds.

I would preserve that separation rather than pooling the exploratory and confirmatory runs into one primary statistic.

Prefer paired same-session comparisons over literal cross-machine value matching

Absolute floating-point results can move with hardware/software details. PyTorch’s own reproducibility notes explicitly caution that completely reproducible results are not guaranteed across releases, platforms, or CPU/GPU execution.

So for this kind of test I would record at least:

model revision
tokenizer revision
GPU
CUDA
PyTorch
Transformers
PEFT
dtype
attention backend
TF32/determinism settings
seeds
exact target list
actual trainable params

and interpret a same-session paired arm contrast as more portable than demanding byte-identical per-seed numbers on another machine.

The SmolLM2 Notebook exports the environment and raw per-seed values for exactly that reason.

Where this leaves me

The new v12 control is a real strengthening of the central observation: in the reported Llama experiment, where the adapters are placed contains substantial useful information beyond merely knowing the projection-type counts.

The SmolLM2 one-swap result points to a different boundary: the useful diagnostic signal does not automatically make the current summed score a reliable local configuration ranking function. In that neighborhood, all six same-type one-swaps improve retention, and five of them do so while moving the score in the nominally worse direction.

The independent Llama-3.2 probe then adds one more qualification: a local placement difference can survive an important pre-B matching control while still failing to generalize to another Task-A family.

So the formulation that currently makes the most sense to me is still:

interference-aware measurement
        ↓
candidate generation
        ↓
cheap configuration-level validation
        ↓
final acquisition/retention decision

That preserves the part of TIM for which the evidence has become substantially stronger, while leaving additivity, scalar calibration, and top-quartile optimality as independently testable questions rather than assumptions.

[Acedo]

This is the most useful thing that has happened to this work since I published it, and I want to start with the correction rather than with the agreement.

I described the v12 control as isolating depth selection. That over-claims, and your reading is the right one. The measured and reader arms differ in the entire layer configuration at once: which layers, how they cluster, and whatever comes from training that particular set jointly. What the control establishes is that the full measured configuration carries information the type histogram does not. It does not decompose that information into a per-module depth term.

Second, you picked a better endpoint than the one I led with. Absolute HumanEval after task B, 0.196 against 0.118, winning 10 of 10, is the number that should carry the claim, with the ratio secondary rather than the other way round. The ratio has a denominator that differs between arms; I knew that and reported it as a caveat instead of leading with the measure that does not have the problem.

On the SmolLM2 neighbourhood I have no defence and I am not going to construct one. Six of six same-composition, same-parameter one-swaps reduce forgetting, five of them while moving the summed score the wrong way, and the neighbours learn more of task B rather than less, so the obvious escape route is closed as well. That is a direct test of the additive scalar as a local ranking function and it fails it in that neighbourhood. Thank you for being precise that v7 to v29 is direction-consistent rather than counting it as a sixth counterexample. That kind of care is why I am reading the rest of it as a result rather than as an anecdote.

So the position I am left with is the one you formulated, and I think it is the correct one: useful diagnostic signal is not the same thing as a calibrated configuration utility. Measuring beats not measuring by a wide margin in the setting I tested. Summing per-module scores and taking a top quartile is a heuristic for turning that measurement into a set, and nothing I have shown establishes that it is the right heuristic.

On the Llama-3.2 transport I would emphasise what you emphasised. The pre-B state already favoured B and correlated at 0.917 with the later gap, and once you protected a task-A family from checkpoint selection the effect went to essentially nothing, 3 of 10 with an interval crossing zero. Reporting that against your own result, unprompted, is why I am taking the SmolLM2 finding at face value instead of looking for reasons it does not apply to my setting.

Your four-arm panel is the right shape for the question. Separating a local ranking failure from a configuration interaction from approximate additivity, with everything frozen in advance, is more informative than adding further distant baselines.

The one piece your sketch leaves open is which two sites to swap, and I think that choice can be made mechanically rather than by judgement, which would make the panel considerably harder to argue with. The natural pair is the highest-scoring module in the selected set and the first module below the quartile cut: the two sites at the decision boundary, where the rule is doing its most marginal work. Neither of us picks them; the ranking does. That has the side benefit of testing the criterion exactly where it is least confident rather than somewhere either of us finds convenient.

[John6666]

I think this is a good direction. If we want to make the experiment explicit, probably something like this:

---

I would split the next step into two related questions, rather than making one perturbation answer both:

1. Does the current scalar selector get its own inclusion boundary right?
2. Do nearby module effects compose approximately independently, or is the configuration itself important?

The first one can be tested very cheaply. The second needs the larger factorial.

One tiny wording note first: since the current implementation sorts rel descending, I think the literal quartile boundary is:

rank 31 = lowest-scoring INCLUDED module
rank 32 = highest-scoring EXCLUDED module

So I assume “highest-scoring module in the selected set” above was just referring to the selected side of the boundary, rather than rank 1.

My default route would be:

freeze one full canonical score map
        ↓
C vs literal rank-31 → rank-32 swap
        ↓
if configuration/additivity is still interesting:
two mechanically chosen same-type swaps
        ↓
C / A / B / AB

That keeps the first experiment almost exactly as you proposed, while separating the selector question from the configuration-interaction question.

1. Cheap test first: the literal boundary

Let C be the canonical top-31 placement, and define:

G = C
    - rank 31
    + rank 32

with both sites taken mechanically from the frozen score map, before looking at any downstream result.

Then the interpretation is fairly direct:

G > C
→ local counterexample to the scalar ordering at the inclusion boundary

C > G
→ local evidence in the predicted direction at that boundary

difference small/noisy
→ unresolved at the achieved precision

I like this because it tests the current selector where its decision is most marginal, without choosing an interesting layer or topology after the fact.

There is one GQA wrinkle, but I would not let it block this test.

As you already noted in post 5, equal module count is not necessarily equal LoRA capacity on Llama-3-8B: q_proj/o_proj and k_proj/v_proj have different matrix shapes. Hugging Face’s Llama configuration docs describe the num_key_value_heads/GQA structure, and the PEFT LoRA docs note that LoRA parameter count depends on both r and the original weight-matrix shape.

So if rank 31 and rank 32 happen to cross those cost classes, I would still run C vs G — I would just call it a literal selector test, not a pure same-capacity configuration test.

If they happen to be the same projection type, even better: the cheap test is already budget- and type-matched.

Why I would freeze the complete ranking first

By “freeze the ranking,” I mean slightly more than saving the two module names.

I would save enough information to reconstruct the selector exactly:

full-precision per-module scores
saturated/non-saturated status
score direction
useful-module count
selected N
full ordering
tie/order rule
model revision
dataset/input revision or snapshot
score-map hash

and then derive both the center placement and the boundary sites from that same artifact.

This matters mainly because I would avoid mixing:

historical published top-31
+
rank 32 from a new diagnostic rerun

Those would come from two different realizations of the measurement.

If the original full Llama-3-8B map still exists, that is obviously the cleanest source. If not, I think a fresh prospective run is fine — I would simply let that new map regenerate the whole center and its boundary rather than attaching one new site to the old center.

The current TIM-OmegaS repo is enough to recover the selected placement I was looking at, but I could not reconstruct the historical rank-32 site with enough provenance to want to guess it.

For a fresh run, pinning Hub inputs to a commit is also cheap. huggingface_hub accepts a full commit hash via revision; the Hub download documentation has the exact mechanism.

One other small implementation detail worth freezing is module-name normalization. If the diagnostic emits something like:

model.layers.3.self_attn.o_proj.weight

while PEFT targeting uses:

model.layers.3.self_attn.o_proj

I would normalize that once, deterministically, before ranking/target comparisons.

2. If we want the interaction question too: use two same-type swaps

A single rank-31 → rank-32 exchange gives a clean C/G selector test, but it cannot by itself give a C/A/B/AB interaction panel.

For that we need two independent atomic substitutions.

I would choose them from the same frozen score map, mechanically, with no topology-based hand selection:

eligible pair:
    selected module → excluded module
    same projection type

priority:
    1. nearest to the global cutoff
    2. smallest total cutoff distance
    3. smallest absolute score gap
    4. deterministic rank/name tie-break

swap A = first eligible pair
swap B = next eligible pair with disjoint endpoints

Then:

C  = canonical placement
A  = swap A only
B  = swap B only
AB = both swaps

Exact-type matching is useful here because it automatically preserves the q/k/v/o composition and the GQA-aware LoRA parameter budget.

That gives two quite different tests with very little conceptual overlap:

panel	main question
C vs literal G	does the scalar cutoff ordering survive a downstream perturbation?
C/A/B/AB same-type	do two local placement changes behave approximately additively?

There is also an intermediate case if the global boundary changes projection type but stays within the same LoRA cost class (q↔o or k↔v). I would probably just report that fact rather than build another main experiment around it unless the distinction becomes important.

How I would read the factorial

For a larger-is-better endpoint:

I = Y_{AB} - Y_A - Y_B + Y_C

I would treat that as a local interaction diagnostic, not as a universal model of the whole selector.

Roughly:

lower-ranked A or B > C
→ local failure of scalar ordering

A and B small, AB clearly non-small
→ local configuration interaction

A/B move in the predicted direction
and AB is close to their additive expectation
→ compatible with approximate local additivity

everything small / intervals broad
→ unresolved, not proof of additivity

There is a useful broader connection here: Xu et al., “Understanding and Guiding Layer Placement in Parameter-Efficient Fine-Tuning of Large Language Models” explicitly separate layer-local signal from layer coupling in their PEFT analysis. I do not mean that their quantity is TIM’s quantity; just that “useful local diagnostic” and “how multiple selected locations interact” are naturally separate questions.

So I would see this factorial as extending the measurement to configuration-level behavior, rather than as a test that needs to invalidate the measurement for one side to be interesting.

One small protocol scout I tried

I also tried this mechanical procedure on a smaller Llama-3.2-3B setup mainly to see whether the protocol itself had an obvious failure mode.

The interesting result was not really the downstream score; it was the ranking behavior.

Across two independent diagnostic samples I got approximately:

full-ranking Spearman:      0.972
top-quartile Jaccard:       0.807

so globally the rankings looked quite similar.

But the exact two modules at the cutoff were different between the two probes.

In other words:

global ranking stability
!=
decision-boundary identity stability

The four endpoints selected from the primary map still remained on their respective included/excluded sides in the secondary probe, so the panel was not obviously meaningless. But it made me think it is worth reporting cutoff stability separately from a global rank-correlation number.

I would not transport any of those 3B module identities to Llama-3-8B; this was only a protocol scout.

The one-seed downstream scout also did not produce a boundary rescue, and the factorial interactions were small. With n=1 I would not read much into that, except that the procedure is at least capable of returning an uneventful result rather than being constructed to find a reversal.

There is a somewhat analogous evaluation issue in Du et al., “On the Stability of Prompt Ranking in Large Language Model Evaluation”: overall rank correlation can remain reasonably high while the identity of the item selected at the top changes. Different problem, but the distinction between ranking stability and decision stability seems useful here too.

If rerunning the TIM diagnostic is cheap relative to downstream training, an optional low-cost check would therefore be:

independent diagnostic samples
        ↓
global rank correlation
top-N overlap
rank-N / rank-(N+1) identity
membership of the proposed swap endpoints

I would treat this as a measurement/readout, not necessarily a pass/fail gate.

Minimal run receipt

I would keep the preregistration fairly small. Something around this seems enough:

canonical score-map artifact + hash
model/dataset revisions
selected N and cutoff definition
swap-selection rule
exact C/G or C/A/B/AB target lists
LoRA r / alpha / dropout
actual targeted modules
actual trainable parameter count
Task-A state-matching rule
Task-B acquisition criterion
primary endpoint
secondary endpoints
seed list
environment
paired comparison rule

PEFT already exposes most of the runtime checks needed here. The PEFT custom-model guide recommends checking the actual adapted layers and documents both:

peft_model.print_trainable_parameters()
print(peft_model.targeted_module_names)

So the requested target list, effective target list, and actual parameter budget can all be left in the run output as a simple receipt rather than becoming another methodological discussion.

For the behavioral endpoint, I would also carry forward the distinction you made above: absolute Task-A capability after Task B is the stronger primary readout here, with Task-B acquisition as a validity condition and retention ratio as useful supporting context.

So, in compact form, my preferred version would be:

1. freeze one full score map
2. take its literal rank-N / rank-(N+1) boundary
3. run C vs G first
4. if interaction/additivity is still worth resolving,
   mechanically select two disjoint same-type boundary-near swaps
5. run C/A/B/AB with matched actual PEFT capacity

That seems to preserve the useful part of the proposal: the next sites are chosen by the selector itself, not by whichever topology looks interesting afterward.

And it keeps the claims separable: the measured configuration can carry useful information even if the scalar cutoff is imperfect; the scalar cutoff can work locally even if interactions exist elsewhere; and the factorial can test those interactions without requiring either earlier result to be reinterpreted.