Thank you for actually running it. That is more than I expected from a forum post, and the parameter-budget point is a hit. You are right, and it is worse than a nitpick.
I matched the arms on module count, which is not the thing that needs matching. Llama-3-8B uses grouped-query attention, so q_proj and o_proj are 4096x4096 while k_proj and v_proj are 4096x1024. Running the numbers on the exact lists in the repo, at r=8:
- measured placement (17 o_proj, 10 q_proj, 4 k_proj): 1,933,312 trainable parameters
- conventional (16 q_proj, 15 v_proj): 1,662,976
So the arm I report as better carries 16.3% more trainable capacity. Equal module count, unequal budget. The paper does not state that and it should; the numbers are above for anyone reading this thread.
One piece of luck: the control arm I designed and pre-registered uses the published type composition (17/10/4) with layers drawn at random, one draw per seed, precisely because the paper reports composition but not depth. That arm is parameter-matched to the measured one by construction, so this confound does not reach it. Not foresight, but I will take it.
On composition, I think you have found the real boundary of the method and I do not have a defence. The scores are measured per module, independently, and then I take a top quartile as if they added up. Your one-swap result says they do not. The diagnostic can carry signal while the selection rule built on top of it is not optimal, and those are separable claims that I conflated. Your framing, TIM as interference-aware candidate generation followed by a cheap configuration check rather than as a final selector, is more defensible than mine, and I would rather adopt it than argue for the stronger version.
The two sanity checks are cheap and worth having, and I will add them when I next touch the harness: targeted_module_names and print_trainable_parameters() are a real verification, where I had a string check on a log line. Same for reporting acquisition alongside retention rather than retention alone. Retention is a ratio, so it can flatter an arm that simply learns the new task less, and that should be visible rather than inferred.
Your historical framing is also more accurate than mine: q+v came from a fixed-budget attention-only ablation, and QLoRA already found that matching the 16-bit baseline needed all linear layers. I had been treating the default as unexamined, when the record is that it was examined under conditions that no longer hold. That is a better sentence than the one in my paper.
What I would ask, if you are willing: which base model and which task pair, and what the one-swap actually changed. The collapsed sections did not come through in the email notification. Even the raw numbers would help, and I would report them as yours. A configuration effect large enough to show up in a small test is exactly the kind of thing that should bound the claim, and at the moment I only have your description of it.
The retention control is designed and the module lists are frozen. Whether it gets run depends on compute I have not committed yet. If I run it, the result goes here whatever it says.