I think the overall picture is much clearer now:
I think your proposed per-example failure analysis is the right next step, and I would probably do that before adding another training sweep.
The one change I would make is to split the current:
evidence retained
vs
reader failed
into a slightly more explicit chain:
paragraph selected
↓
required evidence still survives paragraph/token preprocessing
↓
reader recognizes / prioritizes that evidence
↓
reader composes the multi-hop answer correctly
↓
training gain appears
↓
gain is worth the selector cost
That distinction seems especially useful now because the latest result is no longer just a coverage result.
At the same 60% paragraph budget, the query-aware semantic policy retained much more complete gold support than random/cheap selection, and in the controlled QLoRA pilot that also coincided with a materially higher held-out full-context task score.
I would still be cautious about the numerical semantic > full result itself. I would read it as an interesting mechanism question, not yet as evidence that selected context is generally better than full context.
So my default next route would be:
existing adapters/checkpoints
↓
save exact eval IDs + predictions
↓
save selected paragraph IDs + post-cap inputs
↓
run the official MuSiQue metric path
↓
paired per-example failure analysis
↓
add a cheap BM25 boundary
↓
only then decide whether another training experiment is needed
That seems likely to extract more information from the runs you already paid for than another immediate QLoRA sweep.
A minimal per-example record like this would already support most of the analysis:
example_id
seed
training_policy
gold_support_paragraph_ids
selected_paragraph_ids
original paragraph token lengths
post-cap paragraph token lengths
prediction
reference answer + aliases
EM / F1
selector timing / caching status
If the existing policy×seed adapters are still available, I would also consider evaluating them on a larger common 2-hop held-out set before retraining anything. The current 80-case result is useful as a controlled pilot, but a broader inference-only pass would make it much easier to tell whether the difference is broad or concentrated in a small number of examples.
I would split evidence retention one step further
One thing I would make explicit in the failure analysis is:
support paragraph retained
is not necessarily the same as:
usable evidence retained
if the paragraph is subsequently capped or otherwise transformed.
If the QLoRA path is still using the 128-token paragraph cap described in the current experiment setup, that looks worth auditing directly.
I did a small independent check on the public MuSiQue-Ans 2-hop dev data using the Qwen2.5 tokenizer. This was not a reproduction of your train/eval split or selector; I only wanted to see whether the cap could be a visible variable at all.
On that public set, roughly 37.6% of gold-support paragraphs exceeded 128 tokens.
I then used a deliberately weak diagnostic: for each supporting paragraph where a decomposition-step answer string could be found verbatim in the original paragraph, I checked whether that string was still present after the 128-token cap.
Most survived individually, but not all. At the example level, roughly 10.8% of the auditable 2-hop cases lost at least one such mapped answer string after truncation.
I would not interpret that as:
10.8% of examples are broken by the cap.
The string check is much too crude for that. A reasoning dependency can survive without the exact answer string, and the string can survive without the full useful reasoning evidence.
But it was enough to convince me that the clean failure decomposition is probably:
gold paragraph selected?
↓
required content survives preprocessing?
↓
reader succeeds?
rather than stopping at paragraph IDs.
This also connects nicely to the distinction in Failure Modes in Multi-Hop QA: The Weakest Link Effect and the Recognition Bottleneck.
That paper separates failures in recognizing/locating required evidence from failures in integrating it, and shows on MuSiQue and other multi-hop settings that one poorly visible required fact can bottleneck the whole reasoning chain.
I would use that mostly as an evaluation-design precedent, not as a claim that your model is exhibiting exactly the same mechanism.
A useful table might therefore look like:
| Gold paragraphs selected |
Evidence survives preprocessing |
Answer correct |
Likely boundary |
| No |
— |
No |
Selection |
| Yes |
No |
No |
Preprocessing / cap |
| Yes |
Yes |
No |
Recognition / synthesis |
| Yes |
Yes |
Yes |
Successful path |
And I would keep a fifth bucket for:
proxy says evidence is absent
but answer is correct
because that can expose either alternate evidence, shortcuts, or limitations in the proxy itself.
The semantic > full result is interesting, but I would test it pairwise before interpreting it
The current mean F1 pattern is more interesting to me when written relative to the common untrained adapter:
step 0 0.3803
full 0.4119 (+0.0316)
random 0.3421 (-0.0382)
cheap 0.3460 (-0.0343)
semantic 0.4516 (+0.0713)
So the observation is not only:
semantic > full
It is also:
generic evidence-poor shortening
→ fell below the common starting point
full context
→ learned somewhat
task-aware semantic selection
→ learned more in this pilot
That makes dependency preservation a plausible explanation.
But there are still several other explanations consistent with those means:
dependency preservation
hard-distractor filtering
cleaner / easier small-data curriculum
position or ordering changes
compact-position geometry
optimizer / training-horizon interaction
ordinary finite-sample variation
I would therefore avoid choosing one yet.
The cheapest discriminator is probably paired data from the runs you already have:
per-seed F1 / EM
+
same-example semantic vs full results
+
win / tie / loss counts
+
paired bootstrap over examples
The exact statistical machinery is less important than seeing whether the advantage is:
many small improvements
or:
a handful of large flips
and whether its sign is reasonably consistent across the three seeds.
This seems especially useful in a 96-example fine-tuning setting. Measuring the Instability of Fine-Tuning is a useful general reminder here: fine-tuning on small datasets can be seed-sensitive, and aggregate standard deviation alone does not capture every useful notion of instability.
My decision boundary would be roughly:
semantic > full persists
across seeds
+
on a broader inference-only evaluation
+
in paired per-example comparisons
↓
mechanism investigation becomes quite worthwhile
versus:
semantic > full weakens or becomes unstable
↓
the stronger supported claim remains:
semantic > random / cheap
at the same reduced context budget
That second outcome would still be a useful result.
I would not treat full context as a theoretical quality ceiling either.
Lost in the Middle gives a good general reason not to assume that simply supplying more relevant context guarantees the model will use it robustly: long-context QA performance can change substantially with the position of relevant information.
That does not establish the cause of your current result, but it makes semantic > full something worth investigating rather than dismissing automatically.
A BM25 row now looks unusually informative
Now that the experiment has a real task query, I think BM25 has become a particularly clean attribution control.
The comparison would be:
random
→ no task signal
BM25
→ cheap lexical task signal
semantic
→ richer semantic task signal
I ran a separate public-data sanity check on the official MuSiQue 2-hop dev set at the same 60% paragraph budget.
Again, this did not reproduce your selector, your 96/80 split, or the QLoRA experiment. It was only a selector-level check.
The approximate both-gold-support retention rates were:
matched random 36.6%
BM25 over question → context 56.6%
MiniLM semantic proxy 75.1%
The exact random expectation under the common 20-paragraph / choose-12 / two-required-paragraph structure is about:
(12 / 20) × (11 / 19) ≈ 34.7%
so the random result behaved roughly as expected.
The more interesting part was:
semantic proxy > BM25 > random
on the public task.
I would not compare the MiniLM percentage directly with your 72.5% or 80.2% numbers. The public proxy used all-MiniLM-L6-v2, which is simply a convenient public semantic-search model, and its own model card notes that longer inputs are truncated by default.
So I see that result only as independent evidence for this evaluation distinction:
semantic ≈ BM25 >> random
→ query-awareness is doing most of the work;
the semantic scorer may not be necessary
semantic >> BM25 >> random
→ stronger evidence that semantic matching adds something
beyond lexical query matching
semantic retains more evidence
but BM25 and semantic task scores are similar
→ the extra coverage is not translating into reader utility
That seems like a lot of attribution information for a very cheap extra row.
I would only use BM25 in the task-aware branch, though.
It is not an obvious baseline for the generic continuation setting where there is no meaningful query or task anchor.
If semantic stays above full, I would treat that as a mechanism question
MuSiQue makes the distractor-filtering hypothesis somewhat more plausible than it would be on an arbitrary collection of long text.
The MuSiQue paper was designed specifically around connected multi-hop reasoning, and its context construction deliberately includes difficult distractors rather than merely padding questions with unrelated text.
That matters because semantic selection may be doing something closer to:
remove task-confusable distractors
+
preserve the two dependent supports
rather than merely:
make the prompt shorter
There is also an interesting precedent in the official MuSiQue repository.
On the released MuSiQue-Answerable dev results, the repository reports:
End2End answer F1 0.423
Select+Answer answer F1 0.473
and the corresponding step-execution variants are also numerically higher for the select-then-answer route.
Those systems are quite different from SpiralCoreAttention, so I would not use this as evidence for your method.
But it is a useful task-specific precedent that filtering/selecting context before answering can be beneficial on MuSiQue.
There is another relevant direction in Tackling Distractor Documents in Multi-Hop QA with Reinforcement and Curriculum Learning.
In that setting, training examples with fewer distractors allowed citation and reasoning skills to be learned more sample-efficiently, and the models could later generalize to noisier retrieval contexts.
That suggests a plausible small-data interpretation of your current result:
semantic selection
→ cleaner dependency-bearing training examples
→ stronger learning signal over 96 examples / 192 steps
But I would still call that a hypothesis, not the explanation.
A very cheap way to investigate it would be to inspect:
semantic correct / full wrong
cases and record:
where the two gold paragraphs occur in the full prompt
how many distractors precede / separate them
whether both survive the cap
their order in the semantic-selected prompt
whether the full prediction looks like a one-hop / partial-answer failure
If full-context failures visibly cluster around position or distractor patterns, the recognition/filtering explanation becomes more plausible.
If they do not, finite-sample or optimization explanations stay very much alive.
Coverage and reader success should remain separate columns
I think the gold-only result remains one of the most useful diagnostics in the whole sequence of experiments.
Your frozen-model bridge had:
gold evidence only
>> noisy selected/full contexts
but gold-only was still far from perfect.
That implies at least two separable limits:
selector limit:
did the right evidence survive?
reader limit:
can the model use it?
So even a hypothetical selector with 100% support recall does not automatically solve the task.
I would keep the layers separate:
selection
are the required passages chosen?
preprocessing
does the useful content survive caps / joins?
recognition
does the reader find and prioritize it?
synthesis
can it combine both hops correctly?
This is also why I would not respond to every retained-but-wrong example by immediately modifying the selector.
If the evidence is clearly present and usable, the next useful experiment may instead concern:
evidence ordering
position
reader prompting
multi-hop composition
distractor sensitivity
The Weakest Link / Recognition Bottleneck work is particularly useful as a conceptual map here because it explicitly asks whether long-context multi-hop failures come from locating evidence or integrating it.
I would still keep the cheap and task-aware branches separate
Nothing in the MuSiQue result makes me think the cheap gate should be discarded.
I would still describe the two branches as different operating points:
cheap gate
→ generic low-overhead physical shortening
query-aware semantic selector
→ dependency protection when a meaningful task signal exists
The cheap gate failed to preserve deliberately important multi-hop evidence as reliably as the semantic branch, but that is not necessarily a failure of its original objective.
On the earlier generic local corpus it was extremely cheap and preserved most of the physical-shortening benefit while staying close to the other conditions in held-out LM loss.
So I would keep the distinction:
generic compression objective
!=
dependency-preservation objective
If the project eventually needs both, a later hybrid/dispatcher could be reasonable:
redundant / dependency-poor sample
→ cheap branch
explicit-query / dependency-rich sample
→ task-aware branch
but I would not make that the next experiment.
The current failure decomposition seems more informative.
A later trade-off may be clean skill acquisition vs distractor robustness
There is also a useful counterpoint to the idea that removing distractors is always good.
The curriculum result above suggests:
cleaner context
→ can make initial skill acquisition easier
But NExtLong deliberately goes in the other direction.
It constructs long-context training data by interleaving hard negative distractors between related chunks, with the goal of forcing the model to distinguish genuinely dependent content from plausible distractors.
So I would not turn the present result into:
fewer distractors are always better for training
A later design space might instead look like:
early / small-data adaptation
→ cleaner dependency-preserving context
later robustness training
→ progressively harder distractors / longer context
That is only a future option, but I think it is useful because it keeps two different objectives separate:
learn the dependency efficiently
vs
remain robust when the dependency is buried in noise
I would report semantic preprocessing as an amortization contract
The reported semantic preprocessing cost of roughly 5.8 ms per training example does not look large enough, by itself, to erase the current training-step advantage.
But I would make its lifetime explicit, because the same number means different things depending on the workflow:
A. compute once and cache
B. recompute once per epoch
C. recompute every training step
So future efficiency tables could report:
one-time preprocessing cost
+
recurring selector cost
+
model training cost
=
end-to-end workload cost
If the semantic representation or ranking can be reused, reporting the preprocessing cost separately plus its amortized cost seems perfectly reasonable.
If it is recomputed repeatedly, I would fold that repeated work into the end-to-end number.
I would treat this as a reporting contract rather than a problem with the current pilot.
I would keep full-context loss, but not ask it to carry the task-quality claim
I would still keep the full-context held-out LM loss from the earlier experiments.
It remains useful as a general training-distribution diagnostic.
I just would not make it the primary quality metric once the experiment moves into a dependency-rich task.
What is Wrong with Perplexity for Long-context Language Modeling? gives a useful reason: ordinary PPL averages over tokens and can hide the relatively small subset whose prediction really depends on distant context.
For this project I would therefore keep the roles separate:
full-context held-out LM loss
→ secondary distribution-level diagnostic
gold-support retention
post-cap usable-evidence check
task EM / F1
→ primary dependency-rich task diagnostics
I do not think you need to implement LongPPL itself for the next step; the labeled MuSiQue structure already gives you a much cheaper task-specific diagnostic.
A few reporting details would make the next result much easier to interpret
If you already have the raw predictions and selector outputs, a small reproducibility bundle would make the next comparison unusually easy to inspect:
exact train/eval IDs
per-seed EM / F1
per-example predictions
selected paragraph IDs
post-cap paragraph lengths
paragraph ordering
position policy
join-loss masking policy
semantic preprocessing caching policy
For answer scoring, I would also run the predictions once through the official MuSiQue evaluator.
The official script evaluates each prediction against:
primary answer
+
answer aliases
so this is a cheap way to make the metric contract unambiguous and avoid differences caused only by answer normalization or alias handling.
I see this mainly as making the result easier for future readers to reproduce and extend, rather than as a prerequisite for continuing the experiment.
So at this point, my working picture would be:
generic shortening
→ explains much of the compute / VRAM saving
task-aware semantic selection
→ preserves substantially more dependency-bearing evidence
controlled MuSiQue QLoRA
→ suggests that preservation can translate into training/task gain
next high-information question
→ where do the remaining failures actually occur?
If the paired/per-example analysis shows the semantic advantage is stable across seeds and examples, I think that would make the result substantially stronger: not “60% context is generally better than full context,” but something narrower and more interesting:
at a fixed reduced context budget, task-aware selection can preserve dependency-bearing training signal that generic shortening loses.
If semantic > full weakens under that analysis, I do not think that hurts the main result much. The semantic-vs-random/cheap separation would still identify a useful boundary between generic context reduction and dependency-aware context selection.
And if the evidence is usually retained but many answers still fail, then the project has learned something equally useful: the next bottleneck has moved downstream from which context is selected to how the retained evidence is recognized and composed.