Hmm… I don’t know whether the original poster ever pinned down the final cause, but if you’re asking what I’d consider plausible in a case like this:
First, a small correction to my earlier reply: I am not the original poster, and I cannot see a later message in this thread establishing whether the root cause was the dataset, beta, or something else.
The old TRL issue I linked only showed a superficially similar repetition/corruption symptom in a very different T5, classification-like setup. I would not treat it as evidence that the same mechanism was responsible here.
For a collapse like this, I would not reduce the investigation to “dataset versus beta.” I would separate at least five layers:
- the rendered sequence contract;
- the policy/reference construction;
- the overall optimization strength;
- the preference-data geometry and policy support;
- the saved model and inference path.
Different trainers can still share several of those layers, so reproducing the symptom in a custom trainer and Swift weakens a purely framework-specific explanation, but does not by itself establish that DPO’s objective or the semantic content of the dataset was the cause.
A compact diagnostic path would be:
A. Does merely loading/wrapping the SFT model in the DPO path change its output?
Yes → inspect active adapters, model mode, dtype/quantization,
tokenizer/template, and the inference function
No → B
B. Does a zero-update save/reload round trip change its output?
Yes → inspect serialization, base/adapter pairing, merge state,
tokenizer files, generation config, EOS/stop settings
No → C
C. Are the final rendered prompt/chosen/rejected token sequences,
shared prefix, completion masks, EOS tokens, and truncation positions correct?
No → sequence-contract or preprocessing problem
Yes → D
D. At step 0, does policy/reference behavior match the intended reference design?
No → reference or adapter construction problem
Yes → E
E. What changes first during training?
chosen log-prob ↓, rejected log-prob ↓, margin ↑
→ relative objective is improving while absolute likelihood is falling;
inspect pair similarity / likelihood displacement
chosen log-prob ↑, rejected log-prob ↓, ordinary-chat quality ↓
→ inspect total optimization strength and preference-distribution coverage
only long or multi-turn conversations degrade
→ inspect truncation, role boundaries, trajectory semantics,
and context-length coverage
training-time model is coherent, reloaded model is not
→ return to the artifact/inference path
The highest-information controls, before doing a large beta sweep, would therefore be:
- a load/wrap-only comparison with no training;
- a zero-update save/reload comparison;
- a dump of the actual rendered and tokenized pair, including masks and truncation;
- a step-zero policy/reference log-prob comparison;
- fixed ordinary-chat and capability-boundary evaluations at SFT, step 1, early checkpoints, and final;
- separate tracking of absolute chosen and rejected log-probs, not only reward accuracy or margin.
I would also reproduce the failure first with deterministic decoding and exactly the same inference function. Otherwise, training drift and decoding/configuration drift remain confounded.
1. Sequence contract and shared-prefix checks
“Standard DPO format” at the Python/dataset level does not guarantee that the final token sequences are correct.
As the Transformers chat-template documentation emphasizes, a chat model ultimately consumes a sequence of tokens. The model-specific role and control tokens are part of its learned protocol; using the wrong template, duplicating special tokens, or placing the assistant-generation boundary differently can substantially change behavior.
For at least one random pair and one of the longest multi-turn pairs, I would inspect:
raw prompt messages
raw chosen messages
raw rejected messages
rendered prompt
rendered chosen sequence
rendered rejected sequence
token IDs
decoded token-by-token representation
prompt/completion boundary
chosen/rejected completion mask
all loss labels
BOS location
EOS locations
assistant-start/control tokens
length before truncation
length after truncation
which turns were removed
number of non-masked chosen/rejected tokens that remain
A particularly useful invariant is:
chosen_prompt_token_ids == rejected_prompt_token_ids
up to the exact point where the two completions are supposed to diverge.
I would compare the prefix generated in four places:
SFT training
DPO chosen branch
DPO rejected branch
inference
A pair can look identical when printed as plain text while differing in model-visible control tokens, EOS placement, or assistant-start tokens.
For long conversations, it is not enough to ask whether the example exceeded max_length. The more useful questions are:
- Which historical turn was removed?
- Did truncation remove the part of the context that made the preference meaningful?
- Did it remove the actual chosen/rejected difference?
- Did one branch retain a different number of completion tokens?
- Did an EOS token move into or out of the loss-bearing region?
- Under all-turn loss, how many assistant tokens were actually supervised?
- Under last-turn loss, was the final response still paired with the intended history?
The original report said that system prompts were absent from the DPO data and that inference became even worse without a system prompt. That is a useful clue, but it does not prove that omitting a system prompt caused the collapse. It could indicate format dependence, distribution coverage, or simply that the system prompt partially stabilized an already degraded policy.
Similarly, trying both all-assistant-turn loss and last-turn-only loss weakens the claim that the choice between those two modes was the sole cause. It does not rule out a shared masking, rendering, or truncation problem.
The current TRL DPO documentation recommends an explicit prompt and automatically applies a chat template to conversational preference data. However, the exact behavior is version-dependent, so I would record the trainer version/commit rather than assume current behavior matches a 2025 run.
2. Policy/reference and PEFT construction
DPO depends on four quantities for each pair:
policy log-prob of chosen
policy log-prob of rejected
reference log-prob of chosen
reference log-prob of rejected
Therefore, “the policy and reference were configured from the same checkpoint” is weaker evidence than directly observing their outputs on the same tokenized batch.
If the intended reference is the same SFT policy from which DPO starts, I would expect the step-zero completion log-probs to agree within the numerical tolerance expected from dtype, quantization, and implementation details:
policy_chosen_logp ≈ reference_chosen_logp
policy_rejected_logp ≈ reference_rejected_logp
If a deliberately different reference is being used, exact equality is not expected; the control becomes verifying that the observed difference matches that intended design.
A substantial unexplained step-zero difference would move the investigation toward:
- reference using the base model without the SFT adapter;
- different active adapter names;
- merged policy versus unmerged reference;
- reference adapter not being loaded or activated;
- different token masks or preprocessing;
- policy in train mode versus reference in eval mode;
- dtype or quantization differences;
- stale or incorrectly generated cached reference log-probs.
This is especially relevant with PEFT. The TRL v0.18.1 DPO documentation described multiple ways of constructing an SFT-aware reference, including loading the same adapter twice under separate names. The fact that several valid constructions existed means reference handling was an explicit design choice, not merely an invisible implementation detail.
I would record:
base model repository and exact revision
SFT checkpoint
policy adapter path and name
reference adapter path and name
active adapter before every forward pass
merged or unmerged state
dtype and quantization
whether reference log-probs were cached
exact code/version used to construct the reference
Current TRL has evolved substantially since v0.18.1, so present-day defaults should not be projected backward onto an older Swift or custom implementation. Conversely, reproducing an old run on current TRL without recording the version can silently change the reference and preprocessing path.
3. Preference-data geometry and policy support
“Dataset problem” can mean several different things. I would separate at least four possibilities.
A. Chosen/rejected responses are too similar
Standard DPO optimizes a relative preference margin. It does not require the absolute likelihood of the chosen response to increase.
The DPO-Positive paper demonstrates that the ordinary DPO loss can improve while the likelihood of preferred examples falls, and reports that this occurred particularly often for datasets where chosen and rejected completions had low edit distance.
The later likelihood displacement paper studies a related phenomenon in which probability mass can move away from both members of a semantically similar pair and toward an unintended response, potentially even one with the opposite meaning.
That is highly relevant as a possible mechanism for capability-boundary pairs such as:
"I can do that."
"I cannot do that."
or pairs differing mainly in:
can / cannot
will / may
always / sometimes
yes / no
one number
one modal verb
one safety qualifier
However, it would still be an overreach to infer that this happened in the original run without seeing the pairs and log-prob dynamics.
Useful dataset summaries would include:
token overlap between chosen and rejected
normalized edit distance
length ratio
fraction of pairs differing mainly by negation
fraction differing only near the end
semantic-similarity distribution
difference-token positions
A small manual stratification may be more useful than one global average:
high-overlap / polarity-only pairs
moderately different pairs
clearly different responses
Then compare the training signature and held-out behavior for those groups.
B. The preference label is ambiguous or contradictory
A preference pair may be syntactically valid while providing an unclear learning signal.
Examples include:
- both answers are acceptable;
- both answers are poor;
- the “correct” capability boundary depends on unstated context;
- duplicate prompts have conflicting labels;
- a cautious answer is preferred in one example and rejected in a nearly identical one;
- the label reflects style while the intended target is factual capability;
- the rejected answer contains both the unwanted promise and otherwise useful content.
The Robust DPO paper formalizes the general problem of incorrect or ambiguous preference feedback. I would not jump directly to a different loss, but I would manually classify a small sample as:
clear preference
context-dependent preference
ambiguous preference
contradictory or mislabeled preference
That can distinguish “the dataset is bad” from a more specific and actionable problem.
C. The responses are far from the SFT policy
If the chosen or rejected responses were produced by a different, much stronger, or differently aligned model, the target policy may assign extremely low initial probability to them.
The WPO paper discusses distributional gaps in off-policy preference optimization. Again, this is not evidence that the original case had that problem, but it suggests recording:
which model or process generated chosen
which model or process generated rejected
initial SFT log-prob of chosen
initial SFT log-prob of rejected
If the chosen response is already an extreme outlier under the SFT policy, the pair is different from a local preference correction between two plausible SFT outputs.
D. Length is carrying unintended preference information
Chosen/rejected length should be summarized as a distribution rather than only an average.
I would check:
chosen/rejected token-length ratio
whether chosen is systematically shorter or longer
whether high-overlap pairs also have a length bias
sum log-prob and per-token average log-prob
effective length after truncation
Length is probably not enough to explain severe conversational corruption by itself, but it can confound pair comparisons and the interpretation of sequence-level log-probs.
The important practical point is that “dataset versus beta” is not one comparison. The data side includes at least pair similarity, label clarity, response-source/policy support, length, and the rendered token contract.
4. Optimization strength and early capability degradation
Beta matters, but a beta-only sweep would not isolate the broader optimization problem.
I would record the following together:
beta
learning rate
optimizer and scheduler
optimizer steps
processed preference pairs
processed completion tokens
effective epochs / average example exposure
batch size and gradient accumulation
full fine-tuning or PEFT
LoRA rank, alpha, target modules
number and fraction of trainable parameters
gradient norm
SFT-loss weight
These variables are not perfectly interchangeable, but together they determine how aggressively and where the policy moves.
The current TRL DPO documentation exposes separate metrics for:
logps/chosen
logps/rejected
rewards/chosen
rewards/rejected
rewards/margins
rewards/accuracies
entropy
mean token accuracy
gradient norm
processed token count
Older TRL versions did not necessarily expose the same standard metrics, so an older run may require custom instrumentation rather than simply reading an existing log.
I would avoid using only:
training loss
reward accuracy
reward margin
as evidence that the model is improving.
A more informative timeline is:
SFT baseline
DPO model before the first update
step 1
first few logging/checkpoint intervals
mid-run
final checkpoint
reloaded final checkpoint
At each point, evaluate both:
- a preference-specific holdout;
- a fixed ordinary-chat holdout unrelated to the DPO dataset.
The direct-alignment overoptimization study reports that DPO and related methods can show degradation before completing a single epoch. That does not diagnose this run, but it is a strong reason not to compare only SFT against the final DPO checkpoint.
The result patterns have different implications:
| Observation |
More consistent with |
| Quality collapses almost immediately |
implementation/reference mismatch, very aggressive updates, or severe data-contract problem |
| Preference holdout improves while ordinary chat degrades |
narrow preference distribution or excessive movement away from general-chat behavior |
| Both improve initially, then ordinary chat falls |
overoptimization / checkpoint selection problem |
| Only final reloaded model fails |
artifact or inference-path problem |
| Smaller beta helps only when learning rate/exposure also changes |
not enough evidence for beta as the isolated cause |
The original poster also reported mixing in an SFT loss already. Therefore, “add SFT loss” would not be a sufficient answer here. It is still useful as an ablation signal:
Did SFT mixing prevent chosen log-prob from falling?
Did it delay ordinary-chat degradation?
Did it preserve style but not multi-turn coherence?
Did changing its weight alter the first failing checkpoint?
If the collapse signature is unchanged, that weakens the idea that simple SFT regularization alone addresses the mechanism.
I would treat absolute log-probs as necessary diagnostics, but not as standalone quality metrics. Even a seemingly desirable pattern such as chosen increasing and rejected decreasing does not guarantee preserved general conversation quality. The held-out behavior remains the deciding observation.
5. Multi-turn and capability-boundary semantics
The original task was not generic preference tuning; it was teaching the chatbot its capability boundaries and preventing unrealistic promises.
That makes the unit of preference important.
A chosen/rejected label might refer to:
only the final assistant response
the entire assistant trajectory
whether a refusal was correct
whether the explanation after refusal was useful
whether the model recovered after the user clarified
whether the complete conversation remained coherent
Those are not equivalent objectives.
The original poster tried both loss over every assistant response and loss over only the last round. Since neither fixed the issue, I would inspect what the pair-level preference actually represents rather than treating all-turn versus last-turn as the remaining main question.
Research on direct multi-turn preference optimization treats multi-turn trajectories as distinct from ordinary single-completion preference pairs, with additional state and trajectory-length considerations. That does not mean a specialized multi-turn loss is automatically required here. It does mean that “the final answer is preferred” and “the whole interaction is preferred” should not be silently treated as identical supervision.
For capability-boundary evaluation, I would split the holdout into at least three groups:
| Group |
Desired behavior |
| Clearly possible and acceptable |
comply normally; do not over-refuse |
| Impossible, unverifiable, unsafe, or outside the model’s agency |
decline, qualify, or explain the limitation |
| Context-dependent or underspecified |
ask for clarification or state conditions rather than making a categorical claim |
This is analogous to the contrastive logic behind XSTest, which evaluates both prompts that should be refused and superficially similar safe prompts that should not be refused. Capability boundaries are broader than safety, but the same evaluation principle helps prevent one-sided training.
For multi-turn conversations, I would additionally measure:
Does the model maintain the same capability claim across turns?
Does it explain a limitation without repeating one canned refusal?
Can it offer a realistic alternative?
Can it recover when the user changes the conditions?
Does it switch from refusal to compliance when the task becomes feasible?
Does it continue coherently after a normal unrelated turn?
That can distinguish several failure modes which look similar in a single screenshot:
- over-refusal;
- semantic inversion of a preference;
- repetitive decoding;
- context/truncation failure;
- loss of general conversational ability;
- inability to revise a capability judgment across turns.
6. Save/load, PEFT, and inference-path controls
I would keep the training and artifact paths separate throughout the investigation.
A useful comparison matrix is:
| Model state |
Same deterministic inference function |
| Original SFT checkpoint |
evaluate |
| SFT loaded through the DPO code path |
evaluate |
| SFT wrapped by trainer, no optimizer step |
evaluate |
| Zero-update saved checkpoint, reloaded |
evaluate |
| Early DPO checkpoint in memory |
evaluate |
| Same early checkpoint reloaded |
evaluate |
| Final DPO model in memory |
evaluate |
| Same final checkpoint reloaded |
evaluate |
| Merged export, if used |
evaluate |
This isolates where the first behavioral difference appears.
For a deterministic sanity check, keep constant:
prompt messages
chat template
add_generation_prompt behavior
tokenizer files
BOS/EOS/pad tokens
max_new_tokens
stopping criteria
greedy decoding or fixed sampling seed
dtype
quantization
device/backend
The Transformers generation documentation notes that generation behavior is controlled by GenerationConfig. A different saved or loaded generation config can therefore change output independently of training.
With PEFT, the checkpoint-format documentation is particularly relevant: an adapter checkpoint normally contains adapter parameters, not the base model itself. Reloading therefore depends on the correct base model and configuration.
I would preserve:
base model repository
exact base model revision/commit
adapter checkpoint
adapter_config.json
active adapter name
merged/unmerged state
tokenizer revision
chat template
generation_config.json
library versions
Common artifact-level confounders include:
- loading the DPO adapter over a different base revision;
- loading the base model without the SFT adapter expected by DPO;
- accidentally activating the reference adapter;
- merging an adapter twice;
- comparing a merged model with an unmerged model under different precision;
- saving only the adapter, then evaluating the base model alone;
- tokenizer or chat-template files coming from a different repository;
- generation config changing sampling, EOS, or minimum-length behavior.
A failure here would not mean DPO training was healthy. It would mean the observed deployed collapse cannot yet be attributed to DPO updates, dataset semantics, or beta.
7. How I would interpret the main outcomes
| Observation |
What it supports |
What it does not establish |
| Loading/wrapping changes SFT output before training |
adapter activation, train/eval mode, dtype, tokenizer/template, or inference differences |
dataset or beta as the cause |
| Zero-update save/reload changes output |
serialization, base/adapter pairing, merge state, generation config |
DPO optimization failure |
| Chosen/rejected prompt prefixes differ |
rendering or preprocessing contract problem |
that the preference labels themselves are wrong |
| Completion masks or EOS positions are wrong |
loss applied to unintended tokens |
that correcting masks will necessarily solve every symptom |
| Same-SFT policy/reference differ substantially at step 0 |
reference, adapter, preprocessing, mode, or cache mismatch |
which side is correct, or that the mismatch alone explains all degradation |
| Chosen and rejected absolute log-probs both fall while margin rises |
relative improvement with absolute likelihood displacement |
that pair similarity is definitely the cause |
| Chosen rises, rejected falls, ordinary chat falls |
optimization/coverage problem despite success on preference pairs |
beta alone as the cause |
| Only long conversations fail |
truncation, context coverage, trajectory or role-boundary issue |
insufficient model size |
| Only systemless conversations fail |
format or distribution dependence |
that every DPO example must contain a system message |
| Training-time model works, reload fails |
artifact/inference path |
healthy training in all other respects |
| 7B and 32B both fail |
weakens a simple capacity explanation |
proves an implementation-independent DPO failure |
| Custom trainer and Swift both fail |
weakens a single-framework-only explanation |
rules out shared preprocessing, reference, optimization, or artifact errors |
| SFT-loss mixing does not fix it |
weakens “just add SFT regularization” |
rules out excessive updates or narrow data coverage |
The general rule I would use is:
A symptom can nominate a branch, but it should not be promoted to a root cause until a control separates it from the neighboring branches.
8. A compact case record for comparing post-DPO degradation reports
Since you mentioned researching preference training on open-weight models, these are the fields I would find most useful for comparing cases:
Model:
Exact base revision:
Exact SFT checkpoint:
Trainer/framework:
Exact version or commit:
Relevant custom modifications:
Full fine-tuning or PEFT:
Trainable parameter count/fraction:
LoRA rank/alpha/target modules:
Policy construction:
Reference construction:
Active adapters:
Merged/unmerged state:
Reference log-probs cached or live:
Dataset source:
Chosen response source:
Rejected response source:
Number of pairs:
Clear/ambiguous/conflicting pair sample:
Chosen/rejected token-overlap distribution:
Chosen/rejected length distribution:
Chat template:
Rendered pair example:
Prompt-prefix equality check:
BOS/EOS/control tokens:
Completion masks:
Maximum lengths:
Truncation rule:
Effective completion-token distribution:
Beta:
Learning rate:
Batch size:
Gradient accumulation:
Optimizer steps:
Effective epochs:
Processed tokens:
SFT-loss weight:
Gradient norm trend:
Step-zero policy/reference comparison:
Chosen absolute log-prob trend:
Rejected absolute log-prob trend:
Reward-margin trend:
Entropy/repetition trend:
Ordinary-chat holdout:
Capability-boundary holdout:
Multi-turn holdout:
First checkpoint showing degradation:
Original SFT output:
Trainer-wrapped zero-update output:
Zero-update reload output:
Early in-memory output:
Early reloaded output:
Final in-memory output:
Final reloaded output:
Inference function:
Generation config:
Tokenizer revision:
EOS/pad/stop settings:
Precision/quantization:
Not every report needs every field, but this schema makes “dataset,” “beta,” “reference,” “format,” and “artifact” claims meaningfully comparable.
So my current answer would be: I cannot tell from the thread whether the original cause was ever identified, and the available observations do not justify choosing dataset or beta as the answer.
The default route I would use is:
1. establish identical deterministic inference;
2. compare SFT before and after DPO-path wrapping;
3. run a zero-update save/reload control;
4. inspect actual rendered tokens, masks, EOS, and truncation;
5. verify the intended step-zero policy/reference relationship;
6. track chosen and rejected absolute log-probs separately;
7. evaluate ordinary chat and capability boundaries from the first few steps;
8. only then stratify by pair geometry or sweep optimization settings.
That sequence should usually reveal whether the case is primarily moving toward a sequence/reference bug, an artifact problem, a preference-data geometry problem, or genuine overoptimization—without assuming in advance that “DPO destroyed the model” and without treating a better reward margin as proof that conversational quality was preserved.