Llama-3.2-1B-GRPD

Student checkpoint produced by GRPD in the E3 setting of our paper on reward aware on policy distillation.

Built with Llama.

Setup

student (initialization) meta-llama/Llama-3.2-1B-Instruct
teacher (frozen) meta-llama/Llama-3.2-3B-Instruct
method GRPD
training data UltraChat, 9000 prompts
steps 50, with 128 prompts per step and 8 samples per prompt
learning rate 1e-6
max response length 2048
rollout seed 0
selected step 30

All five methods in this setting share one hyperparameter configuration, the same training data, the same data seed and paired rollout seeds. No method received per method tuning.

Checkpoint selection

The checkpoint was selected on a held out set of 1000 prompts scored by Skywork-Reward-V2-Llama-3.2-3B. That selection set shares no prompts with any reported benchmark, and the benchmark numbers below come from a different reward model, Skywork-Reward-V2-Llama-3.1-8B. Candidate steps were 10, 20, 30, 40 and 50.

Results

External reward model scalar from Skywork-Reward-V2-Llama-3.1-8B, 8 samples per prompt averaged per prompt then averaged over prompts. Values are the mean over 3 rollout seeds with the across seed sample standard deviation; this repository holds the seed 0 checkpoint.

benchmark prompts score
AlpacaEval 805 12.547 ± 0.083
WritingBench 1000 3.789 ± 0.055
WildBench 1024 3.245 ± 0.073

For reference, under the same protocol the untrained 1B student scores 12.991 / 5.043 / 4.295 and the 3B teacher scores 21.085 / 12.992 / 14.019.

Decoding for evaluation: AlpacaEval and WildBench at temperature 0.7 and top_p 0.95; WritingBench at temperature 0.7, top_p 0.8 and top_k 20. Identical for every method and for both reference models.

Files

training_logs/ holds the full record for this run: selected.json with every candidate step and its validation score, metrics.jsonl with the per step training dynamics, and launch.json with the complete hyperparameter configuration and switches.

License

This model is a derivative of Llama 3.2 and is governed by the Llama 3.2 Community License. Use is subject to the Llama 3.2 Acceptable Use Policy.

Downloads last month
39
Safetensors
Model size
1B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for tzwilliam0/Llama-3.2-1B-GRPD

Finetuned
(1,859)
this model