SmolQwen Alpha 30M
A 31.3M-parameter dense language model trained from scratch on dual RTX 4090s, on a compact,
QA-dense corpus (20M unique tokens, ~100M tokens seen over 5 epochs). Second release in the
SmolQwen line after smolqwen-alpha-1m.
Architecture: hybrid attention stack (3 linear-attention blocks + 1 gated-softmax block, repeated), attention sink 2048 + sliding window 6144, RMSNorm, SwiGLU FFN, tied embeddings, RoPE. No MoE β plain dense FFN.
Results (held-out 291-item bench, hand-written items held out of training)
| checkpoint | total | instruction | science | math | commonsense |
|---|---|---|---|---|---|
smolqwen-alpha-30m (pretrain) |
37.5% | 58% | 24% | 25% | 50% |
smolqwen-alpha-30m-sft (+ chat SFT) |
30.9% | 43% | 21% | 25% | 40% |
Reference bar (same bench): SmolQwen 1M mid model 51.5% (pretrain) / 47.1% (SFT).
Files
| file | what |
|---|---|
smolqwen-alpha-30m.safetensors |
pretrained weights (EMA), 155 tensors |
smolqwen-alpha-30m-sft.safetensors |
chat/SFT weights (EMA) |
ckpt_flagship.pt / ckpt_flagship_sft.pt |
same weights as PyTorch checkpoints (cfg + ema + model) |
spm_mid.model |
SentencePiece tokenizer, vocab 8192 |
Usage
import json, torch
from safetensors.torch import safe_open
from smolnet.model import SmolNet # smolnet/model.py in this repo's training code
from sentencepiece import SentencePieceProcessor
with safe_open("smolqwen-alpha-30m.safetensors", framework="pt") as f:
sd = {k: f.get_tensor(k) for k in f.keys()}
cfg = json.loads(f.metadata()["config"])
cfg["layers"] = [tuple(l) if l.startswith("(") else l for l in cfg["layers"]]
model = SmolNet(**cfg)
model.load_state_dict(sd, strict=False) # mnorm/mproj (multi-token-prediction head) are unused at inference
model.eval()
sp = SentencePieceProcessor(); sp.load("spm_mid.model")
ids = sp.encode("What is the largest planet in our solar system?")
Reasonable generation settings at this scale: greedy or low-temperature, stop at . / newline.
- Downloads last month
- 16
Inference Providers NEW
This model isn't deployed by any Inference Provider. π Ask for provider support