SmolQwen Alpha 30M

A 31.3M-parameter dense language model trained from scratch on dual RTX 4090s, on a compact, QA-dense corpus (20M unique tokens, ~100M tokens seen over 5 epochs). Second release in the SmolQwen line after smolqwen-alpha-1m.

Architecture: hybrid attention stack (3 linear-attention blocks + 1 gated-softmax block, repeated), attention sink 2048 + sliding window 6144, RMSNorm, SwiGLU FFN, tied embeddings, RoPE. No MoE β€” plain dense FFN.

Results (held-out 291-item bench, hand-written items held out of training)

checkpoint total instruction science math commonsense
smolqwen-alpha-30m (pretrain) 37.5% 58% 24% 25% 50%
smolqwen-alpha-30m-sft (+ chat SFT) 30.9% 43% 21% 25% 40%

Reference bar (same bench): SmolQwen 1M mid model 51.5% (pretrain) / 47.1% (SFT).

Files

file what
smolqwen-alpha-30m.safetensors pretrained weights (EMA), 155 tensors
smolqwen-alpha-30m-sft.safetensors chat/SFT weights (EMA)
ckpt_flagship.pt / ckpt_flagship_sft.pt same weights as PyTorch checkpoints (cfg + ema + model)
spm_mid.model SentencePiece tokenizer, vocab 8192

Usage

import json, torch
from safetensors.torch import safe_open
from smolnet.model import SmolNet          # smolnet/model.py in this repo's training code
from sentencepiece import SentencePieceProcessor

with safe_open("smolqwen-alpha-30m.safetensors", framework="pt") as f:
    sd = {k: f.get_tensor(k) for k in f.keys()}
    cfg = json.loads(f.metadata()["config"])
cfg["layers"] = [tuple(l) if l.startswith("(") else l for l in cfg["layers"]]

model = SmolNet(**cfg)
model.load_state_dict(sd, strict=False)    # mnorm/mproj (multi-token-prediction head) are unused at inference
model.eval()

sp = SentencePieceProcessor(); sp.load("spm_mid.model")
ids = sp.encode("What is the largest planet in our solar system?")

Reasonable generation settings at this scale: greedy or low-temperature, stop at . / newline.

Downloads last month
16
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support