plasmova/nova-v3

Nova v3

Nova v3 is a decoder-only causal language model with 148.8 million parameters. This repository provides the Transformers checkpoint, tokenizer, and custom model code required to load the model.

Model specifications

Specification	Value
Parameters	148.8M
Vocabulary	32,768 tokens
Hidden size	640
Transformer layers	26
Attention	10 query heads, 5 key/value heads
Maximum context	2,048 tokens
Checkpoint step	3,500

The model supports key/value caching during generation. Its custom implementation supports Transformers 4.57 and 5.x. Review the custom modeling files before loading with trust_remote_code=True.

Load with Transformers

Install PyTorch and Transformers:

pip install torch transformers

Load the tokenizer and model:

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "plasmova/nova-v3"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)

system_prompt = (
    "You are Nova, a helpful and accurate assistant. Answer the user's request directly and concisely. "
    "For simple arithmetic, calculate the result and state it plainly. Do not invent names, scenarios, or equations."
)
prompt = (
    f"<|system|>{system_prompt}<|/system|>"
    "<|user|>Explain why the sky appears blue in one sentence.<|/user|>"
    "<|assistant|>"
)
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False)
output = model.generate(
    **inputs,
    max_new_tokens=160,
    do_sample=True,
    temperature=0.2,
    top_p=0.85,
    top_k=12,
    repetition_penalty=1.12,
    no_repeat_ngram_size=3,
    eos_token_id=[
        tokenizer.eos_token_id,
        tokenizer.convert_tokens_to_ids("<|/assistant|>"),
    ],
)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Recommended starting settings are temperature 0.2, top-p 0.9, top-k 40, and repetition penalty 1.08. These focused settings are a starting point; adjust them for your task. The local inference script applies this concise system instruction by default. Recommended generation defaults are temperature 0.2, top-p 0.85, top-k 12, repetition penalty 1.12, no-repeat n-gram size 3, and up to 160 new tokens. Keep the total prompt and generated text within the 2,048-token context. The example runs on CPU; for GPU inference, move the model and inputs to a supported device and dtype.

Evaluation

The following zero-shot results use the first 500 examples from each task with lm-evaluation-harness 0.4.13 on CPU. These are limited samples, not full validation-set results.

Benchmark	Samples	Accuracy	Normalized accuracy
HellaSwag	500	37.4%	43.6%
ARC Easy	500	51.0%	48.2%
ARC Challenge	500	21.8%	25.0%
WinoGrande	500	52.6%	52.2%
OpenBookQA	500	18.8%	30.0%

Comparison with SmolLM2-135M

In a matched zero-shot run using 500 examples per task, Nova v3 scored 43.6% normalized accuracy on HellaSwag and 52.2% on WinoGrande, compared with 44.2% and 50.6% for SmolLM2-135M. The WinoGrande difference is small relative to sampling uncertainty; treat both results as early directional measurements, not established performance claims. The comparison details and per-run metrics are in benchmark_comparison.json.

On a separate 100-question GSM8K sample with five examples in context, Nova v3 and SmolLM2-135M-Instruct each answered 4 questions correctly (4.0% exact match). This small sample does not establish equivalence. Details are in gsm8k_comparison.json.

Training and license

The v3 training code specifies pretraining on a mixture of FineWeb-Edu, DCLM, Cosmopedia-v2, FineMath, Wikipedia, and CodeParrot, followed by SmolTalk fine-tuning. Dataset contents remain subject to their respective upstream terms.

The model weights and accompanying custom model code are released under the Apache License 2.0. See LICENSE.