WeightBench: Evaluating Language Models as Weight-Space Programmers
Abstract
The WEIGHTBENCH benchmark evaluates the capacity of large language models to perform Weight-Space Programming tasks, which involve reading, modifying, and generating the weights of various neural network architectures.
Recent work shows that large language models (LLMs) can operate directly on the parameters of other neural networks. They can read what a network was trained to do, fix a network that fails, and write working networks from scratch, all without gradient descent. This capability has been termed Weight-Space Programming (WSP). Exploring this new paradigm systematically requires a robust benchmark. We introduce WEIGHTBENCH, a benchmark that evaluates LLMs of increasing capability on 42 WSP tasks across five tracks: reading, fixing and writing weights, recognizing edits that are impossible, and data-free compression. The networks whose weights the LLMs read and edit range from 2K-parameter control policies to 7B-parameter language models. Experimental benchmark results show that frontier agents can already read small networks and write working control policies without ever running them, but cannot yet patch language models, compress them without data, or tell an impossible edit from a hard one. On all 42 tasks, Claude Opus 5.5 reaches a macro score of 0.535 (Sonnet 5.5 0.466, Haiku 5.5 0.402, DeepSeek v4-flash 0.215); on the 38 tasks it was run on, GPT-6.1-sol reaches 0.568 against Opus's 0.546. Code, the task registry, our OS-level sandbox harness, and the live leaderboard are publicly available at https://github.com/Aayush-Marasini/WeightBench.
Community
We built WeightBench to test whether LLMs can genuinely reason about neural network weights (Weight-Space Programming) or if they just brute-force solutions via trial-and-error.
What's inside:
- 42 tasks across 5 tracks (READ, FIX, META, WRITE, SYSTEMS), scaling from 2K-param MLPs to 7B Transformers.
- Auditable OS-level CPU sandbox (bubblewrap / sandbox-exec) with a metered NumPy subclass injected via a
.pthhook to catch hidden forward passes and log everything to a hash-chained ledger.
Key findings (Claude 5.5, GPT-6.1-sol, DeepSeek v4-flash):
- Models can synthesize working control policies (Breakout, Go) completely zero-shot from a blank slate without running the network.
- They still fail at surgical patching without collateral damage, data-free hardware compression, and distinguishing impossible edits from hard ones.
Everything is MIT-licensed—would love feedback from the community and to see other agents tested on the benchmark!
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Replicating and Extending Weight-Space Programming: Frontier LLMs Compile Programs into Weights and Read Input Layers (2026)
- Weights&Caches: LLMs Can Program Tabular Foundation Models (2026)
- Can LLMs Write Backdoors? An Empirical Security Study of Weight-Space Programming (2026)
- Same Fix, Different Weights: Replicating Weight-Space Programming with Claude (2026)
- Can an LLM Actually Reason and Program about Neural Network's Weights or is it just memorization on steroids? Exploring"Weight-Space Programming" (2026)
- Compiling Known Rules into Compact Neural Agents: Weight-Space Programming on Unseen Games (2026)
- Weight Oracles: Reading Neural Network Weights with Language Models (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2611.18553 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper