From Video Frames to Robot Meta-Frames: A Unified Latent Interface for World Action Models
Abstract
MetaWAM injects robot signals into pretrained video diffusion models by packing them into a single frequency-domain meta-frame, improving action and world-model prediction with lower latency.
Pretrained video diffusion models (VDMs) offer a powerful foundation for World Action Models (WAMs), which jointly predict robot actions and future observations. Building such models requires injecting low-dimensional robot signals (e.g. proprioception, actions, and values) into a latent space shaped by natural videos, and a key design choice is the interface through which these signals enter: existing schemes treat it as a formatting detail, spending a separate latent frame on each signal or bolting on dedicated encoders and decoders. We argue that signal injection is instead a distribution-alignment problem: how well a representation matches the prior of the pretrained VDMs during denoising. Using the relative reconstruction error of a frozen VDM denoiser as a probe, we find that conventional spatial encodings sit far off this prior, whereas frequency-domain packing yields latents that the denoiser corrects far less. Motivated by this diagnostic result, we propose MetaWAM, which folds all robot signals into a single unified meta-frame: each signal is assigned to a disjoint frequency-domain channel band and lifted to the spatial latent by an inverse DCT, yielding a decodable interface aligned with the pretrained VDM prior without any added module or architectural change. A fixed band layout together with slot-aware routing lets the same meta-frame support policy, world-model, and value prediction over a shared set of robot variables. On RoboCasa, MetaWAM reaches 68.9% success while collapsing the per-signal frames into one and cutting inference latency by 39%, and attains 85.0% success on three real-world Franka tasks. Code and models are available at https://github.com/showlab/MetaWAM
Community
Thank you for sharing your work. MetaWAM packs action, proprioceptive, and value signals into a frequency-organized meta-frame for joint world, action, and value modeling. Its representation design connects these signals through a shared video-model interface.
We discuss your work in Section 4.5 (“Scaling data construction”), p. 53 of the updated manuscript of our survey, In-Context Learning for Robots: Methods and Applications. We discuss how a common latent interface makes action, proprioception, and value information available for world-action learning. Read the updated discussion.
If you find our discussion of representations for world-model-based control useful for your own work, we would be grateful if you would consider citing the survey.
Get this paper in your agent:
hf papers read 2611.14365 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 3
Ethanliang99/MetaWAM-LIBERO-Predict2-2B
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper