Papers
arxiv:2611.14365

From Video Frames to Robot Meta-Frames: A Unified Latent Interface for World Action Models

Published on Nov 10
Authors:
,
,
,
,

Abstract

MetaWAM injects robot signals into pretrained video diffusion models by packing them into a single frequency-domain meta-frame, improving action and world-model prediction with lower latency.

Pretrained video diffusion models (VDMs) offer a powerful foundation for World Action Models (WAMs), which jointly predict robot actions and future observations. Building such models requires injecting low-dimensional robot signals (e.g. proprioception, actions, and values) into a latent space shaped by natural videos, and a key design choice is the interface through which these signals enter: existing schemes treat it as a formatting detail, spending a separate latent frame on each signal or bolting on dedicated encoders and decoders. We argue that signal injection is instead a distribution-alignment problem: how well a representation matches the prior of the pretrained VDMs during denoising. Using the relative reconstruction error of a frozen VDM denoiser as a probe, we find that conventional spatial encodings sit far off this prior, whereas frequency-domain packing yields latents that the denoiser corrects far less. Motivated by this diagnostic result, we propose MetaWAM, which folds all robot signals into a single unified meta-frame: each signal is assigned to a disjoint frequency-domain channel band and lifted to the spatial latent by an inverse DCT, yielding a decodable interface aligned with the pretrained VDM prior without any added module or architectural change. A fixed band layout together with slot-aware routing lets the same meta-frame support policy, world-model, and value prediction over a shared set of robot variables. On RoboCasa, MetaWAM reaches 68.9% success while collapsing the per-signal frames into one and cutting inference latency by 39%, and attains 85.0% success on three real-world Franka tasks. Code and models are available at https://github.com/showlab/MetaWAM

Community

Thank you for sharing your work. MetaWAM packs action, proprioceptive, and value signals into a frequency-organized meta-frame for joint world, action, and value modeling. Its representation design connects these signals through a shared video-model interface.

We discuss your work in Section 4.5 (“Scaling data construction”), p. 53 of the updated manuscript of our survey, In-Context Learning for Robots: Methods and Applications. We discuss how a common latent interface makes action, proprioception, and value information available for world-action learning. Read the updated discussion.

If you find our discussion of representations for world-model-based control useful for your own work, we would be grateful if you would consider citing the survey.

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2611.14365
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 3

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.14365 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.14365 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.