TCP4D: A Unified Tool-Center-Point Tracking and Generation Model for Robot Manipulation
Abstract
Robot manipulation videos provide a rich visual experience, yet recovering robot action and gripper-state annotations from RGB observations still remains challenging. Using the embodiment-agnostic tool-center-point (TCP) pose as the action space, we present TCP4D, a unified model in which a shared 4D scene representation both tracks observed TCP motion and generates future TCP trajectories. Given a monocular video and initial 2D TCP queries, the tracker directly recovers metric TCP poses in SE(3) and gripper states alongside scene geometry and camera parameters. The generator samples local features from the same 4D representation along the recovered TCP tracks, then combines them with robot observations and language instructions to predict future TCP states. Our new cross-embodiment augmentation pipeline converts existing RGB videos into demonstrations for alternative embodiments by retargeting the recovered motion and synthesizing the corresponding observations, without recorded source-robot states. Experiments show that the tracker achieves absolute trajectory errors of 7.60 and 16.07 mm on DROID and RoboLab and adapts to a new embodiment with only ten demonstrations. The generator achieves 69.19% mean success on RoboTwin and outperforms well-established policy models on two real-world tasks, while joint training on source and augmented demonstrations improves target-robot success by 41.5% over augmented-only training.
Community
Meet TCP4D: A Unified Tool-Center-Point Tracking and Generation Model for Robot Manipulation
Given monocular RGB video and initial 2D TCP queries, it recovers 6-DoF tool poses and gripper states. Its shared 4D representation also supports language-conditioned future action generation.
We turn source-robot videos into target-robot demonstrations without recorded source states.
Highlights:
• 7.60 mm tracking ATE on DROID, 10 episodes few-shot transfer to UR5.
• 69.19% success across 48 RoboTwin tasks.
• ARX-X5 success: 30.5% → 72.0% with source + augmented data vs. augmented-only training.
Hi authors, thanks for the work. A quick question: the abstract says joint training improves target-robot success by 41.5%, but in Sec. 4.3 that is the ARX-X5 gain (30.5% to 72.0%); UR5-WSG goes from 32.0% to 39.0%, and the mean over both target robots is +24.25 points. Details in this GitHub issue: https://github.com/Livioni/TCP4D/issues/1
Thanks for checking the numbers! In the abstract & Intro, we reported the largest gain across the two target robots, which was 41.5 percentage points on ARX-X5. Thanks!
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- PointPilot: Unified Robotic Manipulation via Object-Centric 3D Point Flow Prediction (2026)
- UNITAS: A 3D-Native World Action Model for Embodied Manipulation (2026)
- PDRVLA: A Vision-Language-Action Model with Physical Dynamics Representations (2026)
- Learning the 4D World through Point Trajectory Forecasting (2026)
- WorldFlow: Joint Video and 3D Motion Prediction via Large-Scale Pretraining (2026)
- PointWAM: 3D World Action Modeling for Dexterous Robotic Manipulation (2026)
- Spatial-UMI-Dream: Scaling World Action Models with Metric Geometry for UMI Manipulation (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2611.23143 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 3
HarrisonPENG/TCP4D-stage1-RoboTwin
Datasets citing this paper 3
HarrisonPENG/TCP4D-Benchmark
HarrisonPENG/TCP4D-DROID-Training-Set
Spaces citing this paper 1
Collections including this paper 0
No Collection including this paper