World-Model-Conditioned Diffusion Policy for Visuomotor Manipulation
Diffusion Policy generates strong robot actions from images, but its observation features do not explicitly summarize how a scene evolves over time or how objects respond to contact. This work asks whether a simple, early-stage world-model condition can supply that dynamics-aware cue on a real robot without making the policy harder to deploy.
- Real-robot pick-and-place success rate
- 45.0% → 72.5%
- Gain under identical training conditions
- +27.5 %p
- Added inference latency
- ≈ 70 ms
Approach
WM-DP extends the original Diffusion Policy by concatenating a latent feature produced by a lightweight GRU-based world model with the observation features. The combined condition modulates the denoising network through FiLM layers at every denoising step.
The world model is used only as a conditioning input, not as a simulator for planning or data generation. This keeps the architecture simple and the computational footprint practical for real-robot control.
Experiments
- Real-world pick-and-place on a Franka Emika Panda with objects of different physical properties.
- An additional deformable object whose graspable regions and appearance change depending on its orientation.
- Both success rate and trajectory quality improved under the observed object conditions, without increasing the training dataset.
My role
- Designed the model architecture, experiment plan, and implementation while learning from a senior coauthor.
- Ran the real-robot experiments and analyzed success rate, trajectory quality, and inference latency.
Presentation
I presented the paper in the ICCAS 2026 oral session on October 28, 2026 (Paper WeBT8.3). The full paper and the slides are linked below.
Scope
The reported gains were measured under observed object conditions. The study deliberately tests a simple early-stage conditioning structure; richer world models and unseen-object generalization remain future work.
Resources
Technologies: Diffusion Policy · GRU World Model · FiLM Conditioning · Imitation Learning · Franka Emika Panda