Enhancing Visuomotor Manipulation Control with a Predictive-Dynamics-Aware World-Model-Conditioned Diffusion Policy
The 26th International Conference on Control, Automation and Systems (ICCAS 2026)
Oral presentation, Paper WeBT8.3 · Wednesday, Oct 28, 2026, 16:10–16:25 · Accepted Jul 31, 2026
As an undergraduate researcher at Hanyang University’s Control & Optimization Lab, I studied robot manipulation from reinforcement learning through imitation learning. This work extends Diffusion Policy, a strong imitation-learning policy, by conditioning its denoising network on a latent feature from a lightweight GRU-based world model. The world model is used only as a conditioning input rather than as a simulator, preserving a practical computational footprint for real-robot deployment.
- Addressed the limitation of image-only Diffusion Policy features, which do not explicitly capture temporal scene progression or object interactions.
- Designed the world-model-conditioned Diffusion Policy architecture, experiment plan, and implementation while learning from a senior coauthor.
- On real Franka Emika Panda pick-and-place experiments, improved success rate from 45.0% to 72.5% under the same training conditions, a 27.5 percentage-point gain.
- Kept added inference latency to approximately 70 ms and achieved the improvement through architectural expansion without increasing the training dataset.
Presentation slides
Abstract
Recent visuomotor policies have achieved strong performance in robotic manipulation. Specifically, diffusion policy further improves action generation in various tasks. However, the diffusion policy mainly relies on image-based observation features and does not explicitly learn a dynamics-aware representation that summarizes temporal scene progression or object interactions. Therefore, we propose a world-model-conditioned diffusion policy (WM-DP) for visuomotor manipulation. WM-DP extends the original diffusion policy by concatenating a latent feature generated by a world model. Through this design, we study whether a simple early-stage world model conditioning structure can improve real-world manipulation performance without adding architectural complexity. We evaluate WM-DP on real-world pick-and-place tasks using objects with different physical properties. We also evaluate the method using a deformable object whose graspable regions and appearance change depending on its orientation. The results show that our model improves both success rate and trajectory quality under observed object conditions. Overall, our work suggests that even a GRU-based world model can provide useful cues for physical interaction and improve diffusion policy in visuomotor manipulation.
The DOI will be added once the proceedings are published.