Home / Technology / Robots Learn by Practicing in Virtual Homes
Robots Learn by Practicing in Virtual Homes
23 Jul
Summary
- SceneSmith AI creates 3D indoor environments from text.
- Robots train in virtual spaces, exposing weak action plans.
- Over 1,300 diverse virtual scenes were generated for training.
Researchers at MIT have developed SceneSmith, an AI system that constructs realistic 3D indoor environments from text descriptions, enabling robots to train for everyday tasks. This virtual training environment allows robots to practice and identify weak action plans in simulated settings before encountering real-world challenges.
The SceneSmith system uses AI agents to generate detailed digital rooms, complete with furniture and movable objects that possess realistic physics. This allows robots to interact with virtual environments, such as opening cabinets or placing items, offering a safe alternative to physical testing. Previously, robots struggled with basic chores due to a lack of diverse experience.
More than 1,300 unique training scenes were created, featuring familiar and unusual settings with significant object density, mirroring real-world clutter. This extensive variety helps robots learn to adapt their plans across different layouts and object arrangements, moving beyond single, staged demonstrations.
Human evaluators found SceneSmith's generated rooms to be highly realistic and accurate to prompts, with a 92% win rate for realism. While the system can take hours to generate a single scene, its ability to expose robot policy failures in a scalable manner presents a significant step forward for robotic training.
