Humans make grabbing a book from a shelf look effortless, yet it’s a complex process for the brain involving careful planning and navigation around obstacles. Robotics researchers have long grappled with replicating this human-like movement in their systems. The challenge lies in motion planning, where training a robot to retrieve an object without colliding with obstacles demands significant time and resources, as robots can’t dynamically react like humans in unfamiliar environments.
A team from Carnegie Mellon University’s Robotics Institute (RI) has developed Neural Motion Planning to help improve how robots react in new environments. This innovative approach leverages a single, adaptable artificial intelligence network to revolutionize how robots navigate new environments. From cabinets to dishwashers and refrigerators, this data-driven method promises to elevate robotic motion planning to unprecedented levels.
“Sometimes when you deploy a robot, you want it to operate in unstructured or unknown settings — environments where you can’t assume that you know everything,” said Murtaza Dalal, an RI doctoral student. “That’s where these classic motion planning methods break down. One big issue is that these algorithms are very slow because they have to do thousands, maybe even millions, of collision checks.”
Neural Motion Planning draws inspiration from the way humans learn new skills, starting with cautious, deliberate movements and gradually progressing to confident, fluid actions. This cutting-edge technology empowers robots to navigate unfamiliar environments and adapt on the fly when interacting with objects.
To master Neural Motion Planning, researchers exposed robots to millions of simulated scenarios, including typical household settings with shelves, microwaves, and open cabinets, as well as unexpected obstacles like playful puppies and delicate vases. Through this rigorous training, the robots learned to execute quick, responsive motion planning, resulting in the development of a versatile skill set that allows them to excel in diverse real-world environments.
“We have seen amazing successes in large-scale learning for vision and language — think ChatGPT — but not in robotics. Not yet,” said Deepak Pathak, the Raj Reddy Assistant Professor in the RI. “This work is a stepping stone toward that goal. Neural Motion Planning uses the simple recipe of learning at scale in simulation to produce a large degree of generalization in the real world. It works across scenes with different backgrounds, objects, obstacles, and even entire scene arrangements.”
The incredible capabilities of Neural Motion Planning were on full display as it guided a robotic arm through unfamiliar environments in the lab. Using depth cameras to create a 3D representation of the starting point and being presented with a goal position, the system flawlessly provided joint configurations for the robotic arm to navigate through obstacles such as lamps, plants, bookcases, and cabinet doors.
“It was exciting to see a single model deftly avoid diverse household obstacles, including lamps, plants, bookcases, and cabinet doors while moving the robot arm to complete tasks,” said RI master’s student Jiahui Yang. “This feat was enabled by massively scaling up data generation, following a similar recipe to the success of machine learning in vision and language.”
The research team, including master’s student Jiahui Yang, doctoral student Russell Mendonca, robotics simulation engineer Youssef Khaky, and Professor of Computer Science Ruslan Salakhutdinov, has truly pushed the boundaries of what is possible in the field of robotics.
