Virtual reality headsets may one day let you control a robot anywhere from the battlefield to a surgical suite.

Key Takeaways

  • Mixed reality could soon let you control robots remotely. Mixed reality, or a similar concept known as extended reality, could allow the military to use robots in combat. Large manufacturers are using mixed reality to have a mechanic in one location assist someone across the country on the final assembly of a plane.

Researchers at the Microsoft Mixed Reality and AI Lab and ETH Zurich recently developed a new method that combines mixed reality and robotics. The term “mixed reality” (MR or MxR) refers to the merging of real and virtual worlds to produce new environments, and it’s part of a growing effort to find better ways to use robots remotely.

“Current systems require training and practice for operators to learn the abstracted commands,” Todd Richmond, an IEEE member and director of the Tech + Narrative Lab at Pardee RAND Graduate School, told Lifewire in an email interview. “MxR can provide an operator a more “embodied” control system, and can use more literal body movements for command and control (e.g., human moving an arm to move the robotic arm).”

Virtual Controls

The MR and robotics system devised by the researchers was tested using a HoloLens MR headset. One method is designed to plan missions in which a robot inspects an environment. 

The human user moves in the environment they want to inspect wearing a HoloLens headset, placing holograms shaped as waypoints that define a robot’s trajectory. The user can also highlight specific areas where it wants a robot to collect images or data. 

“The combination of spatial computing and egocentric sensing on mixed reality devices enables them to capture and understand human actions and translate these to actions with spatial meaning, which offers exciting new possibilities for collaboration between humans and robots,” the researchers wrote in their paper. 

Mixed reality robotics will enable manufacturing, surgeries, all kinds of things to be done by robots being controlled remotely by humans.

Mixed reality, or a similar concept known as extended reality, could allow the military to use robots in combat, Gregory Thomas, the director of the Center for Design Research at the University of Kansas, said in an email interview. 

“The robots would be able to not only collect data about the environment, but analyze direct hazards, chart passages, robotically deal with safety issues (think of a bomb detection unit that can move like a human, and provide a huge amount more of info to the controller),” he said. 

More peaceful use of the technology is a current project underway with robots in hospitals. Thomas and his team used a robot at a Kansas hospital to become “friends” with pediatric patients. 

“For a child, we found anxiety, frustration, [and] loneliness are all common, and the robot was able to be programmed with a children’s medical dictionary to be able to explain an upcoming “procedure,” which made them more comfortable,” Thomas said.

Future Robots to the Rescue

If mixed reality takes off, it could eventually help you work from home, too. Large manufacturers are using mixed reality to have a mechanic in one location assist someone across the country on the final assembly of a plane, pointed out Bob Bilbruck, CEO of the technology consulting company Captjur said in an email interview.

“Mixed reality robotics will enable manufacturing, surgeries, all kinds of things to be done by robots being controlled remotely by humans,” he added. 

Researchers at Imperial College London recently developed a malleable robotic arm that can be twisted into shape with the help of augmented reality (AR). The robotic arm can be turned in all directions for use in areas like spacecraft maintenance, manufacturing, and injury rehabilitation.

Mixed reality could one day even allow humans to embody a physical avatar in the form of a robot. 

“These have obvious uses in hazardous situations (e.g., fire fighting, explosive ordnance detail, etc.) but could also lead to more recreational uses such as hybrid sports [and] entertainment,” Richmond said. “The lines will continue to blur between analog and digital, human and machine.”

Get the Latest Tech News Delivered Every Day