Researchers’ robotic system aims to improve autonomy for people with mobility issues

As an undergraduate engineering student in Delhi, India, Amisha Bhaskar took a field trip to a facility for disabled war veterans and met a man who had lost both hands. When she asked him what technologies could improve his life, his reply left an indelible impression: He wanted something so he could take care of himself and not be forced to rely upon others.

Now a second-year doctoral student at the University of Maryland studying computer science, Bhaskar has focused on the wounded veteran’s broad request as her area of study. Working with others in the Robotics Algorithms & Autonomous Systems Lab, she is developing an innovative robotic tool to help people with mobility impairments feed themselves.

The team’s work was recognized last month at the IEEE International Conference on Robotics and Automation (ICRA 2024) in Yokohama, Japan, where a paper Bhaskar presented as lead co-author received top honors in a specialized workshop on cooking and robotics. It is available on the arXiv preprint server.

Existing robotic-assisted feeding technology is very limited, the UMD researchers said. Commercial robotic arms have a fixed, pre-programmed motion that allows them to pick up food only in a specific spot on a plate, and they lack the ability to detect whether they’ve accomplished that task.

“They are not learning on the go, so it will just keep doing this motion no matter if you want to eat it or not, or if the food is picked up or not,” said Bhaskar.

Robotic-assisted feeding can be divided into two steps, she explained: the “acquisition” step involves a utensil picking up the food, while the transfer step is the process of the food reaching a person’s mouth without being dropped or succumbing to some other mishap.

Bhaskar and the UMD team are currently working on the acquisition step, with a lofty goal. While other research groups sometimes count picking up food on a utensil just once as a success, the UMD team’s target is to clear the plate.

The system must be able to recognize and transport a variety of foods served in assisted-care settings—from liquid foods to semi-solid ones like yogurt and tofu to cereals.

One of the most significant challenges for a robot is handling foods with varied textures and consistencies within a single dish, the researchers said. Ramen, for example, presents a complex scenario that includes a liquid broth, squishy tofu, solid vegetables and irregularly shaped noodles that remain the biggest challenge, Bhaskar said. “Every single element requires different strategies, some of which have to be combined,” she said

An interdisciplinary approach has played a key role in the project’s success, said Pratap Tokekar, an associate professor of computer science with an appointment in the University of Maryland Institute for Advanced Computer Studies.

“The technology we’re working on involves computer vision, artificial intelligence, deep neural networks, mechanical engineering and more—it all needs to come together seamlessly so that the robotic system is both safe for users and efficient in accomplishing the task at hand,” he said.

Tokekar is academic adviser to Bhaskar and another graduate student working on the project, Rui Liu, a third-year doctoral student in computer science.

Robotic-assisted feeding is a relatively new area of research for Liu, who had previously focused on computer vision and human-robot interaction. But like Bhaskar, Liu sees the potential here to greatly improve people’s lives, particularly older adults or those with mobility issues that make feeding themselves difficult.

Additional team members include Vishnu D. Sharma, Ph.D. ’24 and Guangyao Shi, Ph.D. ’23, now a postdoctoral researcher at the University of Southern California.

While the project is probably several years away from real-world application, Tokekar is confident in the team’s progress, and particularly in Bhasker’s and Liu’s eagerness and intense focus.

“The best part of this project is that every time we meet, they have 10 new ideas since the last time that we met,” Tokekar said. “Instead of me telling them what to do, they already know what to do. I’m just helping shape their ideas.”

As an undergraduate engineering student in Delhi, India, Amisha Bhaskar took a field trip to a facility for disabled war veterans and met a man who had lost both hands. When she asked him what technologies could improve his life, his reply left an indelible impression: He wanted something so he could take care of himself and not be forced to rely upon others.

Now a second-year doctoral student at the University of Maryland studying computer science, Bhaskar has focused on the wounded veteran’s broad request as her area of study. Working with others in the Robotics Algorithms & Autonomous Systems Lab, she is developing an innovative robotic tool to help people with mobility impairments feed themselves.

The team’s work was recognized last month at the IEEE International Conference on Robotics and Automation (ICRA 2024) in Yokohama, Japan, where a paper Bhaskar presented as lead co-author received top honors in a specialized workshop on cooking and robotics. It is available on the arXiv preprint server.

Existing robotic-assisted feeding technology is very limited, the UMD researchers said. Commercial robotic arms have a fixed, pre-programmed motion that allows them to pick up food only in a specific spot on a plate, and they lack the ability to detect whether they’ve accomplished that task.

“They are not learning on the go, so it will just keep doing this motion no matter if you want to eat it or not, or if the food is picked up or not,” said Bhaskar.

Robotic-assisted feeding can be divided into two steps, she explained: the “acquisition” step involves a utensil picking up the food, while the transfer step is the process of the food reaching a person’s mouth without being dropped or succumbing to some other mishap.

Bhaskar and the UMD team are currently working on the acquisition step, with a lofty goal. While other research groups sometimes count picking up food on a utensil just once as a success, the UMD team’s target is to clear the plate.

The system must be able to recognize and transport a variety of foods served in assisted-care settings—from liquid foods to semi-solid ones like yogurt and tofu to cereals.

One of the most significant challenges for a robot is handling foods with varied textures and consistencies within a single dish, the researchers said. Ramen, for example, presents a complex scenario that includes a liquid broth, squishy tofu, solid vegetables and irregularly shaped noodles that remain the biggest challenge, Bhaskar said. “Every single element requires different strategies, some of which have to be combined,” she said

An interdisciplinary approach has played a key role in the project’s success, said Pratap Tokekar, an associate professor of computer science with an appointment in the University of Maryland Institute for Advanced Computer Studies.

“The technology we’re working on involves computer vision, artificial intelligence, deep neural networks, mechanical engineering and more—it all needs to come together seamlessly so that the robotic system is both safe for users and efficient in accomplishing the task at hand,” he said.

Tokekar is academic adviser to Bhaskar and another graduate student working on the project, Rui Liu, a third-year doctoral student in computer science.

Robotic-assisted feeding is a relatively new area of research for Liu, who had previously focused on computer vision and human-robot interaction. But like Bhaskar, Liu sees the potential here to greatly improve people’s lives, particularly older adults or those with mobility issues that make feeding themselves difficult.

Additional team members include Vishnu D. Sharma, Ph.D. ’24 and Guangyao Shi, Ph.D. ’23, now a postdoctoral researcher at the University of Southern California.

While the project is probably several years away from real-world application, Tokekar is confident in the team’s progress, and particularly in Bhasker’s and Liu’s eagerness and intense focus.

“The best part of this project is that every time we meet, they have 10 new ideas since the last time that we met,” Tokekar said. “Instead of me telling them what to do, they already know what to do. I’m just helping shape their ideas.”