Robotic system feeds people with severe mobility limitations

Cornell researchers have developed a robotic feeding system that uses computer vision, machine learning and multimodal sensing to safely feed people with severe mobility limitations, including those with spinal cord injuries, cerebral palsy and multiple sclerosis.

“Feeding individuals with severe mobility limitations with a robot is difficult, as many cannot lean forward and require food to be placed directly inside their mouths,” said Tapomayukh “Tapo” Bhattacharjee, assistant professor of computer science in the Cornell Ann S. Bowers College of Computing and Information Science and senior developer behind the system. “The challenge intensifies when feeding individuals with additional complex medical conditions.”

A paper on the system, “Feel the Bite: Robot-Assisted Inside-Mouth Bite Transfer using Robust Mouth Perception and Physical Interaction-Aware Control,” was presented at the Human Robot Interaction conference, held March 11–14, in Boulder, Colorado. It received a Best Paper Honorable Mention recognition, while a demo of the research team’s broader robotic feeding system received a Best Demo Award.

A leader in assistive robotics, Bhattacharjee and his EmPRISE Lab have spent years teaching machines the complex process by which we humans feed ourselves. It’s a complicated challenge to teach a machine—everything from identifying food items on a plate, picking them up and then transferring it inside the mouth of a care recipient.

“This last 5 centimeters, from the utensil to inside the mouth, is extremely challenging,” Bhattacharjee said.

Some care recipients may have very limited mouth openings, measuring less than 2 centimeters, while others experience involuntary muscle spasms that can occur unexpectedly, even when the utensil is inside their mouth, Bhattacharjee said. Further, some can only bite food at specific locations inside their mouth, which they indicate by pushing the utensil using their tongue, he said.

“Current technology only looks at a person’s face once and assumes they will remain still, which is often not the case and can be very limiting for care recipients,” said Rajat Kumar Jenamani, the paper’s lead author and a doctoral student in the field of computer science.

To address these challenges, researchers developed and outfitted their robot with two essential features: real-time mouth tracking that adjusts to users’ movements, and a dynamic response mechanism that enables the robot to detect the nature of physical interactions as they occur, and react appropriately. This enables the system to distinguish between sudden spasms, intentional bites and user attempts to manipulate the utensil inside their mouth, researchers said.

The robotic system successfully fed 13 individuals with diverse medical conditions in a user study spanning three locations: the EmPRISE Lab on the Cornell Ithaca campus, a medical center in New York City, and a care recipient’s home in Connecticut. Users of the robot found it to be safe and comfortable, researchers said.

“This is one of the most extensive real-world evaluations of any autonomous robot-assisted feeding system with end-users,” Bhattacharjee said.

The team’s robot is a multi-jointed arm that holds a custom-built utensil at the end that can sense the forces being applied on it. The mouth tracking method—trained on thousands of images featuring various participants’ head poses and facial expressions—combines data from two cameras positioned above and below the utensil.

This allows for precise detection of the mouth and overcomes any visual obstructions caused by the utensil itself, researchers said. This physical interaction-aware response mechanism uses both visual and force sensing to perceive how users are interacting with the robot, Jenamani said.

“We’re empowering individuals to control a 20-pound robot with just their tongue,” he said.

He cited the user studies as the most gratifying aspect of the project, noting the significant emotional impact of the robot on the care recipients and their caregivers. During one session, the parents of a daughter with schizencephaly quadriplegia, a rare birth defect, witnessed her successfully feed herself using the system.

“It was a moment of real emotion; her father raised his cap in celebration, and her mother was almost in tears,” Jenamani said.

While further work is needed to explore the system’s long-term usability, its promising results highlight the potential to improve care recipients’ level of independence and quality of life, researchers said.

“It’s amazing,” Bhattacharjee said, “and very, very fulfilling.”

Cornell researchers have developed a robotic feeding system that uses computer vision, machine learning and multimodal sensing to safely feed people with severe mobility limitations, including those with spinal cord injuries, cerebral palsy and multiple sclerosis.

“Feeding individuals with severe mobility limitations with a robot is difficult, as many cannot lean forward and require food to be placed directly inside their mouths,” said Tapomayukh “Tapo” Bhattacharjee, assistant professor of computer science in the Cornell Ann S. Bowers College of Computing and Information Science and senior developer behind the system. “The challenge intensifies when feeding individuals with additional complex medical conditions.”

A paper on the system, “Feel the Bite: Robot-Assisted Inside-Mouth Bite Transfer using Robust Mouth Perception and Physical Interaction-Aware Control,” was presented at the Human Robot Interaction conference, held March 11–14, in Boulder, Colorado. It received a Best Paper Honorable Mention recognition, while a demo of the research team’s broader robotic feeding system received a Best Demo Award.

A leader in assistive robotics, Bhattacharjee and his EmPRISE Lab have spent years teaching machines the complex process by which we humans feed ourselves. It’s a complicated challenge to teach a machine—everything from identifying food items on a plate, picking them up and then transferring it inside the mouth of a care recipient.

“This last 5 centimeters, from the utensil to inside the mouth, is extremely challenging,” Bhattacharjee said.

Some care recipients may have very limited mouth openings, measuring less than 2 centimeters, while others experience involuntary muscle spasms that can occur unexpectedly, even when the utensil is inside their mouth, Bhattacharjee said. Further, some can only bite food at specific locations inside their mouth, which they indicate by pushing the utensil using their tongue, he said.

“Current technology only looks at a person’s face once and assumes they will remain still, which is often not the case and can be very limiting for care recipients,” said Rajat Kumar Jenamani, the paper’s lead author and a doctoral student in the field of computer science.

To address these challenges, researchers developed and outfitted their robot with two essential features: real-time mouth tracking that adjusts to users’ movements, and a dynamic response mechanism that enables the robot to detect the nature of physical interactions as they occur, and react appropriately. This enables the system to distinguish between sudden spasms, intentional bites and user attempts to manipulate the utensil inside their mouth, researchers said.

The robotic system successfully fed 13 individuals with diverse medical conditions in a user study spanning three locations: the EmPRISE Lab on the Cornell Ithaca campus, a medical center in New York City, and a care recipient’s home in Connecticut. Users of the robot found it to be safe and comfortable, researchers said.

“This is one of the most extensive real-world evaluations of any autonomous robot-assisted feeding system with end-users,” Bhattacharjee said.

The team’s robot is a multi-jointed arm that holds a custom-built utensil at the end that can sense the forces being applied on it. The mouth tracking method—trained on thousands of images featuring various participants’ head poses and facial expressions—combines data from two cameras positioned above and below the utensil.

This allows for precise detection of the mouth and overcomes any visual obstructions caused by the utensil itself, researchers said. This physical interaction-aware response mechanism uses both visual and force sensing to perceive how users are interacting with the robot, Jenamani said.

“We’re empowering individuals to control a 20-pound robot with just their tongue,” he said.

He cited the user studies as the most gratifying aspect of the project, noting the significant emotional impact of the robot on the care recipients and their caregivers. During one session, the parents of a daughter with schizencephaly quadriplegia, a rare birth defect, witnessed her successfully feed herself using the system.

“It was a moment of real emotion; her father raised his cap in celebration, and her mother was almost in tears,” Jenamani said.

While further work is needed to explore the system’s long-term usability, its promising results highlight the potential to improve care recipients’ level of independence and quality of life, researchers said.

“It’s amazing,” Bhattacharjee said, “and very, very fulfilling.”