PITTSBURGH – Most able-bodied people take their ability to perform simple daily tasks for granted—when they reach for a warm mug of coffee, they can feel its weight and temperature and adjust their grip accordingly so that no liquid is spilled. People with full sensory and motor control of their arms and hands can feel that they’ve made contact with an object the instant they touch or grasp it, allowing them to start moving or lifting it with confidence.
But those tasks become much more difficult when a person operates a prosthetic arm, let alone a mind-controlled one.
In a paper published today in Science
, a team of bioengineers from the University of Pittsburgh Rehab Neural Engineering Labs
describe how adding brain stimulation that evokes tactile sensations makes it easier for the operator to manipulate a brain-controlled robotic arm. In the experiment, supplementing vision with artificial tactile perception cut the time spent grasping and transferring objects in half, from a median time of 20.9 to 10.2 seconds.
“In a sense, this is what we hoped would happen—but perhaps not to the degree that we observed,” said co-senior author Jennifer Collinger, Ph.D.
, associate professor in the Pitt Department of Physical Medicine and Rehabilitation
. “Sensory feedback from limbs and hands is hugely important for doing normal things in our daily lives, and when that feedback is lacking, people’s performance is impaired.”
Study participant Nathan Copeland, whose progress was described in the paper, is the first person in the world who was implanted with tiny electrode arrays not just in his brain’s motor cortex but in his somatosensory cortex as well—a region of the brain that processes sensory information from the body. Arrays allow him to not only control the robotic arm with his mind, but also to receive tactile sensory feedback, which is similar to how neural circuits operate when a person’s spinal cord is intact.
“I was already extremely familiar with both the sensations generated by stimulation and performing the task without stimulation. Even though the sensation isn’t ‘natural’—it feels like pressure and gentle tingle—that never bothered me,” said Copeland. “There wasn't really any point where I felt like stimulation was something I had to get used to. Doing the task while receiving the stimulation just went together like PB&J.”
In a series of tests, where the BCI operator was asked to pick up and transfer various objects from a table to a raised platform, providing tactile feedback through electrical stimulation allowed the participant to complete tasks twice as fast compared to tests without stimulation.
In the new paper, the researchers wanted to test the effect of sensory feedback in conditions that would resemble the real world as closely as possible.
“We didn’t want to constrain the task by removing the visual component of perception,” said co-senior author Robert Gaunt, Ph.D.
, associate professor in the Pitt Department of Physical Medicine and Rehabilitation. “When even limited and imperfect sensation is restored, the person’s performance improved in a pretty significant way. We still have a long way to go in terms of making the sensations more realistic and bringing this technology to people’s homes, but the closer we can get to recreating the normal inputs to the brain, the better off we will be.”
Additional authors of this study include Sharlene Flesher, Ph.D., Jeffrey Weiss, M.S., Christopher Hughes, M.S., Angelica Herrera, B.S., and Michael Boninger, M.D., all of Pitt
; John Downey, Ph.D., of the University of Chicago
; and Elizabeth Tyler-Kabara, M.D., of the University of Texas at Austin
TITLE: Brain-computer interface user Nathan Copeland is using a robotic arm to manipulate objects.
CAPTION: Artificial tactile perception allows the brain-computer interface user to transfer objects with a robotic arm at twice the speed of doing it without the feedback.
CREDIT: UPMC/PITT Health Sciences
CAPTION: Jennifer Collinger, Ph.D., associate professor, University of Pittsburgh Pitt Department of Physical Medicine and Rehabilitation.
CAPTION: Robert Gaunt, Ph.D., associate professor, University of Pittsburgh Pitt Department of Physical Medicine and Rehabilitation.