Understanding Music Heard Through Cochlear Implants

Prior to losing his hearing, Goldsworthy liked to jam on his drum set, and he wasn’t about to give up music just because the CI had changed his aural environment. But the rock music he liked involved a lot of different instruments that the CI couldn’t translate appropriately for Goldsworthy’s brain to understand. “It didn’t sound like what I was used to,” he says. So, shortly after he got his CI, he started listening to instrumental solos, often of the jazz variety. Then he tried duets. “[I] gradually worked my way back to something as complex as rock and roll or multiple-instrument jazz pieces,” he says.

Goldsworthy gives credit to the improvements in technology over the years—mostly changes in the algorithm that translates sound into cochlear stimulation—but, “I think just as important or more important is the experience I’ve had rediscovering music,” he says. “You can relearn to appreciate it, but it has to be an active process.”

His personal experiences motivated Goldsworthy to study cochlear implants as a scientist, and he’s always trying to improve the function of his own device. He’ll occasionally upload new software to his CI that he hopes will provide him with new information about what he’s hearing. “When I do that, I have to reenter this period of learning,” he says. Right now Goldsworthy’s working on software that’s better at conveying information about the speed of stimulation, rather than simply where along the cochlea the electrodes are firing. “The spatial location of the nerve that’s being stimulated is a cue, but also the speed with which you stimulate the nerve is a cue,” he says—one that current implants do not use effectively.

Cochlear implant listeners think that they perceive the melody worse, that they perceive the instrument detection worse than
normal hearing listeners.—Waldo Nogueira,
Medical Uni­versity Hannover

Goldsworthy also wants to understand how musical perception—in particular, patients’ ability to discern tones—is tied to the ability to understand speech in everyday conversation. In 2015, Goldsworthy compared pitch and phoneme perception in CI users and found the two measures to be positively correlated (J Assoc Res Otolaryngol, 16:797-809). “Individuals who [understand speech] better in a noisy situation have better pitch perception,” he says.

The exact relationship between music and speech perception remains unclear, however. (See “Musical Roots” here.) Joe Crew of cochlear implant company Advanced Bionics suspects that the correlation may simply reflect general auditory processing abilities, such as focus, fatigue, and working memory. “The link between speech and music is pretty tenuous once you factor that out,” he says. Goldsworthy thinks it’s worth a try, though. His group is now working to train CI users to better hear music in hopes that it might also improve their ability to understand speech.

Waldo Nogueira of the Medical University Hannover, Germany, thinks this type of music therapy will actually become more common among CI users. “It seems that there is more and more evidence—although it’s not accepted completely by the research community—that musicians [or] people having musical training seem to have an advantage in understanding speech in these cocktail party scenarios—in noise,” he says. “This suggests that musical training may [help] CI users to improve speech intelligibility in noise.” (See “Music Therapy” here.)

But it’s not all about improving speech perception. Improving the music-listening experience for CI users is another important goal. And if music is more enjoyable, it will make for a more effective therapy.

One approach to making music more CI-friendly is to remix it—basically, take apart the music’s components (often laid down as separate tracks) and recompose them in a simpler arrangement. Wim Buyens of the Cochlear Technology Centre in Belgium and colleagues have found, for example, that raising the volume on vocals and removing harmonic instruments from pop music improves the listening experience for CI users (IEEE Trans Biomed Eng, 62:2434-42, 2015).

Meanwhile, Nogueira is also putting on music concerts specifically composed for the enjoyment of CI users. He and his colleagues in the Auditory Prosthetic Group at the university’s German Hearing Center organized meetings between composers and CI users, who shared information about the technology and their experiences with music. The musicians then wrote electroacoustic numbers inspired by what they’d learned and by the patients themselves. “We were aiming at music that can be similarly perceived by CI users and normal hearing listeners,” says Nogueira.

At each of two concerts held so far (musIC 1.0 in February 2013 and musIC 2.0 in February 2015), the pieces, often paired with visualizations or other performances, were played over loudspeakers to an audience of 250 normal hearing listeners and CI users. In addition to promoting CI awareness and encouraging CI users to listen to music, the concerts “are also experiments,” Nogueira says. After each musical number, the audience was asked to fill out a questionnaire that touched on the technical aspects of the music, the participant’s subjective impressions of music, and their emotional responses to it. The 90 CI users and 168 normal hearing listeners who responded gave similar scores on the last two counts, “but when it comes to the technical aspects of the music, there was a significant difference,” says Nogueira. “Cochlear implant listeners think that they perceive the melody worse, that they perceive the instrument detection worse than normal hearing listeners.”