Next generation hearing aids can read lips even through masks

Hearing technology has been advancing rapidly since Miller Reese Hutchinson invented the first electronic hearing aid in 1898. Such advances have relieved the deaf community of being probed with tubular devices. Moreover, cochlear implants, visual alert signifiers, and other technology have increased independence for people with hearing loss. Now we are being introduced to the next generation of hearing aids that can read lips through face masks.

The Next Generation of Hearing Aids

An international team of engineers and computing scientists developed a technological system that can read lips, even through face masks. The first of its kind,  radio-frequency (RF) sensing is paired with artificial intelligence to identify lip movements. A study carried out by The University of Glasgow lays out how they will achieve these smart hearing aids. To learn more, Hearing Like Me spoke with the lead author of the study, Dr. Qammer Abbasi.

Read More: A brief history of hearing aids

RF Technology – How it all Works

The new hearing aid uses WiFi and radar signals to scan facial movements. First, an algorithm is trained to sense lip movements. It then interprets the received data, and delivers the translation to the wearer. Researchers asked participants to repeat the five vowel sounds (A, E, I, O, U) with and without a surgical mask. The paper includes results from the WiFi data interpretation accuracy. Without masks, data was interpreted accurately 95 percent of the time. With masks, data was interpreted accurately 80 percent of the time. Finally, for radar data, information was accurately interpreted 95 percent of the time when unmasked. Radar data was accurately interpreted 83 percent of the time with masks. The team found appropriate lighting played a significant factor in accuracy. Dr. Abbasi tweeted a diagram that visually explains the process.

Dr. Abbasi was asked to clarify how the lipreading is relayed to the wearer once the data has been interpreted. A translation will be delivered via text to the user’s smartphone. He explained that they use Bluetooth and Wifi connectivity to share information among RF based hearing devices like smartphones and smartwatches. He also explained that the technology can actually be connected to other devices the wearer might already use.

Substantial time and effort is invested in subsequent research in order to develop comfort, ease of use, and other features.

The Cocktail Party Effect

In addition to eliminating lipreading barriers, it’s believed the system will also help prevent the “cocktail party effect” while socializing. This phenomenon occurs when people with hearing loss are in a large group where multiple people are speaking simultaneously. Digital hearing aids often do not help and instead amplify background noise, turning conversation into undefinable, intolerable nonsense. The brain can have difficulty processing all the information being fed into it and clarity is lost.

Therefore, having a device that tracks lip movement and interprets auditory information will help to relieve stress and listening fatigue. This is an issue that Dr. Abbasi says is troublesome for the hard of hearing community. He and his team are confident these new aids will reduce the effect in a way current ones do not.

Read More: Cocktail party effect in hearing loss

International Accessibility

Next, the paper explains how current camera-based lipreading technologies come with limitations. For instance, in some states it’s not legal to “film” others without their consent. This type of technology could fall under that category. Dr. Abbasi tells us that the RF-based approach can protect the user’s privacy to a great extent. He elaborates that most countries are also more tolerant of RF-based devices than camera-based devices, meaning the new device will be accepted where camera-based technologies were not.

This study used the existing WiFi network to sense the movement of lips. These new hearing aids will be internationally accessible. Although he cannot say when they will be available just yet. “The application of RF sensing in the field of hearing aids is a bold attempt,” Dr. Abbasi says. “Not only do we need to perform complex signal processing and hardware development, but we also need to consider how to keep the system robust in variable and challenging real-world scenarios.”

“The application of RF sensing in the field of hearing aids is a bold attempt.”

At this time, he explains it is hard to predict exactly when RF-based hearing aids will become widely used. As pioneers in this field, the team’s research has demonstrated that the RF-based technology can reliably recognize lip movements and predict articulation.

A Social Issue

In this video, Dr. Abbasi explains that five percent of the world’s population is currently living with hearing loss, and that it has become a social issue. However, it’s not only about statistics, but rather another larger issue too. When the pandemic hit, the deaf and HOH community faced an additional communication barrier to their everyday lives. Facial expressions and lip movements relied on for effective communication were erased by the new mandatory face masks. While the protective coverings played a crucial role in reducing the spread of the virus, the impact they had on every day life for members of the deaf community were devastating.

Hopes for the Future of Hearing Technology

Dr. Abassi shared some of his hopes for the future of hearing technology. “I believe that the future of hearing aids devices should be more natural, low-cost, ubiquitous, and non-intrusive,” he says. “We hope these devices help the wearer get information without affecting their normal activities. This is why we are developing RF-based hearing aids technology that allows people with hearing loss to focus on communicating with the outside world rather than on how to wear and adjust complex hardware devices.”

• Author Details

Author Details

Beth Tingle

Beth Tingle is a 27-year-old English graduate, and currently resides in Bristol, UK. Beth was born with congenital deafness to an otherwise all hearing family, and is currently navigating life with new sound after receiving cochlear implant surgery in April 2021. She is currently learning BSL.