New Netflix doc ‘Audible’ features deaf footballers
June 30, 2021
Smart glasses for the deaf and hard of hearing
July 5, 2021How to learn sign language with AI
Sign language, in all its nearly 300 forms, is still excluded from auto-translation software and apps. However, new and developing technology will help rectify this.
Here are some ways to learn sign language with artificial intelligence (AI).
Fingerspelling XYZ
With a brand new game using machine learning to teach you American Sign Language (ASL), Fingerspelling XYZ is a great place to go to learn the ASL alphabet. It’s fun, free, and easy to use.
Created by HelloMonday in association with the American Society for Deaf Children, Fingerspelling XYZ is a browser-based app that uses your webcam to track the movements of your dominant hand. It gives you the choice of which hand you’d prefer to use. It works by challenging you to quick rounds of words to spell, rather than teaching the alphabet from A to Z (which should be added as a feature). The screen is split down the middle. On one side you see your hand and on the other, an example of the hand movement you’re expected to make.
The best things about Fingerspelling XYZ are the funky interface and various levels of difficulty. There’s lots of room for progression.
MediaPipe
A sign language recognition algorithm developed by Google’s AI Labs named MediaPipe has become the backbone of many new real-time sign language translation services. Startups such as SLAIT are using MediaPipe to build ASL recognition engines.
These new translation programs could be used to learn sign language by copying the signs for words as they’re translated, or by signing into the app or software to test out your signs. However, programs developed with MediaPipe are largely for use as AI translation tools.
If auto-translation software for ASL becomes usable by the general public though, there’s no reason why it couldn’t be further developed for use as a teaching tool, as in Fingerspelling XYZ.
“If auto-translation software for ASL becomes usable by the general public…there’s no reason why it couldn’t be further developed for use as a teaching tool.”
SignAll
One of the first sign language translation tools made available for developers was SignAll SDK. It functions using complex multi-camera systems and gloves. The gloves are labeled with colored markers. Together with the camera system, the gloves make the current technology clunky and expensive for developers to obtain and use.
However, with new ways of reading hand signs accurately, SignAll has broken through at least one huge barrier to auto-translation for sign language. One major issue that developers face is that some signs cover other parts of the hand, confusing the AI system. With SignAll’s system, though, this issue has been overcome, allowing for over 200 signs to be recognized by the engine.
Read more: How I am teaching my deaf baby sign language
Xiao Cong AI, Sign Language News Anchor
Xiao Cong is the world’s first AI sign language news anchor, bringing the daily news to some of the 27 million deaf and hard of hearing people in China. Developed by Sogou, Xiao Cong is a humanoid news anchor who reports the news in Chinese Sign Language (CSL) and looks much like the kind of human news anchor you’d see on television.
Avatars like Xiao Cong are used globally for things like gaming, profile pictures, and virtual meeting personas. The potential this technology has is massive. Imagine auto-translating avatars on your favorite TV programs or on video calls. Sign language translation would be much more widespread, allowing deaf or hard of hearing people better access to many more programs and services. Emergency services could use live signing avatars to communicate with deaf callers, for example.
Xiao Cong could also be repurposed to teach sign language, which could work particularly well with children.
Where Can I Learn ASL Online?
Technology still has a way to go before we can all hop online and use auto-translation or from high-tech avatars. However, there are some great places to learn ASL online:
If you’re already fluent in ASL, grab a hearing friend and help them learn to fingerspell. Or try out a new sign language like British Sign Language (BSL) or Chinese Sign Language (CSL).
With all this new technology to choose from, hopefully doctors, banks, and other essential services will add signing avatars or auto-translation to their services soon.
