Alumni Profile
Rebecca Stewart Adapts Gaming Platform to Assist Deaf People
In the hands of Rebecca Stewart (CS ’12), the Microsoft Kinect for XBOX 360 is being adapted from video game novelty to something far more purposeful. The fourth-year student has been working with the open-source code to utilize the Kinect’s depth-sensing technology to facilitate communication between users and nonusers of Cued Speech, a visual form of communication for deaf people in which hand gestures, hand location and the position of the speaker’s lips combine to signal individual sounds.
In the system Stewart is devising, the Kinect can be used to recognize the movements made by someone fluent in Cued Speech, decode them and then assemble them into phonemes, the smallest meaningful units of spoken language. Developed in the 1960's, Cued Speech assists those with significant hearing loss in successfully differentiating identical lip shapes. The technique employs the manual signing of eight hand shapes in four positions around the mouth to complement the lip movements of spoken language. The method allows children with hearing impairment to blend visual cues about word sounds with other linguistic information, such as printed letters, contributing directly to greater mastery of language.
Stewart’s project adapts the Kinect to aid communication between fluent signers and those unfamiliar with Cued Speech. The device’s infrared dot array provides a depth map that Stewart is using to detect the motion and position of hand shapes and mouth shapes. The system then converts the gestures to audible sounds or text.
“Microsoft’s Kinect motion capture system has opened new doors in interface design and computer usability,” says Jason Lawrence, associate professor of computer science and Stewart’s adviser on the project. “We’ve only scratched the surface in terms of the software applications and interface designs that these devices will enable.”