How Visuals Can Help Deaf Children 'Hear'

John Varrasi is a senior staff writer for the American Society of Mechanical Engineers (ASME). He contributed this article to Live Science's Expert Voices: Op-Ed & Insights. 

The Cooper Union in New York City is combining engineering and acoustics to create a unique learning environment for deaf and hearing-impaired schoolchildren. The college has installed an interactive light studio at the American Sign Language and English Lower School in New York City. The studio, comprising a 270-square-foot space, is equipped with a wall-mounted digital-projection system that works in conjunction with specially designed computer programs to display entertaining images and graphics on an interactive screen. The pre-kindergarten children using Cooper Union's interactive light studio learn through their interactions with the moving images and light pulses — the displays enable the kids to actually understand the intricacies of sound, despite their hearing impairments. 

"We are creating a learning environment in which deaf and hearing-impaired children can explore and appreciate the various qualities of music and sound through the interplay of light and vibration," said Melody Baglione, a professor at The Cooper Union for the Advancement of Science and Art, who advised and mentored seven students on design aspects of the interactive light studio. "We have developed technologies enabling the children to visualize sound." [Science As Art: Soundscapes, Light Boxes and Microscopes (Op-Ed )]

Latest Videos From

Interactive projector system

At the core of the interactive light studio is a series of computer programs specially designed to inspire curiosity and fun among the children. One of the programs is a virtual fish tank, in which images of fish follow and respond to the movements of children in front of the screen. Baglione's students wrote the program, which incorporates an Xbox Kinect sensor along with the em>Open Frameworks programming language to detect the contours of the moving children.

The second program uses sound from a microphone, musical instrument or prerecorded song as inputs. When a child stands in front of a target, a component of a digitized song — such as the keyboards, percussion or vocals — plays. When all of the targets are triggered, the full song plays. In this way, the children can create their own music composition by moving their bodies.

"Both hearing-impaired and deaf children can participate in creating sound inputs and visually seeing the responses," said Baglione, who along with her students used a grant from The American Society of Mechanical Engineers (ASME) to launch the studio. "By creating discrete visual responses to different frequencies and sound levels, the children begin to understand sound and music in quantifiable terms."

Sound to light, talking flowers

In another program, Cooper Union students adapted a wall with images of "talking sunflowers" that transform sound to light. The flowers have embedded microphones that trigger different colored lights, depending on the frequencies of the sound in the room.

After exploring several options for converting audio input to visual input, Cooper Union students selected the "colorganic spectralizer," a type of spectrum analyzer equipped with a microphone and able to operate on standard AA batteries. The Cooper Union students installed seven colorganic spectralizers, modifying each with surface-mount soldering to accommodate five-volt capability that brightens the LED lights. The main benefit of the devices, according to Baglione, is the full interactivity the analyzer provides for the children.

"Deaf and hearing-impaired students, in particular, benefit from the design of a sound-to-light installation employing microphones to provide visual feedback," says Baglione.

Electronic fireflies

One wall in the studio incorporates an interactive, electronic simulation of fireflies that the schoolchildren can move around while observing pulses of light. Each of the fireflies is a self-contained circuit board that synchronizes its flashing with other fireflies in the immediate vicinity, a mode of nonverbal communication enabled via infrared sensors and other electronics. When the flash of a neighboring firefly is detected, the voltage across a capacitor experiences a sudden impulse, advancing the firefly's charging cycle closer to its neighbor's. In interacting with the electronic fireflies on the wall, the children are engrossed in play, arranging and rearranging the fireflies according to the patterns of flashing lights. 

"Interacting with the fireflies entertains the children and teaches the kids about the emergence of visually intriguing patterns and rhythms," said Baglione. "The program encourages the children to move and investigate cause and effect."

Students at The Cooper Union built more than 60 circuits, and introduced children's books relating to the subject, which enabled teachers to incorporate lessons.

Inspiration

The Cooper Union's interactive light studio enables deaf and hearing-impaired children to experience sound in unique ways and to overcome the limits of physical disabilities. But the studio brings other tangible benefits, according to Baglione — it enables the children to experience and appreciate the wonder of science and engineering, possibly inspiring future career pathways in the areas of science, technology, engineering and mathematics (STEM ).

"The strong STEM component in the interactive light studio rendered the project suitable for funding under the ASME Diversity Action Grant program," said Tatyana Polyak, director of Student and Early Career Programs at ASME. 

The studio also had a direct positive impact on the student designers at The Cooper Union.

"The interactive light studio demonstrates the role of technological innovation in helping disabled persons," said Baglione. "The project offered undergraduate engineering students an opportunity to improve their technical and professional skills and develop a broader appreciation of the contribution of engineers in improving society." 

Author's Note: Melody Baglione and most members of the design team on the interactive light studio are members of The American Society of Mechanical Engineers.

Follow all of the Expert Voices issues and debates — and become part of the discussion — on Facebook, Twitter and Google +. The views expressed are those of the author and do not necessarily reflect the views of the publisher. This version of the article was originally published on Live Science.