Talking with Your Mouth Full: Performing with a Gesture-to-Voice Synthesizer

We create DIgital Ventriloquized Actors (DIVAs) that use hand gestures to synthesize speech and song by means of an intermediate conversion of these gestures to articulatory parameters of a voice synthesizer. This requires overcoming technical challenges related to tracking gestures, synthesis quality, and the complex mapping between performance and meaningful, expressive vocal sounds that are easy to learn. We discuss these components contrasting a frequency-based and an articulatory based approach for speech synthesis. The relationship between gesture and voice production embodied in a new type of musical instrument provides a rich means for human expression. We show the latest performance work composed using these DIVAs.