Coordination: The Plaything of Expressive Performance

Spoken communication and musical performance are arguably our most highly skilled activities. In order to analyze speech and music behaviorally, we must find tractable ways to associate complex signal arrays with events of interest. Unfortunately, both sides of this association are problematic. Signals may occur simultaneously within and across multiple channels and modalities, at multiple physical locations, and with the potential for signal correspondence at multiple levels of spatial and temporal coordination – that is, patterns within patterns. Determining what events to measure is limited by technology and by a predisposition to seek familiar structures that may not accommodate the context-specific event structures that emerge in ephemeral behavior. The problem is that if, as we believe, communicative expression is predominately context-dependent, then identifying emergent events is of fundamental importance.

In this talk, we describe an approach to the measurement and analysis of expressive language and musical performance that allows both emergent and familiar events to be quantified in the instantaneous correlation patterns between signals. Our method demonstrates that spatial and temporal coordination within and between performing individuals is ubiquitous and can be accurately assessed so long as temporal fluctuations in the pattern structure are incorporated into the analysis.

We also demonstrate the value of optical flow analysis as a non-invasive and labor-saving means of recovering two-dimensional motion from video recordings. What was previously thought to be a crude method of motion capture, when pooled for defined regions of interest, provides sensitive measures of performance behavior. We exemplify the motion capture and correlation analysis techniques using conversational data from: English, Shona (Zimbabwe), and Plains Cree (Western Canada); the integration of posture, respiration, and vocalization in speech and song; and the expressive coordination between pianist and vocalist in Lieder/Art Song.