Gesture Analysis

The applications concern not only music, but also performances such as dance or theater. The research carried out in the framework of this project is multi-disciplinary and includes engineering, physiology and biomechanics, cognitive sciences, and artistic domains. This work is carried out in synergy with the team's developments on gestural interfaces.

The first axis of research focuses on the study of instrumental gesture and its relationship with both musical writing and the characteristics of the sound signal. Measurements were taken on violinists and trumpet players during a collaboration with McGill University. In the case of string instruments, the movements of the bow, the violin, and the musician's entire body can be measured using optical 3D measurement techniques. The team has also developed systems for complementary capture, compatible with use on stage or in educational settings.

The ensemble of these methods makes it possible to measure and model musicians' gestures. Diverse problematic are also addressed in this study: motor control, learning in the case of the gesture of an expert, characterization of playing styles taking into account sound and gesture parameters, and the modeling of the phenomena connected to the gestural co-articulation similar to those of speech.

The second axis of research focuses on the development of gesture analysis and recognition systems. We favor general approaches that make it possible to apply the team's results to music, dance, or interactive installations. Such systems can be integrated, for example, in the context of a live performance for the synchronization and interaction between the performers' movements and a range of sonorous and visual processes. The recognition is based on diverse temporal characteristics of motion, captured by video or by a set of sensors placed on the performer's body.

The tools that result from this research, such as gesture following, enable the precise recognition and characterization of a range of high-level gestural elements defined by the artists themselves.

Participants