Michelle Agnes Magalhaes : The Gesture of Gesture
Since 2012, Brazilian composer Michelle Agnes Magalhaes has been present at IRCAM several times. She has been a member of the research teams as a doctoral and post-doctoral research fellow, then an artistic research residency in 2018, followed by a Start residency, and currently a guest composer at the institute. These last few years have been devoted to a single research subject—the musical gesture—continuously investigated from sometimes surprising, but always richly instructive, angles.
It all began at the end of 2012, when she joined the Analysis of Musical Practices team to devote herself to a very specific subject. “You often hear composers use the word "gesture". I am guilty of it myself! Gesture has always been important in my music but I realised that each time it was used to designate very different concepts: thematic elements, formal figures (often in Germany), physical gesture on the instrument (in Italy, for example). It is therefore a term that is often difficult to define. Especially since a gesture is never just a gesture; it is part of a context from which it cannot be extracted. Understanding its meaning is therefore more of a cognitive process. Later on, I worked on this subject with Béatrice Sauvageot, a neuroscientist specialising in dyslexia, who is particularly interested in the concept of the musical brain.”
This initial research in 2013-2014 was part of the GEMME project (for "Musical gesture: models and experiments"), and it was on this occasion that Michelle Agnes Magalhaes became aware of the work of the ISMM (Sound Music Movement Interaction) team, and in particular its motion capture technologies.
“For me, it opened up the possibility of a truly practical approach to this subject,” she recalls, “a way of scientifically analysing all this rather complex data on the musician's activity.”
This analysis turns out to be as complex as the concept itself. « By working on the description of gestures, we realise that each person finds his or her own way of performing the same gesture. When you look at the data, you see that the machine doesn't understand either: in the detail, you discover many small, very personal details.»
In-house sensors vs. phone sensors
At the same time as the composer was embarking on this path, the ISMM team was developing two distinct yet complementary technologies for capturing the gesture: « For a while we had been working with a sensor called R-IoT developed by Emmanuel Fléty,» explains Frédéric Bevilacqua, head of the ISMM team. « Basically, it is a small electronic card equipped with accelerometers, gyroscopes and a WiFi transmitter, which can be placed almost anywhere. We were also working on exploiting the motion sensors embedded in mobile phones, through an application called Element, which is based on web tools. Initially, it was just a sound trigger. But over the last three years, thanks to Benjamin Matuszewski in particular, the application has been completely rewritten. Renamed « CoMo Element », it has now become a motion recognition application, thanks to the computing power of the phone exploited by a web page opened in the internet browser. “
« Phone sensors are now very accurate. This doesn't make R-IoT sensors obsolete, they are still better, but phone sensors are good enough not to be limiting. They also allow for more inclusiveness. In our initial discussions with Michelle Agnes, we chose the phone as a good entry point because it is easy for the general public or students to learn. Everyone can join in the game.”
And everyone will be invited: eager to test the device with a wide range of people, Michelle Agnes Magalhaes and the ISMM team are experimenting with performers, musicians, and people from all walks of life.
Constella(c)tions as part of Studio 5 en direct (2019)
A tool for composition and interpretation
At the same time, the composer quickly realised she was not satisfied with only recording gestures, but wanted to develop a real musical instrument from these sensors. And to write for this new instrument. « I had a lot of worries about putting the device into a practical situation, and that pushed me to compose a lot of pieces with it,” explains Michelle Agnes Magalhaes. This implies, on the one hand, continuing to work on the tool, to make it more sensitive and more musical, more controllable in short, for the purposes of interpretation, but also to make it a compositional tool, i.e., capable of arranging sounds, with a certain repeatability of gestures and their results - while adapting to all musicians.
« Technical development and aesthetic development went hand in hand,” says the composer, « in a constant back and forth movement between technical and compositional needs and intuitions. We had to invent a gestural repertoire, with its own vocabulary and syntax, which associates different combinations of gestures. The main challenge is to develop consistency between the gesture and its sonification. It is a purely empirical work, which also depends heavily on the sound materials - each gesture gives us a view of the material and vice versa. For example, in some pieces, the orientation (its inclination: head up, head down, etc.) triggers a bank of sounds, the gesture triggers a mode of sound production within that bank, while its amplitude, quite intuitively, will control the sound energy deployed. The gesture is never treated in isolation, but always in association with what we are trying to achieve, in terms of construction, expression, etc.”
« Depending on the context,” says Frédéric Bevilacqua, « we can adjust the sensitivity of certain parameters: if we focus mainly on energy, independently of a precise form, the direction is less important.”
A notation system for everybody
Many projects are taking shape in this way.Some are intended for a more professional audience - for whom the more autonomous, flexible, finer and more responsive R-IoT sensors are preferred.
« In the case of dancers and circus performers, we see a form of virtuosity in the control of sensors,” says Frédéric Bevilacqua. “We have sensed a strong demand from performers to take hold of the gesture, to make the gesture independent of the instrument, and even to instrumentalise the electronics,” adds the composer, “This can even facilitate access to works for instruments and electronics.”
Others are aimed at a wider audience, usually using their own telephone. This has given rise to a veritable constellation of pieces) - all of which are part of a cycle aptly entitled Constella(c)tions. These are not pieces for the phone: the phone is used as a control and interpretation tool - and, sometimes, for sound diffusion (via its own speakers, Bluetooth speakers or another system), when the sensor data) is not simply collected by a computer which processes it to produce the sounds and diffuse them.
« Our question was: is it possible to convey to people the musician’s experience through the motion sensor,” says Michelle Agnes Magalhaes. An experience that goes from learning the instrumental gesture to learning musical grammar and even interpretation. The challenge of transmitting the gesture as precisely and comprehensibly as possible then immediately arises. Then the need for a form of score intended for the audience/performer: an adapted score, with a new notation that is both abstract and formal, poetic and rigorous. The idea was, once again, to let the audience experience what the formal markers are - the equivalent of the staves, with their notes, key signature and rhythmic indications, which serve the musician.
In vivo danse - Camping, Manifeste 2021 © Quentin Chevrier
« You can see that no matter how accurate the notation of a gesture on the score, there are several ways of doing it,» says Michelle Agnes Mahalhaes. « Everyone finds their own way of doing it. This is also what makes the subject of transmission so exciting! Because the combination of gesture and sound also allows the performer to check the precision of his or her gesture. And, just like on a classical instrument, we can see that all these apprentice musicians are evolving and refining their gestures.»
« It's easy to get the hang of,» says Frédéric Bevilacqua. « But work is needed to make it your own and to make the piece sound musical.»
Some pieces can be played alone: Ex-Tensio, for example, consists of a sound file that is scrolled through according to the gestures made with the mobile phones - a work that is both very reactive and very composed, with the score scrolling in the form of text on the phone's screen. Others are aimed more at a collective game, giving rise to authentic polyphonies. Others even allow for the transformation of sometimes incongruous objects into instruments: like huge ropes stretched across a room, which are manipulated with the phone in hand - the gestures made on the rope cause accelerations on the phone, which in turn produces sound.
« All projects let us test new ideas and add new functionalities to our tools,» concludes Frédéric Bevilacqua.