Play Music with Your Smartphone
Norbert Schnell, from IRCAM’s Interaction Sound Music Movement team, is holding his phone by the edges. Quickly, he shakes it towards the floor twice, then once to the side and then starts the pattern again: the phone lets out percussion sounds characteristic of the beginning of the song We will rock you by Queen. The searcher touches the screen, makes a couple of little movements: he plays the rest of the song, always controlling the rhythm and the sequencing. “This demonstrates the principle of what we are trying to do,” Schnell explains. “Using connected objects, like our telephones, so anyone can play music.”
For the past 10 years the Interaction Sound Music Movement team, led by Frédéric Bevilacqua, has studied systems that make it possible to create music based on motion capture. Dancers and professional musicians were the first guinea pigs. In order to record their movements, the searchers used cameras in the beginning. Very quickly they were able to perfect electronic sensors that were more and more sensitive with each version. Today, a device includes—in just a few square centimeters—an accelerometer, a gyroscope (that measures the rotation of a movement), and a magnetometer (indicates the position in terms of a horizontal axis), a WiFi transmitter, and a battery to power everything.
This sensor was used in the fabrication of “augmented instruments”. Attached to a violin or to the wrist of a musician, it enables the latter to modulate the sounds he creates with his instrument, or add others, via specific movements. But its small size and sturdiness made it possible to use it in far less conventional circumstances, in particular when novices make music.
“In a concert, or a live performance, digital music totally changes the way we watch musicians,” specifies Norbert Schnell. “I think the poetry of a digital musical performance is not only what we see and what we hear, but also what we touch and what we do.”
The team tested the idea in a real-life situation in 2011 with a ball game, Urban Musical Game, during the Parisian festival Futur en Seine.
Two teams of 3 or 4 players competed in a simplified version of basketball (the baskets were on the ground, for example) using a ball equipped with a motion sensor. The music played through the loudspeakers, composed by the Italian Andrea Cera, changed in rhythm with the game. Two musical universes were associated with the two teams: their presence depended on the number of points each team scored.
One of the IRCAM searchers involved in this project, Nicolas Rasamimanana, continued this investigation for commercial applications. He created the company Phonotonic that sells “augmented balls” and associated applications. Another application was created for instrumented juggling balls for the circus artist Jérôme Thomas. These balls were used for a work for juggler, six musicians and electronics written by the composer Henry Fourès (premiere September 30, 2015).
Specific objects have to be made for all of these applications. On the other hand, all of us, or most of us, have a device equipped with powerful movement sensors capable of making sound: a smartphone. These devices are also connected to Internet! And since 2011, the W3C, a consortium that shape web technologies, works on the normalization of an effective protocol for sound processing in real-time online: Web Audio API. This technology is already present in the latest versions of the most common navigators. These elements all made it possible for Frédéric Bevilacqua and his colleagues to give another spin to their work in the framework of the CoSiMa project (Collaborative Situated Media). “The uses of electronics and digital technologies in the domain of music had been developed with two complementary objectives: to improve listening and the restitution of recorded music and to produce music differently,” says Norbert Schnell. “Today, we have the means to bring together the two, offering the possibility of playing recorded music that we can easily change.” The treatment applied to We will rock you is an example. For another experience, called Mojo Guitar, we play the guitar section of a song by M. (Mathieu Chédid), with more or less energy, by shaking the phone slower or faster. The original track was analyzed, cut up, and annotated in terms of harmony. When we shake a phone connected to the project’s Website, the musical segments correspond to the part of the song we are playing and the energy of the movement is sent to these and played.
In February 2015, an experiment called Drops was held in public. Based on sounds borrowed from the application Bloom imagined by the British musicians Brian Eno and Peter Chilvers. Echoes of sounds created by each participant touching the screen of their telephone are repeated on the phones of others. On June 21, 2015, during IRCAM’s ManiFeste festival—and pour the fête de la musique—a “live interactive performance” with the Parisian DJ and producer, Chloé, was held in the Palais Royal garden.
“I played my music live, diffused by loudspeakers that filled the space,” she explains. “But before the performance, I had composed and recorded 4 sound loops that the audience, connected to the project’s Web page, could play and modulate during specific times throughout the performance by touching their phone screen in different ways or by moving the device.” If we add that Chloé herself could send sounds to the audience-members’ phones and control the spatialization of these sounds, we understand the experience of the event, co-created by the audience, was different for each participant. For Norbert Schnell, being able to appropriate a part of the sounds Chloé played, before or after they appear in her music, completely changes the musical experience. An installation at the Gaîté lyrique for the exhibition Paris Musique Club in October of the same year was a continuation of this concept.
Of course these experiments on cutting-edge research and technology are not yet accessible to everyone: only the latest iPhones and Androids support Web Audio API. But the limited life expectancy of these devices will quickly resolve that issue. The Interaction Sound Music Movement team is ready to work with artists who imagine music, but also other systems, gameplays, that make use of the audience’s actions in an artistic and poetic process.
Le jongleur Jérôme Thomas devient instrumentiste
Dels dos principis, premiered on September 30, 2015 during the Musica Fesival in Strasbourg, was written by the French composer Henry Fourès for a unique septet: one of the musicians is the juggler Jérôme Thomas (the other instruments are more habitual: flute, clarinet, piano, percussions, violin, and cello). His instrument: juggling balls equipped with motion sensors developed by the Interactions Sound Music Movement team at IRCAM, connected to computers and electronic sound-generating systems via WiFi. Listeners become—more here than in other concerts—spectators.
“When I juggle, I write visual space in its pure essence, without adding anything. Here, I offer the spectator a transfer from his ears to his eyes; it’s a big responsibility that requires perfect juggling,” says Jérôme Thomas.
by Luc Allemand
