In a few words
Transforming a live performance stage into a participatory Augmented Reality environment.
The previous work done on Augmented Reality projects with Giacinto Barresi resulted in the creation of an AR platform which allows multiple users to share the same virtual environment.
I discussed the potential application of this concept to the performing arts with Victor Zappi, given our previous work together on Virtual Reality, Hybrid Reality Performances, Immersive Virtual Musical Instruments and participatory performances.
After some time, the Augmented Stage (AS) concept was born.
The Augmented Stage Concept
The idea behind the Augmented Stage (AS) is to transforms a live performance stage into an Augmented Reality environment, which can be enjoyed through the cameras of the audience personal smartphones or tablets. Big posters placed on the stage act as trackable AR targets, becoming part of the performance installment. The posters serve as placeholders for AR elements, characterizing the AS. By watching the targets through their devices cameras, the audience can watch both the stage and the AR elements. Features of these AR objects are associated to visual and sonic controls. By manipulating these objects using their devices, spectators contribute to the performance outcome, together with the performers. The changes made to the AS by someone in the audience are perceived by everyone, simultaneously and coherently. Based on these changes, the AR environment controls sonic and visual features of the performance. A fixed camera can be pointed at the stage, watching the performers and the posters. The feed of the camera can be displayed, showing to the entire audience the AS and the interactions taking place within it.
The platform provides the freedom to design different kinds of choreographies and interactions, coherently with performances style and purpose. The simplicity of the setup permits to stage performances in most venues. The use of spectator’s personal devices allows the design of transparent and powerful audience and performer interactions, contributing to the generation of ever-changing performances. This kind of experience increases audience reward and contribution awareness. AR content can be used to improve the transparency of the performers’ actions as well, especially if the artists are using uncommon musical instruments or controllers. The concept of Augmented Stage can be applied to all performing arts, including music, theater and dance.
con i piedi per terra
con i piedi per terra is a performance based on the AS concept, which has been designed and conceived by il GRANDENERO, an acoustic rock duo (Stefano Zarba and me). During the performance, il GRANDENERO played four tracks from its existing repertoire. In addition to the duo usual instrumentation (two guitars and a lead voice), new arrangements were written to be played by an Ableton Live set during the performance, and controlled by the audience. Before the beginning of each song, musical transitions were played by the computer. These intermissions were written using exclusively the audience-controllable instruments accompanying the upcoming track. This allowed the spectators to explore the controls of each track on their own, right before using them together with the band. Each track provided three different audio channels to be controlled by the audience, one for each AR target. Depending on the current instruments and effects, one or more parameters were exposed for audience control. Each parameter was chosen so that its effect could be easily perceived by the spectators as a consequence of their interactions, and they included distortions, LFOs parameters, tremolos, a sequencer, delays, etc. Audio parameters were mapped to the behaviour of AR virtual objects shown in correspondence of the stage posters, resulting in simple choreographies. Audience interaction with AR objects happened through the mobile devices touchscreen: some elements could be dragged with a finger, other behaved like buttons, while other required the touch of multiple spectators at the same time, in order to produce a stronger feedback. As a performance design choice, AR controllable elements were not available for the whole duration of each track. The audience could watch the Augmented Stage on a projected screen, shown from the point of view of the ceiling mounted camera. Non interacting spectators thus enjoyed a shared view of the Augmented Stage, while the interacting audience had an additional way to verify when interaction was available.
Evaluating interactive performances
With the intantion to evaluate our AS performance, Victor Zappi and I defined a generalized set of metrics for the evaluation of concept and platforms used by interactive performances:
-Control Design Freedom: how freely audience interaction can be designed with the platform.
-System Versatility: overall performance setting up simplicity and performer’s comfort on stage.
-Audience Interaction Transparency: clearness of the relation between audience manipulation and its effects.
-Audience Interaction Distribution: to what extent interaction can be located towards the participants (strongly centralized interface vs. every participant holds one).
-Focus: how easily the audience can freely focus on different performance aspects (the stage, their interaction, visuals, music, etc.).
-Active/Passive Audience Affinity: how much the non-interacting audience experience can be similar.
We used these metrics to evaluate a number of interactive performances, and con i piedi per terra as well. More on this can be found in the publication describing the Augmented Stage concept.
Augmented Stage for Participatory Performances, Mazzanti, D., Zappi, V., Caldwell, D. and Brogni, A.
Proceedings of the International Conference on New Interfaces for Musical Expression, 30 June – 4 July 2014, London, UK.