The StoryFutures Creative Cluster, based at Royal Holloway, University of London and its partner Nexus Studios have been working together to create an accessible toolkit to capture and stream live performances using augmented reality to reach audiences at home. We spoke with Executive Producer at Nexus Studios, Colin Davis and StoryFutures researcher Dr Aneta Mancewicz to find out more.
What’s the big idea?
When the theatres go dark how can live performances still be experienced by audiences? What might the experience look and feel like for both actors and audiences? Just as the pandemic closed live venues in March 2020, a research and development project started looking at these very questions. Researchers Dr Aneta Mancewicz and Dr Rebecca McCutcheon at the StoryFutures Creative Cluster, based at Royal Holloway, University of London, partnered with a team led by Colin Davis at Nexus Studios and over the following months they created Live AR.
What is LiveAR?
The LiveAR project has developed a new toolkit to capture and live stream performances from live on stage straight into someone’s home, all using readily available technology to deliver an augmented reality (AR) experience. The team built a prototype kit that captures an actor’s performance via two cameras. A front-facing camera on an iPhone is suspended at head-height, tracking and recording the actor’s face. A second camera captures body movements. These two streams are then merged, along with the audio voice recording, creating a single representation of the actor. The final stream is then delivered to the ‘audience’ at home, via an app on a tablet or smartphone.
The prototype loosely reimagines The Tempest through a journey of five scenes from the play. Watching the performance, the audience sees the actor’s avatar and elements of an imagined world projected onto their room. Viewers can move all the way around the actor, get closer up or move further away. They can also interact with the scene by sculpting the environment around the actor. For example, tapping and moving clouds into new positions or prompting flowers to grow around the avatar.
What do audiences think?
A test audience tried out the prototype and the response was very positive. They found the app simple to use and particularly enjoyed the world-building interactions and getting up close to the actors.
Many assumed the performance was pre-recorded. Researcher Dr Aneta Mancewicz says it has made the team think about what indicates ‘liveness’ to the audience, and if it is important in the virtual context. She asks “what are the necessary conditions to make people feel incorporated and engaged? I think there is great potential to give a sense of play and engagement associated with live performance, without having the actors necessarily performing live.”
Traditional theatre has the thrill of both a shared experience and the uniqueness of each performance. The LiveAR project raised questions of how (and if) to recreate these aspects. For example, the prototype does not yet allow audience-to-audience or actor-to-audience contact, although test audiences expressed an interest in these interactions. Colin notes that “for us, it’s still a very open question of how much interaction do you really want? What works for a group of 10 wouldn’t necessarily work for 100. It could just be a cacophony….We were struggling the whole way through with how the sense of novelty and spectacle could fight against the story and that emotional connection.”
These are all areas to be explored in the future development of LiveAR. However, the researchers all agree that emotional and intellectual engagement with the performance should be at the heart of things.
How did the funding and partnership work?
The LiveAR work was awarded £20,000 from StoryFutures, with Nexus Studios more than matching this in production resources. Due to the pandemic, most work was carried out remotely. As a result, a large live production wasn’t possible, so the prototype was launched with two actors each performing alone.
The partnership has been incredibly positive. Aneta says they immediately clicked with Nexus Studios and liked the way that the team is really interested in big ideas. Colin describes a liberating space for innovation at StoryFutures where failure was a valid part of the creative process. He found the StoryFutures team brought “additional perspectives around business models or audience surveys and can do more qualitative and quantitative research around things so that when we do these kinds of experiments, it’s less anecdotal and more rigorous.”
What next for the future of LiveAR?
There is a great appetite for future collaboration. Nexus Studios and StoryFutures are in the early stages of developing the prototype into a publicly available app. As LiveAR uses readily available, affordable tech, the partners hope the tool can be used by smaller scale theatre venues. There’s also significant future potential in fields such as live music and sports entertainment as well as education. With such a strong relationship and so many routes to explore, Nexus Studios and StoryFutures plan to continue applying for funding together and take LiveAR to the next level.