VR CASE STUDY
STAGE VR
Experimental virtual music app with Meta, Dolby.IO and Move.AI uses the latest technology to make the live gig experience more accessible.
[ 2023 ]
[ VR ]
Draw & Code Stage was born out of a vision to transform the traditional in-venue music experience into a cutting-edge virtual and mixed reality encounter to allow new audiences to access live music. The tech demo aimed to utilise state-of-the-art performance capture and playback technologies including Meta Quest’s mixed reality features, Dolby.IO’s streaming APIs and Move.AI’s unique approach to motion capture. Pushing the boundaries of what was possible in the realm of immersive entertainment was the aim of the project.
We have seen virtual music performance becoming a steadily growing space with Fortnite leading the charge. However, the vision for Stage was to look at replicating real-world music venues and to create a pipeline for performance capture that felt natural in those environments. In short, could we use standard recording equipment common to many venues such as microphones, GoPros and other consumer cameras, to record both sound and motion capture data? Would this be adequate to create a virtual music experience that rivalled those produced in dedicated motion capture facilities?
The concept behind adopting AR for training was that the visual AR output can be used as a guide for setting up specific training practices, while the computer vision processing can allow the training exercise to be evaluated by the app.
The app was being co-developed by Chelsea Digital Ventures and Draw & Code’s specific experience with AR was sought to help create what would be a very advanced and ambitious new concept in training.
A key feature of the demo was the use of a mixed reality introduction scene. The project was developed using a Meta Quest Pro ahead of the release of the Quest 3. Part of the excitement of a live gig is getting ready and heading to the venue – we used mixed reality to bring a little taste of this to Stage as the pre-gig experience starts in your own living room. From a UX point-of-view, virtual gig-goers being able to see their home surroundings makes it a lot easier and more comfortable to get your bearings ahead of the full VR gig.
The demo prominently featured a digital performance by Jake Isaac, captured live on stage at KOKO, the spectacular and historic London venue, using Move.AI’s performance capture technology. This 21st-century motion capture system ensured a lifelike and dynamic representation of the artist’s movements within the virtual music performance. Move.AI’s mocap technology is different from most in that performers can move organically and have their body movements captured without any form of mo-cap suit. The team at Move.AI used iPhones to record Jake, this is truly accessible technology.
Innovative Dolby.IO streaming and communication APIs played a pivotal role, enabling the integration of real-world video content into the virtual realm. This meant that a select few audience members can be seen on stage as part of the video backdrop to the virtual performance. The whole performance could also be downloaded after the experience, showing the audience point-of-view, further breaking the barriers between real and virtual concerts and live-streaming.
Ready Player Me avatars were utilised for the experience. This is one of the slickest and easiest-to-use avatar systems around. For the user it means a quick and easy way to customise your own character and for the artists on stage we can quickly produce a custom avatar that represents them.
The R&D project debuted at SXSW 2023 and was later seen at AWE (Augmented World Expo), both shows seeing us as a prominent part of Dolby’s spectacular expo presence. Later, Draw & Code Stage was shown at a private event with Meta after the makers of the Quest invited us to adapt the project to act as a showcase for the mixed reality features of the Quest 3.
Draw & Code Stage successfully delivered a groundbreaking tech demo that showed how virtual music experiences can be more accessible to artists, venues and their fans. The mixed reality view and full-VR world options provided users with diverse and immersive ways to engage with the virtual stage. Jake Isaac’s digital performance, coupled with Move.AI’s cutting-edge motion capture technology, showed what is possible for authenticity in virtual entertainment without using inaccessible kit.