On 23 and 25 April 2022, the project “Virtual LiVe” (Virtualization of Live Events through Audio-Visual Immersion) has presented a further validation step with a 360 degree keyed broadcast of INDUSTRIE, an experimental musical act by the ensemble PHØNIX16. Virtual LiVe aims at digitally complementing classic event formats but also at elaborating how immersive media technologies (e.g. 3D-Audio, 360°-Video, Light-fields, Volumetric Video) can be utilized to benefit different types of event formats and thus push the envelope in hybrid event realizations.
Performative Industrial Music with an Agenda
The INDUSTRIE by PHØNIX16 is a performative musical act and reflects on the relationship between music and industrialization as well as industrial processes in music. Two composers, who in their individual ways of working with technical extensions or machinization of the performer, are juxtaposed in this piece.
The two works that are designed for VR glasses could not come from more opposite directions. Mexican artist Marisol Jiménez studied composition with Brian Ferneyhough, then turned to multidisciplinary work with self-constructed sound sculptures – experimental electroacoustic instruments that could all sit in a large noise orchestra.
From the opposite corner moves Christophe Guiraud: He comes from Noise, but then discovered his great passion for Ars Subtilior (14th century). Since then, these two inspirations have been part of his work. In his VR work for PHØNIX16, the audience is immersed in an abstract and colorful 3D chapel, where an equally peculiar Renaissance movement, colored by quarter tones, swells, billows and lives. The singers hang from the ceiling like bats. Gravity and balance are vocally and surreally disturbed.
Marisol Jiménez’s work also works with materials: Six gray fringed carpets that form a cube. This cube is lined with an acoustic carpet that the six musicians weave in an entanglement of cables. Sound streams out of machines. A drone blossoms, takes shape and deforms. Only the viewer gives this sculpture its dimensionality. Both works question, problematize, and exhibit the protected space offered by VR glasses. The show provides an immersive and at the same time transgressive experience.
The INDUSTRIE is part of the series Dead On Arrival (D. O. A.) with contemporary music for voices. D.O.A. challenges the impending irrelevance of the voice in current music in society. In its D.O.A. series, PHØNIX16 tries to initiate a paradigm shift in a comprehensible way in order to find a new status of music for voices nationally and internationally.
Fraunhofer Technology: Blurring the Boundaries between Musical Performance, Storytelling and Broadcasting Tools
D.O.A. is discursive and sometimes subversive, but in any case inclusive contemporary music for voices and a possible starting point for social or cultural debate. The team of PHØNIX16 partnered up with the team of the 3IT-hosted Project Virtual LiVe to examine how the use of cutting-edge immersive technology could contribute to their experimental format.
In the production of the INDUSTRIE, the artists performed around the Omnicam-360 and were recorded with this panorama camera from 360 degrees in a green screen studio. The virtual background that was inserted, is constructed based on paintings by Iris Terdjiman.
The OmniCam-360, developed at Fraunhofer HHI is a scalable, mirror-based multi-camera system that allows the recording of live videos in a 360° panoramic format. In addition, Fraunhofer HHI developed the technology to transfer the video material in real-time to end devices – from TV monitors to VR glasses. This Real Time Stitching Engine (RTSE) is a software-based solution for the real-time processing of ultra-HD panoramic recordings.
In the context of the project INDUSTRIE, the end-to-end camera system was especially used to test the practicability and the benefits of a high resolution 360 degree live keying. Keying is the process of removing the green screen background. When the green screen background has been keyed, it will be fully transparent, which allows to fill in that transparent area with a different image or video.
During the production of the musical act the RTSE provided a preview in real-time, in which parts of the 360 degree stage that were not green were masked as green. This created a fully green screen background and allowed for the insertion of the planned video on site. This material in turn was made available to the creative director and operators who were able to immediately identify critical scenes and find a solution on the spot. In order to provide a smooth production flow, a Blackmagic Atem mini was used additionally for the live keying during the production. The Atem mini is able to transmit keyed material in HD for live streaming, whereas the processing of the panorama within the RTSE allows resolutions of 10k by 5k with a color depth of 32 bit per channel.
The broadcast of the two pieces was available online on 23 and 25 April 2022. The real-time keying was tested successfully during the production, adding another layer to the Virtual LiVE collective toolbox of immersive media technologies.
About PHØNIX16
The soloist ensemble PHØNIX16 was founded in Berlin in 2012 and is a collective of singers committed to the expansion and performance of current and contemporary music for/with/without voice, especially by composers still alive. The fact that the singers see themselves as soloists and as part of a collective to the same extent has a decisive influence on the sound, the music-making and the rehearsal work. PHØNIX16 combines voices in an unusual way; the division and instrumentation of the ensemble varies from solo to 16 voices. The repertoire consists of works for voices a cappella, as well as voices plus electronics, feed, instruments, video/film, manipulations, objects and/or machines.
About Virtual LiVe
The Fraunhofer Institute for Open Communication Systems FOKUS, the Fraunhofer Institute for Telecommunications – Heinrich Hertz Institute HHI and the Fraunhofer Institute for Integrated Circuits IIS are collaborating on the project Virtual LiVe – Virtualization of Live Events through Audiovisual Immersion. The project is hosted by the 3IT – Innovation Center for Immersive Imaging Technologies – and supported by the ‘KMU-akut’ program “Research for SMEs” of the Fraunhofer-Gesellschaft. The aim of the project is to digitally complement classic event formats by using new immersive media technologies (e.g. 3D-Audio, 360°-Video streaming, interactive low-latency streaming and advanced stream analytics) and to add value compared to typical video streams via the Internet.
