dock10 and University of York team on AI research project for virtual studios

dock10, the UK’s leading television facility, has teamed with the University of York for a joint research project to explore how AI can tackle a major challenge for programmes filming in virtual studios – simulating realistic light interactions.

The project, which brings together leading AI researchers from the university and virtual production experts from dock10, has been awarded an embedded R&D grant from XR Network+. Led by XR Stories at the University of York, the XR Network+ project provides funding and support to researchers working in virtual production technologies, facilitating collaboration between academia and industry.

It is currently difficult to achieve realistic lighting interactions for live or ‘as live’ TV productions – including sports programmes, game shows and music performances – that film in multi-camera green screen studios. For example, an actor will not automatically cast a shadow or pick up reflections when placed into a computer-generated world. These have to be added in post-production.

In single-camera film or drama productions, these effects can be achieved with an LED volume, but this technique is not suitable for multi-camera productions which need to capture multiple angles simultaneously.

dock10 and University of York’s research project explores how AI can help simulate accurate light interactions between physical actors and virtual environments in multi-camera virtual studios. Senior University of York academics and PhD students with a track record of research in AI and digital creativity will work on the project alongside dock10’s virtual studio experts who will provide industry expertise and production workflows.

The research project aims to lay the foundations for the creation of innovative AI-powered compositing tools that ultimately will be applicable in real-time, making them suitable for live or ‘as live’ productions.

Dr. Florian Block, R&D Lead, AI & Immersive at dock10 studios and a Reader in Digital Creativity at the University of York, said: “Lighting is one of the core elements of creative expression in any TV show, but the lighting options for virtual studio productions are currently extremely constrained. That’s what makes this foundational research project so exciting. We are aiming to create a proof of concept that will ultimately lead to the development of a bespoke AI tool for delivering realistic lighting interactions in virtual studio productions in real-time.”

Richard Wormwell, dock10 Head of Production Innovation, said: “dock10 is the UK leader in virtual studios production and the University of York is the UK leader for research into digital creativity. We are combining our industry and academic expertise to try to crack one of the great challenges for virtual studio productions. To do so, we are exploring innovative, AI driven solutions that have the potential to radically improve virtual production workflows. We are hugely excited to see where this research project might lead.”