Virtual production is a rapidly evolving technology that has revolutionized the way stories are told and visualized on the big screen. By utilizing advanced computer graphics and real-time technology, virtual production allows filmmakers to create complex, photo-realistic environments and characters in a virtual world and then integrate live-action footage seamlessly into the final product. This has broadened the scope of storytelling and cinematography in several ways.
What is Virtual Production?
Virtual production is a cutting-edge technique in filmmaking that combines traditional physical production techniques with the latest advancements in technology. It allows teams to create photorealistic virtual sets in real-time using 3D game engines and display them on large LED walls behind physical sets. This allows for a greater degree of flexibility and creative control in the filmmaking process, as well as enhanced realism and depth of perspective in the final product.
How does Virtual Production Technology work?
Virtual production technology uses real-time 3D engines, such as Unity or Unreal Engine, to create photorealistic virtual sets and environments. Listed below are steps of Virtual Production Technology workflow:
Pre-production
The use of virtual production technology in the early stages of film and TV production allows for the creation of digital scenes that convey the story and tone of the project to investors, producers, and studios. This process, known as pre-visualization, is used during the initial phase of production when the idea for the film or TV series is first visualized.
Virtual Art Department (VAD)
The virtual art department creates assets for a film, from rough 3D to camera-ready prototypes, environments and characters that will be used in the virtual world. The Virtual Art Department team helps to determine which elements of the production should be created digitally and displayed on the LED walls, which should be physical props and set pieces, and which elements will require additional VFX work in post-production.
Visualization
Using game engines, real-time 3D artists create digital imagery to bring a filmmaker’s vision for a sequence or shot to life. This is a key aspect of the virtual production process that filmmakers are well-acquainted with. To create a virtual world, everything must be constructed digitally in 3D using modelling software and a game engine. These 3D models, including props, sets, and characters, make up the virtual worlds projected on LED screens. Mocap cameras also require assets that are superimposed over live-action footage on video screens, helping the crew with framing and timing.
Principal Photography
During this stage of the virtual production workflow, virtual worlds are projected on LED screens surrounding actors, props, and smaller set pieces in the foreground. The Volume’s LED screens create more realistic lighting and reflections, reducing the need for extensive and complex lighting setups on set.
Post-production
Although virtual production technology enables many visual effects during pre-production and production, additional visual effects such as 3D models, compositing, and FX are often still needed. During post-production, VFX artists such as 3D modellers, FX artists, and compositors will create, integrate and render any elements originally represented by placeholders during filming, like stand-ins for digital animals or creatures, into the final footage.
Final delivery
Once all team members have completed their tasks, the film’s final version is processed and ready to be shared with audiences for viewing.
The Impact of Virtual Production on Traditional Filmmaking
Virtual production is redefining the filmmaking landscape by merging the pre-production and production phases, allowing for more iterative and dynamic creative processes. The integration of virtual reality (VR) and augmented reality (AR) further expands the possibilities, offering new ways to engage with content.
Real-World Applications of Virtual Production
From blockbuster movies to live event broadcasts, virtual production is being employed in various sectors to create unparalleled viewing experiences. Case studies from leading productions highlight its potential to revolutionize how we consume media.
Virtual Production Meaning in Different Contexts
Beyond entertainment, virtual production holds significant potential in education, training, and live events, offering immersive and interactive experiences that enhance learning and engagement.
Virtual Production Glossary
The Virtual Production Glossary was created with contributions and support from organizations such as the Visual Effects Society, the American Society of Cinematographers, Epic Games, and Netflix. The Virtual Production Glossary aims to document and define terminology and roles commonly used in virtual production.
It is not intended to establish official terms or roles but to record current terms and roles. The intended audience is the diverse production members involved or interested in working in virtual production. Individuals from various disciplines and technical backgrounds in virtual production should easily understand each term.
LED volume
Virtual productions using LED volumes have become more common in recent years. Actors perform on a physical set in a studio in front of a large array of LED screens displaying images of real-world or computer-generated environments. This setup is similar to the projected backgrounds used in filmmaking since the silent era, but it’s now more sophisticated.
Virtual studio
A multicamera studio that employs game engines, camera tracking technology, and green screens to place presenters and audiences in realistic 3D sets that can be manipulated in real-time. Virtual studios are commonly used for live news and sports programs. They enable the creation of more imaginative sets, save money on transportation and storage, are more environmentally friendly, and can be set up more quickly than real sets.
Hybrid Set
A hybrid set combines elements of both traditional and virtual sets. It includes physical components such as desks, chairs, staging, and a virtual background. This allows entertainment shows to make their physical sets appear larger, providing more value for the budget.
Volumetric capture
3D scanning technology captures a person, object, or place as 3D digital data and reproduces it as a high-quality 3D image. This is done by using multiple cameras that are precisely synchronized to capture the subject from various angles. The recorded video is then processed through reconstruction software, resulting in a 3D avatar of the subject – a solid, moving model that can be viewed from all angles.
Face cap
Facial motion capture refers to recording the physical movements of a person’s face and converting it into a digital model. A head-mounted camera records facial expressions and dialogue to accurately capture the nuances of performance for a digital character, including blinks, smiles, and frowns.
Virtual Reality
A virtual reality environment is a computer-generated three-dimensional space that users can navigate freely, often using a headset. VR environments are typically created using gaming software and transport users to different locations, enabling them to interact with the virtual environment.
Augmented Reality
Augmented Reality in television improves productions by adding 3D computer-generated images, typically in the foreground of a virtual or physical set. These images can include a scoreboard, a Premier League football table, or a representation of an actor or politician. AR can be used in any studio setting, not just those with green screens. The CGI elements of AR do not interact with their surroundings but rather enhance them.
Unreal Engine
Unreal Engine is a powerful game engine developed by Epic Games, which has become a popular 3D creation tool for delivering real-time, photorealistic graphics. Originally developed for the gaming industry, it has since been adopted by various other industries, such as TV, film, and advertising. Unreal Engine is widely used in these industries to construct the 3D worlds used in virtual productions.
Reality engine
Reality Engine is a real-time, node-based compositing platform built on Unreal Engine, developed by Zero Density. It utilizes its proprietary keying technology and control tools to give the broadcast industry control over virtual elements of a production. With the ability to design virtual elements as if they were native to Unreal Engine, they can be easily customized and automated from a centralized hub.
Star tracker
Star tracker is a camera tracking system from Mo-Sys, which uses reflective stickers applied to a studio ceiling to determine the position and orientation of the studio camera in real-time. This information is then reported to the rendering engine. Multiple cameras in the studio can track the star map. Once calibrated, the system operates fully automatically. This is referred to as an ‘inside-out’ tracking solution, where the Star tracker camera is mounted inside the capture area and looks for reflective stickers outside the area.
CG and CGI
The terms CG and CGI are often used interchangeably in the TV industry, and they refer to the same thing: images created using a computer. These images can range from augmented reality scorecards to virtual set backdrops.
Virtual studios developer (VSD)
A relatively new role in broadcast production, a virtual set designer is responsible for designing virtual sets. Many VSDs are from the gaming industry, where they have gained expertise in creating computer graphics using Unreal Engine and now apply their skills in the broadcast sector.
Virtual studios operator
A virtual systems operator works in the studio’s lighting control room and oversees the quality of computer-generated images during live production on a virtual set. They check the keying and masking and collaborate with the lighting director to ensure that physical elements and presenters blend seamlessly with virtual or augmented elements.
Final Thoughts
Virtual production technology has significantly broadened the scope of storytelling and cinematography. It allows filmmakers to create new worlds and experiences that were once impossible or too costly to produce. It gives them the flexibility to explore new styles and techniques and allows them to work more efficiently and effectively. With virtual production, filmmakers can now create visually stunning, immersive, and interactive stories. With virtual production, storytelling and cinematography are becoming more diverse, creative, and exciting than ever before.