Virtual Production and XR

This section is meant to provide some links and resources on things like virtual production and Extended Reality (XR). There are many resources that cover this topic in considerable detail, so this will be very brief.

This area of study is a large synergy across different disciplines like content production, filming, LED displays, motion tracking, and lighting. It felt necessary to ackowledge it as part of the continuum of digital displays.

In a weird way, you could consider XR as a specialized display on its own because of how it utilizes sensor technology and game engines to render a believable environment to the perspective of the camera. In the same way that an AR/VR headset renders a certain perspective to the viewer's eyes, this is just that same concept but unwrapped into a sort of extrernally facing thing. I think that certain kinds of displays will eventually incorporate more and more sensors that are just considered fixed elements of the display.

In very brief terms, many setups for XR utilize a rendering engine like Unreal or Notch to render a scene in real time. The scene is rendered on high resolution LED tiles where actors and other props are used to set the scene. A tracking system (like a Vive Tracker) is attached to the camera and it allows the real-world camera to essentially duplicate the virtual camera position in the rendered 3D scene. There is a lot of details I'm not covering here like the importance of scene lighting, the frame sync/genlock between the LED scene and the camera, and the alignment of the camera intrinsics to the virtual camera so that movement feels natural and smooth for the background of the scene.

Some other resources:

• The difference between XR and Virtual Production

Low budget XR overview: